Methodology¶

1. Introduction¶

In today’s world, data plays a pivotal role in power relations. While it can be exploited for private gain or to restrict liberties, it also has the potential to address social challenges, foster collaboration, drive innovation, and enhance accountability when managed effectively. The Global Data Barometer (GDB) emerged as a crucial tool in understanding and nurturing this potential. The first edition of the GDB provided vital insights into the data foundations necessary for developing and maintaining healthy, inclusive ecosystems worldwide. Building on this foundation, the second edition continues to trace the evolution of these data ecosystems, offering a deeper exploration into governance, digital public infrastructure, AI preparedness, responsible digitalization, and regional data competencies.

Healthy data ecosystems are not just a theoretical concept but a practical necessity in tackling the global challenges we face today. In an era where misinformation, disinformation, and surveillance can erode trust, polarize communities, and threaten democratic values, robust data governance, transparency, responsible data sharing, and protection are critical. The GDB's work is central to this effort. By evaluating and promoting the infrastructure, processes, and strategies that underpin effective data management and use, the GDB helps ensure that decision-making is grounded in accurate, reliable, and trustworthy data. This, in turn, empowers governments, communities, and individuals to make informed decisions, reinforcing democratic values, enhancing public trust, and fostering social cohesion.

The GDB, succeeded by the Open Data Barometer (ODB), assesses the broader data landscapes that extend beyond legal and technical dimensions of open data. It examines governance, availability, and capabilities to provide a comprehensive picture of the essential foundations that support healthy data ecosystems. In this second edition, the GDB has refined its methodology to better highlight bright spots of data use and address relevant cross-cutting themes, offering globally comparable evidence that supports evidence-based decision-making and policy development. By doing so, the GDB promotes positive change on a global scale, providing a benchmark for governments to align their policies with the concept of 'data for public good', fostering healthier data ecosystems that protect democratic values and contribute to a more informed global society. This methodology document offers a detailed overview of the second edition of the Barometer, underscoring its significance in these critical areas.

Looking back to the first edition¶

The first edition of the GDB was based on primary data from a global expert survey carried out in mid-2021 in 109 countries, which was combined with secondary data from trusted sources to generate a range of metrics. The period assessed was May 2019 to May 2021.

The first edition was a pivotal opportunity to unpack issues such as privacy, protection of human rights in the development of technological interventions, and data governance, among other relevant topics, within countries and across them. It provided regional analysis to understand the particular contexts, and the relative strengths and weaknesses of countries within the regions. More information about the results can be explored in the report and downloadable data.

2. The Second Edition¶

The second edition of the GDB represents a culmination of research, coordinated efforts to build upon the first edition's findings, and a dedicated focus on refining and strengthening the survey tool. This edition employs an expert survey, developed in collaboration with thematic partners across various fields, drawing on an open and participatory process to ensure the survey’s depth and relevance. By maintaining a balance between meaningful detail and broad comparability, the GDB ensures consistent evaluations across different regions, making it a robust tool for assessing country performance.

The true value of a multi-dimensional index like the GDB lies in its capacity to function as a learning device, providing insights into each country’s relative strengths and weaknesses across various domains. For instance, the GDB enables a nuanced exploration of countries that may have robust data frameworks but limited capacity to effectively utilize that data. It also allows for a detailed examination within the governance component, highlighting the relative strength of data protection and data-sharing policies—and whether these policies strike the necessary balance to support public good outcomes in our increasingly data-driven society.

Following the study period of the GDB’s first edition, the study period for the second edition is: August 31st, 2022 - September 1st, 2024 (2 years).

Pillars¶

This Barometer is organized around three pillars[^1]: data governance, data capabilities, and data availability. Each GDB indicator is designed to belong to one of these pillars to maintain a logic of distribution and assessment of national data ecosystems; and each of these pillars has its specific domain of analysis to set a comprehensive assessment framework. The figure (1) below presents the subjects of analysis under each pillar.

Fig 1.

This edition will:

Evaluate data foundations crucial for developing healthy data ecosystems.
Assesses the alignment of data frameworks with data availability in specific thematic areas
Track data foundations across several cross-cutting themes and action areas including but not limited to data use, artificial intelligence, equity and inclusion, accountability, and transparency.
Maximize the potential for GDB’s data reuse within the broader research and policy community.

Key Definitions¶

Pillars: The Barometer is organized around three pillars or core areas of assessment—governance, capability, and availability.
Governance: The rules, processes, frameworks, and institutions in place to ensure data is available for the public good and safeguarded against misuse.
Capability: The means, connectivity, skills, and institutional capacity required to create, share, and use data for the public good.
Availability: The presence, sharing, and quality of specific categories of data to allow reuse for the public good.
Thematic clusters are structured groups of related core and thematic indicators that have been adapted to provide a comprehensive assessment of a healthy data ecosystem. Each cluster focuses on a specific aspect or theme within the broader context of data governance, availability and capabilities. Within each cluster, there are action areas, which contain one or more indicators.
Action Areas: Within each thematic cluster, there are action areas, which are specific domains or topics of interest that are essential for evaluating the overall health and functionality of the data ecosystem. These action areas are often using one or more indicators from different pillars to trace intersections of governance, capability, and availability and to provide a comprehensive understanding of the issues.
Cross-cutting themes: they typically draw on a mixture of individual sub-questions and supporting questions, often distributed across multiple indicators, sometimes in conjunction with entire indicators or element subgroups. Cross-cutting themes are thus similar to thematic clusters, but rather than examine topics only in units of indicators, cross-cutting themes reassemble the different components of the survey in innovative ways to surface further insights.
Indicators: The GDB expert survey consists of 27 primary indicators, each composed of several discrete sub-questions and supporting questions designed to generate weighted scores on a 100-point scale. These primary indicators are framed as "to what extent" questions, providing an overarching scope for assessment.
Sub-questions: Indicator sub-questions are organized into two sections: existence (including the extent of existence) and elements.
Existence: Determining whether relevant data, frameworks, or capabilities exist.
Extent: Assessing the breadth and depth of the existence and elements (researchers first answer the existence sub-questions, including the extent sub-questions, followed by the element sub-questions)
Elements: Exploring the specific components or attributes of the data, frameworks, or capabilities.
Supporting questions: A supporting question is designed to gather additional qualitative evidence that helps substantiate or clarify the answers provided in response to a primary question. These questions dig deeper into the context or reasoning behind a response, ensuring that the answers are well-supported and accurate.

Design Principles¶

The Barometer’s structure, components, and weighting respond to a specific set of design principles.

Flexibility for government structure

While some countries set policy nationally, many countries operate within federal systems that mean that aspects of data policy, capability, availability, and use are shaped by sub-national governments. Indicators have been designed to accommodate this reality.

Universality

The highest scores in the Barometer are achieved when governance, capability, and availability can be shown to be universal—when everyone in the country is covered by, or protected by, governance rules; when everyone has access to capabilities or the development of capabilities; when everyone has access to meaningful data; and when data use has impact for the public good across the country.

3. The Barometer Structure¶

The structure of the second edition of the Barometer was developed following intensive consultations with the research advisory committee and thematic partners. As highlighted above, this edition has new changes within the pillars, topic clusters, and cross-cutting themes. The following description offers a snapshot of how the study’s structure addresses all of these.

Figure 2 presents the overall GDB structure. At the center of it there are the indicators. These indicators are carefully and intentionally organized to define action areas, which in turn are organized to define clusters. Also, each one of the indicators is made by discrete sub-questions, which in turn are complemented with qualitative supporting questions.

Fig. 2

Thematic Clusters and Action Areas¶

For the second edition, we have adapted thematic clusters and action areas to provide a comprehensive assessment of the data foundations necessary for a healthy data ecosystem. For example, the GDB's clusters of Governance Foundations and Critical Competencies emphasize robust foundational frameworks while evaluating capabilities and steps to support effective data production, storage, publication, and use. These clusters focus on examining specific topics from various angles, providing a comprehensive understanding of the issues. The governance foundations cluster includes action areas such as data protection, data management, and data sharing. The critical competencies cluster includes the action areas of data literacy, and data reuse.

There is also a cluster on equitable access to assess if data is accessible and useful to all. This cluster addresses two action areas,accessibility and language coverage.

Along these, there are also clusters that were developed in partnership with expert organizations, which contain action areas and sectoral indicators. These clusters focus on the intersection of data within specific sectors, addressing long-standing issues of accountability, power, and money. They provide a nuanced view in these critical areas.

The Political Integrity cluster examines action areas such as political finance, right to information (RTI), interest and asset declaration, and lobbying. The Land management cluster focuses on land tenure and land use, highlighting the need for high quality data availability surrounding property and its use. The Company Information cluster investigates beneficial ownership and company registers, ensuring transparency in business operations. Finally, the Public Procurement and Public Finance clusters scrutinize public procurement processes and budget and spending practices, emphasizing the importance of data for public oversight. The figure 3 (below) presents the various action areas grouped under the thematic clusters.

By addressing these action areas in thematic clusters, the GDB offers a detailed approach to the current state of data foundations aimed at fostering healthy data ecosystems.

Fig 3.

Indicators¶

The GDB's indicators are designed to provide insight into data foundations for healthy data ecosystems. The GDB has primary and secondary indicators. While the primary indicators refer to data collected from the GDB’s expert survey, the secondary indicators refer to data collected from recognized secondary sources that offer complementary context to specific action areas.

The GDB`s primary indicators are composed of several discrete sub-questions designed to generate weighted scores on a 100-point scale. These primary indicators are framed as "to what extent" questions, providing an overarching scope for assessment. Each indicator is answered through specific sub-questions, which are organized into two sections: existence and extent, and elements. While the first group assesses the existence and coverage of the frameworks, capacities, and datasets; the second group assesses the characteristics of those frameworks, capacities, and datasets.

Further, many of the discrete sub-questions inside an indicator are also complemented by what has been defined as supporting questions. These questions allow the collection of qualitative data for further analysis.

By design, the GDB’s indicators are meant to:

Generate scores through discrete elements: Indicators are designed to generate scores through the use of discrete elements. This approach provides more structured, granular data and reduces unexplained variation between assessments.
Anchor in established agreements and practices: Indicators assess countries against benchmarks rooted in international agreements, such as the Sustainable Development Goals, and good practices.
Generate actionable data: We have worked with partners to understand how they might use the primary data generated by the Global Data Barometer, as well as how that data can support improved government practice.

The GDB library of indicators is available for consultation.

Cross-cutting Themes¶

Not all data foundations can be effectively addressed by focusing solely on specific indicators or action areas. Cross-cutting themes like inclusion, use of data, digital public infrastructure, and foundations for AI transcend individual thematic sectors. Integrating these cross-cutting issues into the GDB’s structure provides a comprehensive lens to assess the broader societal implications of data practices. By embedding these themes throughout our survey and analysis, we ensure that our approach reflects the interconnected nature of data in our societies and provide a richer and more insightful analysis of how data is being produced, stored, published and used.

This holistic perspective highlights how robust data foundations are essential for fostering healthy and sustainable data ecosystems as well as the wealth of information that the GDB will provide in this second edition. Some examples of the GDB’s cross-cutting themes are as follows:

Inclusion

Inclusion is a vital cross-cutting theme, focusing on gender and accessibility. This theme adds to the action areas and thematic clusters by addressing how inclusive data policies and practices are, considering the representation and participation of diverse groups, including women and individuals with disabilities. The GDB assesses whether data ensures that all segments of the population can access and benefit from data, thereby promoting equity and reducing disparities.

Use of Data

Use of Data includes sub-questions and supporting questions of primary indicators that assess how data is utilized and the support available for data re-use. This cross-cutting also incorporates a secondary use indicator from the first edition, "Data Use by International Organizations," to assess broader patterns of data use.

Measuring the impact of public data use is challenging, often relying on case studies that lack scalability and generalizability. Previous efforts, like the Open Data Barometer, focus on media coverage and research, which can skew results toward well-documented regions.

The variability in how countries document data use leads to inconsistencies, overrepresenting regions with better documentation. The Global Data Barometer's first edition used representative use-cases, but this limited approach missed many significant data uses. In this second edition of the Global Data Barometer, drawing on the lessons learned from the first edition, we have introduced data use as a cross-cutting theme to ensure our approach more effectively captures the interconnected nature of data in our societies. By addressing data use as a theme that spans across various action areas, we are able to provide a more comprehensive analysis of how data is utilized across different sectors in a qualitative manner. This thematic integration allows us to identify broader trends and gather qualitative information that would be missed by focusing solely on specific action areas. By examining data use throughout the entire survey, we aim to uncover deeper insights into the patterns and challenges associated with data practices globally, enabling us to offer more nuanced and actionable recommendations for policymakers and stakeholders.

Data foundations for Artificial Intelligence (AI)

This cross-cutting theme is examined through sub-questions related to the readiness and implementation of data for AI technologies, including how AI is integrated into fundamental components of existing data ecosystems (data protection, data sharing, data literacy, and data reuse.). Also, to the existing recurring sub-questions, aspects in secondary indicators, such as Digital government, from DGSS and new secondary indicators from sources such as the Global Index on Responsible AI (GIRAI) and other relevant frameworks are added to enhance the evaluation of AI readiness and implementation. By evaluating these aspects, the GDB provides insights into the progress and challenges of data foundations associated with AI deployment in different regions.

Digital public infrastructure

This cross-cutting theme spans all three pillars, integrating several indicators, sub-questions within those indicators, and scores for particular element clusters. In the governance pillar, this includes primary indicators such as Open Data Policy, Data Management, Language Coverage, and Accessibility Coverage. Sub-questions in this pillar cover how governments should share data with other sectors, require a verification process, and empower an agency or official to ensure accurate and timely data collection and publication. In the capability pillar, primary indicators include Open Data Initiative and secondary indicators like Government Online Services, Digital Government, and Data Institutions. The availability pillar focuses on element clusters like Kinds of Data and Data Fields & Quality, and includes sub-questions on whether datasets are available free of charge, openly licensed, in all official languages, supported by accessible tools, timely, updated, historically trackable, machine-readable, and complete without missing required data. This comprehensive approach ensures a thorough evaluation of digital public infrastructure across various dimensions.

Fig 4.

By providing a detailed and standardized assessment of these action areas, thematic clusters, and cross-cutting themes, the GDB offers a valuable tool for governments, policymakers, and stakeholders to understand and improve their data foundations. This, in turn, fosters more robust, inclusive, and transparent data ecosystems that can support sustainable development and innovation worldwide.

4. Data Collection¶

To measure country performance, the GDB employs an expert survey that was developed in partnership with leading organizations in different fields and drawing on an open, participatory process in order to have meaningful depth while also being broadly comparable, ensuring consistent and comparable evaluations across different regions. This tool has been built upon results from its first edition to better speak to data practices and policies, but also to refine and strengthen our survey instrument. The data collection is conducted by a network of local researchers. This primary data is enriched with secondary data from existing sources, ensuring a comprehensive and accurate assessment.

The GDB uses both primary and secondary data sources to collect comprehensive information on data governance, availability, capability, and use.

Primary Data: Collected through an expert survey implemented by a global network of partners and researchers. This survey includes country-wide indicators and thematic indicators focusing on specific sectors or public policy areas. The survey organizes the indicators by action areas to facilitate and run in parallel the data collection and reviewing processes.
The data collection process is carried out through the Survey Solutions software deployed for this study. Along with this document, researchers and reviewers receive a Survey Tool Manual on how to use the Survey Tool for the data collection, with all the details on the technicalities of the Survey Solutions software and workflows for the quality assurance process. Further, to guide researchers in depth through the data collection process, the GDB has the Research Manual, and the Reviewing Manual to guide the different layers of reviewing that provide reliable and quality data.
Secondary Data: Drawn from existing sources and a complementary government survey. This data helps to validate and augment the primary data collected through the expert survey .

Scoring/Calculations¶

The scoring for the GDB is based on a 0-100 scale, where 100 represents best practices as defined against internationally agreed norms or frameworks. Indicators are made up of structured sub-questions and are backed with qualitative evidence to provide a deeper understanding of each country's context .

5. Methodology Governance¶

The GDB methodology is governed by principles that ensure its robustness and reliability:

Transparent Process: The entire methodology, including data sources and scoring criteria, is transparent and open for review .
Inclusive Participation: The methodology is developed and refined through consultations with a broad range of stakeholders, including researchers, policymakers, and civil society organizations .
Continuous Improvement: The methodology is regularly updated based on feedback and new developments in the field of data governance and management .

By adhering to these principles, the GDB aims to provide a reliable and comprehensive assessment of data governance practices around the world.

Research Advisory Committee¶

One of the key accountability systems adopted by the Global Data Barometer is putting in place a Research Advisory Committee. The committee plays a crucial role in guiding and overseeing the research process and methodology. The committee has the responsibility to hold the Global Data Bata Barometer to high standards of methodological rigor.

The committee is responsible for reviewing the proposed weighting of the various survey components and providing feedback. The committee members offer feedback to hold the project accountable and strengthen its methodological stability.

[^1]: Data use remains part of the GDB, but in the second edition is a cross-cutting theme. This theme is further described in section 3 of this document.