Methodology

Overview

This overview is intended both for researchers and reviewers involved in generating and validating the Barometer's survey data, as well as those seeking to understand the overall approach of the Barometer and the data that the study produces.

As the Global Data Barometer investigates data policies and practices across the data spectrum, particularly those that can be influenced by government action, before moving forward into the modules and indicators, it is important to be familiar with foundation concepts behind the Barometer: the Data Spectrum, the Open Data Barometer (ODB) and the Open Data Charter Principles.

Structure & Process

The Barometer is a multi-dimensional index, comprised of components and subcomponents; these components and subcomponents are in turn built from composite indicators that combine primary and secondary data. The goal is to develop an overall comparative assessment of the extent to which countries (or regions) govern and use data for the public good. Consequently, the Barometer breaks down the concept of data for the public good into various individual components and sub-components, each assessed separately. Results are then aggregated into an overall score.

The Global Data Barometer is based around four pillars or foundational areas of assessment: governance, capability, availability, and use and impact.

• Governance is concerned with whether there are rules, processes, and institutions in place both to make data available for the public good and to safeguard that data against misuse. Governance indicators generally look at the country’s legal and policy frameworks that support data ecosystems.
• Capability is concerned with whether the country has the means, connectivity, skills, and institutional capacity to create, share, and use data for the public good. The majority of capability indicators will be drawn from secondary data, but additional capability questions are included in the survey.
• Availability is the largest section of the primary survey and explores whether certain categories of data are available, shared, and of adequate quality to allow reuse for the public good.
• Use and impact is concerned with finding evidence of particular uses of data and their impact in the country.

Structurally, the study is organized into core and thematic modules. Core modules assess the overarching country context, while thematic modules look at the interaction of governance, capability, availability, and use in specific domains, such as political integrity, climate action, and land. Modules vary in size, reflecting both the particularities of different data ecosystems and the pilot nature of the Barometer as we seek to explore both the depth and breadth of assessing data for the public good.

Finally, the Barometer explores several cross-cutting issues directly, typically investigated through sub-questions within indicators. These include: equity and inclusion, COVID-19, and emerging AI practices. The data the Barometer gathers also speaks meaningfully to sustainable development, open data, and data for development.

Indicator Design

We have designed the Barometer’s indicators to:

• Generate scores through the use of discrete elements. This is a change from the Open Data Barometer (ODB) scoring method that asked experts to provide a 0–10 score, and responds both to a desire from partners for more structured, granular data, and to past feedback that the previous scoring method led to unexplained variation between assessments. In the Global Data Barometer handbook, you will see the sub-questions under each indicator used to generate indicator scores. The exact calculation of scores is not included, but will be shared later in the year following further consultation.
• Anchor in established agreements and practices. For the majority of indicators, a review of the literature, international standards, and agreements has been completed to ensure that indicators assess countries against reasonable benchmarks that are rooted in applicable international agreements or commitments. In particular, we draw on the Sustainable Development Goals as the basis for international agreement on public goods.
• Identify bright spots and recognize different systems of government. We have sought to increase the sensitivity of indicators to federal systems and to cases where good practices may exist at an agency or sub-national level, even if the practice is not yet widespread. We have also sought to avoid cultural or high-income country biases in the design of indicators by utilizing examples and evidence from a wider range of settings.
• Generate actionable data. For a number of themes, we have worked with partners to understand how they might use the primary data generated by the Global Data Barometer, as well as how that data can support improved government practice.
• Maintain continuity with the ODB. Certain indicators and selected indicator sub-questions have been included in order to build upon past editions of the ODB and provide continuity with former study metrics, including assessing the presence of open data frameworks, initiatives, and datasets. In some cases, new secondary sources have been identified to replace previous ODB questions; therefore, not all ODB data categories are necessarily reflected in the Global Data Barometer handbook.

Government Survey

To augment the results of the expert survey and support a greater depth of analysis, we are also conducting a complementary government survey. In parallel with the expert survey, governments are invited to provide evidence that a researcher might not find on their own. These answers will be shared with reviewers, who will determine whether government-provided evidence requires researchers to update their assessments.

Running the expert survey and government survey in parallel will allow researchers to move forward with their assessments, while also reducing the impact of potential disparities in the quality and depth of government responses.