General Guidance

The handbook is designed to be a one-stop resource on the GDB assessment process, including detailed information on methodology, general guidance to researchers, sources to be used, as well as detailed question-by-question guidance for researchers. Here you'll find both general guidance, about the research parameters and process overall, as well as guidance specific to each indicator.

Time Period for the Study

Each edition of the Barometer will focus on a particular, defined period of time. Researchers are asked to examine evidence and sources that apply specifically to this period; answers should assess the state of data within the country only during this defined period, unless otherwise stated.

• The 2021 pilot edition of the Barometer covers the state of data policies and practices between May 1st, 2019 and May 1st, 2021.

Therefore, for questions regarding laws, policies, strategies, capacity-building interventions, dataset availability, or uses of data, answers should only consider examples that have either occurred or were current during this two-year period.

Sometimes, determining whether or not an activity remained current during the relevant time period may seem complicated, particularly with regard to questions of capability and use. While we recognize that the effects of successful capacity-building or impactful use are, after all, often long-term by design, for an activity to be assessed for this edition of the Barometer, the activity must have been ongoing during the period of the study.

For example:

• A country that had a Data Strategy 2015–2019 but does not have a more recent strategy document, would not be assessed as having a data strategy in this study period.
• A country that has a Data Strategy 2018–2022, originally published in 2018, but for which there is evidence of continued activity or implementation, would be assessed as having a data strategy in this study period.
• Updates to a country's data portal on June 1st, 2021 would not be counted in the assessment for this study period.
• A government’s data literacy program that was held on April 30th, 2019, would not be counted in the assessment for this study period.
• To answer “Yes” to a question based on a use of data first reported in an article in 2016, a researcher would need to provide a citation or source showing that this use remains active or has meaningfully developed in recent years.
• If COVID-19 has affected data availability at any point during the study period, this would count as a case of COVID-19 affecting data availability, even if changes have been reversed by May 1st, 2021.

Wherever possible, researchers should cite contemporaneous sources from the defined period.

The Research Process

The Barometer uses an iterative process to generate robust, validated data. The following diagram and sequence of steps outline the data collection process.

Research training: Training sessions introduce regional coordinators and country-level researchers to the Barometer’s methodology and help establish consistency.

Research mini-testing: Researchers complete two indicators, which coordinators then review, providing feedback with regard to both form and substance. Researchers make any corresponding updates and apply what they have learned to the rest of the research process.

Research underway: Researchers perform desk research and consult with key individuals to answer the survey questions, providing justifications for each answer.

Research spot-checks: Coordinators monitor the research progress, carrying out spot-checks as responses are completed and providing feedback on areas to improve.

Data collected: Researchers have completed the full survey; coordinators check to make sure all expert survey questions have been answered and the survey tool has been completed properly.

Peer review: To validate the data, researchers then check the answers provided by colleagues in a crossed peer-review mechanism, offering comments or questions where required, or requesting additional detail in the justifications. To inform their review, reviewers will have access to any relevant evidence provided in response to the complementary government survey. The original researcher implements or addresses any reviewer feedback as appropriate.

Thematic review: During the peer review period, the core Barometer team also organizes a followup review of the assessments by thematic experts. Thematic experts may flag certain responses for additional research, correction, or clarification, and provide notes for the researcher to respond to.

Research completion: Reviewers and coordinators check the updated surveys to ensure that feedback has been addressed. The coordinator makes a final check and closes the data collection process.

Research Strategies

Starting points and searching

For each primary indicator, specific research guidance contains suggested starting points; these may include sources to check, searches to run, and individuals to consult.

As much as possible we have linked to existing information that researchers can check for evidence relevant to the indicator and its sub-questions. The search suggestions provided should be run on the most relevant or best available search engine for the country; these will need to be adapted and translated into relevant languages. Before setting up any interviews, researchers may wish to review the entire set of indicators to determine what interviews might be most useful to conduct. Researchers should also use their own expertise to identify other ways to answer questions.

Sub-questions have been designed to support keyword searching of documents and rapid review of datasets. We anticipate researchers will make extensive use of Ctrl-F / Cmd-F keyboard shortcuts to search inside documents and then check surrounding context, rather than reading every source document in full.

Time management

Completing the Barometer survey may take very different amounts of time from country to country, depending on a number of factors:

1. Whether or not the subject an indicator focuses on exists in the country. For example, if a researcher concludes that there is no legal or regulatory framework for publishing political party finance data at all—or that there are no independent political parties in the country, making the questions moot—answers to subsequent sub-questions would all be “No.” In these cases, the researcher only needs to write a research journal entry explaining their answer to the existence question plus a short justification.

2. The availability of online evidence. Some countries publish all of their laws in a single searchable online location and provide comprehensive national data portals; this will make keyword searches and dataset reviews more straightforward. In other countries, finding copies of laws or tracking down datasets may take more time. Similarly, the quality of a government’s websites and search engine indexing may also affect how long research takes. Additionally, when evidence is difficult to find online, researchers may need to conduct interviews with experts, which can affect how long research takes.

3. Levels of centralization or decentralization. For some countries, it may be important to check for each question not only the national government but also a number of sub-national governments in order to locate bright spots. In other countries, only the national government may be relevant to check.

4. Translation requirements. For some countries, researchers will need to identify the correct translation of key terms into the relevant local languages before searching.

We anticipate that, for most countries, these time variations should balance out.

As a general guideline, completing each question should take between 30 minutes and 3 hours. If you’re finding that questions take more than 1-½ hours to complete—on average—you may need to adapt your research strategies. The research journal can be used to indicate cases where it was hard to find a conclusive answer to a question on the first pass of research.

Researchers may also wish to experiment with researching first whether or not the subjects of a range of indicators exist, and then returning to explore other sub-questions in depth; this approach can help you become familiar with possible sources.