Indicator Guidance

Almost all primary indicators in the Global Data Barometer are framed as to what extent questions.

Indicators are made up of a number of discrete sub-questions, which will generate weighted scores on a 10-point scale. In the pilot edition, these scores are not displayed to researchers.

Typically, indicator sub-questions are organised in three sections: existence, elements, and extent, and each indicator has a research journal and supporting evidence section.

Question status and confidence

Status

At the top of each indicator there is a space to indicate the status of your research. Mark it as 'Not started', 'Draft', or 'Complete'. Reviewers will be able to also mark content as 'Reviewed'.

Use this to keep track of your research work. The completed status of indicators is also visible to the research coordinator in their management dashboard.

Confidence

For each indicator, researchers can provide a personal assessment of the confidence in the answers given. This is a useful way of indicating to reviewers whether the answers are highly likely to be accurate, or if there are concerns that all answers have not been able to be confirmed in the time available for research. Use the research journal to briefly record the reasons why, for any indicators, there is lower confidence in the accuracy of the answer.

The research journal

In conjunction with each indicator, the survey tool offers a research journal. Researchers should use this to briefly record the steps they take to carry out their research. The research journal is not published, but is used to support the review process.

For example...

In this example, the research journal entry describes the steps taken to respond to an indicator:

"Followed source guidance to look at DLA Piper Data Protection Laws database & Greanleaf tables of privacy laws: used this to review Part 1 elements.

Searched with Google for 'UK Data Protection Migrants' based on past awareness of 'immigration exemption' and looked at first three results. Sources used included in the supporting evidence section.

Searched for 'Data Protection COVID-19' and reviewed content from the ICO: looked for best source to use, and confirmed dates of documents.

Checked results when talking with civil society informant on May 22nd to discuss a number of different questions. Discussed whether migrants judgement should be 'No' or 'Part' - see justification for conclusion."

Existence sub-questions

For each indicator, the handbook contains a 'Definitions and identification' section, describing the governance arrangements, capabilities, types of data, or use cases to look for. The researcher should check carefully whether anything meeting this definition exists in the country and should locate the best possible examples to assess. The existence sub-question will then ask for an assessment of the nature of the example identified.

Handling null results

If the researcher confirms that no examples that are the focus of the indicator exist, or they cannot locate evidence of existence, then they must include an explanation in the research journal explaining the null result, providing a short justification and source if available. In most cases, a null result to the existence question will mean the other sub-questions do not need to be completed.

For example...

Justification: A data protection law has been proposed a number of times by civil society, and introduced as a draft by opposition parties. However, it has no reasonable prospect of becoming law.

Source: Article describing history of campaign for data protection law in country.**

Locating the best example

Unless otherwise specified in indicator guidance, researchers should first look for national examples (e.g., national law, nationally provided dataset) from the government. If no such example is identified, or the example identified appears weak (e.g., many other sub-questions are answered 'No'), the researcher should check for the presence of sub-national examples, or in the case of datasets, examples of data provided by other sources than by the government.

Element sub-questions

Once the relevant governance arrangements, institutions, datasets, or use cases to assess have been identified, the elements sub-questions invite researchers to carry out a detailed assessment.

In general, elements are written as statements, and the researcher will need to look closer at documents, datasets, or other evidence to confirm whether the statement is true or false. In cases where the statement is only partially true, or the researcher has doubts about how far the statement applies, they may answer that it is 'part' true.

Some elements have supplementary questions that invite structured meta-data, sources, or justifications. Some of these are mandatory and must be completed (marked as “Required”), while others are marked as “Optional” and should be completed if the researcher has the information available.

Extent sub-questions

An indicator's extent sub-question is designed to explore whether what’s being evaluated demonstrates comprehensive coverage or whether it has limitations. For example, a capability question might ask whether a country’s data science training is available across the country or only in a limited number of locations or an availability question might ask whether a dataset that scores highly in the elements checklist is an exceptional outlier in a federal system, or an example of the norm.

Supporting evidence

The supporting evidence section allows you to attach up to five structured links or files that should be published alongside the indicator data, that provide evidence for the given assessment, and that can be checked by reviewers, This might include:

• Links to policies, strategies, or laws for a governance indicator
• Links to datasets for an availability indicator
• Links to reports or discussions on data capability
• Copies of PDF reports where they cannot be linked to online
• Links to academic papers

Evidence should be in the public domain and should not include private documents. Each item of evidence can have a title and either attached file or URL.

Justifications & sources

For every indicator, researchers should provide a list of the supporting evidence used and write a short prose explanation of the example assessed, and how certain judgements have been made.

Justification

The justification should include brief notes to support the assessments made, citing numbered items of supporting evidence to back up each key point. This justification is first used by the reviewers to check your assessment, and is then published alongside the raw data from the survey to support re-users of the data to understand the basis for each assessment.

Justifications should be written in clear english prose that doesn't use first person, but rather is written in a neutral impersonal way. Use a spellchecker or grammar checker if required.

You will need to create your own justifications based on the findings of your research, using content quotations when appropriate to support your argument but not relying solely on them. All justifications need to be self-contained and self-explanatory, with no cross-references between them. While sources must be cited to support justifications, a reader should be able to fully understand the justification without looking at the supporting evidence.

For example:

"Companies House provide a range of open data products, including summary data in CSV form, API access to detailed company records, and accounts data in XML and iXBRL formats [1].

They also provide a URI for each company, which will return data in a variety of formats including RDF, JSON and XML [1].

This data appears to all be under the Open Government License, although license details are not clearly stated on download locations for the data, and the data.gov.uk portal indicates an alternative Creative Commons Attribute license [2].

Data on directors is available through the API which requires a free account, and is subject to rate limiting [3]. However, this does not appear to have prevented bulk use of this data by third parties

A specific accounts data product [4] is available which covers approximately 60% of companies filing electronically.

Shareholder data is available in unstructured form through annual returns, and for shareholders with significant control (>25%) is available in structured form in the Persons of Significant Control data [5]."

You should always provide a justification, even when you answer 'No' to an existence question.

Citation guidance

Citing desk research and interviews

Selected sources should always reflect the project’s study period.

• Exercise professional judgment in determining whether opinions of an interviewee are factual and accurate. We strongly suggest researchers corroborate information obtained in interviews with desk research and do not rely on a single personal opinion.
• Please use exactly the following format when citing interviews: Interview sources must include the full name of the interviewee, the name of the interviewee’s employer, and job title. Example: (Interview with Jane Doe, Ministry of Justice, Director General).
• Anonymity: When it is not feasible to publish the name of an interviewee (out of justified fear for the interviewee’s safety or negative professional ramifications), please include the name of the interviewee in the “Research Journal" box of the survey. In the “Justifications” box, simply state “Anonymous” for the interviewee name while providing as many other details as possible (e.g. Anonymous, Ministry of Finance, government official). Global Data Barometer will maintain that confidentiality and no names will be published that are submitted in the “Additional notes” box.
• Please use the following format when citing media articles: name of author, name of publication, title of article, date published, and a URL/hyperlink (or a digital/PDF attachment if a link is not available or has expired).
• Please use the following format when citing journal articles, written documents, or other third-party desk research: provide the full citation and whenever possible please provide a URL/hyperlink (or attach an electronic copy if a link is unavailable).

Citing laws or regulations

• Researchers must identify the law (full name, article, etc.), case law, specific legal articles, or statutes and provide a direct quote or a detailed explanation of the law in the comments box, as needed.
• Researchers must refer, where appropriate, to case law, specific legal articles, or statutes.
• Researchers are to reflect and describe applicable traditional and customary law when necessary.
• Laws cited must be domestic laws rather than international law.

• Please use this specific format when citing laws or regulations: Name of Law, Article, Section Number, Year, Hyperlink to law. \ Example: Constitution Acts, Part I: Canadian Charter of Rights and Freedoms, Section 6. 1982. http://laws.justice.gc.ca/eng/Const/page-11.html#sc:7:s_6

  Example: Constitution Acts, Part I: Canadian Charter of Rights and Freedoms, Section 6. 1982. http://laws.justice.gc.ca/eng/Const/page-11.html#sc:7:s_6