Availability: Public procurement data

To what extent is detailed structured data on public procurement processes available as open data?

Definitions

Definitions and Identification

Governments enter into many different contracts for the provision of goods, services, and public works. They may publish data about these contracts in tender lists or through contract finder websites, procurement portals, or open data portals.

In some countries, public procurement data may be held in a single system. In others, different stages of the procurement process (planning, tender, award, implementation) may be held in different datasets, and a government may publish "notices"—with or without identifiers—that make it possible to connect data from different stages of the procurement process.

For example:

• Contracts Finder in the UK provides a JSON API and data dumps from a database designed to aggregate tender and award information for all government procurement above a given threshold. It provides machine-readable data and offers an OCDS export. However, a sample export of records reveals no links to spending, that documents are often missing, and company identifiers are only provided in some cases.
• In Portugal, open contracting data is published for public works projects, covering tender, award, and contract implementation; however, checks show that dates are often missing from this data.
• The Zambia Public Procurement Authority hosts a platform that contains data on roughly 1000 procurement processes run through their e-procurement platform.

Research Guidance

Look for services that aggregate data from across government, not just single departmental websites. However, if no such service is available, check a selection of the biggest government departments and note if they publish their contract data in any form.

Starting points

• Sources:
  • The Open Contracting Partnership maintains a map of cities and countries publishing procurement data using the Open Contracting Data Standard (OCDS).
  • The country profiles in the Global Public Procurement Database provide links to procurement agency websites, national e-procurement systems, and, where available, links to OCDS data.
  • Columns EB, EC, ED, and EE of the data spreadsheet from the World Bank Doing Business module on Contracting with Government include links to where public works contracts for roads are posted online for each country (based on data gathered before May 2020). This can be used to check whether public works contract data is in the same portal as other tender information.
  • OCDS Downloads gathers data in the Open Contracting Data Standard for many countries, and shows which sections and fields of the OCDS file are populated with data.
  • The Managing COVID Funds project of the International Budget Partnership, published May 2021, includes information about pandemic-related procurement guidance and information; see particularly the answers to questions 18 and 19 in your country's questionnaire.
• Search:
  • National data portals for "contracts" or "procurement" datasets;
  • The website of the national procurement agency for open data, APIs, or data exports.
• Consult:
  • Transparency experts or journalists writing about procurement; ask about known limitations of contracting data.

What to look for?

To complete the assessment for this question you will need to access and explore the available data. This may involve running queries on datasets to check the variety of procurements included.

Look for evidence of:

• The stages of the procurement process for which data is available—check for details of contact awards and implementation information (spending and performance).
• Goods and services contracts—these can range from low-value to high-value contracts, covering a wide range of supplies to government.
• Public works contracts—these are often higher-value contracts, involving construction work; for example, building schools and hospitals, roads, and other infrastructure.
• Procurement from a range of government departments—does the data appear to contain only procurement from a single department? Or from across government more broadly?
• Procurement from sub-national government units—if you find procurement for sub-national entities in the data, does this appear to be comprehensive, or could it just be voluntary publication by a few local government units?
• Bulk data access via downloads or APIs. Can you, for example, export a search as XLSX and does the resulting data contain relevant data fields? Is there documentation for an API that allows access to full data records?
• Persistent data—can you find data from last year? Or the year before? Does it appear that old data is being archived? Or does old data expire from the platform?
• The data is structured according to an established standard—does the data follow a standard such as the Open Contracting Data Standard (OCDS)?

Dashboards and other public analytic tools may help you to assess the comprehensiveness and coverage of the data.

National and sub-national considerations

Even within federal countries, national governments will carry out significant procurement activities. In some federal systems, national government (or supranational institutions) provide portals that centralize tenders and other procurement data.

Focus on national government first, and then assess whether:

• National datasets also include data from sub-national or local government units;
• Equivalent data exists for a selection of sub-national or local government units, but is not nationally aggregated;

If there is no evidence of procurement data being available at the national level, but there is a strong example of data availability from a sub-national government, or a specific agency, you may carry out an assessment for this data, and use the question on its coverage to note that this only covers a very limited number of procurements.

Question sub-components

Show/hide supporting questions

Existence

• Is this data available online in any form?
  • Data is not available online.

    Supporting questions: Are there other offline ways to access this data in the country? (e.g., attending an office to inspect it).

  • Data is available, but not as a result of government action.

    Supporting questions: If government is not providing access to data, how is this data available? Please provide a URL(s) for where this data can be found.

  • Data is available from government, or because of government actions.

    Supporting questions: Please provide a URL(s) for where this data can be found.

Elements

• Kinds of data:

• Procurement related to goods and services is included. (No, Partially, Yes)

  Supporting questions (conditional)

  If Partially or Yes: If public procurement data can be found in multiple datasets, please provide the URL(s) where data on goods and services is located.

  If Partially: Please briefly explain your 'Partially' answer.

• Procurement related to public works is included. (No, Partially, Yes)

  Supporting questions (conditional)

  If Partially or Yes: If public procurement data can be found in multiple datasets, please provide the URL(s) where data related to public works is located.

  If Partially: Please briefly explain your 'Partially' answer.

• The planning phase is covered. (No, Partially, Yes)

  Supporting questions (conditional)

  If Partially or Yes: If public procurement data can be found in multiple datasets, please provide the URL where data on planning phase is located.

  If Partially: Please briefly explain your 'Partially' answer.

• The tender stage is covered. (No, Partially, Yes)

  Supporting questions (conditional)

  If Partially or Yes: If public procurement data can be found in multiple datasets, please provide the URL where data tender stage is located.

  If Partially: Please briefly explain your 'Partially' answer.

• The award stage is covered. (No, Partially, Yes)

  Supporting questions (conditional)

  If Partially or Yes: If public procurement data can be found in multiple datasets, please provide the URL where data award stage is located.

  If Partially: Please briefly explain your 'Partially' answer.

• The contract implementation stage is covered. (No, Partially, Yes) Answer 'Yes' if you can locate any data from after contract award and signature, such as spending transactions, confirmation that goods or services were delivered, contract amendments, or data on contract performance.

  Supporting questions (conditional)

  If Partially or Yes: What kind of implementation data is available?

  If Partially or Yes: If public procurement data can be found in multiple datasets, please provide the URL(s) implementation data is located.

• Data fields and quality:

• The data contains identifiers or other features that connect together data on each stage of a single procurement process. (No, Partially, Yes)

  Supporting questions (conditional)

  If Partially or Yes: Please briefly explain your answer.

• The data contains names and unique identifiers for companies awarded contracts. (No, Partially, Yes)

  Supporting questions (conditional)

  If No or Partially or Yes: Please, briefly explain your answer.

• The data contains start and end dates for tender processes and/or contracts. (No, Partially, Yes)

  Supporting questions (conditional)

  If Partially or Yes: Please, briefly explain your answer.

• The data contains the value (cost) of each tender, award, or contract (No, Partially, Yes)

  Supporting questions (conditional)

  If Partially or Yes: Please, briefly explain your answer.

• The data contains, or can be linked to, information on spending against the contract. (No, Partially, Yes)

  Supporting questions (conditional)

  If Partially or Yes: Please, briefly explain your answer.

• The data contains a description of the goods, services or works being procured. (No, Partially, Yes)

  Supporting questions (conditional)

  If Partially or Yes: Please, briefly explain your answer.

• The data is published according to one or more relevant data standards. (No, Partially, Yes)

  Supporting questions (conditional)

  If Partially or Yes: Which standards are in use?

  If Partially: Please briefly explain your 'Partially' answer.

• The data contains links to accessible tender, award, or contract documentation (No, Partially, Yes) Answer 'Yes' if it is possible to follow links and download documentation (e.g., tender details, text of contracts) without barriers such as registration or login for all stages of the process covered by the data. Answer 'Partially' if linked documents are accessible, but there are barriers to easy access, or documents are only available for some of the states in the data. Answer 'No' if no links are available.

• Data openness, timing, and structure:

• Dataset is available free of charge. (No, Partially, Yes)

  Supporting questions (conditional)

  If Partially: Please briefly explain your 'Partially' answer.

• Data is openly licensed. (No, Partially, Yes)

  Supporting questions (conditional)

  If No: If there are explicit restrictions placed on re-use of the dataset, briefly describe those here.

  If Partially or Yes: If the data is provided with an explicit open license, please provide the name of the license, or a link to it here.

• Data is available in all the country’s official or national languages. If the country has no official or national languages, data is available in the major languages of the country. (No, Partially, Yes) Assess this against the list of official, national, or in-use languages you provided as part of your response to the governance indicator that asks, "To what extent do relevant laws, regulations, policies, and guidance require that data collection and publication processes be available in the country’s official or national languages?"

  Supporting questions (conditional)

  If Partially or Yes: Please briefly describe the language coverage available.

• There are accessible and open official tools available to help users explore data. (No, Partially , Yes) Answer 'Partially' if tools make it possible to get at extracts of data without having to download a full dataset. Answer 'Yes' if there is an interactive tool that displays user-filtered extracts of the data to answer simple questions without downloading data at all.

  Supporting questions (conditional)

  If Partially or Yes: Please provide URL.

  If Partially : What are the main barriers to accessibility and usability?

• Data is timely and updated. (No, Partially, Yes)

  Supporting questions (conditional)

  If Partially or Yes: When was the most recent update to this dataset?

• Historical data is available that allows users to track change over time. (No, Partially, Yes)

  Supporting questions (conditional)

  If Partially: Please briefly explain your 'Partially' answer.

  If Partially or Yes: For what time period(s) (e.g., start and end dates) is data available?

• Data is provided in machine-readable format(s) (No, Partially, Yes)

  Supporting questions (conditional)

  If Partially or Yes: Please provide a URL where this machine-readable data can be found. (Additional URLs can be included in the justification and supporting evidence)

  If Partially or Yes: Please provide a comma separated list of the formats available? (E.g. csv, json)

  If Partially: What prevents you from assessing this data as fully machine-readable?

• The machine-readable dataset is available as a whole (No, Partially, Yes) Answer no if it's only possible to access individual records; Answer partially if it's possible to export extracts of the data; Answer yes if there are bulk downloads or APIs providing access to the whole dataset without financial, technical or legal barriers.

  Supporting questions (conditional)

  If Partially or Yes: Please provide a URL where bulk download access is available or described.

  If Partially or Yes: If bulk access is provided through an API, please provide a link to where the API is described.

  If Partially: Please briefly explain your 'Partially' answer.

• Negative scoring:

• This information is missing required data. (There is no evidence of data gaps., There is evidence that a portion of mandated data is missing., There is evidence of widespread omissions in mandated data.) In cases where the indicator itself identifies a dataset(s) to assess against or a separate governance indicator has asked you to determine data requirements of a relevant governing framework, assess against that. In cases where there is no such identified dataset(s) or related governance indicator, assess based on the parameters laid out in the publication of the information (e.g., are some fields entirely empty when they shouldn't be?), your local knowledge (e.g., if the data is supposed to include information for all public officials, does the number of total entries look right?), and any broader research you may have done for this theme (e.g., have media articles decried the incompleteness of the data?).

  Supporting questions (conditional)

  If There is evidence that a portion of mandated data is missing. or There is evidence of widespread omissions in mandated data.: Please briefly explain.

• The availability of this data has been affected by government response to COVID-19. (No, Partially, Yes)

  Supporting questions (conditional)

  If Partially or Yes: Please briefly describe how COVID-19 affected the availability of this data.

Extent

• How comprehensive is the data assessed for this question?
  • The data assessed covers a very limited number of public procurements and few other sources of data are available.

    Supporting questions: Please briefly describe the nature of the procurements the data covers.

  • The data assessed covers, or is representative of the data available for, a significant number of public procurements but there are large gaps in coverage.

    Supporting questions: Please briefly describe any notable gaps in coverage.

  • The data assessed covers, or is representative of the data available for, a large proportion of public procurement but some gaps in coverage exist.

    Supporting questions: Please briefly describe any notable gaps in coverage.

  • The data assessed covers, or is representative of the data available for, almost all public procurement.

    Supporting questions: Please briefly describe any notable gaps in coverage.

Indicator Justification

A wide range of stakeholders may use public procurement data, from private firms seeking government contracts, to civil society organizations monitoring procurement processes, to governments using their own data to get better value for money. Numerous agreements, including the G8 Open Data Charter, G20 Anti-Corruption Working Group Open Data Principles, and others, recognize contracting data as an essential open dataset that can bring social and economic benefits.

Many use cases for public procurement data require that the data connects across the different stages of the procurement process. Many use cases also rely on the availability of a number of core data fields, and benefit from the ability to links to other datasets. This indicator draws on the Open Contracting Data Standard, developed to support reusable public procurement data.

The 3rd, 4th, and Leaders Editions of the Open Data Barometer included a data availability indicator on public contracts, focused on award data. Our current, updated indicator supports the disaggregation of data on stages of the contracting process, allowing a more-or-less directly comparable benchmark (availability of award data) to be generated. However, this indicator is also sensitive to the availability of tender, award, and contract performance information, as well as to the completeness of the available dataset (that is, whether it represents just a few procurement processes or all the procurement processes carried out by a country).

Consequently, the GDB version of this indicator may allow countries which only make tender information available as structured or open data to score more highly than they did in the ODB (which would have given a zero score to a country with no award information). This current GDB indicator will also lead to countries that only make award information partially available and don't provide contract performance information achieving lower scores than the comparable ODB indicator.