This post in our technical series explains how to use OCDS to assess the quality and performance of a procurement system. Check our this introduction why MAPS can help us determine how well a procurement system aligns with the principles of open contracting, spot opportunities to introduce open contracting reforms, and track whether they have a positive impact as we’d expect.

If you’re interested in determining the strength of a public procurement system, there’s a great universal tool called the Methodology for Assessment of Procurement Systems (MAPS), which can be used to evaluate the procurement system of any country, regardless of its development level. A new, improved version of the methodology was recently released by its creators, a diverse group of stakeholders. We can calculate the quantitative indicators of MAPS using the Open Contracting Data Standard. This blog explains how, applying a reusable methodology to real OCDS data from Ukraine and Paraguay.

What is MAPS?

The MAPS methodology is built around 4 pillars and 14 indicators that correspond to aspirational characteristics of a public procurement system. The 4 pillars are:

• Legal, Regulatory and Policy Framework
• Institutional Framework and Management Capacity
• Public Procurement Operations and Market Practices
• Accountability, Integrity and Transparency of the Public Procurement System

The indicator framework is complemented by

• a user’s guide
• country context analysis
• 6 supplementary modules that allow focused assessments of specific areas

MAPS can be used as a tool for improvement as well because it identifies the weaknesses of a procurement system. In the last few years, MAPS has been through a revision process and a new version has been published; you can find the updated documentation on its official website.

Many of the required quantitative indicators can be calculated by using data in OCDS format. The advantage of using OCDS data already built and published by a government entity is that most of the data needed is already comprised into one dataset.

Another advantage to standardized data is that the methodology described below can be used to calculate MAPS quantitative indicators for any OCDS publisher. So, another reason that governments should publish OCDS, is that it will be easier for them to assess the performance of their procurement system.

Deep Dive: An introduction to using OCDS to calculate MAPS indicators

Before using OCDS data to calculate the quantitative MAPS indicators, there are some initial considerations you should take into account. We’ll explain these here, then calculate a single indicator with open contracting data from Ukraine and Paraguay, and end with a summary of what we learned in the process, so you can use the information to conduct your own calculations.

According to our research, the following are the MAPS quantitative indicators that can be calculated by using OCDS data:

1. Key procurement information published along the procurement cycle (in % of total number of contracts) (sub-indicator 7(a)(c)):
   1.1. Invitation to bid (in % of total number of contracts)
   1.2. Contract awards (purpose, supplier, value, variations/amendments)
   1.3. Details related to contract implementation (milestones, completion and payment)
2. Annual procurement statistics
3. Uptake of e-Procurement (sub-indicator 7(b)(a))
4. Number of e-Procurement procedures in % of total number of procedures
5. Value of e-Procurement procedures in % of total value of procedures
6. Total number of contracts (sub-indicator 7(c)(d))
7. Total value of contracts (sub-indicator 7(c)(d))
8. Public procurement as a share of government expenditure and as share of GDP (sub-indicator 7(c)(d))
9. Total value of contracts awarded through competitive methods in the most recent fiscal year (sub-indicator 7(c)(d))
10. Share of contracts with complete and accurate records and databases (in %) (sub-indicator 9(c)(g))

We do not intend to show the calculation of all indicators in here, but if you are interested in more examples, you can find our experiments with open contracting data here:

• Ukraine: Collaboratory Notebook, Github Gist
• Paraguay: Collaboratory Notebook, Github Gist

To download and import the data into a database and then query the datasets we used OCDS Kingfisher. The SQL syntax used here belongs to PostgreSQL 9.6+ with OCDS releases as JSONB columns. However, it should be easy to translate the logic used into other languages and tools you may want to use.

Preparation

Before actually attempting to calculate the indicators, there are a few initial steps we need to take:

• Ensure that we have complete data.
• Plan how we will deal with different currencies.
• Decide a query strategy for OCDS releases, if needed

Dataset completeness checks

The first thing you should do in any analysis is to check that you have a complete dataset: missing data can invalidate all the calculations you make. For the present case, at a minimum, you should check:

• Publication dates: start, end, and patterns along time. Gaps in publication dates along time could mean loss of data: investigate if necessary. Alternatively, if publication dates are missing or unavailable, you can use tender dates to check that public procurement processes happen regularly over time.
• Award and contract values: these should be always present to measure the indicators. If a significant part of the dataset have missing values, you may be unable to proceed with calculations.
• Use of OCDS release tags: verify that at least the tags tender and award are present and if they are used in a way that makes sense.

Also, check if these fields and sections are present in data and if they are correctly used:

• • tender/procurementMethod

• • tender/submissionMethod

• • tender/documents and tender/documents/documentType

If you are missing one or more, you may be unable to calculate some of the indicators; although there might be alternatives as we will see later.

Currencies

In public procurement, chances are that more than one currency is used. As some indicators require us to calculate totals, you will need to decide how to convert all values to a single currency so they can be summarised.

We recommend to start by choosing a currency to convert all values into: usually the local currency is the most used one and therefore, the best option. Then, consider each pair of currencies to convert.

If you want to achieve as much precision as possible or if conversion rates change drastically over time, you may want to use an additional table with conversion rates by date.

On the other hand, if conversion rates change slowly, you could use an average by month, or even a single average value for the whole dataset.

Working with OCDS releases

The OCDS offers two options of schemas to publish OCDS data: releases and records (sets of releases for the same public contracting process). Releases present a versioned history of a contracting process. Seeing how information has changed over time is important for some analysis but not for the quantitative MAPS indicators we are working with here.

Instead, we can use compiled releases: they capture the latest state of (almost) all fields for all phases of the contracting process. Publishers can include an optional compiledRelease field in their Records; but in the case this field is not present or if you have access to releases only, you may be interested in generating compiled releases yourself. Take a look at the OCDS Merge library, the OCDS Kit tool or, if you prefer to write code yourself, read the OCDS rules to merge releases.

Another strategy is to apply the merging rules on each query and for each field of interest. This may not be the best solution if the number of queries is large, but the indicators we have are few. For example: assuming the following releases are in a table called data, we want the latest version of the field tender/procurementMethod for each contracting process:

By reading carefully the rules to merge objects, we can conclude that the key is to find the latest release that specifies the procurementMethod field:

SELECT
  d1.id,
  d1.ocid,
  data -> 'tender' ->> 'procurementMethod' AS procurementMethod,
  data ->> 'date' AS "date"
FROM data d1
JOIN (
	SELECT 
  		ocid,
  		MAX(CAST(data ->> 'date' AS TIMESTAMPTZ)) AS "date"
    FROM data
    WHERE data -> 'tag' <@ '["tender", "tenderUpdate", "tenderAmendment"]'
    AND data -> 'tender' ? 'procurementMethod'
    GROUP BY ocid
) 
AS maxDates ON (
  maxDates.ocid = d1.ocid AND maxDates."date" = CAST(data ->> 'date' AS TIMESTAMPTZ)
);

You can play with the code in this fiddle. Note that for the process with ocid = “123456-1” the latest value has been explicitly set to null, which means that the field has been removed. Using a sentence like

WHERE data -> 'tender' ->> 'procurementMethod' IS NOT NULL

will result in the release #3 to be ignored and a wrong value to be picked, therefore we use the ? operator to find if procurementMethod is a key used within the JSON.

Here is another example that shows how to do the same with array fields like contracts.

If you choose this route, take the time to read the rules and take a look at the fields you may need in the OCDS JSON schema. Especially, look for the options that alter the merge strategy like omitWhenMerge and wholeListMerge.

Calculating an indicator

After the preparation steps, we are ready to show the calculation of a single indicator. Quantitative indicator 1.1

Invitation to bid (in % of total number of contracts)

The language of “Invitation to bid” is not used in OCDS, but is maybe referred to by another term. To ensure that we understand the meaning, let’s read the related MAPS sub-indicator 7(a)(c), which our quantitative indicator supports (emphasis added):

The information system provides for the publication of:

• procurement plans
• information related to specific procurements, at a minimum, advertisements or notices of procurement opportunities, procurement method, contract awards and contract implementation, including amendments, payments and appeals decisions
• linkages to rules and regulations and other information relevant for promoting competition and transparency.

From the bold text, we can infer that our quantitative indicator is probably used to support the assertion that procurement opportunities are announced and/or published. A way to assess this by using OCDS is by looking if the publisher fills the tender/documents section, which should contain all documents related to a tender, and look if they add tender notices or other relevant documents by looking at the documentType field. Specifically, we should look for the tenderNotice type and other custom types that may be used for documents that refer to advertisement or notifications.

Paraguay data

When querying the document types used, we see that Paraguay does not use the tenderNotice type. However, we do see that there are a few custom types that are what we need:

• Carta de invitación a los oferentes (Invitation letter to bidders)
• Publicación en periódicos (Advertisement on newspapers)
• Carta de Invitación (Invitation letter)

Let’s find how many processes used these document types. Assume a table called data with a single column named compiledRelease:

SELECT
 count(distinct compiledRelease->>'ocid') as processCount
FROM
 data
CROSS JOIN
 jsonb_array_elements(compiledRelease->'tender'->'documents') as documents
WHERE documents->>'documentType' in (
  'Carta de invitación a los oferentes', 
  'Publicación en periódicos',
  'Carta de Invitación'
);

Applying the query to our data, we find that 25,969 contracting processes out of 36,033 have these documents: 72.1% of the total.

Ukraine data

Theoretically, Ukraine data is published as releases. But, by looking at how the tags are used (as we recommended in the Preparation section), we see that there is no more than one release per procurement process. Although the situation raises questions about how they are managing the history of contracting processes, in this particular case it allows us to write simpler queries: since there is no more than one release per process, we can use them as compiled releases.

When applying the same procedure as Paraguay to Ukraine data, we find that the tender/documents section has a couple of issues. The result obtained is an extremely low number, which seems inconsistent with the current status of public procurement in Ukraine.

Ukraine uses an e-procurement system called ProZorro, in which open tenders are publicly available and can be viewed without a login. Since their OCDS data comes from ProZorro, it is likely that the tenders in our dataset were published there when active, which is consistent with what the indicator is meant to measure. So, we can approach the calculation by considering that all open tenders were announced.

Usually, selective tenders include an announcement for potential bidders to join to the pre-qualification phase, so we can include them in our calculations as well.

Let’s go to the query. For the following, assume a table named releases with a single column (data) containing a single release each:

SELECT
   data -> 'tender' ->> 'procurementMethod' as ProcurementMethod,
   count(distinct data ->> 'ocid') as NumberOfTenders
 FROM
   releases
 WHERE
   data @> '{"tag":["contract"]}'
 GROUP BY
   data -> 'tender' ->> 'procurementMethod'
;

This gives us the following results:

According to the results, the number of tenders with potential invitations is (535,791 + 18) / 1,520,507 = 35.2%.

Lessons learned

Calculation of quantitative indicators from MAPS with OCDS data is not a complicated process. Just be sure that no important data is missing, and if some fields are unavailable or seem incomplete, remember that indicators may be calculated in alternative ways, as we did in the last section.

Before ending this post, there is another conceptual difference between OCDS and MAPS that affects indicator interpretation, besides the case of indicator 1.1 shown before. OCDS makes a distinction between award and contract: “award” represents the contract being awarded to one or more suppliers, and “contract” is the actual agreement signing between the government entity and the supplier(s): changes may happen between these milestones in the public contracting process. MAPS does not make this distinction and uses the term “contract” to refer to both. Arguably, given this situation, we could choose to use award or contract data for calculations, or we can choose the most appropriate section depending of each indicator.

Hoping that all of this helps you, we wish you luck in your own experiments. Please contact us at data@open-contracting.org  if you have any findings you want to share, or if you want to discuss the experiments we presented in this blog.