Visualizing the new draft EU Procurement Forms

One aspect of the European Union’s common market is a requirement that government open up their public procurement opportunities to companies from other members states. Similar requirements exist under the World Trade Organisation’s Government Procurement Agreement (GPA).

In theory, this should lead to governments getting a better deal on the goods and services they buy, as there are more firms who may compete for contracts – and countries and firms gain the trade benefits that come from different areas of comparative advantage.

However, for a common market in government procurement to operate, firms need to be able to discover opportunities from different countries. And for citizens to understand how liberalised government procurement is impacting their countries, they need information on how contracts are being awarded. And to make sense of that information they really need structured and standardised data.

Part of both the EU and WTO agreements on government procurement is a requirement to publish tender and award information. In the EU, this has taken place over the last decades through publication of contract opportunity and award notices in the Official Journal of the European Union (OJEU) and through a digital form in Tenders Electronic Daily (TED).

Currently, information may be submitted only via XML files, but historically it used to be possible to also fill in and mail paper (PDF) forms. Although TED has been able to build up a rich dataset of billions of Euro’s of procurement, the current forms were not really designed with data analysis in mind, and so making sense of the data from different countries, and building up a clear picture of opportunities and awards can be challenging.

However, there’s an opportunity to change that – the European Commission team is currently working on redrafting the forms. We’ve been engaging with them to explore how learning from developing the Open Contracting Data Standard could be applied in the process.

Visualisation and validation

Developing the EU procurement forms is a complex task. There are a number of different directives describing a number of different stages in a contracting process when notices should be published, and each setting out specific information requirements. To map this out, Jachym Hercher and the team at the European Commission developed a spreadsheet matrix: with a row for each field required in one or more forms, and columns for each form. In the grid created, a simple code was used to indicate when fields were required, optional or repeatable, or conditional upon other fields.

This gives a top-down view of all the forms, and supports efforts to rationalise the requirements from different directives into a consistent set of fields: getting at underlying concepts even when different directives or procurement processes might require slightly different labels for fields.

However, it can be hard to check from this grid what each individual form will contain. So, when we were looking at the drafts to consider how the forms aligned with the Open Contracting Data Standard, we had the idea of writing a parser that would read the matrix and generate example HTML forms for each column. It turns out this wasn’t just useful for us, so with the help of Tom Dimiduk, the European Commission created a full parser which generates draft forms complete with footnotes, much more familiar to lawyers or procurement experts used to reading or filling in the paper forms.

These ‘paper forms’ are generated directly from the spreadsheet matrix. As a result, this visualisation serves another important purpose – checking that the codes and markup used in the spreadsheet is consistent and correctly represents what it is meant to. The parser that generates the visual forms effectively acts as a validator, ensuring that fields in the master spreadsheet are well structured.

This lays key groundwork for the next stage in the development of the EU forms, when an electronic representation for data exchange is created. It is that electronic format which will ultimately shape the quality of EU procurement data for years to come.

As we have been exploring in the ongoing development of OCDS, it’s important for any standard to have a ‘Single Source of Truth’ and to have good technical validation in place. Small inconsistencies between versions of a schema, or between schema and documentation can cause all sorts of problems with data interpretation and interoperability.

Feedback on the forms

The draft forms are open for feedback until January 16th 2016. In particular we’re really pleased to see improved fields for capturing the relationship between different notices, which will hopefully address the current challenges of linking up tender and award stages of a contracting process. There are also improved organisation identification blocks, and a suggested extension on bidder information.

You can find out more about how to give feedback on the forms in Jachym’s blog post here.

Once the updated form design is finalised, we’ll be looking to complete a mapping between the EU forms and the OCDS, ensuring that, with EU extensions to OCDS, the two are well aligned.