What does Australia’s open contracting data look like?

This is part of a series of country data reviews looking at progress in publishing and using open contracting data by our partners. We will focus on:
i) what the goal of using the data is;
ii) what progress has been made so far; 
iii) what’s working and what the challenges have been; and
iv) next steps to get to mass adoption and impact.

This blog looks at Australia’s efforts to implement the OCDS to make government contracting data easier for various actors to access and use. Read on to take a deep dive into the data available on the federal government’s AusTender platform.

Note: The accuracy of the information in this data analysis may be affected by the quality and completeness of the raw data. Any findings should be verified with the publisher.

Why Australia is opening up its contracts

The Federal Government of Australia has begun publishing Open Contracting Data Standard (OCDS) data via its contracting portal, AusTender. The project is part of Australia’s efforts, as a member of the Open Government Partnership (OGP), to promote public participation in government processes and better informed decision-making by improving “openness and accountability in government”.

Australia’s 2nd OGP National Action Plan Commitment: Expand open contracting and due diligence in public procurement  

Publishing open contracting data: progress so far

In January 2019, the Department of Finance updated the contracting portal, AusTender, to include the publication of data in the Open Contracting Data Standard (OCDS). AusTender collates data from central government departments using a custom-built procurement reporting module installed in each agency’s SAP finance system. The OCDS data published from AusTender includes over 450,000 contracts awarded to suppliers across the Australian Federal Government, and includes data on contract amendments too. When a material change to a contract is registered in an agency’s SAP system, the procurement reporting module triggers the publication of a contract amendment notice on AusTender. An OCDS API allows other applications to interact with the database programmatically, making it easier for actors, both in and outside government, to use the data, without needing to request the creation of bespoke reports. 

This update to the platform marks an important step in Australia’s journey to providing comprehensive data on its contracting processes, following the New South Wales OCDS implementation.

One notable achievement of Australia’s OCDS implementation was the introduction of a JSON API to AusTender. Previously, the only way to get machine-readable data from the platform was to search the data.gov.au platform for exported AusTender datasets, or to manually scrape the Contract Notices page on AusTender itself. Neither of these historic approaches are published to an international standard – meaning extra work is placed on the data user to understand the meaning of the fields in the dataset and to produce bespoke tools for their needs as the structure and format of the data is unknown. This also makes it difficult to combine with other datasets.

With the new JSON API, Australia has effectively provided a common interface for multiple applications to “plug in” and consume procurement data with no additional effort from the government other than maintenance of the platform. This enables multiple groups to access and analyse the data easily.

In addition, contracts and amendments to contracts are now linked together through the OCID; a unique identifier for each contracting process which enables its change history to be tracked in a straightforward way. It is also notable that while Australia is not publishing much information on tender and award in OCDS, the fact that they are publishing post-award information (such as rationales for amendments) is best practice. In much of the world, changes made to contracts after signature are under-reported and a common way of hiding corruption through inflating prices. It also gives transparency over delays and changes of the scope of work.

What can you do with Australia’s data?

The OCDS Helpdesk used the open source Kingfisher tool to scrape data from the platform on 17 July 2019, which we used to answer some basic questions about public contracting in Australia. You can use the API on the AusTender platform at https://api.tenders.gov.au/ (an authentication token is needed) but we should point out that OCDS Kingfisher comes equipped with scrapers to help download and process the data.

2019 SPREADSHEETS: Interested in trying out the data but not sure how to use an API or work with JSON data? Access spreadsheets of the 2019 AusTender OCDS data here.

What is the scope of the data?

The OCDS dataset includes 517,454 OCDS releases describing 410,289 contracts and 107,165 contract amendments. The data covers the years 2004 to 2019 though the vast majority of the releases are from 2013 onwards:

A contract release in OCDS represents the signing of a contract, while a contract amendment release represents changes made after a contract is signed. With that as a starting point we can determine that our questions can be geared towards what the government spends, on what, and with whom.

Whilst there is a still a usable corpus of data from 2013 onwards, there is a discrepancy in the number of notices available via the AusTender front end and the OCDS API:

What does the government spend taxpayers’ money on?

When we calculated the total spending for each category, we noticed some issues with identifiers for the items associated with contracts which had been amended, whereby each amendment adds a new item to the contract, resulting in duplication. 

Fortunately we were able to update the queries used in our analysis to remove these duplicates, however this could easily be missed by a user less familiar with the data. We’ve reported this issue, and others noted in this blog, and the Australian government is working to resolve them.

These numbers are very large since we’re taking the entire dataset into account; but this gives us a historic overview of how the government spends its money. Contracts in Australia are classified using the United Nations Standard Products and Services Code (UNSPSC).

We were able to look up the classification codes used in the data using the search-code feature on the UNSPSC website, but this analysis would have been easier if descriptions for the classifications were included in the data, using the ‘contracts/items/classification/description’ field in OCDS.

We can see that many of the largest spend classifications relate to defence spending.

Who benefits most from government money?

There are multiple ways to frame this question. The first, most obvious, one is what are the largest contracts that the government procured?

This shows us the largest individual contracts awarded by the government along with the supplier for each contract. The chart shows the value in AUD, the supplier, and the contract identifier so you can search for them on AusTender.

The largest single contract is with DMOJSF OFFICIAL AUSTRALIAN ACCOUNT PSFD for nearly $14 billion dollars. We can look up this contract in the AusTender front end and see that this is a multi-year contract for ‘Provision of new air combat capability’ from the Department of Defence.

This doesn’t, however, tell us much about which suppliers have benefitted the most historically. For that we can look at how many times a supplier has been awarded a contract with the government and the total value of contracts awarded to each supplier. 

Firstly, we can simply count the number of contracts awarded to each supplier.

We found that organization identifiers weren’t provided for suppliers which makes it harder to total up contracts by supplier, due to the possibility of differences in how supplier names are entered (e.g. abbreviations or spelling errors) or due to changes in trading names over time.

After interrogating an example contract we found that, whilst organization identifiers weren’t provided in the primary `identifier` field, they were in fact being provided, but in the secondary `additionalIdentifiers` field.

We were able to update our queries using this secondary field but this adds an extra layer of complexity to the analysis and could be easily missed by users not familiar with the data.

Thales Australia has the most contracts with the government with 4980 contracts. Thanks to the inclusion of an organization identifier, we can see that the company appears twice on the list with slight variations in its name and also follow the guidance on org-id.guide for the AU-ABN code to look up the company in the Australian Business Register. From there we can find out more about the firm, including its registration date (12th April 2000) and all the different names it operates under (Australian Munitions, Lithgow Arms and others); we can also connect to the Australian Securities and Investments Commission registration for the company to access corporate filings.

What about the total value of those contracts? We can sum the values of contracts per supplier to determine which companies received the most money from the government:

Interestingly, we can see that the top suppliers do not have an organization identifier at all, so we ran another query against the address and contact details in the OCDS data to see what other details were available about these organizations.

We can see that all suppliers have addresses in the United States and the contact names suggest that contracts appearing under these suppliers relate to defence procurement programs with the US government.

What can good identifiers tell us?

We used the Organization Identifier field in the data to link the supplier information to businesses in the Australian Business Register website. This was possible because Australia provide good identifiers for their suppliers in the data following the guidance on org-id.guide website.

One matter that the government is concerned with is the participation of Aboriginal-owned businesses in the procurement process. The challenge presented by this is how to measure participation. Unfortunately the OCDS produced by Australia does not contain any explicit information to measure this yet; but the good quality organization identifiers provided for suppliers (through the ABN number) makes it possible to link this data to other datasets, for example data on Aboriginal owned-businesses. This would be a lot more difficult without the OCDS data and without the identifiers it contains.

With future developments more detailed information on the classification of organizations could be integrated into the OCDS data by using the Party Details section which is currently unused in Australia’s data. This would make it possible to weave details about businesses directly into the OCDS data, such as whether the organisation is Aboriginal-owned or whether it is an SME or large organisation.

What can’t we tell from the data yet?

Even with the progress to date and the achievements of the Australian government so far (and let’s be clear, they are indeed achievements!) there is still some way to go. At the time of writing, there are several data quality issues to be addressed if Australia, businesses and civil society are to reap the full benefits of publishing to OCDS. 

Most prominently, the buyer is not listed correctly in the OCDS data, meaning it may be interpreted by some data users as missing. This seriously limits the possibilities to understand contracting at anything other than a whole of government level. Access to information about buyers is required to answer one of the most basic questions about public contracting – who is doing the buying. Usually buyer information is reported in either the buyer or the procuringEntity fields in the data. Although we did discover some apparent buyer data in the ‘parties’ array with a tag of procuringEntity, finding it requires knowing that it’s there and makes including it in analysis much more difficult. Australia could support data users a lot more by populating the appropriate fields (buyer and procuringEntity) in the data accordingly.

There are also a few other tweaks to parties that Australia should address to better comply with the requirements of OCDS. Furthermore, the way that amendments have been modelled differs from the standard approach to publishing a change history defined in OCDS. In OCDS each amendment should reference the one immediately prior to it to provide a full and incremental change history; in Australia’s data this chain has been broken because each amendment references the original contract. This does allow a data user to see the link between the original contract and the amendment but makes it harder to see more granular changes over time.

We would also love to see this implementation go a lot further than the bounds set by the pilot. There are opportunities here to expand the scope of the data to include information in the ‘tender’ and ‘award’ sections of the contracting process. Doing so would provide a lot of insight and value for civil society, the private sector, and the government itself. It would allow potential suppliers to identify bidding opportunities and improve our understanding of the impact of different procurement methods and timescales on value for money in public contracting. Important data fields related to these sections include dates, values, item classifications for planned and active tenders, procurement method, tender period and number of bidders for each tender. 

The road ahead

Australia’s OCDS data enables us to answer some of the questions we have around government spending and going from unstructured open data to OCDS is definitely progress to widening opportunities for Australians to engage in their government’s procurement. We are hopeful that upgrades to the system will address some of the issues identified here and allow the Australian government to continuously improve the quality and completeness of its open contracting data. 

Interested in trying out the data but not sure how to use an API or work with JSON data?

We’ve exported quarterly spreadsheet (XLSX) and CSV versions of the AusTender OCDS data from 2019 for you to experiment with.

Open the XLSX files using your preferred spreadsheet software, or unzip the CSV files and import them to a database or analysis tool.

We’ve provided two datasets: releases (equivalent to notices in AusTender) and compiled releases (a single release per contracting process), most analysis is easier to carry out on compiled releases, but beware of the issues we noted in the blog.

To learn more about using OCDS data, check out our guidance on using OCDS data and our learning labs on getting and exploring OCDS data using Google Sheets and advanced OCDS analysis with Excel. Get in touch with the free OCDS helpdesk if you have any questions: data@open-contracting.org.

If you’re interested in working with the full dataset, we can also provide access to our hosted version of Kingfisher, our free tool for analysing OCDS data using SQL. Contact the OCDS helpdesk to request access: data@open-contracting.org.