Learning Insights: Measuring results from open contracting in Ukraine
At the OCP, we are inspired by the motto: “We do hard things”. “We” is both our in-country partners and us. We may not always be successful, but we will always try and we will always learn.
What Ukraine is doing with Prozorro and its open contracting procurement reforms is groundbreaking and hard. Measuring and tracking the impacts from those reforms is also difficult.
It’s tricky to quantify impacts from open government more broadly; few have managed to do so thus far.
Yet quantifying impact is important to us and our partners for three reasons:
- Tracking results and reflecting on learning allows us to be adaptive in our project designs and adjust how we collaborate with partners.
- Introducing a stronger focus on measuring and learning may provide a valuable model for others in the wider open government field.
- Most importantly, we firmly believe that sustaining the momentum for open contracting relies on us being able to measure and quantify results.
This is our first blog in a new series reflecting on what we are learning about measuring outcomes and impacts of open contracting and procurement reforms in our projects.
Ukraine is one of our Showcase and Learning (S&L) projects, and we’ve already shared several stories about the success of Prozorro. Each S&L project tests specific theories of change and use cases. Through the Prozorro platform, Ukraine is revolutionizing procurement by digitizing the process and unlocking data to make it available to citizens, CSOs, government, and business. The theory of change for this S&L project hypothesizes that transparency and the implementation of the Open Contracting Data Standard (OCDS), combined with multi-stakeholder collaboration in the design, promotion and monitoring of the procurement system, is having an impact on value for money, fairness and integrity.
The reform introduced other innovations, including electronic reverse auctions and a centralized procurement database that integrates with private commercial platforms. We co-created a monitoring, evaluation, and learning (MEL) plan with our project partners to quantify and measure specific progress and impact indicators, while understanding that it is hard to attribute impacts to distinct aspects of the reform. The indicators featured in this blog are particularly related to our theory of change.
We are at a crucial moment in this S&L project as our first round of comprehensive MEL baseline and progress data are coming in. It’s a good time to reflect on key takeaways and challenges that arose when defining and analyzing these data, and how we are using them to inform the Prozorro reform.
Openness can result in more competition and competition saves money.
One of the benefits of open contracting appears to be improving market opportunity and efficiency. Market opportunity focuses on companies being able to compete for business on a level playing field.
From January 2015 to March 2017, the average number of bids per tender lot rose by 15%, demonstrating an increase in competition. Even more notable, the average number of unique suppliers during that same time grew by 45% for each procuring entity, meaning that agencies are now procuring from more and more diverse suppliers.
How was this improvement achieved? Our partners in Ukraine believed that open contracting was key. First, that everyone should be able to see everything, and that transparency would give bidders confidence in the system to bid. In addition, they understood that if there were consequence to bad behaviour, this would also build bidder confidence. Finally, they could not assume that the private sector knew about the reform, so they made proactive efforts, via multiple channels, to educate business about the system. We are interested to see what happens to these numbers over the next progress period and whether bidder participation will be sustained.
We witnessed a very clear correlation between higher competition and savings, showing that the more bidders there are, the greater savings generated – particularly in the electronic reverse auctions. As an example, tenders with more than three bidders can potentially save around 20%. These higher competition levels led to significant savings – around 9.7% on average. Ukraine spends around 45% of its annual budget through public procurement; 9.7% savings signals that the system could save a significant share of the Ukrainian annual budget.
It’s crucial to understand a couple of points regarding the methodology for measuring savings from Ukraine. The measurement of savings is calculated by looking at cost estimate versus contract award price. As procurement planning in Ukraine is getting better, savings are falling. In this case, savings were around 14.1% in mid-2016, falling to 9.6% in March 2017. Because of the methodology, we assume that over time savings decrease because of more effective planning and value estimates. We will work to confirm this assumption by looking closer at how institutions set estimates and how Prozorro data is helping them do that more accurately. Additionally, it’s important to keep in mind that this methodology does not take contract amendments into account, nor do they account for the fact that budget estimate values can be easily manipulated. Finally, we understand that open contracting is part of a larger reform and that we are tracking the impacts of several reform projects here. So, you can see why the process of determining savings can be hard, and we are interested to explore better methodologies for calculating the impact of open contracting on value for money.
High levels of responsiveness can benefit procuring entities.
Those agencies that leverage their opportunities to interact with business and citizens throughout the contracting cycle, by actively responding to questions and complaints via the online platform, tend to conduct procurement more smoothly, without high levels of amendments or cancellations, than those who don’t. Tenders with a 100% response rate to feedback have a 66% success rate, while those with no response, show a 52% success rate. The portal provides procuring entities with the resources needed to address questions and problems, saving time, effort and money throughout the contracting process.
People are beginning to trust the public procurement process and data more.
According to a survey of 300 entrepreneurs conducted by USAID, most respondents believed that Prozorro significantly (27%) or partially (53%) reduces corruption. Additionally, fewer respondents who participated in procurement said they faced corruption when using the new platform (29%) compared to the old system (54%). These numbers only tell a part of the story, as we do not know what those outside of the procurement system think, but they are a necessary first step towards measuring increased levels of trust for the public procurement process. We will continue looking at trust as one of the proxies for health of an open procurement process.
Citizens are actively seeking out procurement information.
Google search hits grew from 680 in the month of January 2015 to more than 191,000 in the month of February 2017 (tracking 43 related keywords). This means the environment is shifting to one where people are recognizing that this data has value; that there is interest and demand for it. Implementing open contracting processes is just one part of what we want to see happen. We also strive to nurture an environment where open contracting data is seen as something that is worthwhile and necessary.
The newly established www.dozorro.org monitoring platform also shows promising results. Since November 2016, it has been visited by 60,000 Ukrainians; 1,700 of them provided feedback on various procurements. Six procuring entities have institutionalized the use of the portal to monitor feedback on their tenders and react to potential wrongdoing. Most of the feedback comes from business, but in May 2017, DoZorro will launch a comprehensive civil society procurement monitoring module.This will help organizations manage their monitoring efforts, from spotting an irregularity to filing a complaint and tracking an institution’s response. In addition to DoZorro, we’re seeing people harness technology to create several other online tools that utilize the OCDS Prozorro data.
We also saw a few good examples of NGOs and government bodies collaborating to control integrity of procurements. The NGO “Civil Control” works closely with Dnipro Regional Administration to identify so-called “bad” tenders (those with potential wrongdoings or mistakes). Together they analyze tenders and react on inefficiencies or corruption risks. In many cases, findings lead to disciplinary punishment with changes in tender documentation or sometimes cases are taken over by enforcement agencies. In a span of four months from September – December 2016, more than 6,000 tenders were checked, resulting in nearly 700 actions taken by purchasing bodies (tender terms or contracts amended). Kiev City municipality decided to follow these footsteps and are now also closely cooperating with local NGOs to monitor procurements, using www.dozorro.org.
While we’re encouraged by the results, there are some items we’re keeping a close eye on.
The main one is feedback loops. We see that procuring entities’ responsiveness to general questions results in better quality procurement. We also see that only one out of three claims (request to a procuring entity to amend, cancel or modify a tender in question) is successfully resolved. In addition, there are some good individual examples, such as the ones in Dnypro and Kiev. While we do not know if these numbers and instances are sufficient for an effective institutional response mechanism, we do know that business and citizens have to trust redress mechanisms before using them. We will continue trying to identify the ideal level of institutional response to secure trust and develop better metrics to capture that.
We’re also still learning about how we can best help our partners measure and track information that is useful to them, and how to do so in manageable and sustainable ways. It took longer than expected to co-create the methodology and get baselines because of limited capacities and the complexity of the procurement system. We are now partnering with the Kiev School of Economics and Transparency International Ukraine to build up local monitoring and evaluation capacities.
We are encouraged by the change we are seeing. We look forward to digging deeper and continuing to measure impact. As we continue to learn, we will be working to improve our measures and collection processes. We know that we can do more to make our projects more adaptive and agile. This new Learning series will focus on how we do just that, and we welcome your thoughts and ideas in the process.
The work that we have left to do won’t be easy, and we’re okay with that, because, as we said, together with our partners, we do hard things.
