How do you build a network of citizen corruption fighters? Ask Ukraine’s revolutionaries

Community map of Ukraine. All people on the map are caricatures of real DOZORRO activists and their best cases.

In three years, the DoZorro community has uncovered violations in over 30,000 tenders with an estimated value of $4 billion. More than 100,000 people use their procurement monitoring system each month and violations have been fixed in 14% of cases.

In 2016 in Ukraine, when the e-procurement system ProZorro was launched, civil society monitoring of government contracting was already well established. We had great “gurus” like the NashiGroshi investigative journalists and the Anti-Corruption Action Center with their project ukr.aw. Several civil society organizations (CSOs) across the country were monitoring state contracts on a regular basis by browsing tender notices on the government portal. They achieved great results – criminal charges were laid, contracts canceled, companies blacklisted, administrative sanctions were issued to procuring entities and fines to procurers, among others.

But with the creation of the ProZorro e-procurement system, which published tons of new data on everything from the number of bidders on tenders to the phone numbers of procurement officers, we at the anti-corruption organization Transparency International Ukraine wanted to do even better. We wanted to make procurement monitoring ubiquitous and popular, more efficient and accurate, more user friendly. 

We started with a simple e-procurement monitoring pilot with six CSOs that had already been working in the field of procurement monitoring. It was very important for us to understand the whole process and “old” approach to understand how we could improve the situation. After three months, we received the first reports from the CSOs. Some managed to monitor 50 procurement processes, others 2000. What was the difference? Why did we see such different results from CSOs with similar levels of experience, and the same human and financial resources?

The answer was clear: organizations understand and implement “procurement monitoring” differently. For some, monitoring focused on carefully scrutinizing contracts, and then reporting violations to controlling or prosecuting authorities. For others, it was the number of tenders “reviewed” in the procurement portal, regardless of whether or not they were investigated. In all cases, there were no detailed records of which tenders were monitored, nor what violations had been identified etc.

Back then in Ukraine we had a huge problem with the government-run procurement monitoring. There were no detailed statistics released on the types of violations, monitored tenders, or the performance of procuring entities. In short, there was no accountability at all. This created huge corruption risks as the monitoring system could be easily manipulated to influence who was monitored, who wasn’t and the results. 

It was clear that we had to set a higher standard for conducting and reporting procurement monitoring. The reason for selecting certain procedures for monitoring had to be transparent, along with any violations identified, correspondence with respective authorities and the final outcomes. We believe that monitoring had to be accountable and we wanted to change this on a country level.

But how could we do it when there was poor coordination and sometimes competition between CSOs? We had no evidence on the impact of CSO monitoring and we were “competing” with a “state machine” of over 200 people at the State Audit Service who focused solely on ex-ante procurement compliance monitoring all over the country.

We decided to standardize the procurement monitoring process, develop a monitoring management system for the CSOs and build a community of CSOs united by a common goal. Looking at the results today, we achieved this goal and we have lots of lessons to share with the global community. 

Here is a summary of our “path” to accountable procurement monitoring in Ukraine.

Development of monitoring infrastructure based on open contracting data from ProZorro

We were lucky to have tons of open data from planning to contract implementation. The well-established civic tech community didn’t waste any time in developing a range of tools for different purposes. We had comprehensive, publicly available business intelligence tools, scoring systems, red-flags modules, maps to track construction projects, systems to track contracts amendments and implementation, applications to search for the most successful tenders for certain goods, tools to track payments or universal tools which united multiple datasets (like business registry, court decisions, license registry etc.) in one place.

Designing the CSO procurement monitoring management system

Obviously having such a massive infrastructure it was impossible to continue routine monitoring practices by simply browsing through tender notice portals and looking for irregularities. We designed a platform where CSOs were able to see the riskiest tenders according to the red flags methodology with artificial intelligence AI (more detail on that below), and tenders with feedback from business (as the private sector is one of the best watchdogs). 

Having such a list, CSOs could immediately select the procedure for their monitoring and register a case on the same platform. When registering a case, the CSO selects a type of violation from standardized catalogs developed by procuring experts in cooperation with state authorities. If the procedure was monitored, but no violations were found, the CSO reports so. The next step, enforcement. The CSO usually calls the procuring entity responsible, sends them emails and when that doesn’t help, they address the violation to controlling and enforcement agencies. Every step performed by a CSO has to be recorded in the system. Whether it is a phone call, email, or letter, we have all the evidence on the portal with scanned copies. We know when CSOs went to which authority and with what requests  for each case. And when an answer is received, it has to be uploaded to the platform as well. Every step is standardized.

We wanted to collect evidence on monitoring and see which violations are the most typical, which controlling and enforcement agencies ignore CSOs requests, and what outcome we have with every procurement process monitored. Moreover, procuring entities and politicians could not claim CSOs were “swallowing up grants and doing nothing”, as all the results are publicly available and everyone can check them.

Development of a CSO public profile

The monitoring management system was useful. We started getting results and were happy with the opportunity to track every case. But we wanted to help CSOs to promote themselves, build their image, and support them publicly. We developed a public ‘profile’ for eachCSO, which included all their KPIs, achievements, and media activities.

Any business representative or citizen could open the community page of the platform and see the list of all CSOs working on procurement monitoring using our methodology. You can see the name and status of an organization, region and sectors where it operates, Facebook profile of procurement analysts, website, contact information, links to all their blogs and investigations, the number and value of monitored tenders, number of identified and enforced violations, and number of “wins” and “losses” where it managed to fix irregularities or gave up trying. This helps people (including donors) to compare different CSOs and decide who to approach if you have an issue with a tender or contract implementation in your city.

Establishing the community – tenderfest, edutainment

We had data and tools, a very well designed monitoring process and management system, but how could we motivate CSOs to use it? The solution was the community, united by a common goal – to ensure equal rules for all stakeholders in the public procurement system and guarantee the best value for money.

In September 2017 we organized the first DOZORRO_FEST_2017 and invited organizations, journalists and activists from our well-established network. Two days of networking, education through entertainment, procurement contests, and team building activities yielded results. Over 150 people immediately joined the closed Facebook community group and started interacting every day to figure out complex issues, prepare materials for investigation and media and just to discuss how things were doing. Since then, we organized 3 DOZORRO FESTs and it is the most anticipated event in the civil society procurement community of Ukraine.

Gamification to incentivize monitoring

To interact with the community we established a gamification process on our procurement monitoring platform. For every tender monitored, enforcement letter submitted or successful outcome, a CSO receives a certain amount of DOZORRO COINs, which can be traded for subscription to professional procurement magazines, participation in certain conferences, t-shirts or other merchandise. Every organization designates achievements, such as best analyst, best whistleblower, best fighter for the truth. At every DOZORRO FEST we present special achievements and prizes based on the results of that year’s monitoring.

We printed t-Shirts with fun prints and made other merchandise like wristbands, cups, socks, and caps which the community loved. We prepared dozens of media stories about community members to increase their public image and promote their achievements. We also conducted  training sessions and webinars, supporting the community’s demand for knowledge (for example, on healthcare procurement monitoring, or construction peculiarities, or working with data).

Advocacy – one for all and all for one

As the community grew, we gained the power of collective action. With a united voice, over 25 organizations and more than 150 people spoke out about the violations and issues.

And very soon we received our first results. The State Audit Service of Ukraine, which is responsible for procurement monitoring on the government side, was never open and receptive to the society. But it had no choice but to cooperate with the DOZORRO Community. For the first time in the history of the State Audit Service, we had public round tables with the institution’s management, openly discussed problems we observed and soon changed how official monitoring was conducted.

In cooperation with the Ministry of Economy and State Audit Service, with the initiative of the DOZORRO community, we made governmental monitoring accountable. Now it uses the same approach as DOZORRO monitoring: risk-based tenders audit, every action is recorded in the online system and becomes publicly available on the procurement portal and in the API as data. Finally we saw the real results of the monitoring: before, the government procurement monitors reviewed less than 3000 tenders per year with ±200 employees. Thanks to changes in the methodology and constant pressure from civil society, in 2019 this number tripled to almost 9000 cases monitored per year.

Using innovation – red flags with artificial intelligence

The State Audit Service started using red flags and a new monitoring management system in ProZorro, piloted and advocated by the DOZORRO Community. But the approach was not very suitable for civil society, as we have much more flexibility than state authorities. While the State Audit Service’s system focused on compliance and they couldn’t use external data sources, we wanted to use feedback from businesses and CSOs, beneficial ownership data, commercial blacklists and other data. Based on this, the DOZORRO team decided to improve the red-flags methodology.

With the development of machine learning, we decided to apply it for our project. Why not connect the work done by 25 CSOs and machine algorithms? 

First, we created a team of experts from the DOZORRO community who volunteered to train the model. We did not ask them to think about algorithms that may be applied. Instead, we selected a set of tenders with different risk indicators, trying to make a representative sample. Out of this set, we formed pairs of tenders. Then we showed these pairs to our experts, asking them to decide, which tenders out of each pair deserved more attention to investigate. To make it more interactive, we developed a user interface where analysts had to swipe for tenders they perceived as more corrupt. We called it DOZORRO Tinder (or Tinder for Tender) – you are also picking your “beloved one” but in a contracting world=)

Their answers formed the basis for the initial training of the machine. In other words, we asked a machine to identify patterns, used by the ‘collective mind’ of our experts to prioritize different combinations of risk indicators.

This approach showed significant improvement in identifying risky tenders, especially with collusion.

Initial training, based on prioritization of pairs by experts, was a good start. But for a sustainable improvement of the system, another solution is needed. Currently, the DOZORRO team is changing the model to consider real findings of activists, registered in the system.

The moment an activist registers misconduct in the system, the AI system will re-evaluate its model and recalculate weights of risk indicators to increase the accuracy of the identification process for new risky tenders.

Cooperation with authorities and procuring entities

We didn’t do monitoring for the sake of monitoring or to simply identify corruption cases to reveal scandals. Our goal was to ensure fair rules for all stakeholders in the public procurement system and guarantee the best value for money. And obviously having such a goal meant we had to partner with authorities.

But to have a good partnership we had to propose an added value from our activities. We came with a package proposal: the monitoring infrastructure, feedback system, training center, and monitoring development center. How did it work?

1. Monitoring tool – it’s difficult to manage city procurement when you have 500 municipal enterprises under your control. Operational management, risk-management, contract implementation, supplier relationship management, and many more tasks are on your plate every day. To make cities’ lives easier we developed a business intelligence tool to track some key indicators for city procurement (like daily/weekly/monthly red-flags, contracts amendments, competition, complaints etc). It helped to establish proper monitoring on a daily basis. No more paper or excel reports. You open the BI and see an instant report as of today.

2. Feedback system – every day businesses and citizens provided their feedback on tenders and whether everything is fair and efficient. If they flagged bad or good practices it was very hard to track feedback for thousands of tenders from hundreds of entities. We developed accounts for authorities where they could track all the feedback for entities of interest (e.g. all municipal enterprises under city management, or healthcare institutions etc). Moreover, they had the opportunity to immediately react to the feedback and build trust with citizens and business.
3. Training and monitoring development center – having such a great network of CSOs all over Ukraine we decided to work with them not only to monitor tenders, but also to train procuring entities, businesses and citizens on how to prepare tenders, submit bids, monitor contracts or work with analytical infrastructure. Over two years our community delivered hundreds of training sessions for tens of thousands of stakeholders.

We received a lot of requests and more than 50 municipalities and big organizations that manage multiple procuring entities requested this functionality and became active users of DOZORRO tools. Many of them also partnered with the CSOs to train their procurers and audit units how to monitor contracts. Municipalities believe that this approach helps to improve procurement management.

Engaging citizens – DOZORRO Special Force

The community has grown and it is very diverse – civil society organizations, journalists, business, municipalities and procuring entities, even controlling and enforcement agencies. We wanted to attract more citizens to the monitoring process and raise their awareness about new opportunities. Of course this process is very hard and we needed to find an approach to make procurement simple and understandable for citizens.

We decided to visit different cities and do our homework; to understand cities’ problems, identify some procurement scandals and opportunities to improve the process. We were lucky to have partner CSOs all over the country, so we had an “intermediary” between citizens and a complex and boring procurement process. If citizens face any problems with procurement, they can always address them to CSOs in their region. And it is easy to find a point of contact, as every CSO has a public profile on the DOZORRO platform.

Our first visit to Poltava city (with ±300 000 habitants) was very productive. We prepared a map of school repairs in the city and a guideline to show parents how they can track renovations in their schools. This is a real challenge in Ukraine, as parents often pay for goods, works and services which were already covered by the municipal budget. 

We identified some other problems like road construction or university procurement and spent the whole day talking to citizens, giving leaflets with simple guides, delivering workshops, giving interviews to the media and on radio stations.

The visit was successful and the number of the platform users from Poltava almost tripled, from 700 to 2000 users per month.

Raising awareness – gamified journalism

We always worked with media and journalists. We delivered many training sessions on how to use monitoring tools, the DOZORRO portal, how to write about procurement and so on. 

It was good to link regional journalists to CSO members of the DOZORRO community. They always shared some scandals and journalists shared these through their platforms. But we wanted to find more creative approaches to procurement journalism.

One such idea was an Anti-Corruption Sprint. The DOZORRO team worked with the nonprofit Agency for Local Journalism Development to make procurement journalism more popular and fun. They established a regular sprint – each journalist had to produce some procurement-related story from their city. The first week – about local authorities, the second – city landscaping, then education, and healthcare. Journalists have access to DOZORRO experts if they need some advice or leads. For each story they received coins. Additional coins could be earned for every extra view, referrals, official investigations, authority’s reaction etc. Winners receive some financial prizes and access to mentorship.

As a result of the sprint, journalists produced 28 stories from 8 cities across Ukraine. So far they have run 2 sprints already and are not going to stop, as it becomes more and more attractive.

Conclusions

Our journey to develop civil society monitoring in Ukraine was full of thrills and challenges. We tested a lot of different approaches – some of them were successful, some failed. We were lucky to have support from the OCP, Luminate, EBRD, USAID, UKaid and other donors. We always shared the financial resources with the community and often did fundraising for the community.

Today we have more than 20 CSOs actively working on procurement monitoring, 40+ municipalities managing their procurement processes using DOZORRO tools, 2.3 million people have visited the DOZORRO platform and more than 100 000 use it on a monthly basis. More than 7000 people regularly use the complex business intelligence tools. The value of our approach – accountable monitoring – is data on every action made by a CSO. Within a few seconds, anyone can see that over three years CSOs registered violations in more than 30,014 tenders with an estimated value of $4 billion. They helped to fix violations in 4215 tenders with an estimated value of $500 million. In 19,574 tenders worth around $3 billion they initiated official audits, investigations, or other actions. We don’t know whether they will end up with some real results, but the real fix rate of 14%, changing monitoring rules on a country level, and making procuring entities understand that they are being observed by the public – is already a great achievement.

If you’d like to know more, you can read through the DOZORRO story or check the slide deck to see how it was developed and what challenges we faced.