How can we make our algorithms as fair as possible?
Accountability and transparency
Artificial intelligence is a rapidly developing field. Municipal authorities and other government bodies are increasingly using algorithms to make decisions, for instance to track down social assistance benefit fraudsters or to determine the likelihood that children drop out of school. ‘Those developments come with risks attached’, says Tobias van der Knaap, who has now completed his Master’s degree. At CBS, Van der Knaap researched how improvements in archiving and information management could help safeguard transparency, explicability, accountability and fairness in the use of algorithms by public authorities, municipalities in particular. ‘Accountability and transparency are key concepts in the archiving world,’ says Van der Knaap, ‘and fairness and explicability are at the heart of CBS’ “Fair algorithms” project.’Social perspective
Van der Knaap also immersed himself in the relevant literature. ‘I don’t look at the technical details of algorithms. Instead, I look at them from a societal perspective: what impact will municipalities’ use of algorithms have on people? What are the risks? What do people see as important to know and to safeguard?’ In his research, Van der Knaap also devoted attention to the relevant legislation: the Public Records Act, the GDPR and the Open Government Act, which is currently being drafted, and which will update the current Government Information (Public Access) Act. ‘I also looked at the European context’, says Van der Knaap. ‘The European Union is looking for the right way to embrace the opportunities offered by artificial intelligence. The Dutch approach will need to align with the European approach.’Transparency
For a large part the research consisted of interviews, for instance with staff members from the Research and Statistics departments at municipalities and city archives. ‘The interviews helped me understand how people within municipal authorities currently handle information and archiving. Is it a priority? How is the process organised? And: are algorithms used in the decision-making process? If so, how are transparency, explicability, fairness and accountability incorporated into that process? What practical obstacles do people face?’
Thinking ahead
Van der Knaap’s research shows that there is still a lot to be done. ‘As it turns out, little thought is currently being given to how to properly archive information. This can result in gaps in both the process and the data you need to make sure you have the right accountability after the fact. For example, sometimes permits get lost. But I also heard the other side of the story: archive staff not asking project officers to supply documents until after a project had ended. By then it’s too late; if you think carefully beforehand about what information you’ll need, it becomes a lot easier to save and organise that information.’Complex
Van der Knaap acknowledges that it is difficult to draw conclusions about the use of algorithms. ‘A lot of municipalities are still in the testing phase’, he explains. ‘After all, algorithms are very complex to make, which is why municipalities often bring in specialist companies to do it for them. That can cause problems, as municipalities don’t always have the right agreements in place to govern how those companies handle their data. And then there’s the issue - and this is a problem other countries face as well - of companies wanting to protect their intellectual property and refusing to explain how their systems work. That gets in the way of transparency.’Resolving ethical issues
Professor Charles Jeurgens (Archival & Information Studies) was Van der Knaap’s supervisor at the University of Amsterdam during his research. ‘Usually, when people think about archiving, they think it has to do with “old things”. But the new paradigm in archiving is all about how it can improve the transparency and accountability of the government’s current decision-making process. This new paradigm means that archiving will increasingly take place by design, especially if technology - such as an algorithm - is involved in those decisions. You have to work out beforehand what combination of data, algorithms, results of adjustments and context you need in order to offer the right insights into the process and the outcome. That’s where we come in.’ Jeurgens has seen that archiving within the government is becoming more of a priority. ‘But there’s still much too little focus on archiving’, he notes, ‘and there’s still a gap between experts in artificial intelligence and archiving, although I’m seeing more contact and interest from both sides. That’s good - we can enrich each other’s fields. Together, we can make a meaningful contribution to resolving ethical questions around algorithms.’Making algorithms accessible over the long term
Migiza Victoriashoop is a digital information advisor at the Waterlands Archief. She advises municipal authorities on archiving and was interviewed for Van der Knaap’s research. Victoriashoop is a member of a working group of the Information and Archive Knowledge Network and has written a guide to the archiving of algorithms. ‘Our objective is to make algorithms accessible over the long term. We do that through archiving by design. It’s important to keep a careful record of the steps you take in developing an algorithm. The technical steps, but also the documents on which decisions are based, the quality of the input information and the ownership of the data. It was interesting to read Tobias’s thesis. I agree with his comment that of the four values he chooses - transparency, explicability, fairness and accountability - fairness is the most difficult in the case of algorithms. Archiving can help in any case in determining retrospectively whether the algorithm was created fairly. That’s the contribution that archiving makes to transparent government.’CBS project: ‘Fair algorithms’
The ‘Fair algorithms’ project, supported financially by the Ministry of the Interior and Kingdom Relations, is a collaboration between CBS and partners, among others, such as the City of Amsterdam, the Association of Netherlands Municipalities and CodeForNL. It arose from CBS’ statutory task to support other government bodies in their provision of statistical information. In 2019, CBS also entered into a collaboration with the Netherlands Organization for Applied Scientific Research (TNO) to promote the transparent, fair and verifiable use of artificial intelligence in the public sector.
Related items
- Article - Study on fair algorithms for policy-making
- Article - TNO and CBS are joining forces on transparent and verifiable AI use
- CBS privacy regulations - Privacy