Justitia ex machina: The impact of an AI system on legal decision-making and discretionary authority

Abstract

Governments increasingly use algorithms to inform or supplant decision-making. Artificial Intelligence systems in particular are considered objective, consistent and efficient decision-makers, but have also been shown to be fallible. Furthermore, the adoption of artificial intelligence (AI) in government is fraught with challenges which are only partly understood and rarely studied in practice. In this paper, we draw on science and technology studies and human computer interaction and report on a critical case study of the development and use of an AI system for processing traffic violation appeal at a Dutch court. Although much empirical work on algorithms in practice is primarily observational in nature, we employ a canonical action research approach and actively participate in the development of the AI system. We draw on data collected in the form of interviews, observations, documents and a user-experiment. Based on this material we provide: 1. An in-depth empirical account of the tensions between street-level bureaucrats, screen-level bureaucrats and street-level algorithms; 2. An analysis of the differences between decisions made by, with and without the AI system and find that use of the AI systems impacts decisions made by legal experts; 3. A confirmation of earlier work that finds AI systems can best be applied in support of legal-decision making and demonstrate how the decision-making process of the traffic violation cases may mitigate some of the risks of algorithmic decision-making.