Predicting Punishment: Algorithmic Risk Assessment, Domain Distortion, and the Case for Democratic Oversight
Academic Essay | Philosophy of AI and Computing
Abstract
Predictive algorithms, most visibly tools like the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS), are increasingly used in criminal sentencing decisions across the United States. Proponents argue that algorithmic risk assessment introduces consistency and reduces subjective human bias, while critics contend that these systems encode and amplify the very inequalities they claim to correct. This paper examines three problems with the deployment of such tools in carceral contexts: the structurally embedded nature of algorithmic bias, which cannot be resolved through technical adjustment alone; the phenomenon of domain distortion, whereby algorithmic systems reshape legal institutions in ways that undermine judicial reasoning and due process; and the collapse of the traditional boundary between adjudication of guilt and anticipatory punishment. In response to these challenges, this paper proposes a democratic oversight framework grounded in Heather Douglas’s distinction between responsibility and accountability. The framework centers community participation, mandates transparency, and builds in iterative review, not as a technical fix, but as a commitment to redistributing the ethical labor that predictive algorithms currently obscure.
Keywords: predictive algorithms, algorithmic bias, COMPAS, risk assessment, criminal sentencing, mechanical jurisprudence, domain distortion, due process, algorithmic accountability, democratic oversight
1. Introduction
Predictive algorithms have quietly become a fixture of criminal sentencing in the United States. Tools like COMPAS, developed by Equivant (formerly Northpointe), now inform bail, sentencing, and parole decisions across dozens of jurisdictions, often without much public awareness that they are being used at all. The pitch for these systems is not hard to understand. Decades of research have documented how much sentencing outcomes can vary based on who the judge is, what time of day it is, whether the defendant reminds the judge of someone (Spohn, 2015). Human judgment in sentencing is inconsistent in ways that are well-documented and genuinely troubling, and an algorithm that promised to introduce some consistency had obvious appeal. The problem, as this paper argues, is that the consistency these tools introduce is not neutral.
As scholars working at the intersection of philosophy, law, and computer science have increasingly demonstrated, algorithmic risk assessment tools do not stand outside the systems of inequality they are meant to correct. They are trained on data those systems produced, they optimize for metrics those systems defined, and they operate inside institutions those systems shaped. The result, as Pruss et al. argue, is that “the practices that underpin [predictive algorithms in carceral contexts] are rooted in long histories of control and colonization of human populations” and their deployment risks replicating those histories under the legitimating cover of computational objectivity (Pruss et al., 2024, p. 5).
This paper examines three dimensions of that replication. First, it analyzes the sources of algorithmic bias, arguing that bias in these systems is not a correctable technical defect but an artifact of the social contexts in which the data was generated. Second, it develops the concept of domain distortion, the way algorithmic tools reshape legal institutions rather than simply serving them, with particular attention to the erosion of judicial reasoning and due process rights. Third, it examines the normative shift entailed when sentencing becomes prospective rather than retrospective, a move from punishing what someone has done to managing what they might do. The paper then proposes a democratic oversight framework designed to address these concerns. Drawing on Heather Douglas’s distinction between accountability and responsibility, the framework argues that algorithmic governance in carceral contexts requires not merely enforcement mechanisms but shared ethical labor, distributed across developers, legal professionals, independent auditors, and the communities most directly affected by these tools’ decisions. This is not a framework for making predictive algorithms fair, but rather making their use governable and for preserving the conditions under which their use can be contested, revised, or rejected.
2. Algorithms Are Not Neutral: Structural Sources of Bias
The claim that algorithmic risk assessment is more objective than human judgment rests on a category error. Algorithms are implementations of choices about what to measure, what to optimize, and what counts as a good outcome. As Fazelpour and Danks argue, “algorithms implement values because they are almost always optimized for performance relative to a standard,” and the selection of that standard is itself a value-laden act (Fazelpour & Danks, 2021, p. 3). The question is whose values an algorithm embeds, and at whose expense.
Bias enters at multiple stages of system development. At the data collection stage, it arises from real-world inequalities: differential policing, prosecution rates, and conviction patterns that reflect systemic disparities rather than underlying differences in behavior. When COMPAS is trained on historical recidivism data, it learns from a record of who was arrested and convicted, not who actually reoffended. Since Black Americans are arrested at rates dramatically disproportionate to their share of criminal behavior (Alexander, 2010), a model trained to predict rearrest will systematically overestimate risk for Black defendants not because they reoffend more, but because they are policed more. The data launders structural racism as statistical signal.
At the modeling stage, bias emerges from the choice of optimization targets and performance metrics. ProPublica's landmark 2016 investigation into COMPAS found that Black defendants who did not reoffend were nearly twice as likely as white defendants who did not reoffend to be classified as high risk, while white defendants who did reoffend were more likely than their Black counterparts to be classified as low risk (Angwin et al., 2016). The model's aggregate accuracy obscured systematic disparities in how its errors were distributed. Chouldechova (2017) later demonstrated mathematically that when base rates of the predicted outcome differ across groups, as they do when Black defendants face higher rearrest rates due to differential policing, it is impossible to simultaneously equalize false positive rates and maintain calibration. This is not a solvable engineering problem. It is a consequence of the social conditions that produced the data, and choosing which fairness criterion to prioritize is a political and ethical decision that current development practices largely leave to algorithm designers, outside any democratic or legal accountability.
What this means is that the most fundamental sources of algorithmic bias are not located in the algorithm itself. They are located in the social world the algorithm was trained to describe. Technical interventions, reweighting training data, applying post-hoc fairness constraints, selecting alternative metrics, can reduce some forms of measured disparity. What they cannot do is correct for the fact that data was generated under conditions of structural inequality that the algorithm has no mechanism to recognize as such. As Pruss et al. observe, what we call "unbiased software" may simply be software "with biases that we either do not recognize as biased or do not find problematic" (Pruss et al., 2024, p. 10). Treating bias as a calibration problem conceals its social origins and forecloses the more fundamental question of whether these tools should be deployed at all.
3. Domain Distortion and the Erosion of Judicial Reasoning
The introduction of predictive algorithms into legal decision-making does not simply add a new input to existing processes. It reshapes those processes, their logic, their values, the roles of the people within them. Pruss describes this as "domain distortion," the way algorithmic systems alter the domains in which they operate, often in ways neither intended nor anticipated by their designers (Pruss, 2021). In carceral contexts, the distortion is most visible in the transformation of judicial reasoning and the erosion of the interpretive and contextual judgment that is constitutive of law.
Predictive risk assessment tools rest on a formalist premise: that legally relevant facts can be identified, quantified, and combined into a score that captures the correct basis for a decision. Roscoe Pound (1908) called this "mechanical jurisprudence," the treatment of legal reasoning as the mechanical application of rules to facts, with no room for the interpretive judgment that characterizes legal expertise. Pound introduced the term as a critique. Algorithmic sentencing tools have unwittingly operationalized it. By reducing the sentencing calculus to a numerical score derived from fixed inputs, these systems implicitly assert that all relevant legal and moral considerations have already been captured, that there is nothing a judge could know about a particular defendant that the algorithm does not.
That assertion is false, and its consequences are serious. Law is an interpretive practice. It requires sensitivity to context, the ability to recognize morally relevant particulars, and the capacity to reason about how rules apply to situations their authors did not anticipate. As Pruss argues, "laws tend to outlive the worlds of their creators, and mechanically applying laws in our current context can have unanticipated harmful consequences" (Pruss, 2021, p. 1105). When judges are presented with an algorithmic risk score produced by a system whose inputs and weightings are opaque, there is substantial evidence that they anchor on that score, reducing rather than supplementing their own deliberative judgment (Dressel & Farid, 2018). The algorithm does not assist judicial reasoning. In practice, it tends to displace it.
The problem gets worse when you try to actually challenge one of these scores. COMPAS and comparable tools are proprietary, and their developers have consistently refused to disclose the specific variables, weights, and decision rules that produce individual results. In In State v. Loomis (2016), the Wisconsin Supreme Court upheld the use of COMPAS in sentencing while in the same breath acknowledging that the defendant had no meaningful way to examine or challenge the score he received. The court's response to this was essentially: the score was just one factor among many, so it is fine. But that reasoning sidesteps the anchoring problem entirely. If you cannot understand the basis of a number being used against you, you cannot contest it, regardless of how many other factors are supposedly still in play. The contrast with European law is striking and worth sitting with. The EU recognized this as a rights problem. GDPR's Article 22 established a right not to be subjected to solely automated decision-making with significant legal effects, and the EU AI Act (2024) went further, classifying AI systems used in criminal justice as high-risk and imposing requirements for transparency, human oversight, and fundamental rights impact assessments. Nothing comparable exists at the federal level in the United States. That gap is not just a regulatory difference. It reflects something more basic, a disagreement about whether deploying an opaque algorithm in a decision that could determine how many years someone spends in prison is fundamentally a question of rights, or just a question of evidence.
4. The Temporal Collapse: From Punishment to Prediction
There is a third dimension of algorithmic risk assessment in sentencing that receives less attention than bias or due process, but that may be philosophically the most significant. Criminal sentencing in the liberal legal tradition is grounded in retrospective judgment: punishment is calibrated to the gravity of a past act, assessed in light of the defendant's culpability and the harm caused. Risk assessment introduces a prospective logic that sits uneasily alongside this framework and, in practice, increasingly displaces it.
When a defendant's sentence is influenced by a score predicting future reoffending based on factors including age, employment history, neighborhood of residence, and peer associations, most of which the defendant had no real control over, the operative question is no longer what they did but what they might do. That is a significant shift, and it tends to get obscured by the technical language surrounding these tools. Punishment stops being a response to a specific act committed by a specific person and becomes something closer to preemptive management, predicating sentence length on probabilistic inference about a class of people with similar characteristics rather than on individual culpability. The defendant in front of the court is being punished, at least in part, for statistically resembling people who have reoffended.
From a retributivist perspective, this violates the principle that punishment should be proportional to culpable wrongdoing, since one cannot be culpable for a future act one has not yet committed. From a liberal rights perspective, it raises serious concerns about penalizing people for characteristics they did not choose and cannot change. Bernard Harcourt traces this logic to nineteenth-century positivist criminology, which sought to identify the "criminal type" through biological and social profiling (Harcourt, 2007). Contemporary algorithmic risk assessment is its computational descendant, updated with machine learning, but retaining the same fundamental move of inferring individual dangerousness from population-level statistics.
The injustice deepens when the demographic factors driving risk scores are themselves products of systemic inequality. If residence in a high-poverty neighborhood, limited employment history, or prior contact with the criminal justice system are predictors of risk, and if those conditions are disproportionately distributed along racial lines due to decades of discriminatory policy, then the algorithm is not predicting individual dangerousness so much as encoding structural disadvantage as a basis for further punishment. Blinding the algorithm to race does not resolve this. Variables like neighborhood and employment history track race with high fidelity even when race is excluded from the model directly (Dwork et al., 2012). The proxies do the work the variable was removed from doing.
5. Toward Democratic Oversight
The problems identified above are not primarily technical. Bias cannot be engineered away because its sources are social. Domain distortion is a structural consequence of introducing formalist tools into interpretive institutions, and the normative shift from retrospective to prospective punishment raises questions that fall outside the scope of any algorithm's design. Addressing these problems requires governance structures, institutions and practices that make the use of these tools accountable to democratic values and responsive to the people they most affect.
The dominant approach to algorithmic accountability in existing policy discussions focuses on transparency and audit requirements, mandating disclosure of model inputs, external accuracy reviews, and remediation or discontinuation where disparate impact is found. These mechanisms are necessary but not quite sufficient. As Heather Douglas argues in her account of responsible scientific institutions, a framework organized around accountability alone risks "offloading the ethical labor" from those who develop and deploy powerful tools, replacing genuine moral engagement with procedural box-checking (Douglas, 2023). An accountability-only approach to algorithmic governance reproduces, at the institutional level, the same mechanical jurisprudence that algorithmic sentencing tools reproduce in courtrooms.
Douglas's distinction between accountability (enforcement mechanisms) and responsibility (moral obligations that go beyond minimum compliance) provides a more adequate framework. A well-designed oversight structure should establish enforceable floors while creating conditions that encourage genuine ethical engagement above those floors. The framework proposed here has three interlocking components.
Mandatory Transparency and Contestability
The baseline requirement should be straightforward. Any algorithmic tool used in criminal sentencing must fully disclose its input variables, weighting methodology, validation data, and known error rates, broken down by race, gender, age, and socioeconomic status. That information has to be genuinely accessible, not just available to courts and researchers but also provided to defendants and their counsel in language that actually permits meaningful challenge. The current standard, effectively, is that receiving a score counts as due process. It should not be. The relevant question is whether the defendant had a real opportunity to examine and contest the basis of that score, and right now the answer in most jurisdictions is no. The legal precedent for this already exists. Defendants have the right to confront and cross-examine evidence presented against them, and expert testimony is subject to the reliability standards established in Daubert v. Merrell Dow Pharmaceuticals, Inc. (1993). There is no reason algorithmic risk scores should be exempt from analogous scrutiny. Jurisdictions that rely on proprietary tools shielded by trade-secret protections should face a straightforward choice, disclose or discontinue.
Independent Expert Review with Enforcement Authority
Transparency is necessary for accountability but does not guarantee it. The framework proposes review panels with authority to evaluate deployed algorithmic tools against established fairness and accuracy standards, to recommend modifications, and to suspend use where tools fail to meet those standards. Panels should include expertise in law, statistics, ethics, and computer science, and should conduct reviews on a regular cycle and in response to identified disparate impact. What distinguishes this component from existing voluntary audit practices is enforcement authority. A review that produces findings without consequences reproduces the accountability-without-responsibility problem Douglas identifies, developers and jurisdictions note the findings, issue statements of concern, and continue using the tools. Effective oversight requires that findings have binding effect, subject to appeal through established administrative processes.
Community Participation in Standard-Setting and Evaluation
The most essential component of the proposed framework, and the one most likely to be diluted in practice, is the participation of communities disproportionately affected by algorithmic sentencing tools in the processes of standard-setting and evaluation. This is not a gesture toward inclusivity. It reflects a substantive epistemological claim that the people who live with the consequences of algorithmic risk assessment possess knowledge about the accuracy of the factors these algorithms rely on, about the effects of risk scores on their communities, and about the gap between how these tools are described and how they actually function, knowledge that researchers, developers, and courts currently lack access to.
Pruss et al.'s community-centered approach provides a model where community members participate not as consultees but as authoritative contributors to the evaluative process, with structured roles in identifying priorities, evaluating outcomes, and recommending modifications (Pruss et al., 2024). Mayson's work on prediction and racial justice adds another point that meaningful reform requires not just diversifying who sits in the room but changing what questions the room is authorized to ask, including whether a given tool should continue to be used at all (Mayson, 2019). When communities have genuine institutional power to shape how algorithmic tools are designed and deployed, the ethical labor of governance is distributed rather than concentrated. Developers cannot treat compliance with a disclosed specification as the end of their moral obligations. They must engage with the ongoing evaluative judgments of those their tools affect.
Alongside these components, the framework incorporates ethics consultations and training for all actors involved in the development, deployment, and adjudication of algorithmic risk assessment, of the kind Douglas recommends for responsible scientific institutions (Douglas, 2023). Ethics consultations provide context-specific guidance on challenges no general standard can fully anticipate, including situations where the responsible course of action is to refrain from deploying a tool. Ethics training ensures that judges, defense counsel, and probation officers understand the limitations of risk scores and are equipped to exercise genuine rather than nominal independent judgment. Accountability without ethical engagement produces compliance. Ethical engagement without accountability produces aspiration. The framework's goal is to make the two mutually reinforcing.
6. Discussion and Limitations
The framework proposed here will face objections. The most significant is that it may make algorithmic risk assessment tools effectively unusable in jurisdictions without the institutional infrastructure to implement community participation panels and independent review boards. This is not obviously a problem. The costs of using these tools in the absence of adequate oversight are borne disproportionately by communities that are already disadvantaged, and if the framework cannot be implemented, the appropriate response may be a moratorium on use rather than a reduction in standards.
A second objection concerns the tension between community participation and technical expertise. Critics might argue that community representatives lack the knowledge to evaluate algorithmic tools meaningfully, or that their participation introduces forms of bias that undermine the objectivity of the review process. This misunderstands the proposed role of community participants. Their contribution is not technical evaluation, which is the role of expert reviewers, but authoritative input on the values the evaluation should serve and the outcomes it should prioritize. The claim that these judgments require technical expertise is itself a form of technocratic capture that the framework is designed to resist.
It is also worth being direct about what the framework cannot do. It cannot make predictive algorithms fair in any robust sense, because the sources of their unfairness are located in the social conditions that produced the data they rely on. It cannot restore the normative coherence of a sentencing system that has been reshaped by prospective risk logic. What it can do is make the use of these tools governable, subject to scrutiny, contestable by those they affect, and responsive to democratic judgment about whether their use is justified. That is a more modest ambition than fairness. It may also be a more genuine one.
7. Conclusion
Predictive algorithms in carceral contexts are not neutral tools that correct for human bias. They are sociotechnical systems that encode the values of their designers, reproduce the inequalities embedded in their training data, reshape the institutions in which they operate, and shift the normative basis of punishment from retrospective judgment to prospective risk management. These are not incidental features that better engineering can eliminate. They are structural consequences of using actuarial tools in contexts defined by law, discretion, and power, and no amount of technical refinement changes that basic fact.
The democratic oversight framework proposed here is not a technical solution, and it does not pretend to be. What it offers instead is a governance approach, one that tries to make the use of these tools accountable to the people they most affect, rather than just to the institutions that deploy them. The goal is not to make algorithmic sentencing just. It is to make it governable, and to preserve the conditions in which communities can push back, demand changes, or refuse tools that do not serve them.
The broader stakes are harder to contain than the criminal justice framing suggests. Predictive algorithms are already being deployed in welfare determinations, housing decisions, educational placements, and healthcare triage. The same questions arise in all of those contexts… who gets to define what fairness means, who absorbs the costs when the model is wrong, and who actually has the standing to say a tool has failed. Criminal justice is where those questions are sharpest, because the harms are most severe and the affected communities least positioned to push back. The answers worked out there, or not worked out, will reverberate. The decisions being made now about whether these tools can be meaningfully overseen could shape what oversight even looks like in every domain that comes next.
References
Alexander, M. (2010). The new Jim Crow: Mass incarceration in the age of colorblindness. The New Press.
Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016, May 23). Machine bias. ProPublica. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5(2), 153–163.
Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993).
Douglas, H. (2023). Structuring institutions for responsible and accountable science. Philosophy of Science, 90(5), 1207–1225.
Dressel, J., & Farid, H. (2018). The accuracy, fairness, and limits of predicting recidivism. Science Advances, 4(1), eaao5580.
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012). Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference (pp. 214–226). ACM.
European Parliament. (2024). Artificial Intelligence Act (Regulation EU 2024/1689).
Fazelpour, S., & Danks, D. (2021). Algorithmic bias: Senses, sources, solutions. Philosophy Compass, 16(8), e12759.
Harcourt, B. E. (2007). Against prediction: Profiling, policing, and punishing in an actuarial age. University of Chicago Press.
Mayson, S. G. (2019). Bias in, bias out. Yale Law Journal, 128(8), 2218–2300.
Pound, R. (1908). Mechanical jurisprudence. Columbia Law Review, 8(8), 605–623.
Pruss, D. (2021). Mechanical jurisprudence and domain distortion: How predictive algorithms warp the law. University of Toronto Law Journal, 71(4), 1087–1123.
Pruss, D., et al. (2024). Prediction and punishment: Critical report on carceral AI. AI Now Institute.
Spohn, C. (2015). Race, crime, and punishment in the twentieth and twenty-first centuries. Crime and Justice, 44(1), 49–112.
State v. Loomis, 881 N.W.2d 749 (Wis. 2016).
