This article is part of the Hardwiring Justice series on Artificial Intelligence and the Justice System. This is Part 5 in the series examining how AI is shaping policing, prosecution, defense practice, and the courts.*
Artificial intelligence (AI) tools are now embedded across nearly every phase of criminal proceedings, from investigative analytics and risk assessment to case management and predictive analysis.[1] As these systems become more integrated, a familiar concern emerges: opacity, often called the “black box” problem. [2] Judges, lawyers, and justice professionals worry that algorithmic systems produce results that cannot be fully understood or challenged. [3] That concern is legitimate, but it is frequently misdirected. The real issue is not simply whether these systems can be explained, but whether defendants have a meaningful opportunity to contest them.
What Procedural Justice Actually Requires
Procedural justice[4] is not a synonym for accuracy. It is a distinct set of requirements about how legal processes must treat the people subject to them.[5] Research in this field is consistent across decades and jurisdictions: people evaluate the legitimacy of legal outcomes primarily through process, not result.[6] They ask whether the process treated them with dignity, whether their voice was heard, whether the rules were applied consistently, and whether the decision-maker was accountable.[7] A correct outcome reached through an unfair process does not satisfy these requirements.[8]Neither does an accurate algorithm deployed without meaningful opportunity to challenge it.
This matters for AI evidence because the two inquiries can diverge sharply. A facial recognition system might produce a statistically defensible identification rate[9] and still be used in a way that denies a defendant any meaningful opportunity to examine how it was trained, how it performs across demographic groups, or whether it was deployed within its validated scope. At the point of that denial, the process fails on procedural justice grounds regardless of whether the underlying output was correct. Accuracy and fairness are not the same inquiry.
Perceived legitimacy has practical consequences that extend beyond individual cases.[10] Compliance, cooperation with law enforcement, and the social foundations of institutional authority all depend on whether people believe the system treats them fairly.[11] Communities that already are skeptical about the criminal justice system are particularly sensitive to processes that substitute machine authority for accountable human judgment.
How Opacity Defeats Fair Process
The adversarial process is the mechanism through which due process is operationalized in American criminal proceedings.[12] Evidence is tested. Witnesses are cross-examined. Expert conclusions are subject to challenge. When an algorithmic tool enters a case shielded by trade secret protections, that mechanism breaks down.
A defendant who cannot learn how a system works, what its known failure modes are, or whether it has been independently validated cannot mount a meaningful challenge. The adversarial process has not been satisfied. It has been bypassed. Part of what makes this persistent is that courts and justice agencies have rarely defined what disclosure is actually required when these systems influence legal decisions. In the absence of that definition, vendors protect as much as possible and agencies accept whatever is offered. The opacity is not inevitable. It is the predictable result of an institutional failure to set terms.
AI outputs do not arrive neutrally. A facial recognition match carries an implicit claim to objectivity that is difficult for jurors and judges to critically interrogate without access to performance data.[13] The trade secret claim for a risk assessment is therefore not simply an IP protection[14]. In the criminal context, it functions as a structural barrier to the kind of contestation that procedural justice requires.
The doctrinal consequences of opacity run deeper than procedural unfairness. Scholars examining AI through the lens of civil and criminal liability have concluded that black-box systems defeat the foundational legal tests courts depend on to assign responsibility, intent, foreseeability, causation, because the reasoning behind algorithmic outputs cannot be reconstructed even by the systems’ own developers.[15] When legal doctrine cannot reach a tool, governance must.
The Accountability Gap Is Two-Sided
Vendor opacity is the more visible dimension of the governance failure, but it is not the only one. The second is judicial AI literacy, and the absence of it is just as consequential. Procedural justice requires not only that defendants have access to relevant information but that the decision-maker can meaningfully evaluate it. A judge who cannot assess the reliability, demographic performance, or operational limits of an algorithmic tool is not exercising the kind of informed, accountable judgment that fair process demands.
Research examining how algorithmic recommendations interact with human judgment finds that when the two appear together, the algorithm shapes the outcome in the substantial majority of cases, even where the judge formally retains decision-making authority.[16] That is not deference in any meaningful sense. It is delegation without accountability, the formal structure of human judgment without its substance. Addressing vendor opacity while leaving judicial competency unreformed solves only half the problem. The governance failure is institutional on both sides.
What Defendants Are Owed
Framing disclosure obligations around what procedural fairness requires, rather than what vendors are prepared to offer, changes both the scope and focus of the inquiry. Source code is the wrong target. It describes how a system was built. It does not reveal how the system performs in practice, across what populations, and within what limits. Four categories of information are what procedural justice actually demands.
Independent validation. A defendant has a legitimate interest in knowing whether the tool was tested by someone other than its developer, on a population comparable to theirs, under conditions comparable to those in which it was used. Validation data is not a technical nicety. It is what transforms an algorithmic claim into evidence that can be evaluated.
Disaggregated error rates. Aggregate accuracy figures are not sufficient. A system that performs well on average may fail disproportionately for specific demographic groups. Procedural fairness requires that a defendant be able to assess whether a tool’s error profile is relevant to their case specifically, not merely adequate in the aggregate.
Operational scope and actual deployment. Under what conditions was the system designed to operate, and were those conditions present in this case? The gap between validated scope and actual deployment is among the most significant fairness risks in algorithmic evidence. A defendant is entitled to know whether the tool was used as designed.
Human accountability. Who reviewed the output before it influenced the investigation or prosecution, and what qualified them to do so? Procedural justice is not satisfied by a system that generates a result. It requires that an accountable human being stood behind the decision to act on it, someone who can be questioned, challenged, and held responsible.
When “Explainability” Is Not Enough
The AI industry has developed a fluent vocabulary of transparency.[17] Systems are marketed as interpretable, explainable, and auditable.[18] In a governance context these claims require scrutiny, because they describe two distinct things that are frequently conflated.
Explainability as a product feature produces documentation designed for comprehensibility, not accountability.[19] A visualization showing which pixels influenced a facial recognition match is built to make a user feel confident in the output.[20] It is not built to help a defense attorney identify the demographic conditions under which the system fails, or to help a judge assess whether performance characteristics are adequate to the evidentiary weight being placed on the result.[21]
Explainability as a procedural justice obligation is something different: the ability of affected parties to obtain the information they need to contest a decision.[22] That standard is not met by vendor-generated documentation, however well-designed. It is met by independent validation, disclosed error rates, and accountable human review. Courts that treat the former as a substitute for the latter have not resolved the transparency problem. They have formalized the conditions under which it persists.
Governance as a Trust Obligation
What follows from taking procedural justice seriously as the organizing principle is that governance of AI evidence is not primarily a technical project. It is an institutional one. The question is not whether these systems can be made more transparent in some engineering sense. It is whether legal institutions will define and enforce the conditions under which algorithmic tools may influence the liberty of people who appear before them.
Communities with the deepest exposure to algorithmic surveillance in law enforcement are also those with the strongest reasons to distrust institutions that deploy these tools without accountability.[23] When algorithmic outputs are difficult to challenge and easy to present as objective, they do not merely produce individual injustices. They accelerate the erosion of institutional legitimacy in precisely the communities where that legitimacy is most consequential and most fragile.
The question is not unprecedented. Algorithmic and computational tools already operate under structured oversight regimes in medicine, aviation, and financial regulation, fields where the consequences of unreliable outputs are similarly serious. Those regimes define testing requirements, performance standards, error rate disclosures, and limits of use. They do not require revealing proprietary source code. They require the information necessary to evaluate whether a tool is doing what it claims to do, in the conditions where it is being used. Courts face the same task. The difference is that they have not yet done it.
The governance mechanisms addressed this series, ethics, expert protocols, admissibility, professional accountability frameworks, each serve an evidentiary function. But each also serves a procedural justice function more fundamental than the evidentiary one. Getting this right is not only about keeping unreliable evidence out of court. It is about whether the institutions that deploy these tools can claim to be administering justice.
* This article was edited with the assistance of AI in the form of a large language model. It was used solely for grammar, editing, and footnote support. All substantive content and conclusions reflect human authorship.
[1] Clayton Vickers, How AI Risks Creating a ‘Black Box’ at the Heart of US Legal System, The Hill (Apr. 7, 2024, 6:00 AM ET), https://thehill.com/business/personal-finance/4571982-ai-black-box-legal-system/.
[2] Id.
[3] U.S. Dep’t of Justice, Artificial Intelligence and Criminal Justice: Final Report (Dec. 3, 2024), https://www.justice.gov/olp/media/1381796/dl?inline.
[4] Also call procedural fairness
[5] Kevin Burke & Steve Leben, Procedural Fairness: A Key Ingredient in Public Satisfaction, Am. Judges Ass’n White Paper (2007).
[6] Id.
[7] Id.
[8] Id.
[9] El Fadel N. Facial Recognition Algorithms: A Systematic Literature Review. J Imaging. 2025 Feb 13;11(2):58. doi: 10.3390/jimaging11020058. PMID: 39997560; PMCID: PMC11856072.
[10] Burke & Leben, supra note 5.
[11] Id.
[12] Lucia Zedner & Carl-Friedrich Stuckenberg, Due Process, in TREATISE ON INTERNATIONAL CRIMINAL LAW (Kai Ambos et al. eds., Cambridge Univ. Press 2019).
[13] Michael Christopher Naughton, Considering Face Value: The Complex Legal Implications of Facial Recognition Technology, Crim. Just., Winter 2025, https://www.americanbar.org/groups/criminal_justice/resources/magazine/2025-winter/face-value-complex-legal-implications-facial-recognition-tech/.
[14] Lena Chan, Note, The Weaponization of Trade Secret Law, 124 Colum. L. Rev. 859 (2024).
[15] Yavar Bathaee, The Artificial Intelligence Black Box and the Failure of Intent and Causation, 31 Harv. J.L. & Tech. 889 (2018).
[16] Reuben Binns, Human Judgment in Algorithmic Loops: Individual Justice and Automated Decision-Making, 16 Reg. & Governance 197 (2022).
[17] Nagadivya Balasubramaniam, Marjo Kauppinen, Antti Rannisto, Kari Hiekkanen & Sari Kujala, Transparency and Explainability of AI Systems: From Ethical Guidelines to Requirements, 159 Info. & Software Tech. 107197 (2023).
[18] Id.
[19] Christian Kästner, Explainability, in Machine Learning in Production: From Models to Products, ch. 25 (2024), https://mlip-cmu.github.io/book/25-explainability.html.
[20] Id.
[21] Karen McGregor Richmond, Satya M. Muddamsetty, Thomas Gammeltoft-Hansen, Henrik Palmer Olsen & Thomas B. Moeslund, Explainable AI and Law: An Evidential Survey, 3 Digital Soc’y 1 (2024).
[22] L. Metikoš & I. van Domselaar, Procedural Justice and Judicial AI: Substantiating Explainability Rights with the Values of Contestation (May 5, 2025) (unpublished manuscript), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5242905.
[23] Rachel Levinson-Waldman & Ivey Dyson, The Dangers of Unregulated AI in Policing, Brennan Ctr. for Just. (Nov. 20, 2025), https://www.brennancenter.org/our-work/research-reports/dangers-unregulated-ai-policing.
This article is part of the Hardwiring Justice series on Artificial Intelligence and the Justice System. This is Part 5 in the series examining how AI is shaping policing, prosecution, defense practice, and the courts.*
Artificial intelligence (AI) tools are now embedded across nearly every phase of criminal proceedings, from investigative analytics and risk assessment to case management and predictive analysis.[1] As these systems become more integrated, a familiar concern emerges: opacity, often called the “black box” problem. [2] Judges, lawyers, and justice professionals worry that algorithmic systems produce results that cannot be fully understood or challenged. [3] That concern is legitimate, but it is frequently misdirected. The real issue is not simply whether these systems can be explained, but whether defendants have a meaningful opportunity to contest them.
What Procedural Justice Actually Requires
Procedural justice[4] is not a synonym for accuracy. It is a distinct set of requirements about how legal processes must treat the people subject to them.[5] Research in this field is consistent across decades and jurisdictions: people evaluate the legitimacy of legal outcomes primarily through process, not result.[6] They ask whether the process treated them with dignity, whether their voice was heard, whether the rules were applied consistently, and whether the decision-maker was accountable.[7] A correct outcome reached through an unfair process does not satisfy these requirements.[8]Neither does an accurate algorithm deployed without meaningful opportunity to challenge it.
This matters for AI evidence because the two inquiries can diverge sharply. A facial recognition system might produce a statistically defensible identification rate[9] and still be used in a way that denies a defendant any meaningful opportunity to examine how it was trained, how it performs across demographic groups, or whether it was deployed within its validated scope. At the point of that denial, the process fails on procedural justice grounds regardless of whether the underlying output was correct. Accuracy and fairness are not the same inquiry.
Perceived legitimacy has practical consequences that extend beyond individual cases.[10] Compliance, cooperation with law enforcement, and the social foundations of institutional authority all depend on whether people believe the system treats them fairly.[11] Communities that already are skeptical about the criminal justice system are particularly sensitive to processes that substitute machine authority for accountable human judgment.
How Opacity Defeats Fair Process
The adversarial process is the mechanism through which due process is operationalized in American criminal proceedings.[12] Evidence is tested. Witnesses are cross-examined. Expert conclusions are subject to challenge. When an algorithmic tool enters a case shielded by trade secret protections, that mechanism breaks down.
A defendant who cannot learn how a system works, what its known failure modes are, or whether it has been independently validated cannot mount a meaningful challenge. The adversarial process has not been satisfied. It has been bypassed. Part of what makes this persistent is that courts and justice agencies have rarely defined what disclosure is actually required when these systems influence legal decisions. In the absence of that definition, vendors protect as much as possible and agencies accept whatever is offered. The opacity is not inevitable. It is the predictable result of an institutional failure to set terms.
AI outputs do not arrive neutrally. A facial recognition match carries an implicit claim to objectivity that is difficult for jurors and judges to critically interrogate without access to performance data.[13] The trade secret claim for a risk assessment is therefore not simply an IP protection[14]. In the criminal context, it functions as a structural barrier to the kind of contestation that procedural justice requires.
The doctrinal consequences of opacity run deeper than procedural unfairness. Scholars examining AI through the lens of civil and criminal liability have concluded that black-box systems defeat the foundational legal tests courts depend on to assign responsibility, intent, foreseeability, causation, because the reasoning behind algorithmic outputs cannot be reconstructed even by the systems’ own developers.[15] When legal doctrine cannot reach a tool, governance must.
The Accountability Gap Is Two-Sided
Vendor opacity is the more visible dimension of the governance failure, but it is not the only one. The second is judicial AI literacy, and the absence of it is just as consequential. Procedural justice requires not only that defendants have access to relevant information but that the decision-maker can meaningfully evaluate it. A judge who cannot assess the reliability, demographic performance, or operational limits of an algorithmic tool is not exercising the kind of informed, accountable judgment that fair process demands.
Research examining how algorithmic recommendations interact with human judgment finds that when the two appear together, the algorithm shapes the outcome in the substantial majority of cases, even where the judge formally retains decision-making authority.[16] That is not deference in any meaningful sense. It is delegation without accountability, the formal structure of human judgment without its substance. Addressing vendor opacity while leaving judicial competency unreformed solves only half the problem. The governance failure is institutional on both sides.
What Defendants Are Owed
Framing disclosure obligations around what procedural fairness requires, rather than what vendors are prepared to offer, changes both the scope and focus of the inquiry. Source code is the wrong target. It describes how a system was built. It does not reveal how the system performs in practice, across what populations, and within what limits. Four categories of information are what procedural justice actually demands.
Independent validation. A defendant has a legitimate interest in knowing whether the tool was tested by someone other than its developer, on a population comparable to theirs, under conditions comparable to those in which it was used. Validation data is not a technical nicety. It is what transforms an algorithmic claim into evidence that can be evaluated.
Disaggregated error rates. Aggregate accuracy figures are not sufficient. A system that performs well on average may fail disproportionately for specific demographic groups. Procedural fairness requires that a defendant be able to assess whether a tool’s error profile is relevant to their case specifically, not merely adequate in the aggregate.
Operational scope and actual deployment. Under what conditions was the system designed to operate, and were those conditions present in this case? The gap between validated scope and actual deployment is among the most significant fairness risks in algorithmic evidence. A defendant is entitled to know whether the tool was used as designed.
Human accountability. Who reviewed the output before it influenced the investigation or prosecution, and what qualified them to do so? Procedural justice is not satisfied by a system that generates a result. It requires that an accountable human being stood behind the decision to act on it, someone who can be questioned, challenged, and held responsible.
When “Explainability” Is Not Enough
The AI industry has developed a fluent vocabulary of transparency.[17] Systems are marketed as interpretable, explainable, and auditable.[18] In a governance context these claims require scrutiny, because they describe two distinct things that are frequently conflated.
Explain ability as a product feature produces documentation designed for comprehensibility, not accountability.[19] A visualization showing which pixels influenced a facial recognition match is built to make a user feel confident in the output.[20] It is not built to help a defense attorney identify the demographic conditions under which the system fails, or to help a judge assess whether performance characteristics are adequate to the evidentiary weight being placed on the result.[21]
Explain ability as a procedural justice obligation is something different: the ability of affected parties to obtain the information they need to contest a decision.[22] That standard is not met by vendor-generated documentation, however well-designed. It is met by independent validation, disclosed error rates, and accountable human review. Courts that treat the former as a substitute for the latter have not resolved the transparency problem. They have formalized the conditions under which it persists.
Governance as a Trust Obligation
What follows from taking procedural justice seriously as the organizing principle is that governance of AI evidence is not primarily a technical project. It is an institutional one. The question is not whether these systems can be made more transparent in some engineering sense. It is whether legal institutions will define and enforce the conditions under which algorithmic tools may influence the liberty of people who appear before them.
Communities with the deepest exposure to algorithmic surveillance in law enforcement are also those with the strongest reasons to distrust institutions that deploy these tools without
accountability.[23] When algorithmic outputs are difficult to challenge and easy to present as objective, they do not merely produce individual injustices. They accelerate the erosion of institutional legitimacy in precisely the communities where that legitimacy is most consequential and most fragile.
The question is not unprecedented. Algorithmic and computational tools already operate under structured oversight regimes in medicine, aviation, and financial regulation, fields where the consequences of unreliable outputs are similarly serious. Those regimes define testing requirements, performance standards, error rate disclosures, and limits of use. They do not require revealing proprietary source code. They require the information necessary to evaluate whether a tool is doing what it claims to do, in the conditions where it is being used. Courts face the same task. The difference is that they have not yet done it.
The governance mechanisms addressed this series, ethics, expert protocols, admissibility, professional accountability frameworks, each serve an evidentiary function. But each also serves a procedural justice function more fundamental than the evidentiary one. Getting this right is not only about keeping unreliable evidence out of court. It is about whether the institutions that deploy these tools can claim to be administering justice.
[1] Clayton Vickers, How AI Risks Creating a ‘Black Box’ at the Heart of US Legal System, The Hill (Apr. 7, 2024, 6:00 AM ET), https://thehill.com/business/personal-finance/4571982-ai-black-box-legal-system/.
[2] Id.
[3] U.S. Dep’t of Justice, Artificial Intelligence and Criminal Justice: Final Report (Dec. 3, 2024), https://www.justice.gov/olp/media/1381796/dl?inline.
[4] Also call procedural fairness
[5] Kevin Burke & Steve Leben, Procedural Fairness: A Key Ingredient in Public Satisfaction, Am. Judges Ass’n White Paper (2007).
[6] Id.
[7] Id.
[8] Id.
[9] El Fadel N. Facial Recognition Algorithms: A Systematic Literature Review. J Imaging. 2025 Feb 13;11(2):58. doi: 10.3390/jimaging11020058. PMID: 39997560; PMCID: PMC11856072.
[10] Burke & Leben, supra note 5.
[11] Id.
[12] Lucia Zedner & Carl-Friedrich Stuckenberg, Due Process, in TREATISE ON INTERNATIONAL CRIMINAL LAW (Kai Ambos et al. eds., Cambridge Univ. Press 2019).
[13] Michael Christopher Naughton, Considering Face Value: The Complex Legal Implications of Facial Recognition Technology, Crim. Just., Winter 2025, https://www.americanbar.org/groups/criminal_justice/resources/magazine/2025-winter/face-value-complex-legal-implications-facial-recognition-tech/.
[14] Lena Chan, Note, The Weaponization of Trade Secret Law, 124 Colum. L. Rev. 859 (2024).
[15] Yavar Bathaee, The Artificial Intelligence Black Box and the Failure of Intent and Causation, 31 Harv. J.L. & Tech. 889 (2018).
[16] Reuben Binns, Human Judgment in Algorithmic Loops: Individual Justice and Automated Decision-Making, 16 Reg. & Governance 197 (2022).
[17] Nagadivya Balasubramaniam, Marjo Kauppinen, Antti Rannisto, Kari Hiekkanen & Sari Kujala, Transparency and Explainability of AI Systems: From Ethical Guidelines to Requirements, 159 Info. & Software Tech. 107197 (2023).
[18] Id.
[19] Christian Kästner, Explainability, in Machine Learning in Production: From Models to Products, ch. 25 (2024), https://mlip-cmu.github.io/book/25-explainability.html.
[20] Id.
[21] Karen McGregor Richmond, Satya M. Muddamsetty, Thomas Gammeltoft-Hansen, Henrik Palmer Olsen & Thomas B. Moeslund, Explainable AI and Law: An Evidential Survey, 3 Digital Soc’y 1 (2024).
[22] L. Metikoš & I. van Domselaar, Procedural Justice and Judicial AI: Substantiating Explainability Rights with the Values of Contestation (May 5, 2025) (unpublished manuscript), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5242905.
[23] Rachel Levinson-Waldman & Ivey Dyson, The Dangers of Unregulated AI in Policing, Brennan Ctr. for Just. (Nov. 20, 2025), https://www.brennancenter.org/our-work/research-reports/dangers-unregulated-ai-policing.
Get more articles like this
in your inbox
Subscribe to our mailing list and get the latest information and updates to your email inbox.
Thank you for subscribing.
Something went wrong.
