Explanations of Machine Learning

Abstract

There is an unavoidable tension between prediction and explanation: between the idea that predictive accuracy is what matters, and the requirement for explanation. In the law, Holmes drew a distinction between explanation as articulated in a judicial decision using the language of logic, and the real explanation that can be found by looking at the judge’s body of experience, what he called the “inarticulate major premise”; but he went beyond both of these and instead invoked prophecy as the fundamental constituent of law. Holmes prefigured a debate that machine learning has brought to the fore: the debate between machine learning’s “prediction culture” which values case-by-case predictive accuracy, and the scientific “explanation culture” which looks for explanations in terms of rules. The difference between these two cultures was missed by an earlier generation of philosophers of science, who it seems did not conceive of machines whose function is sufficiently advanced that it seems like magic. The prediction culture has made striking advances … and yet, explainability remains a policy priority, because of the impact that machine learning is having on society.

The danger of which I speak is […] the notion that a given system, ours, for instance, can be worked out like mathematics from some general axioms of conduct. […] This mode of thinking is entirely natural. The training of lawyers is a training in logic. The processes of analogy, discrimination, and deduction are those in which they are most at home. The language of judicial decision is mainly the language of logic. And the logical method and form flatter that longing for certainty and for repose which is in every human mind. But certainty generally is illusion, and repose is not the destiny of man.

Oliver Wendell Holmes, Jr., The Path of the Law (1897)

Calls for “explainability ” of machine learning outputs are much heard today, in academic and technical writing as well as in legislation such as the GDPR, the European Union’s General Data Protection Regulation.¹ We are not convinced that very many lawmakers or regulators understand what would need to be done, if the explainability they call for is to be made meaningful.² It might seem to a legislator, accustomed to using the logical language of law and legal decisions, that an algorithmic decision “can be worked out like mathematics from some general axioms.” In the law , Holmes rejected the idea that a logical argument gives a satisfactory explanation of a judicial decision. Instead, in a play upon Aristotle’s system of logic, he invoked the “inarticulate major premise .”³ As a method to account for legal decision making, that idea, as Holmes employed it, left to one side the formal logic that jurists classically had employed to explain a legal output . In this chapter we will suggest that Holmes’s idea of the inarticulate major premise offers a better way to think about explanations in machine learning—and also throws fresh light on a fundamental philosophical stance in machine learning , the “prediction culture.”

6.1 Holmes’s “Inarticulate Major Premise”

The premise behind a decision, it was Holmes’s view, is not always expressed. Legal decision-makers offer an apologia, a logical justification for the decision they have reached, but the real explanation for a decision is to be found in the broad contours of experience that the decision-maker brings to bear. As Holmes put it in 1881 in The Theory of Interpretation, decision-makers “leave their major premises inarticulate .”⁴

Holmes addressed this phenomenon again, and most famously, in his dissent in Lochner v. New York. To recall, the Supreme Court was asked to consider whether a New York state law that regulated working hours in bakeries and similar establishments was constitutionally infirm. The majority decided that it was. According to the majority, the law interfered with “liberty” as protected by the 14th Amendment of the Constitution. Holmes in his dissent wrote as follows:

Some of these laws embody convictions or prejudices which judges are likely to share. Some may not. But a Constitution is not intended to embody a particular economic theory, whether of paternalism and the organic relation of the citizen to the state or of laissez faire. It is made for people of fundamentally differing views, and the accident of our finding certain opinions natural or familiar, or novel, and even shocking, ought not to conclude our judgment upon the question whether statutes embodying them conflict with the Constitution of the United States.

General propositions do not decide concrete cases. The decision will depend on a judgment or intuition more subtle than any articulate major premise. But I think that the proposition just stated, if it is accepted, will carry us far toward the end. Every opinion tends to become a law. I think that the word ‘liberty’ in the 14th Amendment, is perverted when it is held to prevent the natural outcome of a dominant opinion, unless it can be said that a rational and fair man necessarily would admit that the statute proposed would infringe fundamental principles as they have been understood by the traditions of our people and our law. It does not need research to show that no such sweeping condemnation can be passed upon the statute before us.⁵

Holmes rejected the straightforward logical deduction that would have read like this: The 14th Amendment protects liberty; the proposed statute limits freedom of contract; therefore the proposed statute is unconstitutional. He did not accept that this “general proposition[…]” contained in the 14th Amendment could “decide concrete cases,” such as the case that the New York working hours law presented in Lochner. It was Holmes’s suspicion that laissez faire economic belief on the part of the other judges was the inarticulate premise lurking behind the majority opinion, the premise that had led them to offer their particular deduction. Holmes posited instead that the meaning of the word “liberty” in the 14th Amendment should be interpreted in the light of the “traditions of our people and our law” and that applying “judgement or intuition” to the state of affairs prevailing in early twentieth century America reveals that the pattern of “dominant opinion” did not favor an absolute free market. Holmes concluded that the dominant opinion was for an interpretation that upheld the New York state statute limiting working hours. It was only to a judge who had laissez faire economic beliefs that the statute would appear “novel, and even shocking.” It was not a syllogism but the majority judges’ economic beliefs—and accompanying sense of shock—that led them to strike the statute down.⁶

So a judge who expresses reasons for a judgment might not in truth be explaining his judgment. Such behavior is plainly at odds with the formal requirements of adjudication: the judge is supposed to say how he reaches his decisions. Holmes assumed that judges do not always do that. Their decisions are outputs derived from patterns found in experience,⁷ not answers arrived at through logical proof. In Holmes’s view, even legal texts, like constitutions, statutes, and past judgments, do not speak for themselves. As for artefacts of non-textual “experience”—the sources whose “significance is vital, not formal”⁸—those display their patterns even less obviously. Holmes thought that all the elements of experience, taken in aggregate, were the material from which derives the “judgment or intuition more subtle than any articulate major premise.” It might not even “need research to show” what the premise is. How a decision-maker got from experience to decision—how the decision-maker found a pattern in the data, indeed even what data the decision-maker found the pattern in—thus remains unstated and thus obscure.

6.2 Machine Learning’s Inarticulate Major Premise

It is said—and it is the premise behind such regulatory measures as the GDPR—that machine learning outputs require explanation. Holmes’s idea of the inarticulate major premise speaks directly to the problem of how to satisfy this requirement. Holmes said that the logic presented in a judicial decision to justify that decision was not an adequate explanation, and that for a full explanation one must look also to the body of experience that judges carry with them. For Holmes, the formal principle stated in a statute, and even in a constitutional provision, is not an adequate guide to the law, because to discern its proper meaning one must look at the traditions and opinions behind it.

Likewise, when considering the output of a machine learning system , the logic of its algorithms cannot supply an adequate explanation. We must look to the machine’s “experience,” i.e. to its training dataset.

One reads in Articles 13, 14, and 15 of the GDPR, the central loci of explainability, that meaningful information about an automated decision will come from disclosing “the logic involved.”⁹ This is a category error. A machine learning output cannot be meaningfully assessed as if it were merely a formula or a sum. A policymaker or regulator who thinks that machine learning is like that is like the unnamed judge whom Holmes made fun of for thinking a fault in a court judgment could be identified the way a mistake might be in arithmetic, or the named judges whom he said erred when they deduced that working hour limits on bakers are unconstitutional. The logic of deduction, in Holmes’s idea of the law, is not where law comes from; it is certainly not in machine learning where outputs come from. The real source—the inarticulate major premise of law and of machine learning alike—is the data or experience.

If you follow Holmes and wish to explain how a law or judgment came to be, you need to know the experience behind it. If you wish to explain how a machine learning process generated a given output , you need to know the data that was used to train the machine . If you wish to make machine learning systems accountable, look to their training data not their code. If there is something that one does not like in the experience or in the data, then chances are that there will be something that one does not like in the legal decision or the output .

6.3 The Two Cultures: Scientific Explanation Versus Machine Learning Prediction

To explain a decision, then, one must explain in terms of the data or experience behind the decision. But what constitutes a satisfactory explanation? In Law in Science and Science in Law , Holmes in 1899 opened the inquiry like this:

What do we mean when we talk about explaining a thing? A hundred years ago men explained any part of the universe by showing its fitness for certain ends, and demonstrating what they conceived to be its final cause according to a providential scheme. In our less theological and more scientific day, we explain an object by tracing the order and process of its growth and development from a starting point assumed as given.¹⁰

Even where the “object” to be explained is a written constitution, Holmes said an explanation is arrived at “by tracing the order and process of its growth and development,” as if the lawyer were a scientist examining embryo development under a microscope.¹¹ And yet for all of Holmes’s scientific leaning, his best known epigram is expressed with an emphatically non-scientific word: “The prophecies of what the courts will do in fact, and nothing more pretentious, are what I mean by the law.”¹²

Holmes seems to anticipate a tension that the philosophy of science has touched on since the 1960s and that the nascent discipline of machine learning since the 2000s has brought to the fore: the tension between explaining and predicting.

In the philosophy of science, as advanced in particular by Hempel ,¹³ an explanation consists of (i) an explanans consisting of one or more “laws of nature” combined with information about the initial conditions, (ii) an explanandum which is the outcome, and (iii) a deductive argument to go from the explanans to the explanandum. In fact, as described by Shmueli in his thoughtful examination of the practice of statistical modelling To explain or to predict?¹⁴ it is actually the other way round: the goal of statistical modelling in science is to make inferences about the “laws of nature” given observations of outcomes. Terms like “law ” and “rule” are used here. Such terms might suggest stipulation, like legal statutes, but in this context they simply mean scientific or engineering laws: they could be causal models¹⁵ that aim to approximate nature, or they could simply be equations that describe correlations.

In the machine learning/prediction culture championed by Leo Breiman in his 2001 rallying call Statistical modelling: the two cultures , from which we quoted at the opening of Chapter 1, the epistemological stance is that explanation in terms of laws is irrelevant; all that matters is the ability to make good predictions. The denizens of the prediction culture sometimes have an air of condescension, hinting that scientists who insist on an explanation for every phenomenon are simpletons who, if they do not understand how a system works, can’t imagine that it has any value. In the paper describing their success at ImageNet Challenge in 2012—following which the current boom in machine learning began—Krizhevksy et al. noted the challenge of getting past the gatekeepers of scientific-explanatory culture: “[A] paper by Yann LeCun and his collaborators was rejected by the leading computer vision conference on the grounds that it used neural networks and therefore provided no insight into how to design a vision system.”¹⁶ LeCun went on to win the 2018 Turing Award (the “Nobel prize for computer science”) for his work on neural networks¹⁷ and to serve as Chief AI Scientist for Facebook.¹⁸

Here is an illustration of the difference between the two cultures, as applied to legal outcomes. Suppose our goal is to find a formula for the probability that defendants will flee if released on bail: here we are inferring a rule, a formula that relates the features of objects under consideration to the outcomes, and which can be applied to any defendant. Or suppose our goal is to determine whether the probability is higher for violent crime or for drug crime all else being equal: here again we are making an inference about rules (although this is a more subtle type of inference, a comparative statement about two rules which does not actually require those rules to be stated explicitly).

By contrast, suppose our goal is to build an app that estimates the probability that a given defendant will flee: here we are engaging in prediction.¹⁹ We might make a prediction using syllogistic inference, or by reading entrails, or with the help of machine learning . The distinguishing characteristic of prediction is that we are making a claim about how some particular case is going to go.

Making a prediction about a particular case and formulating a rule of general application are tightly interwoven. Their interweaving is visible in judicial settings. One of Holmes’s Supreme Court judgments is an example. Typhoid fever had broken out in St. Louis, Missouri. The State of Missouri sued Illinois, on the theory that the outbreak was caused by a recent change in how the State of Illinois was managing the river at the city of Chicago. In State of Missouri v. State of Illinois Holmes summarized Missouri’s argument as follows:

The plaintiff’s case depends upon an inference of the unseen. It draws the inference from two propositions. First, that typhoid fever has increased considerably since the change, and that other explanations have been disproved; and second, that the bacillus of typhoid can, and does survive the journey and reach the intake of St. Louis in the Mississippi.²⁰

In support of this second proposition, Missouri put forward rules, formulated with reference to its experts’ observations, stating how long the typhoid bacillus survives in a river and how fast the Mississippi River might carry it from Chicago to St. Louis. If you accept the rules that Missouri formulated from its experts’ observations, then you could express the situation like this:

• Let x = miles of river between downstream location of outbreak and upstream location of a typhoid bacillus source.

• Let y = rate in miles per day at which typhoid bacillus travels downstream in the river.

• Let z = maximum days typhoid bacillus survives in the river.

• If x ÷ y ≤ z, then the bacillus survives—and downstream plaintiff wins;

• If x ÷ y > z, then the bacillus does not survive—and downstream plaintiff loses.

Expressing the situation this way necessarily has implications for other cases. Justice Holmes drew attention to the implications: the winning formula for Missouri as plaintiff against Illinois might well later have been a losing one for Missouri as defendant against a different state. “The plaintiff,” Holmes wrote, “obviously must be cautious upon this point, for if this suit should succeed, many others would follow, and it not improbably would find itself a defendant to a [suit] by one or more of the states lower down upon the Mississippi.”²¹

Missouri was making an inference of the unseen in a particular instance, which in machine learning terminology is referred to as prediction . Missouri used general propositions to support this prediction, and Holmes (with his well-known distrust of the general proposition) warned that such reasoning can come back to bite the plaintiff.

The difference between finding rules and making predictions might seem slight. If we have a rule, we can use it to make predictions about future cases; if we have a mechanism for making predictions, that mechanism may be seen as the embodiment of a rule. Hempel did not see any great difference between explanation and prediction . To Hempel, an explanation is after the fact, a prediction is before the fact, and the same sort of deductive reasoning from natural laws applies in both cases.

But what if it is beyond the grasp of a simple-minded philosopher—or, for that matter, of any human being—to reason about the predictive mechanism? This is the real dividing line between the two cultures. The scientific culture is interested in making inferences about rules, hence a fortiori practitioners in the scientific culture will only consider rules of a form that can be reasoned about. The prediction culture, by contrast, cares about prediction accuracy, even if the prediction mechanism is so complex it seems like magic.

Arthur C. Clarke memorably said, “Any sufficiently advanced technology is indistinguishable from magic.”²² Clarke seems to have been thinking about artefacts of a civilization more advanced than that of the observer trying to comprehend them. Thus, a stone age observer, presented with a video image on a mobile phone, might think it magical. It would take more than moving pictures to enchant present-day observers, but we as a society have built technological artefacts whose functioning we struggle to explain.

The prediction culture says that we should evaluate an artefact, even one that seems like magic, by whether or not it actually works. We can still make use of machines that embody impenetrable mechanisms; we should evaluate them based on black-box observations of their predictive accuracy. A nice illustration may be taken from a case from the U.S. Court of Appeals for the 7th Circuit in 2008. A company had been touting metal bracelets. The company’s assertions that the bracelets were effective as a cure for various ailments were challenged as fraudulent. Chief Judge Easterbrook, writing for the 7th Circuit, recalling the words of Arthur C. Clarke that we’ve just quoted above, was dubious about “a person who promotes a product that contemporary technology does not understand”; he said that such a person “must establish that this ‘magic’ actually works. Proof is what separates an effect new to science from a swindle.”²³ Implicit here is that the “proof,” while it might establish that the “magic” works, does not necessarily say anything about how it works. Predicting and explaining are different operations. Easterbrook indeed goes on to say that a placebo-controlled, double-blind study—that is, the sort of study prescribed by the FDA for testing products that somebody hopes to market as having medical efficacy—is “the best test” as regards assertions of medical efficacy of a product.²⁴ Such a test, in itself, solely measures outputs of the (alleged) medical device; it is not “proof” in the sense of a mathematical derivation. It does not require any understanding of how the mechanism works; it is just a demonstration that it does work. True, a full-scale FDA approval process—a process of proof that is centered around the placebo-controlled, double-blind study that the judge mentions—also requires theorizing as to how the mechanism works, not just black-box analysis. But Easterbrook here, focusing on proof of efficacy, makes a point much along the lines of Breiman: a mechanism can be evaluated purely on whether one is satisfied with its outcomes, rather than on considerations such as parsimony or interpretability or consonance with theory.²⁵ A mechanism can be evaluated by seeking to establish whether “this ‘magic’ actually works.”

Holmes made clear his view that a judicial explanation is really an apologia rather than an explanation, and that the real explanation is to be found by looking for the “inarticulate major premise ” that comes from the jurist’s body of experience. Holmes shied away from asking for logical or scientific explanations as a way to understand the jurist’s experience. He instead invoked prophecy. Holmes went beyond logic (because simple mathematical arguments are inadequate), and beyond scientific explanation (perhaps because such explanation would either be inaccurate or incomprehensible when applied to jurists’ behavior), and he came finally to prediction. In this, Holmes anticipated machine learning .

6.4 Why We Still Want Explanations

The inarticulate major premise , starting immediately after Holmes’s dissent in Lochner, provoked concern, and it continues to.²⁶ Unexplained decisions, or decisions where the true reasons are obscured, are inscrutable, and, therefore, the observer has no way to tell whether the reasons are valid. Validity, for this purpose, may mean technical correctness; it also may mean consonance with basic values of society. Testing validity in both these senses is an objective behind explainability. We turn here in particular to values.²⁷

Eminent readers of Holmes conclude that he didn’t have much to say about values.²⁸ But he was abundantly clear that, whatever the values in society might be, if they form a strong enough pattern, then they are likely to find expression in law: “Every opinion tends to become law.”²⁹ Whether or not one has an opinion about the opinion that becomes law, Holmes described a process that has considerable present-day resonance. Data from society at large will embody opinions held in society at large; and thus a machine learning output derived from a pattern found in the data will itself bear the mark of those opinions.

The influence of opinions held in society would be quite straightforward if there were no conflicting opinions. But many opinions do conflict. Holmes plainly was concerned about discordance over values; it was to accommodate “fundamentally differing views” that he said societies adopt constitutions.³⁰ Less clear is whether he thought that certain values are immutable, imprescriptible, or in some fashion immune to derogation. He suggested that some might be: he said that a statute might “infringe fundamental principles.” He didn’t say what principles might be fundamental.

A law, if it embodied certain biases held in society, would infringe principles held to be fundamental today. Examples include racial and gender bias. In Holmes’s terms, those are “opinions” that should not “become law.” Preventing them from becoming law is a central concern today. The concern arises, mutatis mutandis, with machine learning outputs . Where machine learning outputs have legal effects, they too will infringe fundamental principles, if they embody biases such as racial or gender bias. Preventing such “opinions” from having such wider influence is one of the main reasons that policy makers and writers have called for explainability.

In short, in both processes, law and machine learning, the risk exists that experience or data shaped a decision that ought not have been allowed to.³¹ In both, however, the experience or the data might not be readily visible.³² As we will explore in Chapters 7 and 8, much of the concern over its potential impact on societal values relates to this obscurity in machine learning’s operation.

Notes

• 1. For literature see, e.g., Casey, The Next Chapter in the GDPR’s “Right to Explanation” Debate and What It Means for Algorithms in Enterprise, European Union Law Working Papers, No. 29 (2018) and works cited id., at p. 14 n. 41.

• 2. See Grant & Wischik, Show Us the Data: Privacy, “Explainability,” and Why the Law Can’t Have Both, forthcoming, 88 Geo. Wash. L. Rev. (Nov. 2020). See also, positing a conflict between privacy and data protection regulations, on the one hand, and anti-discrimination regulations on the other, Žliobaitė & Custers (2016).

• 3. In Aristotle’s logic, the “major premise” is a stated element at the starting point of a syllogism. See Robin Smith, Aristotle’s Logic, in Zalta (ed.), The Stanford Encyclopedia of Philosophy (Summer 2019 edn.): https://plato.stanford.edu/entries/aristotle-logic/.

• 4. Holmes, The Theory of Legal Interpretation, 12 Harv. L. Rev. 417, 420 (1898–1899).

• 5. Lochner v. New York, 198 U.S. 45, 75–76, 25 S.Ct. 539, 547 (Holmes, J., dissenting, 1905).

• 6. Differences of interpretation exist among jurists reading the passage in Holmes’s Lochner dissent about “[g]eneral propositions” and judgments or intuitions “more subtle than any articulate major premise.” No less an authority on Holmes than Judge Posner once referred to the passage to mean that certain “statements should be treated as generalities open to exception”: Arroyo v. U.S., 656 F.3d 663, 675 (Posner, J., concurring, 7th Cir., 2011). We read it to mean something more. It means that reasons that in truth led to a judicial outcome are sometimes not expressed in the judgment. That is, stated reasons in a judgment, which typically take the form of a logical proof proceeding to the judge’s conclusion from some major premise that the judge has articulated, are not the real explanation for why the judge concluded the way he did. Our reading accords with a train of thought running through Holmes’s work, at least as far back as The Common Law (1881). A number of judges have read the passage as we do: See, e.g., City of Council Bluffs v. Cain, 342 N.W.2d 810, 814 (Harris, J., Supreme Court of Iowa, 1983); Loui v. Oakley, 438 P.2d 393, 396 (Levinson, J., Supreme Court of Hawai’i, 1968); State v. Farrell, 26 S.E.2d 322, 328 (Seawell, J., dissenting, Supreme Court of North Carolina, 1943).

• 7. Such experience not infrequently includes implicit bias. For an example, see Daniel L. Chen, Yosh Halberstam, Manoj Kumar & Alan C. L. Yu, Attorney Voice and the US Supreme Court, in Livermore & Rockmore (eds.) (2019) p. 367 ff.

• 8. Gompers v. United States, 233 U.S. 604, 610 (1914).

• 9. Emphasis ours. See also in the GDPR Arts. 21–21 and Recital 71. One reads in the legal scholarship, too, that it is because some algorithms are “more complex” than others that they are harder to explain. See, e.g., Hertza, 93 N.Y.U. L. Rev. 1707, 1711 (2018). Mathematical complexity of algorithms is not what drives machine learning, however. Data is. See Chapter 3, pp. 35–38.

• 10. Law in Science and Science in Law , 12 Harv. L. Rev. at 443 (1898–1899).

• 11. Cf. Holmes’s description of a constitution as “the skin of a living thought”, Chapter 4, p. 47, n. 7.

• 12. Path of the Law , 10 Harv. L. Rev. at 461 (1896–1897).

• 13. See, for example, Scientific Explanation, from the Stanford Encycl. Philos. (Sept. 24, 2014): https://plato.stanford.edu/entries/scientific-explanation/. Of relevance here is the inductive-statistical model, due to Hempel (1965).

• 14. Galit Shmueli, To Explain or to Predict? 25(3) Stat. Sci. 289–310 (2010).

• 15. In the social sciences, inference, especially inference about causal relationships, is typically preferred to prediction. But for a defense of prediction see Allen Riddell, Prediction Before Inference, in Livermore & Rockmore (ed.) (2019) 73–89. See also Breiman, The Two Cultures , quoted above, Chapter 1, p. 1.

• 16. Krizhevksy, Sutskever & Hinton (2017).

• 17. https://amturing.acm.org/award_winners/lecun_6017366.cfm.

• 18. https://www.linkedin.com/in/yann-lecun-0b999/. Retrieved 19 April 2020.

• 19. Kleinberg et al. note that judges are supposed by law to base their bail decision solely on this prediction , and they show that a machine learning algorithm does a better job. Kleinberg, Lakkaraju, Leskovec, Ludwig & Mullainathan, Human Decisions and Machine Predictions, 46 Q. J. Econ. 604–32 (2018).

• 20. State of Missouri v. State of Illinois, 26 S.Ct. 270, 200 U.S. 496, 522–23 (1906).

• 21. 200 U.S. at 523.

• 22. Clarke (1962) 21.

• 23. FTC v. QT, Inc., 512 F.3d 858, 862 (7th Cir. 2008).

• 24. Id.

• 25. See Chapter 1, pp. 10–11.

• 26. Since 1917 when Albert M. Kales, “Due Process,” the Inarticulate Major Premise and the Adamson Act, 26 Yale L. J. 519 (1917), addressed Holmes’s famous Lochner dissent, over a hundred American law review articles have addressed the same. It has concerned lawyers in Britain and the Commonwealth as well: see, e.g., the Editorial Notes of the first issue of Modern Law Review: 1(1) MLR 1, 2 (1937). Writings on the matter are recursive: see Sunstein, Lochner’s Legacy, 87 Col. L. Rev. 873–919 (1987); Bernstein, Lochner’s Legacy’s Legacy, 82 Tex. L. Rev. 1–64 (2003).

• 27. For some recent work on the challenge of getting AI to reflect social values in legal operations, see Al-Abdulkarim, Atkinson & Bench-Capon, Factors, Issues and Values: Revisiting Reasoning with Cases, International Conference on AI and Law 2015, June 8–12, 2015, San Diego, CA: https://cgi.csc.liv.ac.uk/~tbc/publications/FinalVersionpaper44.pdf.

• 28. Most comprehensively, see Alschuler (2000). See also, e.g., Jackson, 130 Harv. L. Rev. 2348, 2368–70 (2017).

• 29. Lochner (Holmes, J., dissenting), op. cit.

• 30. Id.

• 31. Kroll et al. (op. cit.) described the matter in regard to machine learning like this:

  machine learning can lead to discriminatory results if the algorithms [sic] are trained on historical examples that reflect past prejudice or implicit bias, or on data that offer a statistically distorted picture of groups comprising the overall population. Tainted training data would be a problem, for example, if a program to select among job applicants is trained on the previous hiring decisions made by humans, and those prevision decisions were themselves biased. 165 U. Pa. L. Rev. at 680 (2017).

  Barocas & Selbst, 104 Cal. L. Rev. at 674 (2016), to similar effect, say, “[D]ata mining can reproduce existing patterns of discrimination, inherit the prejudice of prior decision makers, or simply reflect the widespread biases that persist in society.” Cf. Chouldechova & Roth, The Frontiers of Fairness in Machine Learning, Section 3.3, p. 6 (Oct. 20, 2018): https://arxiv.org/pdf/1810.08810.pdf.

• 32. See generally Pasquale (2015). Though the emphasis in the 2015 title on algorithms is misplaced, Pasquale elsewhere has addressed distinct problems arising from machine learning: Pasquale (2016).