The following is a guest post from Anthony Fowler:
The Negative Effect Fallacy, Gobbledygook, and the Use of Quantitative Evidence in the Supreme Court
The Supreme Court has a mixed track record when it comes to quantitative evidence, and that record was further blemished earlier this month, during oral arguments for Gill v. Whitford, when Chief Justice John Roberts dismissed a quantitative metric as “sociological gobbledygook” without providing further explanation. This kind of convenient but vacuous argument is, unfortunately, a regular tactic for some of the justices. In a recent article in the Journal of Empirical Legal Studies, Ryan Enos, Christopher Havasy, and I analyze another argument commonly used by the Court to justify the dismissal of quantitative evidence, and we call it the negative effect fallacy.
Courts often hear cases in which the answers to difficult empirical questions are relevant for decisions. Within the context of election law, consider Arizona Free Enterprise v. Bennett. A key question was whether the matching funds provision in Arizona’s campaign finance law chilled private political speech. The plaintiffs argued that it did; in other words, they felt that matching funds have a negative effect on private political contributions. Ryan Enos, Conor Dowling, Costas Panagopoulos, and I sought to quantitatively assess whether this claim was true, and we found no evidence to support it. In an amici brief and a subsequent article in the Election Law Journal, we showed that private political contributions did not increase relative to other states after an injunction against the matching funds provision. Chief Justice Roberts, writing the majority opinion, briefly considered this evidence but dismissed it, declaring “it is never easy to prove a negative.”
What’s going on here? Roberts has conflated the arithmetic and philosophical definitions of the word negative. We’ve all heard the adage that it’s difficult to prove a negative, meaning that it’s difficult to prove the absence of something (e.g., can we prove that Santa Claus does not exist?). However, the adage has no bearing on the quantitative estimation of an arithmetically negative effect![1] There’s no reason that negative effects are harder to detect than positive ones.
The negative effect fallacy is not unique to Arizona Free Enterprise or election law. This mistake appears to have originated with Elkins v. United States in 1960. Again, an empirical question played a crucial role in the case. Does the exclusionary rule prevent illegal searches and seizures? In principle, quantitative evidence could be brought to bear on this question, but Justice Stewart rejected even the possibility by asserting that “it is never easy to prove a negative.” Since then, negative effect fallacy has been repeated in many Supreme Court and lower court cases across many legal domains including free speech, voting rights, and campaign finance. Our investigation explains the negative effect fallacy in more detail, documents its use in federal courts, and provides recommendations regarding the use of quantitative evidence in court decisions.
How can the greatest legal minds in our nation fall prey to something like the negative effect fallacy? And why would they insist that relatively simple quantitative methods are gobbledygook? We can only speculate regarding the underlying motivations of the judges. Perhaps they don’t understand the evidence. Perhaps they don’t like the results and look for convenient ways to dismiss them. Perhaps they have legitimate reasons to believe the evidence is uncompelling or irrelevant for the case, but they fail to articulate those reasons. In any case, the pattern whereby evidence is dismissed based on sweeping statements, gut reactions, and logical fallacies is a troubling one. At best, these practices hide the justices’ true reasons for arriving at their decisions, and at worst, the courts regularly make bad decisions because they fail to engage with relevant evidence.
The relevance of quantitative evidence for legal decisions will likely not dissipate anytime soon. Does a new voting technology differentially impact a particular racial group? Does a campaign finance regulation restrict free speech? Does a state’s redistricting plan deviate from a reasonable standard of fairness? Courts will have to consider a lot of questions like these going forward, and social scientists will continue to collect data and develop methods to help them. Whether the justices like it or not, answering these questions and making informed decisions requires engaging with quantitative evidence and evaluating it on its merits.
Anthony Fowler ([email protected]) is Associate Professor in the Harris School of Public Policy at the University of Chicago. He applies econometric methods for causal inference to questions in political science, with particular emphasis on elections and political representation.
[1] Even in the canonical usage of this adage, the word negative is a red herring because we can always write positive statements to be negative, and vice versa. At best, this phrase reminds us that induction does not provide certain conclusions, but in the case of arithmetically negative statements, the adage has no bearing whatsoever.