President Trump’s Fraud Commission will hold its second public meeting Tuesday, September 12th. As has been noted before, the Commissioners include some of the nation’s most notorious advocates of policies that would make it harder for eligible Americans to vote. Heightening the controversy surrounding the Commission, one of those invited to testify at Tuesday’s meeting is John Lott, a figure unfamiliar to those who study elections, but well-known in other circles for his controversial gun policy research. His findings have been questioned by a panel of experts at the National Institute of Science, Engineering and Medicine, and academics like John Donohue of Stanford Law School, Dan Black of the University of Chicago, and Daniel Nagin of Harvard, have challenged Lott’s conclusions.
On Tuesday, Lott will present findings from an unpublished paper that is not peer reviewed, which purports to find evidence of voter fraud. Lott’s presentation of unvetted and unverified research is especially troubling. Unsurprisingly, his paper alleges evidence of voter fraud — a claim that has been disproven countless times.
Lott’s invitation to testify is telling of the Commission’s interest (or lack thereof) in a rigorous inquiry into election integrity. A quick Google search for Lott turns up information that will raise eyebrows, like his penchant for anonymously defending his own work in online forums, while posing as a sock puppet and using the pseudonym Mary Rosh. But, there is no need to focus on Lott’s disputed research in the gun policy arena — the paper he is submitting on voter fraud is problematic on its own terms.
Predictably, the research that Lott will present tomorrow is already the subject of dispute. Here are three important things to keep in mind when reviewing Lott’s work and testimony:
Lott’s statistical analysis includes far too many variables and controls – a tell-tale sign of statistical chicanery
One cardinal rule that Statistics 101 students learn is that when a variable is included in a statistical analysis, often referred to as a regression model, there must be a sound theoretical justification for its inclusion. A regression model measures how several factors simultaneously affect an outcome being measured. If a researcher includes too few factors in a model, there is a threat that the analysis can leave out something important – a problem methodologists refer to as “omitted variable bias.”
On the other hand, a well-reasoned regression analysis will also be careful not to include too many variables. If a researcher were to just slap together a slew of variables without much thought, obviously, the validity of the findings could also be suspect. From a research design perspective, including too many variables can give other researchers the wrong impression, namely that a paper’s author kept adding new variables to a model until the model produced desirable results.
Statisticians tend to strike a balance between having too few and too many predictors in a regression model. When statisticians see a suspicious research design with an unreasonable number of variables or controls, it can be a sign that someone is trying to gloss over the results. For those who regularly read academic papers, 10, 20 or even 30 predictors could conceivably pass the smell test.
Lott’s analysis sets off the “too many variables” alarm because it purports to measure the effects of more than 50 causal variables on a dependent variable: voter turnout (and 50 is a conservative number, since his model controls for other factors like county and year). Such unwieldy regression models can detect statistically significant relationships that, in reality, are just random noise caused by the interaction of so many variables.
Moreover, the results of oversaturated regression models like Lott’s can be extremely sensitive to minor changes in research design, like dropping one or two variables. In fact, Lott’s work was singled out as being especially sensitive to these kinds of minor changes. When the National Research Council reviewed Lott’s gun research, a panel noted that his findings were “highly sensitive to seemingly minor changes in the model specification and control variables.”
He doesn’t rethink his model in the face of implausible and erroneous results.
Ultimately an oversaturated research design will produce empirical findings that are not valid. The interactions between, and correlations among, so many variables can produce unreliable and imprecise findings. Naturally, Lott’s analysis produces estimates that are simply implausible. His paper finds that certain ballot amendments, those relating to business regulation, decrease turnout by a whopping 12 percent. He also finds that labor reform ballot measures increase turnout by almost 19 percent. As Michael McDonald at the University of Florida noted, anyone who studies elections should be immediately suspect of such findings.
He doesn’t have enough relevant observations.
While he purports to be examining the effect of voter ID laws on turnout, the time period he chose and state policies he studied will not reveal much. For example, Indiana was the first state in the country to pass a strict photo-voter ID law and the first Indiana election with the law in place was the state’s 2006 primary. Lott’s paper looks at the data from 1996–2006, and therefore can say little about the effects of the nation’s strictest voter ID laws because it only includes one election where a strict ID law was in place. To be fair, Lott makes this distinction clear in the abstract of his paper. But his paper and presentation, whether intentionally or not, muddy the waters. Lott’s research only evaluates state ID laws that have fail safes to make sure voters can access the ballot, like an affidavit option for voters that lack the necessary ID or laws that permit voters to bring various forms of ID, like a utility bill, which will not predict how strict ID laws affect turnout.
~
The Fraud Commission has been mired in controversy since its inception. If the Commission had any hopes of acquiring credibility, it should have steered far away from this paper by Lott given his controversial record.