POLICING FORENSIC EVIDENCE

If there had been no video of Derek Chauvin killing George Floyd on May 25, 2020, the legal and public response might have been quite different. Before a series of videos captured by witnesses and security cameras surfaced, the Minneapolis Police Department issued a statement: “After Floyd got out of his car, he physically resisted officers. Officers were able to get the suspect into handcuffs and noted he appeared to be suffering medical distress.”¹ The Chief Medical Examiner for the county did determine that Floyd’s death was a homicide but also speculated that Floyd suffered from “significant conditions,” such as drug use and a heart condition, that may have increased his likelihood of death.² At Chauvin’s trial, the medical examiner testified, “In my opinion, the law enforcement subdual, restraint, and the neck compression was just more than Mr. Floyd could take by virtue of those heart conditions.”³ Attorneys for Floyd’s family commissioned an independent autopsy that found that there were no such contributing factors.⁴ More specifically, they found that the cause of death was not a heart attack but rather “mechanical asphyxia”—meaning the flow of oxygen was cut off so Floyd, as he told the officers, simply could not breathe.⁵ As Martin Tobin testified at the criminal trial, given the force placed on Floyd’s neck, “[a] healthy person subjected to what Mr. Floyd was subjected to would have died.”⁶

It was not an aberration that the medical examiner’s determinations regarding cause of death were hotly disputed. There is a long, troubling history of prosecution, in media releases, labeling people killed by police as criminals or drug users or persons with preexisting medical conditions.⁷ Forensic evidence, in which medical examiners or coroners set out conclusions in death certificates, has played a troubling role in providing faulty support for such claims.⁸ All too often, questionable findings have been shared with the public and our courts, including by omitting entirely the role of police in cases of police killings.⁹ The coroner and medical examination field is rife with poor methods. A landmark 2009 National Academy of Sciences report (“NAS Report”) called death investigations in the United States “fragmented” as a system that is “deficient” in its practices and called for the replacement of local coroners' offices with a medical examiner system.¹⁰

The Floyd case is just one high-profile example of a larger problem: police evidence collection, including when it is conducted with outside experts and crime labs, lacks adequate scientific safeguards and as a result is prone to bias and error.¹¹ A central function of policing is to gather evidence during criminal investigations. Law enforcement receives a great deal of deference when it carries out those investigative functions. While detailed constitutional criminal procedure rules regulate traditional policing functions, such as searches and seizures, there is very little regulation of forensic evidence of any kind, much less regulation informed by community priorities, research, or policy.¹² To be sure, a growing number of police policies and guidelines reflect the recent effort to emphasize accurate and impartial evidence collection. The International Association of Chiefs of Police has taken an active role in promoting the consideration of ways to improve the “accuracy and thoroughness” of police investigations.¹³ Far more is needed to ensure such accuracy and thoroughness.

A second example of forensics gone wrong illustrates the problem, not in the medical examiner setting but in the crime laboratory setting. The Department of Forensic Science (DFS) in Washington, D.C., was supposed to be a model lab that would change lab work on crime scene evidence for the better. The D.C. government passed the Department of Forensic Sciences Act of 2011 to make the lab more independent of police and require it to be accredited.¹⁴ It paid tens of millions of dollars for a large new facility.¹⁵ Unfortunately, while accredited crime labs may have more in the way of quality control than a typical policing agency, they do not have the quality controls present in a scientific or clinical lab.

In a murder trial, Judge Todd Edelman ruled in August 2019 that after hearing from several experts at an extensive hearing, he was concerned that published studies had not been able to show that DFS examiners could reliably make a “source attribution.”¹⁶ The judge held that the most an expert could say for the government was that “the recovered firearm [could not] be excluded as the source of the cartridge casing found on the scene of the alleged shooting.”¹⁷ A new case involving firearms evidence analyzed in 2017 and charges against two men for two killings was in front of Judge Edelman.¹⁸ The firearms group at the lab had reported that the same weapon fired the cartridges in the two shootings.¹⁹ Perhaps because Judge Edelman had been skeptical in the past, the prosecutors wanted to make sure; they took an unusual step and asked an independent examiner to take a look at the evidence.²⁰ That examiner was sure that it was clear that two different firearms were involved.²¹ The lab defiantly insisted the outside examiner was wrong and that in fact the examiner’s report was an “administrative error” and the evidence was now deemed to be “inconclusive.”²²

The lab went into damage control mode. The lab director called it a “darn good lab.”²³ Management told the American National Standards Institute National Accreditation Board, which accredited the lab, that it had made an “inconclusive” finding after initiating a review of the incident. In April 2020, the lab’s accreditation was suspended.²⁴ The lab entirely stopped doing work as a result. Prosecutors opened a new probe into the firearms unit and disagreed with the lab in eleven cases. A scathing audit report found that lab managers sought to conceal the problem in the case before Judge Edelman and pressured others to report that the finding was “inconclusive.”²⁵ One examiner recalled how he tried to “push back as long as he could” and wished he had resigned in protest.²⁶ The auditors concluded that this illustrated “very serious” problems with management that indicated lack of adherence to “core principles of integrity” and ethics. The lab director resigned. A letter alleged “potential concealment” of errors in the fingerprint unit as well.²⁷ In September 2021, the lab disbanded the firearms unit. A criminal investigation of the lab is pending, and the lab remains closed.²⁸

As these high-profile examples both show, forensic evidence has, to varying degrees, left the police station for the laboratory in a manner that represents a highly incomplete and much-needed transition to sound science. In recent years, the scientific community has called for forensic work to be independent of police so it can adhere to sound scientific standards. The central recommendation of the NAS Report, which brought together leading scientists to consider the path forward for forensic science, centered on independence and scientific methods.²⁹ At best, some crime labs have obtained financial but not structural independence from law enforcement with the accountability and accuracy that should accompany an embrace of scientific methods.

Many are asking new questions about how to define public safety.³⁰ Even in the area of forensic evidence, where there is a greater expectation that science will be used in service of public safety, questions about the police’s role in collecting and testing evidence have remained unasked. Far too often, untrained police, rather than trained scientists, collect and potentially alter or contaminate evidence at crime scenes.³¹ Further, we allow law enforcement oversight of most crime labs, or we allow law enforcement and prosecutors to direct and influence their work, practices that would never be tolerated in a scientific experiment, much less a scientific lab.³² As police deploy new technologies, these questions will hopefully continue to be asked, but little scrutiny has resulted when old technologies, like pathology examinations or fingerprint comparisons, have been used. One reason these practices remain under-examined is that neither courts nor the public can readily access what procedures crime labs employ or what results they obtain.

However, the movement to bring scientific standards to forensics, which is increasingly embraced by the forensic community and legal actors, has the potential to address these long-standing needs. As I discuss in this essay, that movement also has promise for policing more generally. We should be using quality controls to regulate crime labs, and policing more broadly, to evaluate the fairness and public safety of outcomes. The focus here is on three specific core principles that can improve the role of forensic evidence in policing: (1) independence, (2) accuracy, and (3) oversight.

The NAS Report warned that a “lack of independence” of crime labs from law enforcement can damage the objectivity of forensic science and harm criminal justice.³³ Traditionally, forensic evidence has largely been a law enforcement function, not a scientific function, in the United States. Early crime labs were small and operated by police departments, following the early example of the Federal Bureau of Investigation (FBI) lab.³⁴ By the 1960s, every state had crime labs run by law enforcement.³⁵ Today, there are over four hundred publicly funded crime labs with over 14,300 lab employees.³⁶ Very few labs in the United States are independent from law enforcement.

Nor is the work done by professionals necessarily independent of law enforcement. This problem begins at the crime scene. Such work may be conducted by a variety of professionals and roles, including officers working for police departments but also crime-lab analysts, technicians, and others.³⁷ Evidence collection may be conducted by law enforcement. Law enforcement priorities and resources may affect the initial decision whether to test evidence. Recent inquiries and audits have resulted when labs have failed to test forensic evidence, particularly regarding untested rape kits. One study found that forty percent of unanalyzed rape and homicide cases were estimated to have testable DNA.³⁸ In Los Angeles County, following a 2009 Human Rights Watch report and public protests, police and sheriffs began to work to address years of backlogs.³⁹ In response to the serious problem of non-testing, and with federal grant support,⁴⁰ twenty-six states have enacted statutes requiring inventories of untested sexual assault evidence.⁴¹

Police begin the process of examining forensic evidence when they collect evidence at the crime scene. Police often do not have policies regarding sound and scientific collection of crime scene evidence, which then affects the work done by a lab.⁴² There has been a recent focus on developing policies and procedures for crime scene evidence collection, including to promote the use of forensic professionals at the crime scene.⁴³ Not only do police often lack the resources or training to adequately collect evidence at the crime scene, but they often fail to collect evidence that could be highly probative.⁴⁴ Police may also preemptively test evidence themselves using inaccurate field methods. For example, commercial drug-testing kits are often not carefully validated; the few studies conducted on this issue have shown that these kits can have high error rates. The field tests are supposed to be followed up with a more rigorous lab test, but in the meantime, a person may be arrested and face great pressure to plead guilty. In Harris County, Texas, an audit by the District Attorney’s Office Conviction Integrity Unit uncovered hundreds of such cases. The Texas Forensic Science Commission later found these tests too unreliable for use and the convictions were subsequently reversed.⁴⁵ In 2017, Houston police ceased use of field drug tests, but they remain widely used in other jurisdictions.

In other areas, bias in police evidence collection can be of constitutional concern. For example, the U.S. Supreme Court has found highly suggestive and unreliable practices in the area of eyewitness identifications that may violate the Due Process Clause. The Court’s ruling in Manson v. Brathwaite set out a due process test using a series of so-called reliability factors to assess eyewitness identifications in which police used suggestive procedures.⁴⁶

No such inquiry exists for forensic evidence. A large body of research has now documented how cognitive biases can affect forensic work. Psychological research has shown how where forensic techniques involve some degree of judgment and interpretation, experts are vulnerable to cognitive bias.⁴⁷ In the laboratory, testing is rarely blind, and a range of communications with law enforcement can bias the test results. Traditionally, beginning with the submission of evidence, the forensic analysts communicate with law enforcement about evidence in the case and hear task-irrelevant and biasing information.

In one of the most famous studies in all of forensics, five highly experienced fingerprint examiners reviewed a set of prints in the course of their ordinary work.⁴⁸ The examiners did not know that Professor Itiel Dror and colleagues had given these examiners prints that each of the five had looked at years before in a real-life case and had found then to be from the same source.⁴⁹ Other experts had separately looked at those prints and agreed that they were from the same source. Dror and colleagues informed the five examiners that the prints were from the infamous Madrid bombing case, in which three senior FBI examiners mistakenly implicated an innocent Portland, Oregon, lawyer.⁵⁰ After having been told that these prints were from that case, in which such a high-profile error had been made, four of the five reached a different conclusion: one persisted in reporting that the prints came from the same source, three now concluded these were “definite non-match” prints, and the fourth called the evidence “inconclusive.”⁵¹

Addressing cognitive bias is only now becoming a central part of quality control at crime laboratories. Adoption of procedural protections can reduce such bias. For example, a linear sequential approach has been developed for comparative pattern disciplines. The 2016 President’s Council of Advisors on Science and Technology (PCAST) Report recommends that approach for fingerprint comparisons: an examiner “should be required to complete and document their analysis of a latent fingerprint before looking at any known fingerprint” to prevent circular reasoning where looking back and forth between prints encourages an expert to disregard differences.⁵² A second protection is to ensure that potentially biasing task-irrelevant information is not passed on to lab analysts.⁵³ The PCAST Report also recommended that a fingerprint examiner “should separately document any additional data used during their comparison and evaluation.”⁵⁴ Information is task-irrelevant if it is not necessary for drawing conclusions about the propositions in question, using the method in question. Separation of roles in a laboratory can help keep task-irrelevant information from biasing analysts.⁵⁵ Such procedures are much needed to define the relationships between law enforcement and crime labs, and they should be a hallmark of functionally independent and sound science.⁵⁶ Those reforms are only slowly being implemented, however.

A wide range of scientific and legal actors, including scientists, forensic practitioners, lawmakers, lawyers, and judges, have called for greater reliability in forensics. The National Academy of Sciences Report in 2009 emphasized that “some forensic science disciplines are supported by little rigorous systematic research to validate the discipline’s basic premises and techniques.” Slowly, as such research has begun to be conducted, we have learned just how often forensic methods go wrong.⁵⁷

The legal response has been relatively piecemeal. Many states now follow standards based on the U.S. Supreme Court’s ruling in Daubert v. Merrell Dow Pharmaceuticals⁵⁸ and its progeny, which task judges with greater gatekeeping responsibilities to assess the reliability of scientific evidence. Those rules are widely understood to be ineffectively used in criminal cases. Leading reports, such as the influential 2009 National Academy of Sciences Report, have noted that lawyers and judges have not taken an active or effective role in reviewing the reliability of forensic evidence in criminal cases.⁵⁹ Constitutional criminal procedure bars the failure to disclose exculpatory evidence and the outright fabrication of evidence by law enforcement. The U.S. Supreme Court has long held that fabricating evidence or knowingly using perjured testimony violates the Due Process Clause.⁶⁰ In addition, officers and prosecutors must supply “any favorable evidence known to others acting on the government’s behalf in the case, including the police.”⁶¹ No constitutional rules regulate the reliability of forensic evidence, apart from those elemental disclosure and non-fabrication rules.

For decades, forensic analysts of different types testified they were one hundred percent certain of their conclusions.⁶² As federal judge Harry T. Edwards put it, “The courts had been misled for a long time because we had been told, my colleagues and I, by some experts from the FBI that fingerprint comparisons involved essentially a zero error rate, without our ever understanding that’s completely inaccurate.”⁶³ Yet, no one had carefully tested the basic assumptions that experts have relied upon for decades. More recently, the NAS and PCAST Reports highlighted the need for scientific testing of the reliability of core forensic disciplines. The PCAST Report detailed how for a more objective method, like a drug test, you can evaluate each step in the process by seeing whether it produces accurate results.⁶⁴ However, for subjective techniques like pattern evidence comparisons, there are no clearly defined steps. The person is the process: an examiner’s mind is a “black box” that reaches judgments based on training and experience.⁶⁵ To test such a “black box” examiner, one can give tests with samples of realistic difficulty and with the correct answer known in advance.⁶⁶ In forensics, that type of proficiency testing is not routine in labs, as part of quality control or performance testing of examiners, except using very elementary tests that are not designed to be rigorous.⁶⁷ Entire fields have lacked such testing to establish reliability. And, as the PCAST Report underscored, “without appropriate estimates of accuracy, an examiner’s statement that two samples are similar—or even indistinguishable—is scientifically meaningless: it has no probative value, and considerable potential for prejudicial impact.”⁶⁸ For example, the PCAST Report described that, while several early studies had been conducted and documented high error rates, just two properly designed studies of the accuracy of latent fingerprint analysis had been conducted to date.⁶⁹ That alone is deeply disturbing, given the prominence of fingerprint evidence in criminal investigations and prosecutions over the past hundred years. It was generous for the report to say that just two studies would be enough to permit a technique to be used in criminal cases. Indeed, the people participating in the studies knew that they were being tested and that it was important for the field. They were likely very cautious in their work, and they could drop out of the study, including if they found a question challenging and they feared making an error. In the larger of the two studies—the only one published—a huge group, eighty-five percent of the examiners, made at least one false negative error.⁷⁰ These findings provided another overdue wake-up call. It would shock jurors to hear of such error rates, and it would affect their convictions.⁷¹

The 2009 National Academy of Sciences Report similarly concluded that there needs to be more research “to confirm the fundamental basis for the science of bite mark comparison.”⁷² It said that the uniqueness of human dentition has “not been scientifically established.”⁷³ The scientists who wrote the PCAST Report concluded that since no valid studies of error rates had been done, the analysis lacked validity and should not be used.⁷⁴ What we do know about reliability is disturbing, as the PCAST Report detailed.⁷⁵ None of these troubling findings blunted the force of standard testimony on bite marks delivered in court, nor did forensic dentists make a habit of describing these studies in reports or testimony; indeed, courts continue to allow the evidence in criminal trials.⁷⁶

Of particular importance to police investigations, in part because firearms violence is a major problem in the United States, is that firearms comparisons are in great demand. To complete such analysis, examiners seek to link crime scene evidence, such as spent shell casings or bullets, with a firearm.⁷⁷ The assumption is that manufacturing processes leave markings on the barrel, breech face, and firing pin. When the firearm discharges, those components contact the ammunition and leave marks on it.⁷⁸ Experts have assumed that different firearms should leave different toolmarks on the ammunition such that the spent ammunition can be definitively linked to a single firearm.⁷⁹ By the late 1990s, experts premised testimony on a “theory of identification” set out by a professional association, the Association of Firearms and Tool Mark Examiners (AFTE), which instructs practitioners to use the phrase “source identification” to explain what they mean when they identify “sufficient agreement” when examining firearms.⁸⁰ The AFTE’s theory is circular: an identification occurs when the expert finds sufficient evidence defined as sufficient to find an identification.⁸¹ In recent years, scientists have called into question the validity and reliability of such testimony. In a 2008 report on ballistics imaging, the National Academy of Sciences (NAS) concluded that definitive associations like “source identification” were not supported.⁸² In its 2009 report, the NAS followed up and stated that categorical conclusions regarding firearms or toolmarks were not supported by research and that, instead, more cautious claims should be made.⁸³ The report stated that the “scientific knowledge base for tool mark and firearms analysis is fairly limited.”⁸⁴ The AFTE theory of identification “is inadequate and does not explain how an expert can reach a given level of confidence in a conclusion.”⁸⁵

By 2016, only a single black box study on firearms comparisons had been done, and it showed an error rate as high as 1 in 46.⁸⁶ This single study is unpublished, and it raises many of the same concerns as other studies in which participants were aware that their assignment was a test and in which some could and did drop out after reviewing the materials.⁸⁷ The rate of inconclusive errors was over thirty-three percent.⁸⁸ To this day, firearms examiners use terms like “source identification” in court—and avoid reporting what is known about error rates. Some judges have begun to step in and require more cautious wording for firearms conclusions.⁸⁹ However, crime lab regulation is needed, as described next, just as other scientific laboratories are regulated.

There are some ways in which crime labs can be a model for policing agencies. Forensic evidence should bring with it better documentation, scientific methods, and quality controls, which are broadly needed in the criminal system. Forensic examiners must follow lab protocols to document their work; police detectives and patrol officers should be held to higher standards regarding documentation of their investigative work. Forensic labs may be accredited and are required to adopt certain quality control standards. In addition, forensic examiners are subjected to periodic proficiency testing, which, in theory, examines the quality of their work. However, those tests are understood to be elementary and are often not taken individually or under realistic conditions.⁹⁰ Crime labs offer a model for more accurate policing, even though they themselves traditionally have fallen short of the independence, oversight, and quality controls in clinical and scientific laboratories outside the criminal system. Far more rigorous regulation of crime labs and forensic evidence is needed, and such a model offers lessons for how to improve policing more generally.

The National Academy of Sciences summarized the state of affairs facing forensics in the United States in 2009: “Forensic science facilities exhibit wide variability in capacity, oversight, staffing, certification, and accreditation across federal and state jurisdictions.”⁹¹ “Rigorous mandatory certification and accreditation programs, adherence to robust performance standards, and effective oversight”⁹² are lacking. Moreover, what policies labs do have may be nonpublic.⁹³

In contrast, public regulations for medical laboratories have developed over time to support public health goals. After World War II, medical laboratories that conducted an experiment to assess clinical labs in Pennsylvania found serious errors, such as failure to correctly identify diseases. In 1967, Congress passed the Clinical Laboratory Improvement Act (CLIA) to ensure that medical labs conduct accurate tests.⁹⁴ A second wave of reform began in the mid-1980s, after reporters at the Wall Street Journal wrote about misdiagnosed cancer and lax standards at labs in a Pulitzer Prize–winning series.⁹⁵ In 1988, Congress passed amendments to the CLIA extending the regulations to practically all clinical laboratories, public or not. The law required that proficiency testing reflect “to the extent practicable … normal working conditions” to make tests realistic.⁹⁶ The law also permitted “announced and unannounced on-site proficiency testing of such individuals.”⁹⁷ After all, the lawmakers concluded, “regular proficiency testing [is] vital evidence of a laboratory’s competence.”⁹⁸

The swift responses by federal lawmakers to clinical lab failures was completely different from the tepid responses to crime lab failures. We need similarly serious federal legislation and quality controls imposed on crime labs. At the state level, there are a few state forensic science commissions, but only the Texas Forensic Science Commission has done audits and investigations in even a modest way.⁹⁹ Maryland began licensing crime labs in 2007 after a discredited state police ballistics and firearms expert was found to have falsified his academic credentials, resulting in a review of over 4,000 cases.¹⁰⁰ In response, Maryland adopted a model based on the CLIA, extending clinical regulations to crime laboratories.¹⁰¹ The statute and accompanying regulations provide a model for detailed regulation of crime laboratories; however, the effectiveness of the regulatory body that enforces the statute and accompanying regulations is unclear, and problems have persisted, including in firearms examinations, which had originally prompted the regulations.¹⁰²

As a result of the nonexistence of federal quality controls, many crime laboratories have long “lacked quality control measures that would have detected” uses of scientifically “questionable evidence.”¹⁰³ No federal or national accreditation system exists in the United States. Some states have required that their labs be accredited; otherwise, accreditation is voluntary.¹⁰⁴ The National Commission on Forensic Science strongly recommends that all forensic science service providers (FSSPs) become accredited to promote compliance with industry best practices, promote standardization, and improve quality of services.¹⁰⁵ The American Bar Association, similarly, has made this recommendation: “Crime laboratories and medical examiner officers should be accredited, examiners should be certified, and procedures should be standardized and published to ensure the validity, reliability, and timely analysis of forensic evidence.”¹⁰⁶ Accreditation has become far more common among crime laboratories.¹⁰⁷ This is a good step to ensure that minimal standards exist on paper, but accreditation does not ensure that reliable and consistent work is done in practice. Thus, the American Bar Association called for “demanding written examinations, proficiency testing, continuing education, recertification procedures, an ethical code, and effective disciplinary procedures” for all forensic analysts.¹⁰⁸ Scientists and lawyers have called for comprehensive quality controls for laboratories in the criminal legal system, where a person’s life and liberty are at stake, but so far none have been established.

The central problem in forensics is that the function has been treated as a law enforcement rather than scientific function. The result has undermined both public safety and fairness. Innocent people have been convicted, and guilty people have gone free. Quality control failures have caused enormous harm to criminal investigations, victims, and the lives of persons wrongly convicted. We should not allow untrained officers to collect and potentially alter or contaminate evidence at crime scenes instead of enlisting trained scientists. We should not allow people to be arrested and incarcerated based on largely unregulated forensic tests that have never been scientifically tested. Law enforcement should neither run crime labs outright nor otherwise direct and influence lab work. These reforms also provide a model for police more broadly. These standards will likely have to come from lawmakers and not from the courts, which have provided little guidance regarding sound evidence collection during criminal investigations. Fortunately, there are good models for policing forensic evidence, including from labs that have made large-scale changes and from clinical laboratories. Independence, accuracy, and oversight of forensic evidence when it is handled by police, medical examiners, and crime laboratory staff have the potential to dramatically improve the quality of justice in the United States.