IT'S A MATCH IF I SAY IT IS

The always interesting Grits for Breakfast has this important new post on cognitive bias which is, in turn, inspired by this important article in Miller-McCune.com.

Oversimplifying almost to the point of being misleading, cognitive bias is the reason that when you know what you're looking for, you're likely to find it. It's a major issue in the hard and social sciences. As relevant here (and as Grits talks about it), it has serious dangers in crime labs doing fingerprint matching.

What happens, and what Grits discusses, is that when fingerprint examiners have case information saying, for instance, that the suspect is in custody, they're more likely to find that the latent print is the suspect's. Similarly, they're more likely to find matches when they think they're solving a terrorism case than a simple robbery. That's dangerous and frightening given juror's belief in the talismanic power of fingerprints.

The easy remedy for much of the cognitive bias in crime lab work is simply to remove crime labs from police departments and insure that the people examining the evidence don't have information about the context of what they're looking for. Detectives and investigators need that. Lab folks mostly don't.

But it's more than just cognitive bias and it's more than just fingerprints. All the forensic experts who testify that can make visual matches are telling stretchers, whether they know it or not. Fingerprints are especially problematic because we've been hearing for over a century that they're the gold standard, the infallible forensic, and the examiners believe it. So a typical fingerprint testimony, spoken in complete if altogether inaccurate honesty, and I'm not not making this up,

There is no error rate. It's 100 percent accurate. I compare the prints and when I'm certain it must be this person and can't be anyone else in the world, then it is.

These folks claim to be scientists, but they're really trained lookers who've never had their work validated in any rigorous way because there is no rigorous validation for what they do.

The truth is that we have no idea whether fingerprints are unique, and no idea how to tell that this print (let alone this fuzzy, smudged, damaged, partial, latent print) and that print over there are a perfect match. How much identity in the prints do you need? Nobody knows, and the little testing that's been done doesn't suggest that the results should be trusted. (See, for instance, this article in the New Yorker).

And what we don't know and shouldn't trust about fingerprints is the tip of that iceberg. Tire tracks, ballistics, fiber comparison, it's all nonsense. That's not to say it's all wrong, just that there's no telling when it's right and when it isn't. One study of bite mark analysis showed that the forensic odontologists who tried to make matches were wrong more than half the time. So if they say it's you, it's probably not. (Just ask Ray Krone, an innocent man who spent a number of years on death row in Arizona because of a faulty bite mark match.)

And what's true, by the way, of trained lookers is also true for those working with the harder science of DNA analysis. The science of DNA is perfectly sound. Interpretation of the scientific results, the analysis, is something else.

The recent exhaustive study by the National Research Council, Strengthening Forensic Science In The United States: A Path Forward goes a long way toward identifying the problem and offering some ideas for fixes. But the first step is probably the hardest: Convincing the public that CSI (and Quincy, to those of us of a certain age) is nonsense.