Because One Size Doesn't Fit All

I heard a judge once, sentencing a guy to death, explain that he was a serial killer even though they only knew of one person he'd killed.  Sure, he'd never got caught before.  Sure, there were no cases in which he was even a suspect.
But the judge knew.  He just knew.
Maybe.
Then again, maybe not.
For years, really for decades, everyone assumed that fingerprints were essentially infallible, the perfect identifiers.  When they first started doing DNA testing in criminal cases, prosecutors would refer to it as "DNA fingerprinting" because that would give this new technology the imprimatur of absolute certainty.
If it's like fingerprints, it has to be good.
Except, and as I've talked about before (see, e.g., here and here), it turns out that fingerprints ain't so good.  It's just that nobody knew. For decades there were no false positives (or false negatives, for that matter), because they were infallible.  Except that since they aren't, we have this question.

How many innocent people were convicted based on false fingerprint matches?  

Nobody knows. More problematic,

Which innocent people were convicted based on false fingerprint matches?

Damn, we don't really know the answer to that, either.
But now we have DNA.  Which nobody much refers to as "DNA fingerprinting" anymore because DNA is so much better.  It's the gold standard.  Infallible.  The perfect identifier.  If the testing says the DNA is Sally's, then she's guilty.  If the testing says it wasn't Fred's DNA, he's off the hook. Case dismissed.  Freed.  Declared innocent.  Given the big bucks for his trouble.
Well, maybe.  Sort of.  Some of the time.  Often not.  But I digress from the point I haven't yet really started to make.
In yesterday's Times, Cyrus Vance, Jr., New York County (that's Manhattan) District Attorney, has an op-ed touting the wonders of collecting DNA from everyone convicted of anything, no matter how trivial. 

WE have a tool that can prevent hundreds of murders, rapes and robberies each year at minimal cost to taxpayers. But we’re not using it in a majority of cases because a state law restricts its use.
DNA evidence solves crimes. Since 1996, when New York State’s DNAdatabank opened with strong support from my predecessor, Robert M. Morgenthau, the bank’s DNA samples have been linked to more than 3,500 sexual assaults, 860 murders, 1,100 robberies and 3,400 burglaries. Thousands of criminal convictions have resulted. Today, however, we are hamstrung by a law that does not authorize the collection of DNA following convictions of certain misdemeanors. This has meant that we can’t use DNA technology in more than half of our cases. By expanding the collection of DNA to include those convicted of all crimes in New York State’s penal law, as Gov. Andrew M. Cuomo has called for, we will be better able to identify the guilty, exonerate the innocent, bring justice to crime victims and prevent additional crimes from occurring.

Identify the guilty.  Exonerate the innocent. Bring justice.  Prevent Crime.  Wow!  That's good stuff.
See, bad guys, really bad guys, sometimes don't get convicted of really bad crimes.  Sometimes they get convicted of trivial ones.  But if we have their DNA, we can tie them to really bad crimes.  And that will get them off the streets so they don't do more bad stuff.
And it will also free the innocent people who are charged with and convicted of really bad crimes because we didn't bother to test their DNA against the evidence because since they'd never been convicted of trivial crimes we couldn't get it to test against the evidence before locking them up forever.  Wait.  No. That's not right.  I don't know how it exonerates anyone. 
But sure it prevents crimes.

Whom does DNA bring to justice, and how does it prevent future crimes? The case of Curtis Tucker is instructive. Following his conviction in 2010 for robbing and assaulting a 74-year-old Manhattan man suffering from Parkinson’s disease, Mr. Tucker was required to submit a DNA sample, which typically is obtained by swabbing the inside of a person’s cheek. Three days later, that $30 test produced a match linking Mr. Tucker to a brutal 2004 assault and attempted rape of a 15-year-old girl in the stairwell of her Manhattan apartment building.

Got it.  Tucker committed a terrible crime in 2010.  And DNA links him to a really serious crime in 2004.  So if he'd been arrested and charged and convicted of the 2004 crime, he'd have been locked up and his DNA could have maybe showed that he didn't commit the 2010 crime while he was in prison.

Wait, maybe that's not a really great example of how DNA can prevent crimes.

And it has nothing to do with misdemeanants, which is really what this is about. So maybe this.

[S]amples collected from people convicted of petty larceny have been linked to roughly 48 murders and 220 sexual assaults. Clearly, the 2006 expansion of the DNA program — which passed with only six dissenting votes in the State Assembly — confirmed that collecting samples from offenders convicted of minor crimes helps solve and prevent more serious crimes.

OK, solving more crimes.  248 maybe.  From how many samples?  What's the percentage?  And what was the prior status of those folks?  Were they suspects anyhow?  Were they charged with those crimes?  Any of them?  Still, 248 links.

Look, I get it that law enforcement types want to know everything about everyone.  (DNA from all kids at birth?  Why the hell not?)  Scott Greenfield is right when he observes

His proposal is relatively modest, given that others argue for the collection of DNA from every person arrested, rather than convicted.

He's right, too, about the danger of error.  Oh, sure, Vance acknowledges that it's possible, but just barely.  And never in New York.

To date in New York, we have never had a “false positive,” or the misidentification of DNA from one person as the DNA of another.

By which, of course, he means that they've never found a mistake.  Just like all those major felons who are only known to be misdemeanants because their more serious crimes have never been found.

It's true.  DNA is great stuff.  It really can help to properly convict and exonerate.  But it's also subject to all the mistakes you can imagine.  Scott talks about that, too, but he's after a different point - Vance's recognition that other sorts of evidence routinely relied on (eyewitness testimony and confessions) are often convict the innocent.  Me, I want to keep talking about DNA.

The science is good, but that doesn't make the results of every test reliable.  Even the best labs screw up.  Even the best analysts get it wrong. I've written about this before, starting with part of a blog post by Walter Reeves.

Itiel Dror and Greg Hampikian recently published a paper titled Subjectivity and Bias in forensic DNA mixture interpretation. They wanted to address the potential problems in interpreting evidence when there may be multiple suspects. The situation most commonly occurs in sexual assault cases where there are multiple perpetrators. There may not be enough to conclusively identify one person, but there may be enough to say an individual cannot be excluded - which in the minds of most jurors means you must have done it.

They obtained an actual case out of Georgia that involved a gang rape. One of the alleged suspects identified and testified against the others. The results of the DNA analysis - by examiners who knew who the suspects were - could not exclude the others, which corroborated the co-defendant testimony.
The actual data was submitted to 17 qualified analysts who routinely do forensic work. No other facts were sent, so the examiners did not know who the suspects were. The results obtained without that contextual information were startling. Only one of the 17 agreed with the original examiner. Even more startling is that 12 examiners would have EXCLUDED the suspect they looked at. The remaining 4 would have called the results inconclusive.

Now, Reeves goes on to talk about how this is serious and how it shows the need for independent testing and not trusting the government's expert to have got it right because, after all, it's DNA.  All that is surely so.  But it's important to be clear about just what is so.
The science of DNA is in the technology.  Done properly, with properly preserved and protected samples, the results are exactly what they are.  But along with the science is the art. (Or the bullshit, depending on how you want to discuss it - Truman Capote famously said of Jack Kerouac's books, "That's not writing, it's typing.")  That's the part where someone tries to figure out what the science revealed.
The science of DNA is good and reliable and actually science if it's done properly and if the samples tested were properly taken and preserved.  The bullshit of DNA is in making sense of the science.

And there's this.  No matter what Vance says, there's no such thing as secure data.  And DNA isn't like a fingerprint that you can store and that maybe serves as an identifier sometime and maybe not.  DNA doesn't just serve as an indication that Joe Blow is really Joe Blow.  DNA tells a whole story about Joe.  Because it's not just that Joe is Joe but it's every genetic fact about Joe.  More and more of which we're learning to unravel.  And more and more of which will be used in ways we never imagined - or authorized.

Because that's the one sure thing we've learned about data.  It escapes from the confines we erect for it and imagine will successfully cabin it.

The dirty truth, which I've suggested before, is that they just can't be trusted.  Even when they're trying to do the right thing and their hearts are in the right place.Si