Scientific Testimony: An Online Journal


People v. Bokin

An Internet Discussion


Listed below are comments regarding Bokin that were posted in May, 1999 on the internet discussion list, Forens-L. Readers are invited to add their own comments to the discussion. Send comments by e-mail to openforum@scientific.org.

Forens-L is an active list for people interested in forensic science. To subscribe to this list, send an email to mailserv@acc.fau.edu. In the body of the message type "subscribe Forens-L [your name]." (Without the quotation marks.)

Read the Bokin opinion. You may download the Adobe® Acrobat® Reader to view this file.


I agree with Simon Ford that the Bokin ruling is significant.... What I don't understand is why the prosecutors and forensic scientists quoted in the SF Chronicle article reacted to the opinion with such outrage (e.g., calling the judge an "idiot"). It seems to me a logical application of the Frye standard in a case where the proponent of a new form of DNA evidence could not prove that the underlying method had been adequately validated. The opinion makes a number of important points that forensic scientists should keep in mind (whether they agree with them or not).

First, the judge held that a DNA testing procedure must be validated in accordance with TWIGDAM guidelines in order to pass muster under the Frye standard. In my view, this is a no-brainer. What is the point of the TWIGDAM validation guidelines if no one needs to follow them? Is TWIGDAM nothing more than window-dressing?

Second, the judge held that the validation research must show the reliability of the specific method that was used for DNA testing. It is not enough to show that STR testing in general is accepted, the proponent of the evidence must show adequate validation of the particular method used for STR testing (in Bokin it was the ABD Green One Kit in conjunction with the ABD 310 genetic analyzer). Again, who can disagree? The fact that some other method of STR analysis is accepted does not prove that this particular method is reliable, does it?

Third, the judge held that the proponent of novel DNA evidence cannot rely on secret validation studies that are shielded from discovery by claims of proprietary interest. In Bokin, the prosecution sought to rely on studies purportedly conducted at Perkin-Elmer (the company that developed the Green One Kit) that were reported only in Perkin-Elmer's user's guide for the kit. When the defense sought discovery of the underlying data, P-E refused, claiming the studies were proprietary information. The judge sustained P-E's objection to the discovery request, but decided to give no weight to these "secret" studies when evaluating the admissibility of the method.

Here I think the judge was too kind to the prosecution and Perkin-Elmer. I see the claim of proprietary interest (in validation data!!!!) as nothing but a smoke screen designed to shield P-E from outside scrutiny. Consequently, I think the judge should have drawn an adverse inference about what discovery of the underlying data would have shown. His decision simply to ignore the secret studies was an unwarranted sop to the prosecution.

Fourth, the judge held that, apart from the secret studies, the validation presented by the prosecution was not adequate to establish the reliability of Green One Kit/310 Analyzer method. The only studies the prosecution presented had been performed at the Cal-DOJ lab. This is a fine laboratory, but their studies involved a gel-based electrophoresis system rather than the 310 Genetic Analyzer (which employs capillary electrophoresis), the studies were not peer-reviewed or published, and experts on both sides of the case testified that the studies did not meet the TWIGDAM standards for developmental validation of a new method. Is it surprising, then, that the judge found the validation inadequate?

Finally, the judge made some negative comments about the lead DNA analyst at the San Francisco Crime Laboratory. The court pointed out that this individual's credentials do not meet the DAB requirements for supervising a DNA lab. Among other facts, the court expressed concern about this individual's admission that he routinely deviated from the manufacturer's suggested protocols when using P-E equipment. That is a bit scary, isn't it? (Or does everyone deviate from the P-E protocols when using the 310?) The court also criticized this individual for bias, saying his stance "was beyond advocacy—it indicated a critical attitude toward the defense function in a criminal case." (The individual had filed declarations that made personal attacks on the defense lawyers and defense experts. Hence, it required no psychic powers for the judge to detect an attitude that deviated from scientific neutrality).

In sum, I find nothing in the Bokin opinion to justify the cries of outrage I am hearing from forensic scientists. In my view it is exactly what forensic scientists should expect if they go to court with inadequately validated methods. It is a correct application of the law to the facts of the case. In the long run, opinions like this will have a positive effect on forensic DNA testing. The only "idiocy" I see in connection with this opinion is way some have reacted to it.

Bill Thompson
wcthomps@uci.edu


Why call them "guidelines" if what is meant is "prescriptive rules"? And why cannot a method of forensic DNA analysis attain general acceptance unless it follows this particular set of prescriptive rules? Are the STR testing procedures practised in the UK, for example, not deserving of general acceptance in the scientific community, insofar as forensic scientists in Britain appear to take no notice of TWGDAM's rules? Nor, I think, do they make for themselves any similar rules. It cannot be seriously suggested, though, that British STR tests have not attained general acceptance in the scientific community and are not reliable. Insofar as the Bokin court would rule a British STR analysis inadmissible, therefore, its view that TWGDAM compliance is irreplaceable and non-negotiable is both parochial and fallible.

It may be a shortcoming of the TWGDAM rules that no distinction seems to be possible between substantial and trivial modifications to a protocol for achieving analytical objectives which are known to be achieved reliably by the general protocol. For example, given that chemiluminescent labelling of DNA fragments can be demonstrated to be possible, is it not absurd that "developmental validation" of the replacement by chemiluminescence of radiolabelling should be required by the Courts to the same degree as was (and rightly so) the development of forensic RFLP testing generally? For this is my understanding of the Bokin court's view (and that of the Court in U.S. v. Lowe) at page 6 lines 18 to 21.

Although indeed no weight should be given to secret validation studies, should no weight be given to the apparent failure by any expert who testified on behalf of the Defense to advance any claim that the Green I kit and the 310 instrument do not perform as advertised? Of course, in a legalistic sense, I'm arguing here for a displacement of the burden of proof. But on the level of common sense, does not the absence of any claim of underperformance (in the very forum where such a claim could be expected to be vigorously pressed) itself indicate general acceptance? Why is it necessary to litigate that which is not disputed? For it does not seem to be disputed that the method works, only whether the guidelines cum rules have been satisfied. Thus does TWGDAM offer up hostages to fortune and serve justice (not).

Tom Fedor
federini@nothingbutnet.net


Tom Fedor writes:

Why call them "guidelines" if what is meant is "prescriptive rules"? And why cannot a method of forensic DNA analysis attain general acceptance unless it follows this particular set of prescriptive rules? Are the STR testing procedures practised in the UK, for example, not deserving of general acceptance in the scientific community, insofar as forensic scientists in Britain appear to take no notice of TWGDAM's rules? Nor, I think, do they make for themselves any similar rules. It cannot be seriously suggested, though, that British STR tests have not attained general acceptance in the scientific community and are not reliable. Insofar as the Bokin court would rule a British STR analysis inadmissible, therefore, its view that TWGDAM compliance is irreplaceable and non-negotiable is both parochial and fallible.

I think we all recognize the games that are played with TWIGDAM. Those who meet TWIGDAM requirements claim that TWIGDAM is the gold standard, those who do not meet TWIGDAM requirements say they are mere guidelines and hence should not be taken too seriously. From my perspective, TWIGDAM offers a rather minimal set of validation guidelines that are designed to be easy to meet. On many points that I consider essential, the guidelines are vague and forgiving. Hence, I think it is quite proper for a court to look askance at a method that does not meet these standards. I agree that TWIGDAM cannot and should not be the sine qua non. And I am not sure that Bokin held that it was (I may have overstated this point in my brief summary). But certainly it is relevant for a court to consider whether a method has been validated according to TWIGDAM's guidelines, don't you think? And if it has not, that should weigh against admissibility, shouldn't it?

With regard to British STR testing, my impression is that it has been validated far more rigorously, and is carried out far more rigorously, than TWIGDAM would require. Hence, I think your implication that the STR protocol of the Forensic Science Service would "flunk" the Bokin test is both wrong and silly. However, if you disagree I would like to hear which specific TWIGDAM requirements you think the British system fails to meet.

It may be a shortcoming of the TWGDAM rules that no distinction seems to be possible between substantial and trivial modifications to a protocol for achieving analytical objectives which are known to be achieved reliably by the general protocol. For example, given that chemiluminescent labelling of DNA fragments can be demonstrated to be possible, is it not absurd that "developmental validation" of the replacement by chemiluminescence of radiolabelling should be required by the Courts to the same degree as was (and rightly so) the development of forensic RFLP testing generally? For this is my understanding of the Bokin court's view (and that of the Court in U.S. v. Lowe) at page 6 lines 18 to 21.

You raise a crucial point that has long bedeviled the courts. How should a court distinguish between a substantial modification of a protocol, which calls for entirely new validation, and a trivial modification, which does not? This problem is not limited to TWIGDAM's guidelines, it is a general feature of legal systems that condition admissibility of scientific evidence on meeting a particular standard. The proponent of new evidence will always claim that it is merely a trivial modification of previously accepted procedures. The opponent will claim that the any variation of protocol is radical departure from what is accepted. In order to resolve the dispute, the court must have the scientific sophistication to understand which changes are truly important, and that is not an easy determination to make.

In Lowe the defense presented declarations from academic molecular biologists who argued that the shift to chemiluminescent labeling might change the operating characteristics of the FBI's gel system for RFLP analysis, thereby affecting such things as the appropriate matching standard, appropriate bin sizes, and so on. Based on this showing, I think it was proper for the trial court to require the FBI to present some validation on the chemiluminescent system in order to show that it worked pretty much the same as the radiographic system. And the FBI was able to make this showing rather easily. The court did not require, for example, that the validation research be published, merely that some validation had actually been performed. I see nothing absurd about the Lowe decision. I think it shows a legal system working properly.

With regard to Bokin, my opinion is the same. During the past few years, prosecutors in California have been arguing with considerable success that any DNA testing method that uses PCR is generally accepted because PCR is generally accepted. By their account, any variation in the system for detecting alleles is a trivial matter that should not affect the admissibility of the test. In my view, that position is absurd (unless one thinks there should be no admissibility standards for DNA evidence). I find Bokin refreshing and encouraging because it is the first time in recent memory that a trial judge has taken a critical look at whether the specific method employed (rather than some generic approach) has actually been validated. In my view a method like "Green One with the 310 analyzer" is sufficiently different than, say, CTT with silver staining (or PM+DQA1, for that matter), that it requires separate validation. Don't you agree?

Although indeed no weight should be given to secret validation studies, should no weight be given to the apparent failure by any expert who testified on behalf of the Defense to advance any claim that the Green I kit and the 310 instrument do not perform as advertised? Of course, in a legalistic sense, I'm arguing here for a displacement of the burden of proof.

You certainly are. Under your approach, the party that opposes scientific evidence would have the burden of proving it is unreliable. While I can understand the appeal of such a standard to forensic scientists, it is not the law, nor should it be.

But on the level of common sense, does not the absence of any claim of underperformance (in the very forum where such a claim could be expected to be vigorously pressed) itself indicate general acceptance? Why is it necessary to litigate that which is not disputed? For it does not seem to be disputed that the method works, only whether the guidelines cum rules have been satisfied. Thus does TWGDAM offer up hostages to fortune and serve justice (not).

My understanding is that the defense in Bokin did raise questions about the reliability of the P-E system through the testimony of two molecular biologists. In any case, I disagree with your suggestion that validation is unnecessary where no one has pointed to a specific problem. The major problem of undervalidated systems, from my perspective, is that their operating characteristics are not fully documented and understood, which can make it chancy to interpret the results correctly. For example, with STR systems it is very important, when interpreting mixtures, to know whether peak heights are consistent within and across loci under various conditions. If this information is not available, I don't see how the system can be considered reliable for dealing with mixtures. And this problem exists whether or not the opponent of the evidence can point to a specific instance where there was a problem.

Thank you, Mr. Fedor, for some very thoughtful comments on the Bokin opinion. Although I disagee with you on many points, I find your analysis intelligent and respectful. It stands in stark contrast to the public comments of another forensic scientist, who was quoted in the San Francisco Examiner Friday as saying the Bokin opinion was "ignorant and preposterous" and who attacked the judge personally as follows: "What you have here is an arrogant, ignorant, stupid judge who is unable to get through the smoke thrown at him by a very skilled lawyer." We will get a lot farther through respectful exchanges like this one than we will by publicly declaring that those who disagree with us are ignorant, stupid or callous.

Bill Thompson
wcthomps@uci.edu


CLOSE