Volume 1 Number 2 - December 2003

How Not to Lie with Statistics

Next year will mark the fiftieth anniversary of a classic book: Darrell Huff's How to Lie with Statistics. With his tongue firmly planted in cheek, the author wrote a "primer on the ways to use statistics to deceive." His real intention was to encourage statistical literacy in a population continually bombarded by statistical half-truths and over-simplifications.

Five decades later, the kinds of problems he addressed are even more widespread. Newspaper, television, and the internet are filled with statistical "factoids" that are often misleading, lacking in context or downright wrong. In the courtroom, statistical evidence has become increasingly prominent, and the potential for misinformation is great.

Fortunately, recent case law has begun to address the issue of standards for scientific evidence (see article below on Daubert and Statistics). So, abuses of statistical reasoning should in theory be on the decline, as unqualified experts and unfounded opinions are weeded out by inceasingly knowledgable judges. Before wading into the murky waters of statistical methodology, however, judges could do worse than to peruse Professor Huff's insightful little book.

Cluster or Coincidence?

A few weeks ago I was attending a parents' night at my son's high school. His English teacher remarked on the coincidence that often at least one parent in each class happened to sit in his/her child's seat. This statement got me thinking.

A simple calculation revealed that the probability is actually over 60% that such an occurrence would take place by chance. Now, at first glance, this might seem odd, considering that in a class of 30 students, the chance that any particular parent would sit in the right seat is only one in 30. The paradox is easily resolved, however, by realizing that the likelihood is much higher that at least one out of the 30 possible "coincidences" will occur.

The failure to take into account the multiplicity of occurrences that might lead to an apparently rare outcome is quite common in legal and regulatory settings. For example, there are many cases in which a cluster of illnesses (e.g., specific types of cancer) are attributed to some identified potential exposure (nuclear power plant, toxic waste site, manufacturing facility). The large number of cases noted in a small geographic area leads to the belief that the proximate source of exposure must be the cause.

A recent well-publicized example involves over 200 lawsuits filed against IBM Corporation by workers in microchip assembly plants who contracted a variety of rare forms of cancer. IBM contends that the link between exposure to chemicals in the workplace and cancer is entirely spurious. Epidemiologists and biostatisticians on both sides have been embroiled in arguments over whether this apparent cluster of cases is statistically significant and, if so, can be attributed to the IBM plants.

Without attempting to weigh in on the merits, we simply observe that the true rarity of any event must be evaluated in a broad context. In estimating the probability of observing a particular unusual occurrence, one must take into account all patterns that could have occurred and might similarly have been regarded as anomalous. In some cases, we may find that the occurrence is not so unusual as we may have thought.

Daubert and Statistics

Like any scientific testimony, statistical evidence must be relevant and reliable to be admitted by the court. In today's world, this means that the evidence presented must meet the criteria established in the famous case of Daubert v Merrell Dow Pharmaceuticals, Inc. As is now well known, the U.S. Supreme Court in Daubert and placed a gatekeeping responsibility on judges, who must determine whether the proposed expert testimony is sufficiently reliable and relevant to meet the standard articulated in Fed. R. Evid. 702.

The Daubert decision was stimulated in part by the complex statistical evidence confronting the court in Daubert, and the attendant battle of experts. It is clear that statistical analysis can be relevant and admissible in product liability cases like Daubert, as well as other similar situations. However, to pass the test, the proposed use of statistics must satisfy the following criteria:

1. Whether the theory or technique has been or can be tested, i.e. falsified;

2. Whether the theory or technique has been subected to peer review and publication;

3. Consideration of the known or potential rate of error of the method used;

4. The existence and maintenance of standards controlling the technique's operation; and

5. Whether the theory or method has been generally accepted by the academic community.

Statistical methodology certainly meets these criteria in general. Statistical techniques embodying the above principles have become the backbone of many scientific disciplines. So, there will often be a presumption that appropriate statistical analyses can pass the Daubert test. The real question, then, is whether the statistical techniques are being applied in a manner that is both technically correct and relevant to the issues being litigated.

So how often does statistical testimony pass the test? This is a difficult question to answer in general. Our research has failed to reveal any large-scale studies pertaining to the frequency of Daubert challenges and the proportion of these challenges that are successful. If our readers know of any such data, we would be grateful for information on how it can be obtained.

Some limited statistics on the subject are available at www.daubertontheweb.com. For cases that reached the appellate level, data pertaining to many areas of expertise are presented. Unfortunately, testimony by statisticians accounts for only seven of the challenges assembled in this impressive database. However, in five of the seven challenges, the admissibility of the proposed expert testimony was upheld. While the sample is obviously small, and possibly biased by restriction to appellate level cases, this result provides evidence that well-prepared statisticians are often surmounting the Daubert hurdle.

Herbert I. Weisberg, Ph.D.
61 Pheasant landing Road
Needham, MA 02192-1000

Phone (781) 455-6850

Fax (781) 444-9563

email: hweisberg@correlation.com
Please let us know what you think about our newsletter. Thanks.