P.S. Cato has now posted my Commentary article from last year on federal pressure for universities to reduce the procedural rights of accused students and faculty.

“The data also shows 6 of every 1,000 students are victims.” is a post from Overlawyered - Chronicling the high cost of our legal system

P.S. This supposedly corrective piece at Slate is if anything worse than the chart it purports to correct, straining to minimize false accusation as even rarer than portrayed. (It’s worth remembering that its author, Amanda Marcotte, has a bit of a history herself when it comes to credulity on this subject.) Bennett again provides a needed corrective: “Forensic DNA typing laboratories — as numerous commentators have noted — encounter rates of exclusion of suspected attackers in close to 25 percent of cases.” (& Greenfield; and an informative followup from Bennett regarding the incidence of false accusation.) Yet more: Washington Post ombudsman says mistakes were made.

An in-faux-graphic on rape statistics is a post from Overlawyered - Chronicling the high cost of our legal system

Sleight of hand and litigation statistics is a post from Overlawyered - Chronicling the high cost of our legal system

I wonder what he’d think of the latest statistics about case filings and dispositions in our federal courts. [Update: The statistics were just released on August 20th.]  We just got our printed version in the office on Friday (and the stats are now available here online). Could the days of “Overlawyered” be numbered with an actual decline in certain types of lawsuits? Nah, but some of the numbers sure are curious.

In Connecticut, for example, new lawsuits are down over 20 percent in the last ten years or so. Employment discrimination lawsuits are down almost 25 percent nationwide since 2000.

But as Twain hinted, stats aren’t always what they seem. While certain areas have seen decreases, others have seen increases. Wage & hour claims are up 25 percent since 2000 and claims filed in parts of Florida have skyrocketed over the last year or so.

So, is litigation up or down in federal courts? Yes and no. It just depends how you crunch the numbers.

Lies, Damned Lies and Court Statistics… is a post from Overlawyered - Chronicling the high cost of our legal system

Physicians who made three or more malpractice payments between 1990 and 2006 – accounting for no more than 4 percent of New York’s doctors – were responsible for nearly half (49.6 percent) of medical malpractice dollars paid out on behalf of doctors in the time period.

This is technically true, but wildly misleading; we previously refuted this precise statistic as a natural statistical consequence of any randomly distributed set of payouts–and given that doctors in high-risk professions such as neurosurgery or ob/gyn are far more likely to be sued than dermatologists or gerontologists, the random concentration effect is going to be even more pronounced, so the Public Citizen statistic is meaningless without a showing of speciality-adjusted correlation between time periods–something no study has ever found.

But note how blogger Eric Turkewitz writes an op-ed in a small-town New York newspaper that isn’t even satisfied with simply misleading the public, and says something that is out-and-out false:

4 percent of the state’s doctors contribut[e] to half of the malpractice suits [emphasis added]

Not remotely true. “Nearly half of payments” has been turned into “half of malpractice suits.” Justinian Lane, who knows or should know that the latter statistic isn’t true, because his blog posted about the original statistic, then repeats the lie either thoughtlessly or deliberately:

Maybe doctors should discipline the four percent of doctors that make up half of all malpractice claims.

Will either of them retract the false claim with the same fanfare that they made it? Stay tuned. (They certainly won’t explain that there’s nothing damning about the accurate statistic–though I have been refuting this for over three years, Public Citizen and trial lawyers and their fans continue to regurgitate the data as if it means something.)

How trial lawyer urban legends get started is a post from Overlawyered - Chronicling the high cost of our legal system

Some small X% of doctors responsible for large Y% of payouts is always going to be true simply by random chance. It’s going to be true over any time period: the problem is that if you take that time period and divide by two, the X% in the first half of the time period are going to be almost entirely different than the X% in the second half of the time period. Even if you were to fire every single one of those doctors in the tail in the first time period, all you have is X% fewer doctors; the very next year, it’s going to be a different small A% of doctors responsible for large B% of payouts, and you’ve solved nothing. With very rare exceptions medical malpractice payouts have absolutely nothing to do with the quality of the doctor, and everything to do with the risk profile of their practice.

It’s worth noting Eugene Volokh’s excellent explication of the issue:

I often see these sorts of statistics that purport to show that some fraction of the X’s are disproportionately prone to event Y. One paper I read, for instance, reported that 10% of all police officers in a department account for 25% of all abuse complaints, and used that as evidence for the proposition that some police officers are especially prone to misbehavior. One can imagine similar claims when, say, 10% of all holes on a golf course account for 25% of all hole-in-ones, or 10% of all slot machines account for 25% of all jackpots, and so on.

The trouble is that this data point, standing alone, is entirely consistent with the X’s being equally prone to Y. Even if, for instance, all the holes on a golf course are equally difficult (or all the police officers equally prone to abuse complaints), and hole-in-ones (or complaints) are entirely randomly distributed across all holes (or officers), one can easily see the 10%/25% distribution, or 20%/80% distribution, or whatever else.

Consider a boundary case: Say that each police officer has a 10% chance of having a complaint this year. Then, on average 10% of all officers will have 100% of this year’s complaints. Likewise, say that each police officer has a 1% chance of having a complaint each year for 10 years, and the probabilities are independent from year to year (since complaints are entirely random, and all the officers are equally prone to them). Then, on average 9.5% (1 – 0.99^10) of all police officers will have 100% of the complaints over the 10 years, since 0.99^10 of the officers will have no complaints.

Or consider a less boundary case, where the math is still easily intuitive. Say that you have 100 honest coins, each 50% likely to turn up heads and tails. You toss each coin twice. On average,

* 25 of the coins will come up heads twice, accounting for 50 heads.
* 50 of the coins will come up heads once and tails once, accounting for 50 heads.
* 25 of the coins will come up tails twice, accounting for no heads.

This means that 25% of the coins account for 50% of the heads — but because of randomness, not because some particular coins are more likely to turn up heads than others.

Likewise, we see the same in slightly more complicated models. Say that each police officer has a 10% chance of having a complaint each year, and we’re looking at results over 10 years. Then 7% of all officers will have 3 or more complaints (that’s SUM (10-choose-i x 0.1^i x 0.9^(10-i)) as i goes from 3 to 10). But those 7% will account for 22.5% of all complaints (that’s SUM (10-choose-i x 0.1^i x 0.9^(10-i) x i) as i goes from 3 to 10). And again this is so even though each officer is equally likely to get a complaint in any year.

Now of course it seems very likely that in fact some officers are more prone to complaints than others. My point is simply that this conclusion can’t flow from our observation of the 10%/25% disparity, or 7%/22.5% disparity, or even a 20%/80% disparity. We can reasonably believe it for other reasons (such as our knowledge of human nature), but not because of that disparity, because that disparity is entirely consistent with a model in which all officers are equally prone to complaints.

If you have more data, that data can indeed support the disproportionate-propensity conclusion. For instance if nearly the same group of officers lead the complaint tallies each year (or nearly the same group of slot machines leads the payouts two months running), that’s generally not consistent with the random model I describe. Likewise, if you have more statistics of some other sort — for instance, if you know what the complaint rate per officer is, and can look at that together with the “X% of all officers yield Y% of the complaints” numbers — that too could be inconsistent with a random distribution.

But often we hear just a “10% of all X’s account for 25% of all Y’s” report, or some such, and are asked to infer from there that those 10% have a disproportionate propensity to Y. And that inference is not sound, because these numbers can easily be reached even if everyone’s propensity is equal.

UPDATE: (1) Some commenters suggested this phenomenon “depends on the sample size; if the sample size is large enough, the inference is sound.” That’s not quite right, I think.

The sample size in the sense of the number of police officers / golf holes / coins does not affect the result. I could give my coin example, where 25% of all coins yield 50% of all heads, with a million tosses.

The sample size in the sense of the number of intervals during which an event can happen (e.g., the length of time the officers are on the force, if in the model there’s a certain probability of a complaint each year) does affect the result. But if the probability per interval is low enough, we can see this result even when there are many intervals.

Say, for instance, that there’s a 1% chance of a complaint per month for each officer, and we look at 240 months (representing an average police career of 20 years). Then even when all officers have precisely the same propensity to draw a complaint, 9.5% of all officers would have 5 or more complaints, and would account for over 21.5% of all complaints. So a 9.5%/21.5% split would be consistent with all officers having an identical propensity to generate complaints, even with a “sample size” of 240 intervals. If the monthly complaint probability was 0.005, then 12% of all officers would account for over 33% of all complaints.

(2) More broadly, this isn’t a matter of “sample size” in the sense that we’d use the term when discussing significance testing, and talking about “statistical significance” wouldn’t be that helpful, I think. If you have a lot of data points, you can determine whether some difference between two sets of results over those data points is statistically significant. But here I’m talking about people’s drawing inferences from one piece of (aggregated) data — 10% of all X’s account for 25% of all Y’s. Statistical significance testing is not apt here.

“4% of doctors responsible for 50% of payouts” is a post from Overlawyered - Chronicling the high cost of our legal system