A case of misleading analysis, inaccurate statements, and bogus evidence
For those unfamiliar with the case, American 1500 meter runner Shelby Houlihan revealed in a press conference on June 14th, 2021 that she had been banned from competition for four years by the Court of Arbitration for Sport (CAS) as a result of testing positive for the steroid, nandrolone. On September 1st, CAS released its 44-page decision in which the majority of the 3-person Panel found she had committed a doping violation. Although disappointed in the result, I was relieved her lawyer’s previous statements had been accurate and nothing new had emerged to fundamentally undermine the pieces I’d previously written on the LetsRun.com message board in her defense (The Railroading of Shelby Houlihan and Burritogate All Comes Down to Corn). For a better understanding of all the issues involved, I would highly recommend Twoggle’s comprehensive review of the case, Houlihan, Burrito-Gate and the Problems of Sports Drug Testing.
There were several new aspects to the story in the report which grabbed my attention:
First, there can be no longer any reason to doubt Houlihan ordered food from a Mexican food truck on that fateful night, a story which many observers had described as being fabricated. Second, the manager of the truck was revealed, along with information about the source of the pork in question, and that pig stomach (buche) and sausage (chorizo) burritos had been on the menu. Houlihan’s team believed she had ingested buche in a burrito, explained how it happened, and was able to convince the Panel the pork had originally been purchased frozen by the food truck company in September of 2020. Third, I also learned that pig diets were altered by many American farmers to include more soy (and less corn) during the processing slowdown that occurred during the pandemic. Lastly, the Panel was unable to decide which version of the WADA testing rules applied (2019 or 2021), but concluded it didn’t matter to the outcome and opted to follow the TD2021NA technical document.
So how did the legal team for World Athletics convince the Panel that Houlihan’s positive test result was certain enough to impose a harsh 4-year ban? They employed the following time-tested rhetorical tricks and logical fallacies to advance their arguments, the first of which sets the stage for the others.
The Fake Court
Despite appearances and commonly held assumptions:
- The Court of Arbitration for Sport is not a court.
- An arbitration hearing is not a trial.
- Arbitration does not mean both sides have necessarily consented to arbitrate the dispute. And under CAS rules:
- An athlete is not innocent until proven guilty.
- Allegations do not need to be proven beyond a reasonable doubt.
CAS was created in 1984 primarily to provide the speedy adjudication needed for Olympic competition, which it failed to do in this case until being forced to act by Houlihan’s legal team. Many observers understand correctly that the question of cheating in sport is not, generally speaking, a good matter for government courts to decide. However, these same observers fail to make allowances for the much lower standards of evidence, proof, and judgment that are relied upon in private adjudication, which allow weaker arguments to be used than those in a government court of law.
The statistical sleight of hand
The explanation presupposes a cascade of factual and scientific improbabilities, which means that its composite probability is (very) close to zero:
-World Athletics, page 7
World Athletics’ argument that an ingestion of pork-laden nandrolone could only have happened as a freak accident, and is therefore unworthy of the Panel’s consideration, does not stand up to statistical scrutiny. Since WADA performs ~278,000 tests per year, it is a mathematical certainty their net will capture such freak accidents on a regular basis. The “1 in 10,000” standard, for example, would result in 28 freak-accident positive results per year. Coincidentally, the annual number of traffic accidents in New Jersey is also 278,000 and the statistical probability of getting into an accident on any given drive is vanishingly small, about 1 in 13,000. Yet, somehow, every day there are approximately 760 NJ drivers involved in an accident, each one highly improbable; how can that be? Was the fact the drivers got into an accident sufficient proof their driving was bad enough to have their licenses suspended for four years? It is obviously not, and yet that is essentially what CAS has done, by denying Houlihan’s result could have been the result of any number of plausible – though individually unlikely – circumstances, despite a massive sample size of hundreds of thousands of tests performed on athletes across the globe, and dozens of negative tests over the years on Houlihan alone.
The Hearsay Evidence
The case made by World Athletics rests on a single assertion that the source of the pork was from an IBP/Tyson meatpacking plant (p. 30). The only person who could have possibly known this information was someone working for the food truck business itself, presumably the owner, who provided a statement about the menu that day (p. 28). As this was adjudicated in arbitration rather than a real court, this person was not a witness, did not take an oath, and would suffer no penalty whatsoever for providing a false statement. In an American court of law, such an assertion would be considered hearsay and disallowed from the proceedings. This is not meant to suggest in any way that the food truck owner has been dishonest or unhelpful, but simply that such a statement cannot be considered reliable evidence, even as Houlihan’s team has every reason to avoid antagonizing them by questioning their credibility. Nevertheless, a rational observer must recognize that every person in the pork supply chain (the farmer, the slaughterhouse, the butcher, the distributor, the purchaser, and the food preparer) would have a strong disincentive to reveal any knowledge that their processes could allow tainted pork to pass through under certain circumstances. The reality is that restaurants source their ingredients from a wide variety of sources, including just downstream from my local sewage plant, as learned on a run recently:
The Red Herring
Expert witness, Professor John McGlone, stated that “in order for uncastrated boar to end up in the normal pork supply chain, the boar(s) in question must have been cryptorchid” (i.e., pigs with hidden testes). By excluding boars and other types of pigs which may contain naturally occurring steroids, this assertion allowed World Athletics to focus their subsequent arguments and analysis on a much smaller range of possibilities and avoid considering other scenarios.
In addition, CAS’s quasi-legal standard that requires Houlihan’s team to prove she ingested tainted pork meant they had to choose one explanation even while others existed. The focus on pork stomach as the source of nandrolone allowed World Athletics to ignore previous research and analysis related to the ingestion of other pig offal, such as heart, liver, and kidney, which tend to contain much higher amounts of the steroid even though they could well have been ingested as ingredients in chorizo, for example.
The False Premise
McGlone’s cryptorchid deduction was based on the assumption there existed a “normal pork supply chain” when it was well known that, at least for a period of several months, the pork supply chain was anything but normal. In reality, the supply chain experienced a major disruption in the spring and summer of 2020 that created a massive backlog of pigs waiting to be slaughtered. Since the source of the pork in question was ostensibly Tyson Foods, how can the full-page ads taken out in newspapers across the country by its Chairman, John Tyson, on 4/26/2020 be ignored by McGlone and the Panel? Tyson stated: “Millions of animals – chickens, pigs and cattle – will be depopulated because of the closure of our processing facilities. The food supply chain is breaking.”
The food supply chain is breaking.
John Tyson, Chairman of Tyson Foods
In addition, the only Tyson Foods slaughterhouse in the American northwest, located in Wallula, Washington, was among those plants that was forced to close its doors with 147 Covid cases and 3 deaths among its workers. At the same time, hogs were suddenly being imported to Oregon and Washington in unprecedented numbers that were overwhelming the system. The Oregon Department of Agriculture warned on May 22, “These are hogs we cannot track and that leads to serious concerns about this new swine population…” and on June 30, the Oregon passed a law allowing in-state USDA processing to increase capacity. In response, the policy director of the Friends of Family Farmers said, “A big problem … is that there are a lot of custom-exempt or smaller meat processors in Oregon, but in order to get them up to USDA standards will take considerable resources.”
One has to ask, if the pork supply chain was still considered “normal” even after the chairman of Tyson declared it was broken, and the state admitted it could not track the hogs, under what crazy scenario would it ever be considered “not normal” by the WADA lab?
The argument McGlone provided was that meat processing had pretty much returned back to normal by the fall, and yet, the Panel admitted it was likely the pork in question had been purchased as a frozen product in September (p. 30). Pork is routinely frozen for periods up to 4 months or more and it is probable the pig in question was slaughtered and packaged at the height of the disruption. As to how this chaotic situation led to a piece of pork containing nandrolone is an open question, but like the many sides of a die from which only one will be chosen, there are a number of possible explanations:
- It may well have been a cryptorchid that slipped through the system, which due to the delays in processing, was older than the usual 6-month-old pig, and subsequently had higher androgen levels. Or similarly, it could have been an intersexual pig (not quite the same as cryptorchid) which are relatively common (0.5% of the population) and may also produce nortestosterone (see Intersexuality in pigs: Impact on veterinary public health and food safety, 2009).
- Maybe the manager of the food truck was mistaken about the pork used that night, or the cook had replaced it for some reason with another product. Perhaps the pig was in fact a boar that had been butchered at one of the many local slaughterhouses and was somehow packaged as a Tyson product for expediency during their shutdown/slowdown. In normal times, Tyson Foods relies on 2000 independent pork producers to supply its plants and it’s reasonable to guess a few of them raised their pigs slightly off-manual during the pandemic.
- Perhaps the source of pork was a gilt or sow with elevated androgen levels, and/or the burrito contained sausage from its kidney or liver (for possible nandrolone levels, see Table 3 in Poelmans, Endogenous occurrence of some anabolic steroids in swine matrices).
- Perhaps the source was an immunologically-castrated barrow that, due to the delay in processing and lack of tracking, was slaughtered after the temporary effect of the vaccine wore off and its testosterone levels had risen to normal levels. Beyond that, the use of its testicles in sausage (see Formulation of sausage with added pork testicles as an offal ingredient) from such a barrow would certainly also result in high levels of nandrolone.
- Or perhaps the slaughterhouse thought they were looking at an immunologically-castrated barrow when it was actually a boar. Such a barrow’s testicles might not look much different to a worker unaccustomed to such barrows. A PowerPoint presentation, definitely not for the faint of heart, by the Food Safety and Inspection Service (FSIS), shows the difference between the two (FSIS Requirements associated with Slaughter of Immunologically Castrated “Barrows”).
- There is no FSIS rule against boars per se, and the law which slaughterhouses must follow (Sec. 311.20) requires them to check carcasses for a “pronounced sexual odor”. If it is “pronounced” the meat must be discarded; if the odor is “less than pronounced”, the meat may be chopped up and used for sausage and the like. There can be little doubt this job of sniffing heated pork carcass samples, requiring removal of one’s mask to perform properly, became even less desirable when their colleagues were dying from Covid. To repeat, the federal rule that USDA slaughterhouses must follow to prevent boar taint (which goes hand-in-hand with nandrolone production in pork) would have been dangerous to perform during the height of the epidemic, and impossible for any worker who’d lost their sense of smell due to Covid.*
*Hyperbole? One third of all Covid cases in the state of Arkansas were Tyson employees (At Least 9,000 Arkansas Workers Caught Covid-19 as Pandemic Overwhelmed Regulators, Companies).
The Monday Morning Quarterback
With respect to due process, careful observers will also note that every bit of McGlone’s testimony was irrelevant to the question of whether the Montreal WADA lab and its director, Christiane Ayotte, gave adequate consideration to the possibility the nandrolone came from food. There is no indication the lab consulted with McGlone in January 2020, or that it contacted the food truck to learn more, even though farm-to-table restaurant arrangements are common. Ayotte’s decision to ignore that possibility and issue an Adverse finding instead of an Atypical finding was based on the shaky presumption that the food truck procured its meat through mainstream commercial sources, when it could have been acquired through a custom meat processor. In Oregon, there are over a dozen custom slaughterhouses and nearly 80 custom meat processors, in addition to ones in nearby Washington. (A guide to non-mainstream meat processing and rules can be found at nichemeatprocessing.org).
If a positive finding had come from an athlete in a country in which male pigs are not usually castrated, would Montreal have issued an Atypical finding then? One would hope so, however, it is unreasonable in the first place to expect a WADA lab director to have knowledge of every local food supply chain on the planet, in addition to the biochemistry and testing expertise needed to perform the precise analysis required. Ayotte, who is also serving as an expert witness to this arbitration, claims to have ample evidence from the literature and her own studies of the United States and Canada that show corn feeding practices are so consistent that any possible pork specimens Houlihan ingested would have been free of nandrolone and would have exhibited δ13C values (i.e., carbon isotope signatures) in line with corn-based diets (well above -23‰ on the PDB scale). It is doubtful such evidence exists.
While mainstream pork products in the US certainly share those characteristics on average, the possible RANGE of those values at the consumer level of supermarkets and restaurants has never been adequately tested for research purposes, let alone legal ones. In fact, there appear to be no studies on the presence of boar taint on the consumer side, and only one study that measured δ13C values in US pork. That study (Application of Stable Isotope Ratio Analysis for Origin Authentication of Pork) was performed in Korea on pork samples exported to that country, and thus hardly representative of the variety raised in small farms and hog factories across the US. As expected, the δ13C values of the 30 American samples were the highest of the bunch (-14.78‰ ±1.34), but the Canadian pork — which Ayotte claimed had similar corn feeding practices to the US — was quite different. The seven samples of pork from Canada had δ13C values that averaged -22.87‰ ±0.92, while all nine European countries had pork δ13C values that ranged from -23.44‰ to -27.17‰. Thus, it is easy to see δ13C values are quite sensitive to diet and there is little reason to believe American pasture-raised pigs who eat less corn than confined pigs would have δ13C values much higher than -23‰ (the value found in Houlihan’s sample). The 2019 study Isotopic carbon turnover in pig hoof and rib (Figure 2) also demonstrates the range and sensitivity of δ13C to diet.
Added on to that is the possibility that pork δ13C values changed in response to American farmers’ desperate actions to slow the growth of the pigs in their care during the pandemic – to avoid killing them – as slaughterhouses across the country became backed up. In part, this was done by reducing the percentage of corn in their feed and replacing it with soy which has higher δ13C values. This could also explain why the pork eaten by Houlihan, even if it did come from a mainstream source, may have had lower δ13C values than usual.
Whatever the reason, it is reasonable to think that the particular circumstance which led to the pork in question containing nandrolone was the same circumstance which led to it having low δ13C values. For example, a pig raised at a small farm is much more likely to be pasture-raised (resulting in lower δ13C values) and be left uncastrated (resulting in the presence of nandrolone) than one raised in a Concentrated Animal Feeding Operation (CAFO). That is one reason why the “composite value” of probability for something like this happening should not be deemed anywhere near zero as the Panel decided.
Misleading Analysis
The World Athletics legal team also misrepresented the research that has been done on nandrolone and pork consumption to give a false impression the 19-NA values were suspiciously higher than they actually were.
First, they did this by highlighting the higher tested raw values (6.9 and 7.8 ng/ml) instead of the proper values adjusted for specific gravity (as they are reported in the research literature) found in the urine of a dehydrated Houlihan, which were actually 5.2 and 5.8 ng/ml.
Second, they designated stomach maw as “meat” rather than offal (is heart meat not offal then?), and referenced only those few instances in the research literature where boar meat alone was tested (not offal). This allowed the team to ignore the much higher excretion rates that have been observed to occur after consumption of pig offal, as well as meals consisting of mixed meat & offal. As admitted by the Panel, there is no research showing levels of nandrolone in pork stomach, but it is certainly considered offal, and it is remiss to exclude references in the scientific literature where high levels of 19-NA excretion were observed after the ingestion of pig offal.
Third, the team failed to recognize that the Cologne study’s wild boar meat could have been either male or female, and pointed to 7 of 12 subjects testing negative, as if that were a sign boar meat doesn’t contain much nandrolone when probability alone would predict half of the samples being negative.
Fourth, Ayotte seemed to contradict statements from her own studies, when she describes the concentration of 19-NA in the Athlete’s urine as being ‘2-3 times higher than the highest values reported in the scientific literature’. Yet it has been repeatedly shown that in the hours following ingestion of boar meat and offal, 19-NA values in urine samples “could be in vast excess of the threshold” (Excretion of norsteroids’ phase II metabolites of different origin in human, Conclusion) and “with levels even reaching 160 ng/ml in one case” (Significance of 19-norandrosterone in athletes’ urine, page i27).
Fifth, since Houlihan is accused of taking a synthetic form of nandrolone, it is odd to omit the range of 19-NA values found in the literature for individuals who were known to have taken such drugs orally. The range of those values is very wide, indeed, and the highest concentrations astronomical (e.g., in Guay, 19-NA values as high as 433, 3540, and even 36500 ng/ml were observed) as compared to Houlihan’s 5.2 and 5.8 ng/ml readings.
Lastly, Ayotte stated “there is no evidence suggesting that the North American population is involuntarily consuming edible parts of uncastrated pigs.” In previous postings I have provided evidence from farmers themselves that uncastrated pigs still get eaten, often unawares. One farmer wrote, “We’ve steadily been harvesting boars here on our farm, and thus far have had no complaints… Our first set of hogs that we brought to the USDA processor were more like 8-9 months old…” while another commented about processing tainted pork, saying “You can always make chorizo!”
There is no USDA regulation against harvesting boars and it would be worthwhile to test a wide-ranging set of pork samples from the consumer side to observe how many were actually from males (and if scientifically possible, boars vs. barrows), and how many (not just males) had taint. The kind of study needed would be something like what Consumer Reports did in 2012 when it discovered that most pork (69%) is contaminated with Yersinia bacteria. However, it is likely the pork industry would not welcome another study like this.
Inapplicability of Past Research
The truth is, the measured value of orally consumed nandrolone metabolites, if present, will depend primarily on the TIMING of the test, according to how well a particular urine sample happens to capture the peak of excretion that occurs after nandrolone ingestion. More importantly, all previous research studies on detecting nandrolone after consuming pork cannot be applied to Houlihan’s case because the sampling methodology differed in one very important aspect: their subjects were permitted to urinate (and drink) MULTIPLE times before the 10th hour, whereas Houlihan presumably urinated only once shortly after consuming the pork, and then slept a full night before being tested in the morning. That is to say, if previous research subjects had been tested in the same way as a dehydrated Houlihan was, they would likely have exhibited much higher levels of nandrolone metabolites than they actually did.
Illustrating this effect was WADA’s sampling of 800 meter runner, Gomathi Marimuthu, who in 2019 provided two separate In-Competition samples that later tested positive for nandrolone. The reason for two collections was that, according to the World Athletics Disciplinary Tribunal decision, she hadn’t quite filled the second of two bottles to the top and the WADA agent demanded it be redone. According to the Decision (Paragraph 94), the first sample that day, taken at 19:15, was found to have a concentration of 187 ng/ml. (Again, compare these values to the Panel’s misleading assertion that 5 ng/ml represented suspiciously high amounts.) The second sample, taken at 19:50, had a concentration of 16.2 ng/ml. Why the vast difference of 187 vs. 16 ng/ml if the test had been taken only 35 minutes apart? The obvious reason was that she was starting from an empty bladder after the first urine sample after having consumed fresh fluids, and the peak phase of excretion, which the first sample captured, had already passed. So, to reiterate, all past research on nandrolone ingestion via pork allowed participants to urinate multiple times before testing at the 10-hour mark, whereas the timing of Houlihan’s urination and test would have maximized the amount of metabolites found. Expressing the measurement in nanograms per milliliter and adjusting it for specific gravity would not get around this basic fact.
For all the above reasons and more, it is my hope the CAS decision will eventually be overturned and that the approach to testing athletes for forbidden substances will be reformed.
Comments, questions, and corrections are welcome (on the LetsRun.com Messageboard, or in the moderated comments below).
Let it be noted I have no connection to any athletic organization or any party related to this case whatsoever. With respect to my motives, I am firmly against the common practice of mandatory binding arbitration, as well as CAFO farming which is a great public health risk (through antibiotic resistance, etc.). For all the wonderful small-scale farmers out there, I am offended that elite athletes, as role models for health and fitness, cannot safely eat food from local and natural sources.
Update 9/26: Fixed δ13C units to ‰ (instead of %) and reworded introductory sentence to read, “September 1st, CAS released its 44-page decision in which the majority of the 3-person Panel found she had committed a doping violation.” It formerly read, “On September 1st, CAS released its final 44-page report on the case which found her guilty by a 2-1 vote.”
Update 9/29: Since a FSIS training document suggests slaughterhouse taint testing is performed only on boars, and another FSIS doc suggests it is performed also on barrows, I have removed mention of testing on females in explanation #6 and the question ‘why is there even a need to check for sexual odors?’ (even though 311.20 clearly applies to all pigs).
The lab director has already been found to be dirty in a prior case.
Nothing she says can be believed.