Racing and training in carbon-fiber plate and foam “super-shoes” has been a rewarding and undeniably faster experience for the author. Nevertheless, their use has been controversial in competitive running circles, as they are credited with giving runners an unfair advantage over those not competing with them, and making it a challenge to compare the records of today with those of the past.
[View discussion on LetsRun.com]
Numerous studies and anecdotal accounts show runners have achieved major gains in running efficiency and speed while wearing super-shoes. As far as maintaining the integrity of the sport, however, complaints against the supposed advantage of the shoes have been almost entirely focused on the record-breaking performances of its elite competitors, whereas few if any have voiced concerns about the integrity of records being set at the local Turkey Trot or parkrun. For this reason, examining the performances of the top 100 runners each year, which encompass nearly all the record-breaking performances in question, is the most appropriate and meaningful analysis to undertake towards answering these concerns. These are not the top-100 times, but the best time of the top-100 runners in their particular events, the majority of which reached the top 3 podium spots, and nearly all of whom were in the top-20 positions of the race timed. As winners and top finishers of their respective races, super-shoes should have provided a massive boost to all who were wearing them, with an effect on times that should be easily discernible in yearly statistical comparisons.
Regardless of where one stands on the issue of super-shoes, it is paradoxically in everyone’s interest to believe super-shoes do provide advantages:
-The journalist who wants a sensational story.
-The curmudgeonly commentator who claims the times of his favorite runners were not truly beaten by today’s youthful stars.
-The athlete who races in the shoes hoping they will benefit not only from their technology, but their psychological boost as well.
-The plain-soled runner who lost to a super-shoe-bearing competitor and wants to believe the loss was due in part to the lackluster response of their sponsor’s shoe.
-And of course, the store salesman who sells the shoes, and especially the company which manufactures them, that wants to show they are worth the exorbitant prices being asked for them.
- “This generation of shoes has created a seismic shift in performance, dissociating times achieved in them from all performances over the last 25 yrs & more, which had previously stood up to mutual comparison.”
- “While Nike’s Vaporfly Next% has survived a ban from the 2020 Olympics, any future versions of the shoe will be outlawed. And the trainers have changed running forever.”
- “I think that running is undergoing a new phase, lets call it before and after super-shoes phase. Old records cannot be compared with the new ones.”
- “My times have improved by an astonishing 4.5 per cent, which is almost exactly the figure that Nike claims the £240 shoes – please don’t tell my wife – improve one’s ‘running economy’ by.”
- “I had talked to people on the inside in late 2016 who said the shoes were a performance enhancer on par with EPO, but I didn’t know that they truly worked until Nike released their data in March of 2017.”
- “Athletics is playing catch-up to police running-shoe regulations adequately, which have distorted results and times to such an extent that it’s a similar crisis to the gains made by doping.”
- “Super shoes… ripping the heart out of distance running since Rio 2016. Records are now almost totally meaningless.”
- “And now Nike has produced an even more advanced shoe, known as the Alphafly, which Nike says meets World Athletics’ new shoe guidelines and which some have suggested could be twice as effective as the Vaporfly, yielding 7-8% gains in running economy vs. 4-5% for the Vaporfly.”
- “This isn’t about Nike v the rest; it’s about the integrity of the sport overall, fairness & a level playing field for everyone. Shame an honest, objective & transparent debate is so difficult, bc the shoes have totally changed athletics.”
- “Is that fair?”
So it may come as a surprise to many that an analysis of 20 years’ worth of race data shows that elite men have shown no improvement in race times beyond prevailing trends since the adoption of super-shoe technology in 2017. The same definitive conclusion cannot necessarily be drawn for the elite women, which is a bit more ambiguous, though similarly suggestive.
Data Sources
The events studied for the years 2001-2021 were the: 10,000 Meters, 10km Road race, Half Marathon, and Marathon, based on data from World Athletics. For each event, the average time of the top 100 runners was calculated for each year and placed on a graph showing the progression of times before and after the adoption of super-shoes in 2017. (There were a few cases of super-shoes appearing before then, but their use did not become statistically meaningful until 2017.) Based on a few different sources, the percentage of the top-100 runners wearing super-shoes was estimated to be, for women:
11% in 2017, 38% in 2018, 59% in 2019, 88% in 2020, and 92% in 2021, and for men:
13% in 2017, 45% in 2018, 65% in 2019, 88% in 2020, and 92% in 2021.
The estimates for the first three years were derived from data downloaded from the 2020 Cornell study, “An Observational Study of the Effect of Nike Vaporfly Shoes on Marathon Performance,” whose authors painstakingly examined race photographs of 578 runners in 22 well-known marathons in the US and Canada to record their times and determine what type of shoes they were wearing. They found that roughly 9%, 31%, 47% of women runners were wearing Vaporflys, as were 12%, 41%, 59% of the men, respectively, during 2017 to 2019. Naturally, shoes worn by the top-100 distance runners in the world would have differed significantly from those worn by the mix of American elite and amateur runners in the Cornell study. To get a sense of the difference, race photographs of the 100 women in the top 2019 world marathon times were searched for and examined for this analysis. Of the 87 that could be identified, 51 were wearing super-shoes, or 59% (about a quarter more than the 47% in the Cornell study).
A second major source was a Running World article, “What were people wearing on their feet for the World Half Marathon Champs?” that precisely counted the shoes all 116 men competitors of the World Athletics Half Marathon Championships were wearing in 2020 at Gdynia. The researchers estimated approximately 90% of the men were wearing carbon-plated shoes. A few might not fall under the category of “super-shoes” defined here1 so the estimate is cut back to 88%. Another assumption is that the number of women’s shoes had reached similar proportions as the men by 2020, and that races for the top-100 were composed of athletes similar to those at Gdynia. Then, for 2021, it is assumed the use of super-shoes among the elite increased slightly to 92%. Based on these measurements and a spot-checking of other sources, the estimates above for the last five years are hoped to be slightly conservative, and as long the percentages of elite runners wearing super-shoes are not much lower, the conclusions should remain valid.
The Cornell study also attempted to determine how much a runner’s time would improve after changing over to Vaporfly shoes. Although the range varied quite a bit between individuals, they found on average that men’s times improved about 2.1% (+/-.7%) and women’s times less, about 1.4% (+/-.8%). However, these numbers are at odds with other studies finding that women seemed to benefit more, not less, from the super-shoes. For example, Senefeld’s 2021 study, “Technological advances in elite marathon performance” found that the ” the magnitude of improvements in performance for males (2.0%) and females (2.6%) are similar to the 2% faster performance predicted using models based on metabolic savings in running.”
If the performance advantage of super-shoes is only 1.4% for women or lower, it naturally becomes easier to claim that any lowering of times is due to the shoes. On the other extreme, if the performance advantage for men is not 2% but 2.6% (which would be astronomical), it becomes almost impossible to reconcile the more modest performance gains with those predicted by the shoe technology. The original research on super-shoe performance, by Wouter Hoogkamer et al. concluded in 2017: “a 4% average energetic savings observed should translate to ~ 3.4% improvement in running velocity at marathon world record pace (20.59 km/h).” For simplicity, and because the number most often cited with concern by critics is 2%, the following graphs for both men and women have been calculated using that figure.
The Graphs
Each graph can be interpreted in four main ways:
- The primary blue line with diamond-shaped points is simply the average time of the top-100 athletes worldwide in that event for the years indicated.
- The black and red trend line is a (“best fit”) linear regression based on the times between the slowest year and the year 2016, as marked by the arrowhead. The red segment after the arrowhead shows the projected trend after 2016, following the slope from earlier years. Blue diamond points located on or near the trend line indicate those performances were likely not impacted by shoe technology.
- The orange line with asterisks shows what average times would have been in 2017 through 2021 if the 2% performance increase of the shoes was subtracted from racers’ times, and adjusted for the increasing percentage of runners wearing them each year. If super-shoes had an impact, the asterisk line should follow the general trajectory of the blue line established prior to 2017. If, on the other hand, the asterisk data points do not seem consistent with the blue diamond line, it is likely the shoes were not a significant factor.
- The green triangle shows what the top-100 average runners’ time would have been in 2021 if the improvement from the slowest year to 2016 had continued at the same rate each year AND if 92% of runners had been wearing shoes that decreased their times by 2%. If super-shoes were providing such a boost, one would expect the blue diamond line to intersect the triangle, or come close to it, either above or below it.
Consider first the men’s performances:
Men’s 10,000 Meter & 10K Road Races
In the 10,000 Meters event (on the track), the slowest year in the twenty-year period was 2001. During the years 2001 through 2016, the average time declined by 1.6 seconds per year3, and if that trend continued through 2017-2021, one would predict an average time of 27:31 in 2021. The actual time was almost identical, 27:32, so it is very difficult to say that super-shoes were responsible for the time. If we also consider the advantage super-shoes were supposed to have on the result, the 2021 time should have been 27:01 which, as the green triangle on the graph shows, is wildly off the mark. Likewise, the orange asterisk line (that models what the men’s performances would have been without the advantage of super-shoes) shows a rather stark rise that is inconsistent with the slope of times in the years before it.
The range and slope of the Men’s 10K Road race times are similar to the Men’s 10,000 Meters. Starting from 20042, the average time declined by nearly 1 second per year, and the 2019 average time was a bit below the trend line. However, by 2021 the average time had reverted back to what the trend would predict, and was nowhere near the mark promised by the shoes. Like nearly all the graphs, there was a major spike upward in 2020 which is most likely explained by the large number of race cancellations from the pandemic and the resulting smaller pool of times.
Technical Notes:
1Super-shoes are defined here as any carbon-and-foam running shoe which has been shown in the lab to increase running efficiency, primarily consisting primarily of Nike’s Vaporfly and Alphafly models, Asics’ Metaspeed Sky, and to a lesser extent, Adidas’ Adios Pro models, Saucony’s Endorphin Pro models, and a few others.
2With respect to the 10K Road race trend line, astute observers may object it starts from 2004 rather than the 2001 time used in the 10,000 Meters graph. The purpose (aside from 2004 being the slowest year in the period) is to determine, as much as possible, the prevailing trend in effect during the latter, not earlier, years of the period, since the years immediately preceding 2017 will better capture the forces impacting contemporary times than ones existing in the early 2000s.
3The average improvements per year stated are not calculated based on the visual trend line but are, for simplicity’s sake, simple averages based on the number of years between the slowest time in the period and the 2016 time.
*The scale of the Y-axis varies among graphs, thus making visual comparisons between graphs difficult.
Men’s Half Marathon
The slowest year was 2001 (same as 2003) and the average decrease in times through 2016 was 5 seconds per year, leading to a projected time of 1:00:08. The men ran exactly that in 2020, and 3 seconds slower in 2021, nowhere near the 59:02 projected by the super-shoe model.
Men’s Marathon
The slowest year was again 2001, and right in line with the Half Marathon, the time improved 10 seconds per year up through 2016. The 2:06:02 result for 2021 is very close to what the trend line predicts. The first three years of the orange asterisk line (the times expected if there were no super-shoes) are basically flat with the preceding years, but the 2020 and 2021 values seem improbable, and even more so the low projected time (represented by the triangle) that takes into account the super-shoe effect as well as the simple average decrease in times.
Given the historical evidence, it is very difficult to discern a fundamental shift in men’s distance running times in the five years following the adoption of super-shoes.
Women’s 10,000 Meter & 10K Road Races
The overall women’s race times show greater improvement than the men’s times. In the 10,000 Meters the slowest year was 2010 and the average improvement in the six years following was nearly 7 seconds per year (compared to 1.6 seconds for the men), with a 2021 result of 31:22 which (like the Men’s 10,000 Meters) is nearly identical to the trend line prediction. The asterisked values are somewhat consistent with the natural variability of the times stretching back to 2001, but the performances expected by a 2% increase due to a change of shoes (as represented by the triangle) are clearly not there. Perhaps super-shoes are less advantageous on the responsive smooth surface of a track.
Despite being the same distance, the trajectory of the women’s Road 10K race results is rather different. The average improvement since the slowest year in 2004 to 2016 was 2 seconds per year, and the average times in 2019 and 2021 were well below what the trend line or simple average would predict. The large spike of nearly a minute in 2020 is almost certainly the result of 10K races being cancelled during the height of the pandemic. The 2019 result is of particular note because the adoption of super-shoes was not complete and yet was in line with the type of performances predicted by the technology. In addition, the estimate of 59% super-shoe wearers in the 2019 10,000 meter events may be a little high since there is a tendency to use more minimalist footwear in these shorter events. The 2021 result of 31:41 is not quite down to the projected super-shoe figure of 31:32, but it is close and we must consider that shoe technology may have played a role in this instance.
Women’s Half Marathon
The Women’s Half Marathon times improved on average by a whopping 13 seconds per year from 2004 through 2016, with impressive times in 2017-2019, and returning to times typical with the trend line in 2020 and 2021. The average time observed for the Half in 2021 was 1:08:05 vs. the projected super-shoe estimate of 1:07:23. The times of the last five years can probably be better explained simply by pre-existing trends, but shoes might also have played a small role.
Women’s Marathon
In the Women’s Marathon, an average decrease in times was observed to be 15 seconds per year through 2016. The final result in 2021 was 2:23:44, which is very nearly identical to the result predicted by both the simple average decrease and the trend line. The year 2019, was again, a standout year in women’s marathon, but the times in 2020 and (more importantly) 2021 did not continue the trend, and the actual result last year was two minutes off the time that would be predicted by 92% of women runners wearing super-shoes.
Discussion
It has been demonstrated in multiple studies that carbon fiber plates paired with thick lightweight foam in precisely designed super-shoes can improve the metabolic efficiency of runners measured in the laboratory. Thus, it would seem the greatest benefit should be in the marathon event, where successfully managing energy deficits are a crucial aspect of the race. The 10K Road race on the other hand, does not deplete the body’s energy stores anywhere near the same extent, and yet there were improvements to the women’s times that indicated super-shoes may have provided an advantage. Given all the studies and statements pointing to a universal improvement in running economy due to the shoes, it would be very odd for the shoe to show its true colors in only one or two events, and in only one set of runners. If there were an effect of the magnitude described, it should show up without ambiguity in all long distance races for all participants.
So what is really going on; how can all these studies and experts be wrong? Some possible answers:
1. With respect to the women’s 10K Road race and banner year of 2019 in general, prime conditions existed for a placebo effect to manifest itself in the performance results. By that time, it was considered a known “fact” among runners that super-shoes worked, and crucially, a large number of runners were still not wearing them. Researcher Chris Beedie suggests it would not be surprising if the performance advantage from a placebo was in the 2 to 3 percent range, for example. If true, when nearly all runners are seen to be wearing super-shoes, the placebo effect should fade away since each competitor would “realize” they no longer had an advantage, which seems evident in the 2021 results (except perhaps the Women’s Road 10K).
2. One can surmise that with respect to studies comparing runners’ times with and without super-shoes, they may have failed to properly account for the tendency of runners to optimize their choices and select the highest-performing shoes when they were more likely to be at the top of their form.
3. With respect to experts’ false interpretation of contemporary fast times, it appears nearly everyone has failed to take into account just how large the improvements have been over the last 20 years. Times are fast, of that there can be no doubt, and may be shocking to someone who grew accustomed to times from a slower era.
4. Another possibility is that super-shoes do increase running economy (which does seem fairly certain), and do have a positive effect on performance for average and even outstanding runners, but not the elite-of-the-elite top 100 runners of the world. Perhaps they, as a function of their training and natural gifts, have a surplus amount of “energy to burn” whereby improved running economy does not aid them in a significant way. If true, this would be the best of all possible outcomes, since it would allow amateur runners the thrill of better times and perhaps fewer injuries through the use of super-shoes, while at the same time providing assurance that the times of today’s elites can still be meaningfully compared to past athletes.
Uncertainties
There remain some uncertainties which could lead to this analysis being invalid, beyond those already mentioned.
It could be the performance increase of super-shoes in real world conditions is much lower than 2%, although of course that begs the question whether such an improvement is then no longer objectionable.
Another is the possibility the calculations used to generate the graphs and comparisons were incorrect in some way. Replication of these results, preferably in a program other than Excel, would be welcome.
It might be true the race courses or conditions during the last five years have made them slower than usual, thus masking the effects of super-shoes.
Another significant possibility is that the number of races in 2021 was still well below the number of races organized in a typical year, thus shrinking the pool of times and their average.
Further Study
One might expect a similar analysis of so-called “super-spikes” would show little linkage between improved times in shorter track events and shoe technology. However to perform such an analysis requires an estimate of super-spike prevalence during 2017 to the present, which seems currently lacking, as well as a laboratory-informed baseline of performance improvement for such shoes.
It would also be very interesting to compare these results of the world’s top-100 runners with more local results. Perhaps at more regional levels, with fewer world-class competitors, the effects of super-shoes would be more pronounced.
