Why crank journals are unreliable

By AnthonyLast updated: Thursday, May 29, 2008 • Save & Share5 Comments

This is a post which has sprung up because of a discussion at Melanie Phillips’ blog at The Spectator. It is probably longer than the subject matter really deserves, but illustrates an important point.

Recently a news article that daisy-chained a number of very weak vaccine-autism hypothesis stories together was published at The Daily Telegraph. This was picked up by Melanie Phillips, who said:

Good to see that the Telegraph today picks up on developments I wrote about here in the US, where a head of official steam is building behind the perception that there is a troubling relationship between certain childhood vaccines, including MMR, and autistic symptoms and other damage in a small subset of particularly vulnerable children.

The subsequent discussion lead to comments about a 2002 paper by Madsen et al. in the New England Journal of Medicine which showed a relative risk of autistic disorder in the group of vaccinated children, as compared with the unvaccinated group, of 0.92 (95 percent confidence interval, 0.68 to 1.24). In short, MMR vaccine was not associated with autism.

It was suggested by John Stone, a supporter and voice of JABS, that this study was incorrect, and that the Madsen study really showed a 45% increase in the risk of autism associated with MMR vaccine:

When the Madsen paper was published Prof Samy Suissa, an epidemiologist from McGill university, wrote to NEJM suggesting that correctly calculated, the vaccinated subjects, instead of being 8% less likely to be autistic, were in fact 45% more likely:


One of the tactics of members of the anti-MMR lobby is that attempt to twist studies which have failed to confirm a link between MMR vaccine and autism to show the exact opposite. For example, I have seen John Stone write in an anti-vaccine forum that the The Honda study, which showed that MMR vaccine withdrawal in Japan did not prevent autism cases continuing to rise, “tended to support Wakefield”.

The anti-vaccine movement feeds off a steady stream of poor quality research that either appears in posters at conferences not subjected to sufficient scientific scrutiny, in gray literature, or in dubious journals. The Journal of American Physicians and Surgeons (JPANDS) is the house journal of The Association of American Physicians and Surgeons (AAPS). The AAPS are a right wing fringe medical organisation in the US described by the New York Times as “ultra-right-wing… political-economic rather than medical” group. Their journal carries material denying that HIV is linked AIDS, that gay lifestyles shorten life by 20 years, and is pro-creationist. It is not listed in Medline/Pubmed or the World of Science. It does make the Quackwatch list of non-recommended periodicals which lists publications which are “untrustworthy because they promote misinformation, espouse unscientific theories, contain unsubstantiated advice, are insufficently skeptical, and/or fail to distinguish between reliable and unreliable sources of advice.”

However, this is not enough for John Stone, back at Melanie Phillips’s blog, who retorts:

You introduce a further red-herring. Irrespective of the controversial status of JPandS, Prof Suissa allowed his letter to be re-produced there. Are you disputing the letter?

Stone is obviously correct. Arguments can be right, even if produced in a biased and unreliable house journal of a right wing political group. Prof Suissa’s letter is reproduced in a commentary by Andrew Wakefield, Mark Blaxill, and Carol “Try me, shit head” Stott. They put great store in the Prof Suissa letter, which they note the New England Journal of Medicine declined to publish. Here is his letter:

Madsen et al. observed an adjusted rate ratio of autistic disorder after vaccination of 0.92 relative to no vaccination, when the crude rate ratio (my computation) was 1.45 (95% confidence interval 1.08-1.95). Moreover, the rate by time since vaccination increases to a high of 27.3 two years after vaccination (rate ratio 2.5) and decreases thereafter to 11.4 per 100,000 per year (Figure 1). It is stated that adjustment for age eliminated these rate increases, but the corresponding data are unusual. Indeed, the rates of autistic disorder by age at vaccination, although not the age at follow-up, are 18.9, 14.8, 24.6, 26.9 and 12.0 per 100,000 per year respectively for ages <15, 15-19, 20-24, 25-35 and >35 months. These rates are all above the overall rate of 11.0 for the reference group of no vaccination, over all ages. It is then somewhat implausible for the adjusted rate ratio to fall below 1, unless the risk profile by age in the unvaccinated is vastly different than in the vaccinated (effect-modification). In this case, the adjustment for age could have been artificial. It would be useful then to present rates on subjects 24-29 months since vaccination and on the unvaccinated (crude rate ratio 2.5) stratified by age. Otherwise, one could be tempted to conclude that the figure is in fact suggestive of an association between MMR vaccination and the risk of autism.

From this, Wakefield, Blaxhill and Stott conclude that “The data of Madsen et al., unadjusted for age, support an autism-MMR association.” Yes, that’s the exact opposite of what the Madsen paper said.

I do not profess to be an expert in epidemiology, so I passed the Suissa letter over to Epiwonk, who has kindly allowed me to reproduce his views on Suissa’s letter here.

There’s nothing wrong with the Madsen paper that I can see. It’s easy for me to figure out what Suissa did. In Figure 2 of the Madsen paper he divided 263/1,647,504 by 53/482,360 to get an unadjusted relative risk of 1.45 (or autism “45% more likely.”) In other words, a a relative risk unadjusted for the confounding effect of age. Suissa then goes on to argue that it’s “somewhat implausible for the adjusted rate ratio to fall below 1, unless the risk profile by age in the unvaccinated group is vastly different than in the vaccinated (effect-modification).” Well, the reason for adjusting for age in the first place was because the risk distribution of unvaccinated children is much younger than that of vaccinated children — this is confounding, not “effect modification.” The rest of Suissa’s argument has the same problem, except it’s compounded by (1) his misunderstanding that you can’t calculate “rates per 100,000 per year” from the Madsen study — Madsen calculated rates per person time, which is what the Young-Geier study should have done. (2) in the Madsen study the n of autism cases vaccinated >20 months age is just too few to quibble about: 30 out of 316.

So there may be a very good reason the New England Journal of Medicine declined to publish Suissa’s letter. It is wrong. That is why it appears in a right-wing crank house journal under the names of three prominent anti-MMR campaigners with an axe to grind. That is why those who rely on papers published in JPANDS, like Melanie Phillips, to support their views about MMR vaccine are exposing their lack of credibility in the debate.

5 Responses to “Why crank journals are unreliable”

Comment from TheProbe
Time 29/5/2008 at 3:01 pm

Kathleen Seidel wrote an extensive article at Neurodiversity.com showing exactly what JPANDS and the The Association of American Physicians and Surgeons are NOT a medical journal nor a medical group.

Comment from Anthony
Time 29/5/2008 at 3:14 pm

Yes, it is rather good.

Comment from dvnutrix
Time 30/5/2008 at 12:42 am

So there may be a very good reason the New England Journal of Medicine declined to publish Suissa’s letter. It is wrong.

It is remarkable that so many people default to accusations of conspiracy before considering the possibility/probability that the criticism is wrong. (Shades of Holford feeling forced to doubt the integrity of the BMJ and the authors of the omega 3 review, sooner than consider that he might be wrong in his interpretation of the blobbogram.)

It is remarkably useful and timely to have this fine rebuttal of the attack on the Madsen findings. I hope that I’m wrong, but I’m apprehensive that Kirby might enjoy some more credulous coverage in the UK press over the next 10 days or so.

Comment from PJ
Time 30/5/2008 at 8:50 am

Bah, that list doesn’t even include Medical Hypotheses (and that _is_ indexed by medline bizarrely).

Comment from Dr Aust
Time 31/5/2008 at 12:32 am

Nice work, Anthony and EpiWonk.

Hadn’t see the Quackwatch List of non-journal journals before. Delighted to see that “Cargo Cult Classic” the Journal of Alternative and Complementary Therapy there – richly deserved.

Many of the other Medline-listed Journals of Alt Nonsense should be there too, IMHO.

Re Mad Mel P, I strongly suspect she gets all her MMR info straight from John Stone and the JABS crew. I find I have to turn off Radio 4 if Mel is on the The Moral Maze – otherwise I find myself uncontrollably grinding my teeth .