Wednesday, November 17, 2010

Jabberwocky of the Day : Imaginary Data

I feel as if I have said this before, but science is all about data.  First you come up with an idea, then you go collect data, and finally you analyze the data to see if it supports your idea.

This process is formally known as the scientific method and, for the most part, it works well.  Sure, you will occasionally run across something where the researcher seems to have started out with the conclusion and, working backwards, selected only the data that fit the conclusion, but that doesn't happen all that often.

But, every once in a while, you will run across a paper where the researchers seemed to feel that the whole collecting data thing is overrated and went straight to the conclusion without bothering with any data.  I ran across an example of this last type recently and couldn't help but point it out.

The paper in question is "Prevalence of Pervasive Developmental Disorders Among Children at the English Montreal School Board" (open access, go read it) and it dealt primarily with measuring autism prevalence in a group of schools in Canada.  Overall, the paper wasn't bad and it showed what almost every recent study of autism prevalence has shown - the prevalence of autism increasing every year because of reasons unknown.  This part of the paper had real data behind in in the form of school records.

But then, there is this second part to the paper and that is where things get a little, well, strange.  Lets go through it from the top.

In the abstract of the study, you will find this objective listed -
Our objectives were to determine prevalence rates of PDDs among school-aged children, and to evaluate the impact of discontinuation of thimerosal use in 1996 in routine childhood vaccines on PDD rates.
Pay attention to that second part (yes, sorry, vaccines again), and what it implies.  The goal is to measure what impact - if any - the removal of thimerosal had on autism prevalence. To do determine that, you need some measure of exposure to thimerosal.

With that in mind, look at the "Conclusion" section of the abstract -
Our study provides additional evidence that the PDD rate is close to 1%. We estimate that at least 11 500 Canadian children aged 2 to 5 years suffer from a PDD. The reasons for the upward trend in prevalence could not be determined with our methods. Discontinuation of thimerosal use in vaccines did not modify the risk of PDD.
And, in the main body of the study, look at the "Interpretation" section -
As in other studies where the effect of the discontinuation of thimerosal in childhood vaccines was examined, no change in the underlying population trends for PDD rates could be observed in relation to thimerosal discontinuation.
I think it is obvious that the authors feel that their data shows that removing thimerosal didn't impact the rate of autism.  So, clearly, the paper should have measure of thimerosal exposure, right?

Wrong.  Look in the section of the paper entitled "Exposure to TCVs" -
Individual immunization data were not available for study subjects.
Wait, data about the actual exposure to vaccines and thus the exposure to thimerosal wasn't used?  Then how did the authors support their conclusion?
In Quebec, thimerosal was removed from vaccines used as part of the recommended childhood vaccine schedule in 1996. In previous years, exposure to thimerosal varied from 125 to 200 micrograms for birth cohorts included in our study.
Oh. So every child born before 1996 was assumed to have exposure to thimerosal but every child after that was assumed to have no exposure?
The prevalence in each individual birth cohort born in or after 1996 was consistently higher than that in cohorts born prior to 1996 (Table 1). ... Further, the inclusion of a dummy predictor variable indicating exposure or not to thimerosal (before or after 1996) to the model predicting prevalence with birth cohort did not improve the model and was not significant.
OK, no thimerosal data, just a "dummy predictor variable".  Hmm.  If thimerosal was completely removed from all vaccines in Quebec, then what should we make of this "Statement on Thimerosal" from the "National Advisory Committee on Immunization" in 2003 -
In Canada, the vaccines currently used in routine infant immunization do not contain thimerosal (see Table 1). Some hepatitis B vaccines licensed in Canada do, but one formulation with no thimerosal and another with only trace amounts are now available in Canada, and NACI recommends their use in infants preferentially. The two hepatitis B vaccines in which thimerosal is added as a preservative are gradually being phased out. Influenza vaccine also contains thimerosal but is only recommended for use in Canada for those infants > 6 months of age. The other vaccines licensed in Canada that contain thimerosal are primarily used for people travelling to developing countries and are not routinely administered to infants.
So, thimerosal was completely gone in 1996 - except that in 2003 it was still in some hepatitis B vaccines, the flu vaccines, and "other vaccines"?  I guess it is possible that Quebec did remove it completely from all vaccines in 1996 and it is only the rest of Canada that didn't.

But, even if that were the case, you would still have people moving from other parts of Canada or from other countries that would still have had exposure to thimerosal, so the "dummy" variable with the arbitrary cut-off date has no real relation to actual exposure.  And if there is no measure of exposure, how can the researchers conclude that removing that exposure had no effect?

Note to the researchers - next time actually have data to support your conclusion.

9 comments:

  1. This study is often held up as solid research against a connection between thimerosal and autism. Oh well.

    ReplyDelete
  2. Minority,

    You may be confusing this study with another one. This one was just published this week.

    ReplyDelete
  3. Knowing the lay of the land matters. I live in Ontario, Canada. That's where the flu vaccine is free to everyone over 6 months of age, but they only routinely vaccinate against Hepatitis B in Grade 7. Yes, almost everywhere in the world they vaccinate all kids at birth (rare) or at the time of the DTaP vaccination (6j-10 weeks).

    Guess what Quebec's policy? I know what it was in 2008-2009. See
    http://www.cmaj.ca/cgi/content/full/180/2/196 and I'm sure it was the same or stricter in the late 1990s.

    As to influenza, I don't the story, but my sense is that uptake in kids was pretty low at that time. In addition, it only started at 6 months of age.

    You could check with authors for more details on influenza.

    Kinda takes the wind out of your sail, doesn't it?

    www.vaccineswork.blogspot.com

    ReplyDelete
  4. Sheldon,

    If I an not mistaken, Quebec has a substantial immigrant population. That would mean a good sized part of the population would not have followed the same schedule.

    However, I did assume that Quebec gave the Heb B at the same time (infant) as most of the world and you are most likely right about the uptake of the flu shot.

    But, it doesn't really matter either way because there is no actual measurement or even an attempt to measure and that is the problem. Nor is there an attempt to control for confounders like immigration status.

    I found this particular study's lack of actual data to be troubling. You know that someone, somewhere will cite it as evidence that thimerosal plays no role in autism and that statement simply is not supported by the data in this study.

    And yes, I do know that vaccines work and no, I don't think that all cases of autism are caused by vaccines.

    ReplyDelete
  5. I'm thinking we've already had hep B once before... I'd have to double check but it's in the basement. The one in Gr 7 would be the booster. It's amazing how the anti-vax group in the years I've been online have gone from MMR vaccine - bad... now it's thermisol - bad and... Had I known then what I know now... I'd probably do what I've done. No flu vax, basic shots yes, no chicken pox - caught that and meningitus yes.

    Actually, I have little respect for science over the years. The quality is decreasing. The use of social science and venn diagrams is NOT science. When autism parents lobbying for services talk about "best practices" and list the same types of poorly designed studies which have no controls... if they list any at all.... I simply keep doing what I've always done... Ignored it all, made my own decisions and fight for services we require.

    ReplyDelete
  6. Thanks MJ, I was thinking of an earlier study. But it was on the same population.

    ReplyDelete
  7. Sheldon, you just don't get it. The article didn't collect any accurate data on whether the group of children in question actually got TCV or didn't get TCV.

    Without data the article doesn't make sense.

    Which leads to the next question: Why do articles that defend vaccines get a pass through peer review? This is such an obvious bit of nonsense...

    ReplyDelete
  8. Hepatitis B vaccination is turning out be as effective as hoped so that there doesn't seem to be a need for a booster.

    As to immigration to Quebec mattering, you would have to look at the incoming versus outcoming numbers and the sources of immigration.

    Frankly, I'm not that impressed by the information on TCV in the study. But it is additional evidence that the thimerosal causes autism claim is as bogus as all the other studies have shown.

    The case control study recently published had very accurate information on the vaccines that those in the study received and there just wasn't a relationship between the amount of thimerosal and the frequency of ASD.

    As to present practice, the nice people at UC David Mind Institute are about as extreme as experts get when it comes to believing that autism can be caused/contributed to by events after the child is born. And even they recommend the current vaccination schedule with 2 exceptions. One, flu vaccination with thimerosal-free vaccine for pregnant women and infant and kids. Two, talk to your doctor if their is a strong family history of immune disorders or the kid has a history of immune disorders or gets persistent infections.

    And that's the extremists.

    ReplyDelete
  9. Sheldon,

    The immigration into Quebec seems to be substantial -

    http://www.canadavisa.com/about-quebec.html

    "Since the end of World War II, more than 650,000 immigrants from over 80 countries have moved to Québec, particularly to the multicultural city of Montreal."

    "With one of the lowest birth rates in Canada, nearly all of Quebec’s future growth can be expected to come from immigration. Today immigrants come to lives in Quebec from all over the world, with the majority settling in either Montreal or Quebec City. Many Canadian immigrants from francophone countries choose to settle in Quebec, notably from France and a sizable Haitian population."

    and

    http://www.micc.gouv.qc.ca/publications/fr/recherches-statistiques/FICHE_syn_an2009.pdf

    "In Quebec, the 2006 Census counted 851,560 immigrants, representing 11.5% of the Quebec's total population, the highest proportion ever in the history of province and this proportion was 9.9% in the 2001 census. This proportion is about 19.8% for Canada, 28.3% for Ontario and 27.5% in British Columbia"

    (The translation is via Google).

    I don't know how that would translate to children, but I suspect it is enough to skew the results.

    "Frankly, I'm not that impressed by the information on TCV in the study. But it is additional evidence that the thimerosal causes autism claim is as bogus as all the other studies have shown"

    While I would not argue with the thimerosal alone not causing autism point, I would disagree that this study adds any new information because it has no information to add. All it shows is what every other study has shown - an increasing rate of autism.

    "The case control study recently published had very accurate information on the vaccines that those in the study received and there just wasn't a relationship between the amount of thimerosal and the frequency of ASD."

    That study was well done, but the problem with it is that not everything shows a clear relationship between exposure amount and outcome (i.e. more thimerosal means higher rate of autism). There are some things that any exposure increases the risk of the outcome.

    Take smoking for example. It doesn't really matter if you smoke a half a pack a day or two packs a day - you have an increased risk of cancer. And more than that, the people who smoke two packs a day don't necessarily have a greater risk than those who smoke half a pack. The increased risk doesn't pop out as much until you compare smokers to non-smokers.

    Or take low level exposures to lead. Recent research has shown that any exposure to lead is harmful but no one has been able to really quantify the difference between, say, 1 ug/dl and 5 ug/dl. It is only when you get to higher exposures the difference become more pronouced.

    But, having said all of that, I think the idea that just thimerosal alone can autism has been put to rest - except perhaps in extremely rare cases ( less than .01%)

    "As to present practice, the nice people at UC David Mind Institute are about as extreme as experts get when it comes to believing that autism can be caused/contributed to by events after the child is born."

    Why do you think that autism isn't caused/contributed to by events after birth?

    "Two, talk to your doctor if their is a strong family history of immune disorders or the kid has a history of immune disorders or gets persistent infections."

    Heh, such as my children. They have shown abnormalities with their immune system for years. But we only found out about it after they were diagnosed. Before that we gave them all the shots the doctor recommended whenever they recommended them - even if they were sick at the time. Now we know that isn't a good idea and wait until they are feeling better first.

    ReplyDelete