Tuesday, March 24, 2009

When simply ignoring the evidence isn't good enough

Earlier this month there was a study published in BMC Pediatrics called "Hyperbaric treatment for children with autism: a multicenter, randomized, double-blind, controlled trial". This study looked at whether Hyperbaric oxygen therapy (HBOT) can help treat autism.  

The short answer is that it seems possible that it can.

The study is a definite improvement from earlier studies as it a multicenter, randomized, double-blind, placebo controlled trial.  However, it still isn't perfect and there are potential issues.  And most importantly, it needs to be replicated by other groups.

There has been much written about the study itself, so I am not going to go rehash what has been covered elsewhere.  Rather I wanted to write about the reaction to the study, specifically the analysis that appeared on Left Brain Right Brain in the post Autism, HBOT, and the new study by Rossignol et al.

In this post the blogger DoC attempts to argue that the pressure achieved at one of the study locations (Center for Autism Research and Education in Phoenix, Arizona) was not the full 1.3 ATA as was stated in the study but rather might only have been 1.24 ATA.  If you want the full detail about the math go read the original post but the thing that struck me is that when you look at the actual figures used DoC does not have accurate data for his criticism.  Specifically he is missing a key data  point about what the actual air pressure is at the study location.  If his assumed figure was off even 0.6% then his conclusion is incorrect.

I pointed out as much in the comments and he conceded the point while at the same time saying that the study had the same problem. 

Call me silly, but I am thinking that the people at the center in Phoenix would be aware of how to use their equipment properly to achieve the results desired.  And according to the information provided by DoC about the equipment is rated by the manufacturer for the 1.3 ATA quoted.  

But set that aside for a minute and assume that DoC is correct and that the pressure wasn't as high as as reported; that it only got up to 120% instead of 130% of sea level pressure.  What would that mean?  To my mind it would only make the study stronger since the benefits of HBOT were achieved with a lower pressure than was stated.  So if that is the worst problem with the study then that isn't too bad.  

But wait, there's more.  

If you look at the comments you come across some, ahem, interesting additional theories about where the study went wrong.  

Lets start with a comment from "A Proud Father":
I’m sorry, but all you have to do is look at the authors of this study: Rossignol, Neubrander, Mumper. Names just loaded with credibility. I’ll read such a study when it comes from respected members of the research community.
Ok, I guess the whole peer-reviewed journal doesn't mean too much when you instantly know that the authors faked the whole study.  Good catch there, good thing you weren't too proud to say anything.

Joseph suggests that:
Does Dr. Rossignol in particular or HBOT proponents in general recommend that children who live at high altitude should move to the coast? That would seem to be a straightforward conclusion of his hypothesis if it is true.
Showing that he is completely missing the point of increased pressure (1 ATA = sea level).  Perhaps it would have been better to suggest that the families go live underwater since that is where you found find pressure greater than that of sea level.  Or maybe they should dig a really big hole in the ground and live in that...

Continuing on Author says that: 
Despite the blinding attempt, it would have been trivial for someone inside the inflated “nylon” chamber to guess wether (sic) it was inflated at 1.03 or 1.3 ATM, by touching the side of the chamber.
I have to wonder if he has ever been inside a chamber and tried the theory out for himself.  I think not.

Joesph jumps back in with :
The statistics of the study also seem unconvincing. There’s only 1 statistically significant between-groups effect among 9 measures taken using ATEC and ABC.
And finally Sullivan jumps in with the knock-out punch:
The diagnostic instruments (if I can call them that) are inadequate for drawing any real conclusions.
So there you have it, the study has been demolished because of a lack of specified data, the authors are crooks, the chamber gave away the the pressure, the statistics were bad, and even if all the rest were fine the diagnostic tests weren't valid.  Therefore there is nothing to the study and no possible improvements from HBOT.

If only the study's authors had not relied on their own professional experience and had asked this group of people they could have been spared the time and effort of actually performing the study.

(In case you couldn't guess, that was sarcasm)

The only point I keep wondering about is why a group of people who claim that they believe in hard cold facts feel the need to pontificate and come up with wild theories in an attempt to discredit a study.  

Sounds to me like they are afraid of something.

6 comments:

  1. Excellent comment. I enjoy reading Autism Street regularly for the amusement value. DoC and Sully routinely pose as experts on any number of technical subjects. Joseph is a self proclaimed expert on everything period. Hilarious stufff.

    ReplyDelete
  2. Thanks. It never ceases to amaze me how a group of people who claim that they believe only in evidence go to great extremes to discredit anything that don't agree with their philosophy.

    ReplyDelete
  3. "The short answer is that it seems possible that it can."

    Hi MJ. I'll concede this point as well. It's absolutely *possible* that it can. Now, please explain for your readers why they shouldn't pursue simple O2 therapy (at 10% of the cost of pseudo-HBOT in an inflatable chamber as done in this study) instead. Simple O2 therapy can provide identical increased oxygen tension. The laws of physics that explain partial pressures do not know the difference. Why should parents pay $100 per "treatment" for something that costs 10$ and would be easier to do for most people?

    ReplyDelete
  4. DoC - I think you may misunderstand what I was trying to say.

    My post was about the analysis of the study that happened at LBRB.

    My post was not advocating for running out and treating anyone with HBOT. I think this is still a very experimental treatment that should only be tried under appropriate medical supervision.

    It is entirely possible as you say that simple O2 therapy can have many of the same benefits. I thought there was something about the high pressure that changed how the body could process the higher level of O2 but I haven't looked into the subject enough to have an educated opinion.

    So you could very easily be correct in that simple O2 can have many of the same benefits at a fraction of the cost.

    ReplyDelete
  5. "My post was about the analysis of the study that happened at LBRB."

    "Call me silly, but I am thinking that the people at the center in Phoenix would be aware of how to use their equipment properly to achieve the results desired."


    Then you should have understood why they couldn't have achieved the pressure claimed in the study in Phoenix. It's not rocket science, and you were paying attention to the math! What happened to your ability to understand what happens after the "equals" sign?

    "And according to the information provided by DoC about the equipment is rated by the manufacturer for the 1.3 ATA quoted."

    Really? Go back to the document and read about operating pressure - but not in the rounded parentheses. Why not include the FDA link for your readers? Or the phoenix doctor's website information on treatment pressure?

    "To my mind it would only make the study stronger since the benefits of HBOT were achieved with a lower pressure than was stated. So if that is the worst problem with the study then that isn't too bad."

    Perhaps you haven't considered the alternative explanation - that the results of this study are placebo effect by proxy, and the reason the pressure may not matter is that it is irrelevant.

    "If only the study's authors had not relied on their own professional experience and had asked this group of people they could have been spared the time and effort of actually performing the study."

    Professional experience?

    Which of the authors of this “professional” HBOT for autism study are trained, and board-certified in:

    A: Developmental-Behavioral Pediatrics
    B: Undersea and Hyperbaric Medicine
    C: Both of the above

    Go ahead, while you're appealing to "profession experience", just substantiate theirs by listing the answers to A, B, and C right here.

    I am glad that you clearly understood that the actual total treatment pressure was critically dependent upon the ambient atmospheric pressure, though. That very basic and painfully simple point was apparently lost on others - good on you!

    Thanks for taking my comments.

    ReplyDelete
  6. "Then you should have understood why they couldn't have achieved the pressure claimed in the study in Phoenix. It's not rocket science, and you were paying attention to the math! What happened to your ability to understand what happens after the "equals" sign?"

    As I said before, you lack the data to make that statement. You are taking a proxy for a variable that you simply do not have. And so I would defer to the people who have set up the room and are running the equipment since they are the ones who have the specific knowledge of the conditions.

    So go talk to them and see if you can find your missing piece of information.

    "Really? Go back to the document and read about operating pressure - but not in the rounded parentheses."

    Look at the first page of the product document - look at the upper right corner, see the little 1.3 there? Now that may very well be a rounded number and only apply under some conditions but the device is being marketed as 1.3. Are you suggesting that the manufacturer is advertising their product as something it is not?

    "Why not include the FDA link for your readers? Or the phoenix doctor's website information on treatment pressure?"

    Because I assumed that anyone who was interested would click through to LBRB and read the discussion and links for themselves.

    "Perhaps you haven't considered the alternative explanation - that the results of this study are placebo effect by proxy, and the reason the pressure may not matter is that it is irrelevant."

    That is possible but not really relevant to the point at hand. The study measured HBOT not simple O2. So while it is possible as you suggested earlier that O2 would have the same effect that is completely outside the scope of your initial criticism.

    If that was meant to be part of what you were saying then perhaps you should have actually said that?

    "Which of the authors of this “professional” HBOT for autism study are trained, and board-certified in:"

    Which commenter on LBRB is trained in any of these areas? Anyone over there even an MD?

    Furthermore since the authors managed to get the study published in a peer reviewed journal I am assuming that it was in fact peer reviewed.

    Think any of the comments from LBRB would pass the same muster?

    You still seem to be missing the main point here - the analysis of the study by the people commenting at LBRB was absurd.

    You were the only one that provided any sort of real information to back up what you were saying and even what you said was flawed because you simply lack the appropriate data.

    The rest of the comments were just silly and pointless and had nothing to do with any real criticism of the study and everything to do with attempting to discredit it in their minds.

    This was the point I was attempting to get across - that a group of people who pride themselves on an evidence only approach don't really follow that approach.

    It may have started out that way but it has become a poster child of group think and how to dismiss any idea that doesn't fit into the established dogma. It has turned into something almost like a religion.

    ReplyDelete