CT FEAT: Families Helping Children Achieve Their Full Potential
  Print This Article    

How Can We Judge Autism Treatment Claims?

By Richard Irwin (CTFEAT member)

Thus far there is no one universally accepted and recommended treatment for autism. That is not to say that all treatments are equally effective. Deciding which treatment is appropriate for your child can be an exceedingly difficult and stressful choice. The first question that must be answered before making this choice is: What standard should we use when evaluating treatments for autism? A treatment can only be deemed effective if it is based on sound, scientifically validated principles and supported by empirical data. In simple terms this means that treatments for autism must be backed by the same quality of research that we demand from other fields of science, such as medicine, chemistry, and engineering.

What questions should we ask when evaluating treatment claims?
Unfortunately, parents are most often left on their own to read the research literature and draw their own conclusions as to the effectiveness of a particular treatment. The best allies a parent has in their search for a treatment for their child are a healthy dose of skepticism and a willingness to ask questions. The following is a short list of questions that should be answered when evaluating claims about a particular treatment. For a comprehensive guide to evaluating treatment claims, the reader is referred to the chapter "Evaluating Claims about Treatments for Autism" in Behavioral Intervention for Young Children with Autism, edited by C. Maurice, G. Green and S. Luce, (Pro-Ed, 1996).

What were the goals of the treatment program?
Children with autism exhibit cognitive, social, behavioral and communicative deficits. A treatment program designed to focus on only one or two narrowly defined areas shouldn’t be considered sufficient. For instance, the goal of a proposed treatment program may be only "to increase the use of language." However, even if language usage improves, that would not necessarily remediate the other core deficits typically associated with autism. A child can make tremendous gains in his speech and still exhibit severe behavioral, social and cognitive problems. A treatment program should address all the core deficits found in children with autism.

Some programs appear to subscribe to the philosophy that "the best we can do is manage the child’s maladaptive behaviors." Such programs often set up very structured environments and routines to minimize troublesome behaviors, but accomplish little else. These programs underestimate the enormous gains in every domain that many children can make with effective intervention.

Were there specific measurements made pre- and post-treatment?
Very often, words such as "good to outstanding" or "significant" are used when describing the gains made by autistic children in a treatment program. While these descriptors sound good, in reality they tell us nothing about the effectiveness of the treatment unless they are defined in advance and based on data derived from specific measurements. There are currently many accepted standardized tests that can be used to measure the progress of autistic children in a treatment program. Some examples are the Intelligence Quotient (IQ) test, the Vineland Adaptive Behavior Scale, the Childhood Autism Rating Scale (CARS), the Peabody Picture Vocabulary Scale and the Personality Inventory for Children. These tests are used to measure such things as cognitive function, language development, adaptive behavior and social skills. As children with autism spectrum disorders exhibit a broad range of deficits, a combination of these tests should be used to establish base lines and document progress. A meaningful determination of progress can only be made if these tests are administered before and after treatment, and the results carefully compared.

One common but misleading measure used to demonstrate the effectiveness of a treatment program is the percentage of its children that transition to "regular school placements." But this measure often has very little, if any, significance. A child meeting this measure of "success" could require extensive supports to function in a regular school placement - or none at all. The child's "regular school placement" could be due more to local school policies - such as mandated "full inclusion" - than to the child 's progress.

Who made these measurements and were they potentially biased?
Tests such as those mentioned above should only be administered by well trained, experienced professionals. Steps should be taken to minimize the possibility that the data collected is biased by the desires of the investigator to prove treatment effectiveness. One method of eliminating investigator bias is to have those administering the tests and collecting the data be completely independent of those administering the treatment. Those performing the tests should have no knowledge of the treatment program or its predicted outcomes. This is referred to in the literature as a "double blind" model. Some measures, such as parent testimonials or satisfaction ratings, can be biased and should only be one part of a comprehensive evaluation of a program. It is also wise to be skeptical of professionals designing, researching and evaluating treatment studies outside their area of expertise. For instance, we wouldn’t want a podiatrist developing and reporting on a new dental procedure.

Was the progress made due to the treatment?
While this seems like a silly question, it is necessary to demonstrate that the progress made is due to the treatment and would not have occurred in the absence of the treatment. One method used is to collect data on two identical (all test measures equivalent prior to treatment) groups of children, one group receiving the treatment in question and the second group not receiving the treatment. Differences in progress between the two groups can then be attributed to the treatment under study. There also must be controls to ensure that no other plausible explanation for the results exists. No matter the approach used, there should be some explanation of the experimental controls used and why they were appropriate. One type of study of a treatment’s effectiveness that is fairly common in the autism literature is the "chart" review. A chart review is a summary of the data and clinical observations taken from patient or student records. In general, a chart review does not contain experimental controls and should be viewed with skepticism. Usually chart reviews are used to identify potential trends that can be investigated experimentally.

Were the gains permanent?
A treatment can only be considered effective if the gains made are long lasting and permanent. The best method to determine this is to perform follow-up testing, employing the same measures used in the treatment study, at intervals substantially after the discontinuation of the treatment.

Were the results of the study reported in a reputable scientific journal?
The vast majority of reputable scientific journals are peer-reviewed. This means that before an article is published, the article is reviewed and commented on by other experts in the field. While this by no means guarantees the validity of the data reported, it does provide some measure of scientific scrutiny on the quality of the science published, the interpretation of the data and the claims made. Articles published in newsletters, flyers or books are generally not peer-reviewed, and so the potential for inflated or false claims is greater. Practically every day we see flyers that make ridiculous claims about weight loss procedures or hair growth. We do not accept these at face value and the same skepticism should be applied to treatment claims for autism.

Have the results been reproduced?
All scientific data should be viewed with skepticism until it has been reproduced by independent investigators. As an example of why this should be the case, we only have to consider recent history. Some years ago, two scientists published a report claiming that they had achieved cold fusion - essentially the ability to produce nuclear energy from simple water. This energy source would be clean, safe and inexhaustible. The news was so exciting that the nation and worldwide media reported on it immediately. However, when other scientists attempted to reproduce the original findings, they found they couldn’t. There had been errors made in the original study and it turned out that they had not actually achieved cold fusion. We do need to be aware that studies in the field of autism are generally more difficult to reproduce than are those in some other fields of study. Besides requiring an ample number of autistic children and the potential ethical questions involved, these studies require enormous resources, time and trained personnel to complete.

Where can I turn for help?
The families of CT FEAT have each gone through the process of choosing a treatment program for their children. If you would like to learn from their experiences, please feel free to contact CT FEAT by phone at 860-571-3888 or by E-mail at ctfeat@ctfeat.org.

Back to Top


CT FEAT, P.O. Box 7063 Prospect, CT 06712-0063
Copyright 2022
CT FEAT    Contact Us