In the first of a two part feature, Peter York begins by questioning why randomized controlled trials are the gold standard of impact measurement.
The Holy Grail for anyone who develops, invests in, funds and/or runs a social program is to prove that their intervention creates a positive change for all who are served. The randomized controlled trial (RCT) is the industry gold standard amongst all evaluation methods currently used in the social sector. In this, the first of a two part feature, I will be looking into the validity of this level of standard.
The experimental design method is used in the rapidly-scaling world of pay-for-success and social impact bond markets, while also maintaining its top-level status amongst government and foundation grantmakers. When RCTs cannot be applied, the next-level or “silver” standard is the non-randomized controlled trial – an evaluation design that mimics the rigor of random assignment by applying some form of comparison group matching. “Controlled study findings are not applicable in the “real world.”
The problems with controlled trials have been documented and communicated by many leaders in the social sector, for many years. The most concerning problems regarding controlled trials are that they:
- Require a large financial and human resource investment, for over a long period of time;
- Require an unethical and/or infeasible adherence to the research standard of people in the control or comparison group NOT contaminating the experiment by seeking similar treatments, elsewhere – also known as, treatment “diffusion” or “imitation”;
- Because of this lack of “social” control over the comparison group’s independent problem solving efforts, these studies rarely prove that the intervention group achieved “significantly” greater levels of outcomes, and when they do, effect sizes are often small; and
- When a positive finding is found, the intervention cannot be applied elsewhere because change agents in the “real world” cannot precisely adhere to implementing the program exactly as provided during the experimental condition.
RCT’s top status has been challenged by medicine for over a decade, so why does social sector still consider it The Gold Standard?
Controlled study findings are not applicable in the “real world.”
The primary reason we haven’t replaced the controlled study as the gold standard is the belief that any and all next-best methods compromise too much when it comes to rigor and validity. We borrowed our gold standard method from the field of medicine. So, does the whole field of medicine still believe that controlled trials are best? Nope! Back in 2000, the New England Journal of Medicine published an article where the authors conducted a comprehensive review of numerous studies, comparing the validity, effects and findings of RCTs versus non-experimental “observational studies.” The finding, according to the authors, was a definitive, “one method does not give a consistently greater effect than the other.” And, this is not the only study, report, article or expert saying the same thing.
RCT’s top status has been challenged by medicine for over a decade, so why does social sector still consider it The Gold Standard?
If RCTs are NOT the only best method for evaluating much more “controllable” and way more risky medical treatments, why on earth should controlled studies be the gold standard for evaluating far less “controllable” and somewhat less risky social treatments? In an attempt to delve deeper into this question, in my follow up piece to this article, I shall be proposing what I believe to be a genuine alternative to RCTs.
Many thanks to Peter York for his examination into randomised controlled trials and the value we place on them. We look forward to his follow up piece on his alternative to RCTs, scheduled to be published next week.
To stay up to date with the latest Markets For Good articles and news, sign up to our newsletter here. Make sure that you are also following us on Twitter.