How do we understand the world around us? As individuals, we have five senses that will reveal some truth about the world, and they serve us pretty well when we’re trying to survive. But how do you see an atom? Can you touch a burgeoning social movement? Can you hear the migratory behavior of the humpback whale? (One guesses that the answer is no.) To gain a deeper understanding of the world around us, we have to gather data that lie beyond what we can directly observe.
The origins of scientific inquiry
In 1747, James Lind conducted what is widely considered to be the first medical clinical trial in his quest to treat scurvy. He gave 12 of his scurvy patients (“Their cases…as similar as I could have them”) differing treatments over the same time period and found that the ones who were fed a steady diet of oranges and limes improved rapidly. This comparison of a group that receives a special treatment (diet of oranges and limes) against a group that receives the standard diet hinted at the rough structure of what we today call a Randomized Controlled Trial (RCT).
The modern era
Since then, RCTs have become the “gold standard” for evidence across myriad domains. In the BETA Project, we conducted RCTs at two of our three partner sites, Accion Texas and Cleveland Housing Network . The modern RCT is more rigorous than Lind’s design, but it is motivated by the same desire to evaluate the effects of trying something new. During an RCT, individuals in a population are randomly assigned to receive a treatment, and they are evaluated in comparison to another group in the same population who (randomly) did not receive the treatment.
In the academic world, doing an RCT is a common way to design a study, but more recently, social impact organizations and government agencies have increasingly begun turning to RCTs to evaluate their programs. The Institute of Education Sciences, an office of the Department of Education, has supported 175 RCTs to date in their effort to evaluate interventions in the school system.2 Their efforts are part of a broad push to formalize and quantify the effects of new initiatives in public policy.
Why randomized controlled trials?
Why are randomized controlled trials so valuable for evaluating new ideas? If you ask a statistician, they’ll tell you that experimental designs allow us to isolate the effects of a given intervention. The idea of an “intervention” is a broad one – scurvy patients are given citrus; students are taught with a redesigned curriculum; cancer cells are treated with a new chemical agent. With a large enough sample, we can assume that individuals who got the treatment have the same innate characteristics and are affected by the same environments on average as the individuals who didn’t, which means that any differences in outcomes we observe come from the intervention applied.
If you ask a behavioral scientist, however, they’ll be happy to tell you why we need to systematically evaluate interventions for behavioral reasons. First, observers are subject to biases in their intuition and perception—rigorous evaluation provides solid evidence that cuts through these biases. Second, humans behave in ways that are hard to predict. Careful testing provides us with reliable evidence on human behavior, and this evidence allows us to feel confident that social impact interventions are having real effects.
Patterns in the noise
The behavioral research suggests that people are exceptionally adept at seeing patterns in the noise, especially when those patterns happen to fit neatly with their motivations. In one study, participants were told that they would be asked to drink either some delicious orange juice or a much less appealing vegetable smoothie depending on what a computer display showed them.3 Some were told that they would drink the orange juice if they saw a letter, while others were told they would get the juice if they saw a number. However, every participant actually saw something like this:
Is this a letter or a number? It’s unclear. But participants who were told that a letter meant they would drink the OJ overwhelming saw a “B”, while those participants who were told that a letter meant they would drink the gross vegetable smoothie saw the number 13 instead. Their motivations fundamentally changed how they perceived the world around them.
This lab study echoes things we see every day – political junkies are sure that the media is biased against their candidate, the refs have made an obviously terrible penalty call when it’s against your favorite team, and your child’s art is objectively the most beautiful in the class. But all of these examples serve to highlight the importance of rigorous evaluation. Without testing, we’d be likely to perceive our interventions as effective in every case – an assumption that can be irresponsibly inaccurate.
Next BETA project post: testing in the social sector
Randomized controlled trials help us determine what works, and figuring out what works is especially vital in the social sector. Our next post on the BETA Project will discuss using RCTs in the social sector. This post and other helpful insights from the BETA Project are available on CFED’s Behavioral Economics blog and BETA Project website.
