What organization doesn’t want to describe itself as “outcome oriented” and “data driven”? These two buzz phrases highlight a growing interest in the social sector for measuring and tracking concrete outputs in order to demonstrate organizational impact.

Beyond the usual metrics, however, is the need to measure the impact of specific program changes or initiatives. One useful tool for testing the effects of a change is the randomized controlled trial (RCT), the “gold standard” for evidence across many domains. In our last blog post, we discussed how RCTs are useful because they cut through the bias introduced by our motivations and allow us to assess the impact of a specific idea. Many good ideas just don’t have an impact in the real world, and testing is important to make sure that resources are spent in ways that have real effects on welfare. In the BETA Project, we conducted RCTs at two of our partner sites, Accion Texas and Cleveland Housing Network.

Change is hard

In order to learn about the best ways to effect change, we need to test specific programs or interventions. But there are cases all over the social impact world where programs aren’t evaluated, so we have no idea whether they had any impact at all. In cases where rigorous testing has been done, the results have often suggested that the conventional wisdom about “what works” is off the mark. Sociologist Peter Rossi summarized the problem with his “Iron Law of Evaluation”: The expected value of any net impact assessment of any large scale social program is zero. In other words, we could save a lot of effort and have the same effect by never trying out any programs at all!

While the Iron Law is perhaps a dramatization of the difficulty of effecting large-scale change, it encourages us to be careful with our time, our money, and our optimism. Three recent RCTs have produced results that push back on the conventional wisdom of the social impact world.

The miracle of microfinance

In 2006, microfinance was on the top of the world. Dr. Muhammed Yunus and the Grameen Bank received the Nobel Prize – not for Economics, but for Peace. In the prize announcement, the Nobel committee stated that “Yunus and the Grameen Bank have shown that even the poorest of the poor can work to bring about their own development.”

Microcredit makes a lot of sense. Many economists believe that lack of access to credit is one of the most important barriers keeping people impoverished. But some economists were skeptical. While microcredit borrowers saw huge gains compared to non-borrowers, it was possible that previous studies were simply catching the effect of being entrepreneurial. The poor with an entrepreneurial spirit were both more likely to succeed and more likely to get a loan.

In 2009, these economists put microfinance to the test – and found that, if microcredit was distributed in a randomized trial, there were no effects on poverty a year and a half later. Of course, this does not mean that microfinance did not work. Many people who received microcredit seem to have been investing in their business. But realizing that investment may take them much longer than we once expected.

Location, location, location

How much of poverty is driven by “neighborhood effects? The sociologist Julius Wilson has theorized that harmful effects of neighborhoods could be responsible for high dropout rates in school and in the labor market. For example, in high poverty areas it might be harder to find peers and role models that would encourage study. In 1994 the department of Housing and Urban Development tested this theory with the “Moving to Opportunity” experiment. The experiment gave some families vouchers to help move them from high poverty areas to low poverty areas, with the hope that such a move would substantially improve outcomes.

But the Moving to Opportunity findings were mixed. While the effects on schooling and jobs were lower than expected, there were surprisingly strong effects in health. Obesity rates dropped by 40%. Neighborhoods effects are important – but not in the ways we initially thought.

The effects of financial literacy

In the aftermath of the Great Recession, many have called for personal financial education to become a standard part of the high school curriculum. Surveys show that individuals who know more about personal finance are also more likely to build their savings and limit their debt, allowing them to better weather financial storms.

But once again, randomized controlled trials tell a different story. Several studies have randomly assigned some individuals to receive financial education, while control individuals did not. These studies did not find any changes in savings or debt. This suggests that providing information alone did not put people on the path to financial security, and might encourage us to address other problems limiting financial success, such as limited self-control and attention.

Past research informs future design

The results above may seem surprising, but it’s not all bad news! Careful evaluation also lets us know when we’ve gotten it right, and can steer us towards new applications of proven solutions. For example, ideas42 has an ongoing project where we are working to redesign financial education curriculum using “rules of thumb” (a paper on this project was released in early 2014). In the BETA project, our intervention designs were based on ideas that have worked in the past. For example:

• At Accion Texas, we reminded borrowers to be prepared to pay their monthly bill via text and
email reminders. This approach that has been shown to be effective in helping the poor build
assets by reminding them to save.

• In our work with the Cleveland Housing Network, we designed a raffle for borrowers who paid
their rent by the 1st of the month. In the past, a raffle-based approach has been shown to help
heart patients who are especially at risk of forgetting to take their medication.

• At Neighborhood Trust, we helped clients plan their account usage by prompting them to
make a plan as to how they would do so. This intervention was inspired by research that
shows that helping voters make a plan with information about where, when, and how they
plan to vote makes them more likely to actually follow through.

Next BETA project post: findings and implications from the BETA project

Our next post on the BETA Project will be final post based on the work completed in 2013 with partner sites Accion Texas, Cleveland Housing Network and Neighborhood Trust. This post and other helpful insights from the BETA Project are available on CFED’s Behavioral Economics blog and BETA Project website.