The Challenge of Measuring Impact
Tony Cadena is an Encore Fellow with the Children's Book Bank through Social Venture Partner's Encore Fellows Program.
I started my work as an Encore Fellow with The Children’s Book Bank (CBB) in November, 2014. The vision of this remarkable organization is a society in which all children, regardless of means, are surrounded by and experience books during their early years of development. In low income homes, the ratio of books to children is 1 book for every 300 children. Yet early exposure to books is important to developing prerequisite literacy skills and strong reading skills correlate highly with later school success. In pursuit of its vision, the CBB solicits book donations, cleans and sorts approximately 100,000 books per year and works thorough early literacy partners and schools to deliver a number and variety of high quality books. Partners, families and children alike report great interest and excitement in the books, both when they are newly distributed in their distinctive green bags and after children have owned the books for some time.
And yet, the Executive Director, Dani Swope, has not been content to rely solely on anecdotal stories of the children and families the CBB has impacted, or the glowing gratefulness of literacy partners, but has pushed to really understand and measure the impact of the CBB program.
As a former financial manager, performance measurement, especially beyond the financial statements, was a key aspect of my job. As a contributing faculty member for the Atkinson Graduate School of Management, I ask my students, in their capstone project course, “How will you measure success?” I am a firm believer that everything can be measured. More about that later. In my Encore work with the CBB, building the capability to measure impact is a key part of my work.
I’ve talked with many people, both within and outside non-profits who often express the belief that “you just can’t measure that” (meaning how much impact a non-profit is making). And as part of my work with Project Impact PDX, along with a cohort of early literacy non-profits, I have learned that non-profits often rely upon “effort” measures given the difficulty of getting to “impact” measures. Organizations may focus on clients served, or books distributed (in the case of the CBB), or programs completed, etc. Such measures, while important, do not quite get to the thing all organizations would like to measure: sustaining impact in the lives of those they serve. Even if organizations can measure an outcome in the present moment – say, an increase in the time spent reading with a child – how do we know that that translates into a child developing a lifelong love of reading and an interest in books?
So back to “all things can be measured.” The challenge of measuring impact in my work, as I see it, is twofold:
Not all things can be measured, and
Measuring sustained change is very difficult.
To the second point, many non-profits are desirous of enabling long lasting change in the lives of those they serve. I believe that for change to be sustained, people need not only the knowledge of a new skill or way of operating, but also they need to practice that skill and have an emotional connection to the desired change that keeps them inspired and motivated to sustain the new skill or way of being. This is hard to measure directly as sustained change occurs over time.
To the first point, even measuring change in the present time, we can ask people to report behaviors or feelings, but is it accurate? We could “test” people on a newly acquired skill, but that can change over time, improving with practice or fading with neglect. Even with well-designed ways to measure impact, the practical realities of cost and time may make measurement undoable.
So, what to make of “all things can be measured”? I have learned, both from experience and in my Encore work, that the process of trying to develop and implement measures of impact alone is worthwhile work, even if true accurate measurement remains out of reach. Why? Focusing in a rigorous way on impact helps an organization to examine its activities in light of their intended impact. This often leads to changing or improving those activities in powerful ways. Developing indicators of impact is like starting from the “customer’s” perspective and aligning programs, processes, tools, etc. to most effectively achieve the desired results. This approach also creates opportunities for talking with a sample of clients in a structured way leading to insight into an organization’s program. Lastly, the focus on impact indicators forces one to think about the theory of change. The organization wants to achieve a certain outcome and it engages in a set of activities. The theory of change requires an organization to understand what links those – “We do these activities, and in this way (theory of change) we see our intended impact.Simply put, I know sustained change requires knowledge, behavior and affect. Measuring our efforts is good, but measuring impact is best. If we believe we can measure our impact, then we will embark on the fruitful process of developing those indicators of impact. That process is hugely valuable. Even if not all things can be measured, the process of trying to do so improves our work.