Just yesterday, I was discussing the framework for one of those sessions with a board member who will be at this retreat and has been thinking carefully about this topic. The retreat I’m preparing is designed to help program staff and board members learn to love evaluation — okay, maybe not love yet, but appreciate the importance of.

We both agreed that the conversation about measurement needs to shift away from evaluation, which conveys judging, to learning, which is about a desire to get better at what you do for the sake of the people or community you serve.

So let me say I was cheering when I read the  three principles that The Gates Foundation says guide its approach to measurement:

1. “Measurement should inform specific decisions and/or actions.
2. “We do not measure everything, but we do strive to measure what matters most.
3. “The data we gather help us learn and adapt our initiatives and strategies.”

Hallelujah.

Their formula for actionable measurement is just brilliant in my book:

“Planned collection, analysis and synthesis of data + time devoted to development of reflection and insight + willingness and ability to change = Informed Decisions & Actions”

With the growing drumbeat to rate and rank nonprofit outcomes, it is refreshing to hear such an important funder talk about reflection, insight, adaptation, learning.  I’m also impressed that this foundation that has more money than any of my clients could ever dream of makes a point that they are judicious about what they measure as they can’t (and shouldn’t) try to measure everything.

They also note that the initiatives they fund are one of many factors that lead to large community change. Thus, while they track those top level  indicators of what they call “strategy-level”  outcomes and impact, they don’t try to account for how their specific investments contribute to that impact as they know they are one of many partners pushing to the same objective.  That should come as a relief to many smaller organizations whose funders are asking them to show how their limited programs are ending poverty or hunger, etc. Now you can cite the Gates Foundation.

My advice: download the guide, read and discuss it with your Board and staff. It may give you many ideas for how to approach the idea of evaluation and measurement in your organization.

There is a snowball gathering momentum and mass rolling down the hill in the USA. Your board needs to pay attention to it now.

That snowball is the growing movement by independent intermediaries to develop simple rating systems for the very complex world of nonprofit performance and social impact.

As charitable giving has grown to over $300 billion  annually in the US, it seems that the business world is now taking great interest. Those individuals who brought you the financial crisis have now decided that most charitable giving is misdirected (see this Fox Business News clip and you’ll get the picture). What the public really needs, they believe, are unbiased “intermediaries” to redirect charitable giving to  the “best performing” nonprofits.

Funders are focused on measuring and publicizing nonprofit outcomes

A 2008 study released by the William and Flora Hewlett Foundation, called “The Nonprofit Marketplace: Bridging the Information Gap in Philanthropy,” outlines the desired objective:

“…a more efficient and effective nonprofit market would direct more funds to solving the world’s social problems faster and at a lower cost, thereby helping more people sooner. Reallocating just 10 percent of the current $300 billion annual fund flow to the best performers would have a similar effect as raising billions in new funds — with nowhere near the same cost in fundraising time and energy.”

Now, as much as I want to debate those assumptions, the train has already left the station.

For example, The William and Flora Hewlett Foundation has already provided seed funding to an organization called GiveWell which they are promoting on their website. They describe GiveWell as

“the sometimes controversial … independent, nonprofit evaluator of nonprofit organizations. A self appointed watchdog, it performs in-depth research to help people accomplish as much good as possible with their donations.”

Who are GiveWell?  As listed on their web profiles, their staff are former hedge fund employees and recent college graduates (who appear to have little-to-no nonprofit experience). They boast on their website that they have already evaluated 500 nonprofits and only found four worthy of their top ranking!

While you may not have heard of GiveWell, you’ve probably heard of Charity Navigator. Ken Berger, its thoughtful President and CEO, has heard the criticisms of its 4 star ratings which assess only financial indicators and not nonprofit program performance.

Mr. Berger is aware of the criticisms of Charity Navigator’s limited perspective and determined

“to transform our evaluation system of charities to include two additional dimensions (beyond financial health) – accountability (including transparency) and outcomes.”(Emphasis added).  Read more here

Charity Navigator has appointed an advisory group to help it design its system.

Is your organization attempting to measure outcomes at all?

We all know that measuring outcomes is one of the hardest tasks that most nonprofits and social benefit organizations face. And one of the most controversial.

Just look at the maelstrom stirred up by the US government’s No Child Left Behind Act which tests students to assess public school performance. Critics of the testing process point out many shortcomings including school cheating, lack of similar standards across states, the failure to test the same students over time, the wide disparity of the student starting line, a too narrow focus on test scores that excludes other factors that contribute to student success (like social-emotional intelligence), or the narrowing of the curriculum to focus on only what is measured.

Yet, I hope we all agree that we have a responsibility to our investors and our donors to use their money responsibly.

And we have an even bigger responsibility to the people whose lives we touch, or the world we live in, to produce something very good with those dollars.

Unless we attempt to measure our performance, how will we ever know that we are moving toward our desired vision of societal change?

Unfortunately, way too many nonprofits use the excuse of “it’s too hard” to avoid any outcome measurement at all. Under these public rating systems, you won’t be able to use that excuse much longer.

Your organization will be especially penalized if it lacks any attempts at all to assess its performance and social impact.

A public airing of your performance measurement (or lack thereof) may be right around the corner

In a conversation I had this summer with Ken Berger, he indicated that Charity Navigator’s goal is to include all nonprofits with budgets over $1 million, though they intend to start with the largest nonprofits that consume most of the charitable dollars and have the most capacity for measurement. (Yet may be some of the most complex and multilayered as well.)

One of the instruments that Charity Navigator is reviewing as a prototype for its ratings was developed by Hunter Consulting LLC for social and human service organizations. Its name is very revealing of its intent: “Protocol for assessing Investment Risk – with regard to the likelihood that an organization is producing Social Value.”

My intent is not to be a how-to on performance measurement. Instead, my goal is to give you a heads up before you find your nonprofit listed without a star for all the world to see.

I’d especially suggest that you add your voice to the discussion about how to rate nonprofit performance, sharing your own experiences and what you’ve learned about what works, what doesn’t and unintended consequences.

And though I am absolutely a cheerleader for every nonprofit developing a vision of the societal change they are trying to create along with measures and assessment of their progress toward that vision, I have deep, deep reservations about intermediary developed rating systems and their attempt to influence donor dollars.

For example,

• I worry about the smallest nonprofits, like neighborhood associations, small watershed organizations, food pantries or community advocacy groups which won’t even be on the radar of these rating systems, yet perform invaluable services at the micro-level.

• I worry about a focus solely on individual organizations and how that overlooks the critical networks of organizations that only create effective change as a system themselves.

• I worry about funding for advocacy organizations that take up the most difficult of causes, like ending the death penalty in the US or protecting human rights around the world, and how a national push to rate effectiveness may result in the abandonment of life-saving activities that don’t meet funder expectations for large numbers of “successes.”

• I wonder what timeframe will be used for measurement, as many of us know that social change takes decades, or that the consequences of actions are not often revealed for years.

• I worry about the loss of funding for experimental organizations, especially in the arts, culture and humanities which may be forced even more than they have already to justify their work only by non-artistic measures like economic impact because those are the easiest to assess.

• I also wonder how these intermediaries and their supporters will develop and confidently proselytize their measurement formulas in mere months while some of the best minds in academia and government and the nonprofit sector have been struggling for decades to isolate agency actions and measure social outcomes.

But regardless of my reservations or yours, change is on the way. So dust off those logic models, start your indicators and bring on the evaluators.

******************************************************************************************

This article first appeared in the September 23, 2009 edition of Nonprofit Boards and Governance Review at CharityChannel.com

In her latest op ed piece in The Nonprofit Quarterly, “Mom and Pop Giving Over to Big Brother?,” editor-in-chief Ruth McCambridge, shares our concern about national trends that overlook the value of “small, locally controlled organizations to civic life.”

She goes on to note that small businesses have disproportionately shed jobs in this recession and cites examples that indicate that this may be the result of government policy directing recovery funds toward the biggest corporations and away from the small guys. Ruth worries whether this will be repeated for nonprofits as this Administration works with “large philanthropic organizations to craft … approaches to ’social  innovation.’”

We have raised the same question in different forms in this blog . See Now I’m Worried – Who decides what is effective and who should be funded or Are nonprofits only safety nets? among other entries.

If we get too caught up in focusing funder attention on “taking programs to scale,” we are destined to overlook the critical community building that can only be done by small, in-the-neighborhood organizations. Or, those scale-ups may overlook the impact of design that is an adaptation to local circumstances that doesn’t scale well or shouldn’t be scaled but should be redesigned for a new locale or new population. Or, even more likely, we may tend to  forget that social change depends on a continuum of organizations, people and actions to finally tip power balances and produce desired improvements.

Thanks to The Nonprofit Quarterly for using its national platform to continue to remind the top of the nonprofit infrastructure that this is a complicated world and that the contributions of the little guys can’t be dismissed or ignored.

This reminds me of a quote in Boards that Make A Difference by governance guru John Carver that has always stuck in my head. “A crude measure of the right thing beats a precise measure of the wrong thing.”

All this was stirred up for me by the recent buzz within the world of philanthropy for measures to better direct donor giving to “what works.”

There is a real danger in oversimplifying what works.

While I’m completely in favor of focusing the attention of our sector toward processes that produce real community results, I’m wary of reliance on simplistic nonprofit rating systems (e.g. GiveWell) that attempt to duplicate for mission effectiveness the same style of rating formulas that Charity Navigator and others use to rank nonprofits by their financial metrics. We already know that judging a nonprofit solely upon the percentage of program expenses tells us nothing about community results and, in many cases, not even a terribly lot about nonprofit financial effectiveness.

How can we better use the indicators that do exist to influence whole systems change and not just randomized philanthropic endeavors?

There are already some pretty powerful indicators out there. Hats off to the Annie E. Casey Foundation for its funding of KIDS COUNT data nationwide and in every state. (We are very fortunate to count the superb Rhode Island KIDS COUNT among our clients). Or thanks to UNICEF for the State of the World’s Children.

Having data such as this helps us understand where we are starting and helps focus our attention on the progress that we’d like to see made.

Throughout RI, we heard that publishing credible and sophisticated data sets on current conditions has led to significant changes in the way that RI government and nonprofits think about policy for kids.

But in addition to its data, Rhode Island KIDS COUNT is known for researching and bringing forward examples of practices that have been shown to make real progress for children. By bringing all parties to the table, they help entire systems develop legislation, policies and practices that better serve children.

UNICEF articulated GOBI-FFF, now considered the basic elements of child survival, to reduce infant mortality worldwide. (Growth monitoring, Oral rehydration therapy, Breast feeding, Immunizations with  supports from Female literacy, Family spacing, and Food supplements).

Yes, we can and should hold many individual organizations accountable for where they spend their dollars and the quality of their investments. But if we get distracted and only focus our attention on what nonprofits donors should invest in, as a community we distort the massive, interlocking systems changes needed to dramatically move the needle for all.

Funding the best of charter schools won’t change public education as long as charters enroll miniscule numbers of kids. Funding a few individual organizations that adopt child survival isn’t enough to eliminate communicable disease when 100% of kids need to be immunized.

We must hold accountable whole communities, states and countries for the investments they make in the outcomes we profess to desire. Our society must invest in practices at the scale needed to reach every kid or every adult and not just the fortunate few who may win the nonprofit assistance lottery.

While I believe in the power of nonprofits to change lives, I also know that our institutions are a small part of the picture.  The easily measured usually serve as band aids or incubators. It’s a lot harder to measure the efforts of the advocates or catalysts for widescale change.

I’d hate to see philanthropy distracted from enabling big system societal changes. Let’s not invest excessive amounts of energy in measuring and evaluating individual nonprofits in isolation, and miss the bigger systems that need our attention.

For some time I’ve been thinking that it is a shame that all that great learning is locked away, inaccessible from others who might put those lessons to good use.

Call it Open Source grant reporting.

My dream is to see all those reports made available on the web. Imagine that you’re sitting in Iowa and thinking about launching an arts mentoring program for high school students. Wouldn’t it be great if your initial googling would not only surface the names of other nonprofits that run arts mentoring programs for you to call, but also produced links to the dozens of reports reflecting on arts mentoring programming from start up to roll out?

I can picture researchers mining these reports and preparing national “lessons learned” papers that can be shared across the industry. Or program officers skipping right away to implement “what worked” rather than rehashing the same missteps and dead ends.

Kudos to those foundations and others who make their final reports available to their colleagues throughout the US. But why wait for the foundations. What if nonprofits posted their own grant project reports on the web to share with their colleagues?

Insanity you say? Who would want to expose themselves in this way?

Even just having short summaries on the web to tempt us would be a great start. Then we could “call for a copy of the full report.”

What do you think? Doable?