Return on Involvement: Creative Ways to Measure the Impact of Community

Measuring return on investment (ROI) in community is hard. It's OK to admit and acknowledge that.

The fact that it's challenging is actually a good sign; this is a bit of a paradox, but my theory is that if it were easy to measure, then in some ways, it would also mean that the impact isn't as far-reaching. Community cuts across traditional silos, departments, and stakeholders; that's why it's so powerful and challenging to measure.

I'll be honest; if you're reading this and hoping to walk away with some magic formula that will solve all your ROI or reporting challenges, it's not going to happen because it doesn't exist.

Measuring the return on what we do — the output and performance of the tactics we try — is a constantly evolving and challenging puzzle. So we're going to explore a few creative ways to measure it in a way that makes sense for your specific business and community.

But first, let's look at some examples of why communicating ROI can be difficult across industries and the creative ways some companies have found their way around those challenges.

Google's research on Media Mix Modeling

Community is not alone in the dilemma of communicating ROI. Even well-supported stakeholders like performance advertising still find it extremely challenging to measure something as basic as the return their clients are getting from sending out ads on their platforms.

Media mix models are commonly used by companies such as Google and Meta to understand the return on their client's investment in advertising on their platforms. Of course, advertisers need to understand the effectiveness of their media spend, and these models usually help them do that really well — but they're not without limitations. Google’s paper outlines the various challenges such models may encounter in consistently providing answers to advertisers' questions on ROI.

The community industry has even fewer resources to truly explore how we can improve the ways we communicate ROI. That is also not to say that we shouldn't invest in trying to measure the value of communities or that it's a lost cause. It's important to acknowledge that communicating ROI is a constantly evolving, constantly improving, and endlessly complex process in itself.

Knowing this is a complex process, community builders must help the organization they work for decide what is essential to measure. And that is precisely what the community in my following example did.

How the Oregon Food Bank measures love

The Oregon Foodbank is a 38-year-old statewide organization serving over 1,200 food distribution sites with an annual operating budget of $100 million.

In 2020 they had an extensive review with staff, stakeholders, donors, and partners. As a result, they decided they weren't measuring the most important thing for their organization — love.

They decided that in the next year, they would figure out how to measure love throughout the organization and in all their activities. The goal makes sense and seems really important, but how do you actually do it? How do you measure it?

This is very similar to how we would try to measure sentiment in community — it's a challenge, and it was for the Oregon Food Bank too. Luckily, they documented their process pretty well.

Just because they measured love didn't mean they were at odds with traditional financial outcomes. In fact, they hypothesized that if they were able to measure and optimize love successfully, it would improve some of the non-profit's financial outcomes. So far, it has proved to be at least directionally accurate.

From the very beginning, they asked important questions such as: how do they define love, what goes into love, and what does it mean when they talk about love in an organization?

What are the different ways that love shows up for individuals, staff, and donors? What are the attitudinal and behavioral indicators of that love?

Below are the 12 dimensions of love they came up with.

To measure impact, they put these questions into the annual, monthly, and weekly surveys and assessments sent to donors, staff, and partners. Here's just one example.

Their survey was a mixture of quantitative and open-ended qualitative questions that would give them more context on how their interviewees were thinking about these complex human stories around hunger and food scarcity.

It's interesting to think about the challenge of an almost 40-year-old organization, from centering everything around traditional financial measures to something as abstract and powerful as love. It is an excellent example of how you can really change an organization's focus by using something that is admittedly difficult to measure.

I liked that story because I heard and said these things myself. I've listened to many community builders, my boss, CEOs, and clients — they all want a clear, straightforward way to measure ROI.

The question is, how do you measure the ROI of a sense of belonging? How do you measure the ROI of a relationship, a connection of shared knowledge or feedback, and all of these complex things that happen in the communities we manage?

It's really up to us to figure out what is important to measure for our specific organizations; it can't all be the same.

For the Oregon Food Bank, their creative way was to measure love. In the same way, community leaders have to devise innovative, thoughtful ways to test and measure the most important metric for their specific communities. Those measurements can differ from what the platforms we use box us into, such as the traditional measures of clicks and engagements. There are other ways to measure the output of community that show how individual activities can ladder up to concrete business results — and Shell Oil did just that .

How Shell Oil's Community of Practice saved them $24 million in one year

One of my favorite articles on measuring communities was written in 2002, almost 20 years ago. It was in a print magazine called Knowledge Management.

This particular article focused on internal communities of practice (CoP), that were for the most part, offline. Still, this framework works well for the modern super, online communities we have today.

This example is about as far away as you can get from the Oregon Food Bank because it's about Shell Oil and fossil fuels. Yet, they managed to use their CoP to achieve business results in very concrete terms.

Shell Oil had (and still has) a committee and CoP of geologists. The experts in this community included people with vast knowledge of which coastlines to destroy and where Shell would need to start, stop, or continue drilling for oil.

Management wanted to know if having this community was worth it and if its existence warranted the company's investment. So they started with interviews with members of this CoP and found some interesting results.

In one year, there were sites that the company did not test and did not drill based on the knowledge freely shared by the geologists in this CoP through daily conversations. They found that for each of these wells they did not mine, it saved the company approximately $20 million — roughly $120 million in savings per year.

So they did further research and talked to a few executives and experts who confirmed with about 80% surety that $120 million is around the amount they saved by not drilling these wells.

They then devised a formula to calculate ROI (shown below) and multiplied the $120 million by 80%.

Researchers then realized that the geologists may have figured out not to drill in these sites on their own because they were experts in the field and knew their work, but they may not have shared it if it wasn't for the existence of the CoP. The researchers estimated there was a 25% chance that the information shared in the CoP contributed to these geologists' recommendation not to drill.

So again, they multiplied that by 25%, coming up with a very conservative approximation that the CoP has led to $24 million in annual savings for the company. That’s just over $40 million + if today’s dollars if we include inflation.

This was just one sort of activity and potential outcome from the CoP, but it was enough for executives to believe that the CoP was a worthwhile investment for the company.

It didn't have to be a super hard, fast, precise, and accurate number or show how the CoP affects 100 different parts of the company. This one example was enough to show that they should continue investing this money in the community.

In the same way, when proving ROI, you don't have to exhaust all the different ways your community makes a difference. Instead, find a metric that truly matters in your company and a creative way to measure and report on it.

‍

A framework to find business result

Within this same article was a framework that the author used to determine the value of community. Again, this is not a magic formula or something that you would necessarily present to your stakeholders or the C-suite, but it is a framework that can help you figure out what to measure and how to measure it.

‍

Input: These are all the things that happen in your community daily, including introductions, onboarding messages, events, etc.

Outputs: This is what your members or your company gain through the input. Do they gain access to something? Financial compensation, status in their fields, access to information, knowledge or a network referral?

Value types: Then you find the value of those outputs and categorize them. What's the value of this output for the customer or your organization as a whole? What's the value for the success of sales teams in your company?

Business result: These should help you see more clearly the business result all these activities ultimately yield. They could be results such as brand affinity, increased customer retention, or even attracting talent if your company wants to hire from within the community.

How to present your community's ROI

Now that you have all of these different ways that activities within the community can create different outputs, types of value, and, ultimately, business results — your goal should be to find one or two threads that show the ROI of community investment.

1. Analyze the data

You'll want to look at the activities, the outcomes, and the type of value, not just for the company but for specific teams, individuals, or departments within your organization.

2. Find a result that has the most impact on business

Create a strategy around what you want to measure and why it's essential for your company. This could be revenue (how to increase or decrease costs like Shell Oil) or something as abstract as love (like The Oregon Food Bank) — whatever it may be, figure out how it fits into your strategy.

3. Tell your story

Finally, talk to your members through interviews or surveys to find stories that show the connection between community activities and business results. When presenting, these stories will anchor your theory and clearly show how the community has affected the company and how it can continue to do so in tangible ways.

This blog post is a snippet of a talk by Commsor's Head of Community-Led Erik Martin on Return on Involvement: Creative Tactics to Measure Community at The 2022 CoLab event. Watch the full talk above or on YouTube here.

‍