Community Experimentation: How to Do It (and Why You Should)

Community building is more or less one big experiment. Think about it: you're launching initiatives and building programs and often just... hoping for the best.

This is another beautiful similarity to the product development world, where a good chunk of what you build is based on assumptions, experimentation, and iteration. Your assumptions may be based on past experience, learnings from others, or even just gut feeling, but everything from what events to host to what your community rules are could be considered different types of experimentation.

Like many things in the community world, you're likely already experimenting, even if you aren't actively thinking about it. But one of the best ways to get leadership buy-in is to take what you've been doing innately and add some order and process. You'll have an easier time explaining why you've made those assumptions, how things have turned out, and whether to keep investing in the initiative or shift gears.

But enough of the generalities — I wanted to share some concrete examples from my own experience. I hope you'll find some inspiration in my learnings!

Experimentation: a few lessons learned

Rules and reddiquette

When I started working at reddit, we had very few rules. It worked for us at the time because users mostly acted in good faith, and there weren't as many people causing chaos on the internet as a whole back then. Our assumption was that what we had in place would be enough, since it hadn't been a problem before then.

Of course, things change! Our original assumption was proven incorrect, and we had to come up with a new hypothesis and plan to accommodate new learnings. This led us to create the first official "rules of reddit" page, that had five core rules (and reddiquette, of course).

Based on past learnings, some research, and predictions on future behavior, we narrowed down what we thought would cover the most bases and released it to the community. We knew, however, that eventually we'd need to update them again, and tried to plan as best we could for what that "success/failure indicator" might be (spoiler alert, it ended up being really difficult to predict that with reddit in particular!).

To hangout or not to hangout?

A more recent example comes from The Community Club, in the form of our bi-monthly hangouts. We wanted a way for members to meet each other and build relationships, but we didn't want it to feel overly formal or for there to be a high barrier to entry. Our hypothesis was that if we host casual hangouts in the community, members would make important connections with peers.

So we came up with a basic plan to roll it out and started holding frequent events. But what we didn't account for in our initial plan was the number of international community members who were interested in hanging out! With this new nugget of information, we adjusted our plan and did one US-focused event and one EU-focused one. This change increased participation in the events, which was a great success indicator!

But over time participation has dwindled, and now we're at the "go/no-go" decision point because our success indicators are showing us they're not working as well as they once were. Taking into consideration all of the potential variables impacting these findings (summertime naturally has less online participation, vaccination rates are up, many cities are reopening, people are exhausted from online events, etc.), we're able to conclude that how we had previously structured these events no longer makes sense.

So now, with the same problem/purpose as before, we're back to the hypothesis step of the process — our new hypothesis is that if we host more structured hangouts that accommodate different time zones, members will make important connections with peers and get direct, actionable help so they feel more supported in their work. We'll let you know how this goes. 😉

3... 2... 1

A fun little experiment we ran (which those in our Slack community will likely be familiar with) was changing out Slack workspace icon to a countdown timer before launching one of our programs. We skipped a lot of the "official" steps in the planning process for this one, but hypothesized that by doing a countdown it would drive additional excitement around our announcement that wouldn't necessarily have received the same reception otherwise.

Dear reader, our hypothesis was correct! Success indicators were: people immediately noticing and asking what it meant, which led to an increase in overall activity in the community; members returning each day to see what was happening; and multiple members reaching out to ask if they could take inspiration from it and use it in their own communities.

We've now used this countdown strategy for a few other exciting launches within the community and seen a similar response, so we'll likely continue to utilize this going forward. If we reach a point where this tactic is no longer driving results, we'll take those learnings and try something new.

This brings me to another point — it's totally fine to emulate what you see in other communities! Borrow away (but it's always nice to ask first, just in case), and be sure to make note of what was successful in the other community so you can do a comparison of how things turn out in your own. It's entirely possible and, frankly, normal for different engagement plays or initiatives to work in one community but not in another. But if you approach this with an experimental mindset and document results, it's still worth the time to give it a try even if it doesn't work out as expected.

How to experiment within your own community

I tend to treat most of our community initiatives and projects like experiments. Here's why I think you should too:

• You'll have something concrete to point to when talking to stakeholders or leadership
• It's a handy record of past attempts so you don't go down the same rabbit hole of things that didn't actually work
• It may even help you with the prioritization of projects/initiatives since you'll have a better understanding of what best drives business objectives

My step-by-step process

Here's a basic template you can use when kicking off any projects. If you're struggling to come up with an answer for any of these points, you may need to do a little more research!

1. Problem/purpose — What's the issue you're trying to fix, or what do you want to achieve?
2. Hypothesis — What do you suspect is going to happen, and why? If X, then Y.
3. Plan — What's your approach? How are you going to test your hypothesis?
4. Success indicator(s) — How will you know your experiment is working (or not)?
5. Go/no-go deadline — When is it time to call it quits?
6. Conclusion — Whether it was a success or not, your experiment should have given you some insights into your community. It's worth documenting them.
7. Next steps — Was your experiment a success you'll be repeating and if so, are you making any tweaks? Or are you dropping it entirely, in favor of a new plan altogether?
8. DRI — Who is directly responsible? And who makes the go/no-go decision?

You may not need to go through this exercise for every single thing that you do in your community, but it's a good starting point for getting a better understanding of what is worth your time and what brings value to both your community members and your org.

Have you run any experiments in your community lately? Whether they were a roaring success or failed miserably, we want to hear about them. Share them over on our forum!