Measures: Will You Get What You Want?

I recently spoke with a new client who shared with me their innovation measures.  When I looked at their measurement system, I immediately saw flaws.

But before addressing these imperfections, let me first provide you my perspective on innovation measures.

In general, there are three types of measures associated with “challenge-based” innovation (be sure to read this article if you are unfamiliar with the concept of challenge-based innovation):

1. Process Measures - These measure the activity associated with your challenges (e.g., 500 registered solvers, 40 submissions per challenge, 80 votes per challenge, etc)
2. Solve-Rate Measures - These subjectively measure how well you solved your challenges (e.g., 82% of challenges were partially solved, 61% of challenges were completely solved, etc)
3. Value Measures – These measure the actual value accrued (e.g., increased revenues by $25M, reduced costs by $35M, etc)

The last measure (value) is where the rubber meets the road.  This is your ultimate goal.  But sometimes, value realization can take years (or in the case of pharmaceutical companies, decades).  Therefore, the second measure (solve-rate) is a good way to monitor progress with your program.  But what about process measures?

Process measures are leading indicators that can be useful in measuring trends over time for things like community engagement, effectiveness of internal communications, and quality of challenges.

Let’s look at one common process measure: the number of ideas/solutions submitted for a given challenge.  This was one of the measures that my new client used.

Imagine that you are using crowdsourcing to find a solution to a challenge.  You post the challenge on your website or intranet.  A month later you check to see how many responses you get.  In this scenario…

Which is better:

• getting 100 ideas/solutions?
  or
  
• getting only 2 ideas/solutions?

Most people intuitively think that 100 solutions is better than 2.  In fact, most organizations believe that more ideas equates to greater success.  The reality is, however, that 100 is not necessarily better than 2.

Let me re-frame the question…

Which is better:

• getting 100 ideas where only 2 of them were exactly what you needed and the other 98 were duds?
  or
• getting 2 ideas where both were exactly what you needed?

Now the correct answer is a bit more obvious.  In this situation, the latter is probably better.  The amount of work needed to sift through the solutions is a lot less when you have only 2 submissions.  Imagine if you received 10,000 ideas of which only 2 were good.  You can see now that the effort to find the best solutions/ideas might be overwhelming.

Although activity is good, too many submissions can indicate that you have a poorly defined challenge.  Therefore the ratio of good ideas to duds might be a more interesting measure.

The key is, make sure you understand the unintended consequences of your measurement system, especially when it comes to process measures. If done properly, process measures can help you drive higher solve rates (measure #2). And often, higher solve rates lead to greater value (measure #3) in the long run.  But not always.

High solve rates with low value can also indicate problems with your innovation program:

• Poor implementation – You are unable to convert solutions into finished products/services
• Poor commercialization – Your solutions do not meet the needs of the market/customers and therefore do not generate revenue
• Poor relevance – Your challenges, although solved, are not important enough to “move the needle” of the organization’s innovation efforts

Measures are important for helping tracking your innovation efforts.  And they can help diagnose potential issues.  But it is important to measure the right things.

There is an old expression: “You will get what you measure.”

But the bigger question is, “Will you get what you want?”

Measure Innovation NOT InnovationS

Measuring innovation is very different than measuring innovationS (“s” capitalized for emphasis). Let me explain.

In our goal-obsessed society, we want to measure everything that moves. In doing so, we feel as though it gives us a grasp of the real world. But is this true? Do measures really provide a lens into reality?

A colleague of mine used to say, “You’ll get what you measure, but will you get what you want?”

This poses a very interesting question. What should you measure in order to get the insights you want and the behaviors you desire?

In the past I wrote a few blog entries on this topic.

In “How Do You Measure Success?”  I talk about how quantitative measures can be misleading indicators as many factors impact the numbers. Maybe qualitative measures provide more accurate insights. But how do you accurately capture this information?

In one of my favorite blog entries, “Innovation Lessons from the Apprentice,” I discuss how measuring hot dog sales, stopped one team from being creative. As a result, their competitor, who was not constrained by the numbers, made three times the profits of the numbers-driven team. Read this article. I suspect most companies are measuring hot dogs, not results.

When companies try to measure innovation, they are really measuring innovationS.

InnovationS have a start and end. You can measure the amount of effort put into developing these ideas and the value they produce. Some common measures are:

• Number of ideas generated
• Number of people who contribute ideas
• Number of ideas selected and implemented
• Economic value generated from implemented ideas
• % of revenue from products introduced in the last year

The big question is: How do you know if your organization is innovative enough? How do you measure innovation (without the “s”)?

The answer: When your business achieves its overall performance targets. This is real innovation – innovation as a capability – and not just an innovation pipeline. This is embedded innovation that happens real time, and, as a result, is much more difficult to directly measure. The only real measures for this are business results.

At the end of the day, these are the only measures that really matter.

Are you measuring innovations or innovation? Or, in other words, are you measuring hot dogs or results?

P.S. Be sure to read my blog entry entitled “Is Your Organization Anorexic?” It explains why I chose the picture I did. In short, companies are obsessed with measures which causes them to get poor results. People too can become obsessed with numbers (e.g., weight) and miss out on what matter most (e.g., health).