Measures: Will You Get What You Want?

May 10, 2010  

I recently spoke with a new client who shared with me their innovation measures.  When I looked at their measurement system, I immediately saw flaws.

But before addressing these imperfections, let me first provide you my perspective on innovation measures.

In general, there are three types of measures associated with “challenge-based” innovation (be sure to read this article if you are unfamiliar with the concept of challenge-based innovation):

  1. Process Measures - These measure the activity associated with your challenges (e.g., 500 registered solvers, 40 submissions per challenge, 80 votes per challenge, etc)
  2. Solve-Rate Measures - These subjectively measure how well you solved your challenges (e.g., 82% of challenges were partially solved, 61% of challenges were completely solved, etc)
  3. Value Measures – These measure the actual value accrued (e.g., increased revenues by $25M, reduced costs by $35M, etc)

The last measure (value) is where the rubber meets the road.  This is your ultimate goal.  But sometimes, value realization can take years (or in the case of pharmaceutical companies, decades).  Therefore, the second measure (solve-rate) is a good way to monitor progress with your program.  But what about process measures?

Process measures are leading indicators that can be useful in measuring trends over time for things like community engagement, effectiveness of internal communications, and quality of challenges.

Let’s look at one common process measure: the number of ideas/solutions submitted for a given challenge.  This was one of the measures that my new client used.

Imagine that you are using crowdsourcing to find a solution to a challenge.  You post the challenge on your website or intranet.  A month later you check to see how many responses you get.  In this scenario…

Which is better:

  • getting 100 ideas/solutions?
    or
  • getting only 2 ideas/solutions?

Most people intuitively think that 100 solutions is better than 2.  In fact, most organizations believe that more ideas equates to greater success.  The reality is, however, that 100 is not necessarily better than 2.

Let me re-frame the question…

Which is better:

  • getting 100 ideas where only 2 of them were exactly what you needed and the other 98 were duds?
    or
  • getting 2 ideas where both were exactly what you needed?

Now the correct answer is a bit more obvious.  In this situation, the latter is probably better.  The amount of work needed to sift through the solutions is a lot less when you have only 2 submissions.  Imagine if you received 10,000 ideas of which only 2 were good.  You can see now that the effort to find the best solutions/ideas might be overwhelming.

Although activity is good, too many submissions can indicate that you have a poorly defined challenge.  Therefore the ratio of good ideas to duds might be a more interesting measure.

The key is, make sure you understand the unintended consequences of your measurement system, especially when it comes to process measures. If done properly, process measures can help you drive higher solve rates (measure #2). And often, higher solve rates lead to greater value (measure #3) in the long run.  But not always.

High solve rates with low value can also indicate problems with your innovation program:

  • Poor implementation – You are unable to convert solutions into finished products/services
  • Poor commercialization – Your solutions do not meet the needs of the market/customers and therefore do not generate revenue
  • Poor relevance – Your challenges, although solved, are not important enough to “move the needle” of the organization’s innovation efforts

Measures are important for helping tracking your innovation efforts.  And they can help diagnose potential issues.  But it is important to measure the right things.

There is an old expression: “You will get what you measure.”

But the bigger question is, “Will you get what you want?”

Leave a Reply

Old Comments

2 Responses to “Measures: Will You Get What You Want?”

  1. Innovation « Generate demand through Permission Marketing on January 1st, 2011 7:26 am

    [...] – Measures: Will you get what you want? [...]

  2. Expert Blog on July 31st, 2013 8:58 pm

    McKinsey’s Technology Enabled Business Trends – New KPI Strategy?…

    There is promising news in the ROI justification department for collaborative technologies. In Clouds…