The Open Innovation Formula?

July 29, 2009

Today we have a guest blogger. In an earlier entry, I held a contest to see who could guess how many ideas were submitted as part of the LG Electronics competition. One of the respondents provided a detailed assumption analysis and complex equations. Although the assumptions proved inaccurate (read the actual answers), I felt that his thought process provided an interesting perspective on crowdsourcing and wanted to share it with you.

Introducing Rishi Bhalerao…


Steve and I have exchanged e-mails on the methodology to estimate the number of entries and related metrics in the LG challenge. Even though my numbers ended up quite far from the observed data, Steve thought the process was worth describing because it raised some interesting questions. Here is the “Cliff Notes” version of the method; a more detailed explanation is included in the attached pdf file.

Steve asked for an estimate of the number of entries and the number of unique participants. Here are the steps I followed:

  1. I assumed that the larger the payoff, the more entries a contest will attract. The public data available on crowdSPRING (the online platform used to run the LG Electronics competition) is too limited to allow a regression analysis, so let’s simplify and just assume “Pro” projects—those with a minimum $1000 payoff—attract more entries than “Non Pro” projects with smaller payoffs.
  2. The blog entry crowdSPRING By The Numbers indicates that, since launch, 4700 projects have attracted 370,002 entries (average: 79 entries/project). The website lists active projects (158, when I checked on July 23) and also indicates the number of entries received by each Non Pro project. From this sample, we can calculate the average number of entries per Non Pro project. We can then deduce, assuming the sample is representative, that the average number of entries for a Pro project is ~97.
  3. The next big assumption (which turned out to be flat wrong) is that each payoff attracts a separate set of entries. So, if this project has 43 payoffs, it will attract 43X the number of entries compared to an average Pro project with a single payoff. There were 3 big prizes and 40 consolation prizes. First place received $20,000, second place received $10,000, third place received $5,000 and 40 entries got $1,000 plus an LG phone.
  4. The number of unique participants was somewhat tricky, because the average entries/participant cannot easily be guessed from the sample. I took a SWAG from my experience in customer service: a single customer generates 1.2 transactions. I didn’t have a better benchmark–particularly not one that was relevant to crowd-sourced graphic design. Turns out the actual entries per participant are ~2.6 (834 entries from 324 participants) which suggests the marginal cost of a second entry is much lower than that of the first.

How accurate were the assumptions? Not very! But, they do raise interesting questions:

  • What explains the relatively low number of entries? The max $20K payoff was 45X the average ($444 or $2 million over 4,500 completed projects) but it attracted just 11X the average participation (824 entries instead of 79). 324 participants represent less than 1% of the 36K user base advertised on the site.
  • How does the number of payoffs affect participation? Clearly, entrants weren’t motivated by 43 separate payoffs. More likely, they were looking at the top two or three prizes. Could LG have received the same number of entries with a significantly smaller payoff (e.g. by not offering the 40 $1K prizes)? Do such contests have value beyond solving the stated design challenge? E.g. brand affinity, perception of engagement, mind share, etc?
  • What lessons should crowd-sourcing platforms draw on size of participant networks vs. level of engagement? Are more entries always better?


Rishi, thanks for sharing your thoughts on this. I love your thinking and want to add a few thoughts of my own.

  • Challenge Complexity – The complexity of the challenge has something to do with the number of submissions. Logo competitions receive a lot of entries because they can be created in a matter of minutes. A phone design is much more complex and might be perceived as too time intensive. As a parallel, although I don’t know this to be true, I suspect that the $1 million InnoCentive challenge to find an ALS marker has had fewer entries than challenges with a $10K reward. The complexity to reward ratio must be considered.
  • Solver Base – Many of the people registered on crowdSPRING are logo/graphic designers. They may not be equipped to solve more complex problems. Although the 36K solvers might be great at logos, they may be less skilled at phone design leading to fewer submissions. This may account for why InnoCentive has such a high solve rate on technical challenges that are in their “sweet spot.” They have 180,000 highly technical solvers. However, if you posted a logo design on InnoCentive, I suspect you would have very few entries.
  • Awareness - Of course, awareness of the challenge must be factored in. If people are not aware of the challenge, they cannot respond. LG Electronics did a fantastic job of spreading the word. Having said that, I know of many people who would have entered, but unfortunately were not aware of the competition until it was too late.
  • Challenge Definition – There are other factors that need to be considered. For example, when I did a logo design competition on 99designs (a competitor of crowdSPRING) I had a few entries in the beginning. But as I provided feedback and people saw what I liked, the numbers of entries exploded. The fuzzier the challenge, the harder it is for people to get their hands around it. This is why InnoCentive invests A LOT of time helping frame the challenge before posting.
  • Intangibles – One thought I had is that incentives can sometimes be a barrier to innovation. A saw a fantastic comment on a discussion thread about crowdSPRING from yongfook. He said, “The quality of open source software is high, not because of crowdsourcing, but because of personal motivation to be involved with a project you have some emotional connection to. With (crowdSPRING) the motivation is more monetary… pretty much every professional designer I know wouldn’t touch any of these freelancing sites with a barge pole.” Interesting.

Ok, what do all of you think? What factors are we missing? Can you develop an accurate formula to predict participation in a crowdsourcing/open innovation challenge?

If you found this article useful or interesting, please press the "Like" button and post a Facebook comment below.

LG Open Innovation Insights

July 27, 2009

In a previous blog entry, I discussed how LG Electronics used crowdsourcing and open innovation to help design the next generation phone. One of the individuals at LGE associated with the competition was kind enough to answer some questions I had.

How many solutions were provided?
835 submissions from 324 individual creatives.

What % of the solutions were deemed “good?” (I was looking to get a sense of the “signal to noise” ratio)
I would say maybe around 25%. A lot of ideas had some “nugget” of insight that we can investigate or test further even though the whole concept itself was not new.

What was the process you used to evaluate and select?
We had 4 criteria that all judges scored on. Round 1 judges review all 800+ concepts and narrowed them down to the top 100 for Round 2 judges to review. Round 3 judges only reviewed the top 43 concepts.

What was the selection criteria?
The criteria was a weighted average of the following:

40%: Need Fulfillment

  • 1 point: There is no need for this type of product
  • 3 points: People like this idea but would not pay extra for it
  • 5 points: People want this idea and would pay to have it

30%: Creativity/Originality

  • 1 point: Many products like this already exist
  • 3 points: Idea gives a fresh viewpoint on an idea LG previously explored, but did not implement
  • 5 points: Idea is completely unique & has never been explored by LG or other manufacturers

20%: Feasibility

  • 1 point: Not feasible with any known existing technology
  • 3 points: Feasible, but some technology limitations make it difficult
  • 5 points: Idea is feasible without any major technology hurdles

10%: Polish & Appeal

  • 1 point: Idea & supporting graphics are hard to understand
  • 3 points: Idea is ok to understand but is missing usage scenarios
  • 5 points: Idea is easy to understand & has usage scenarios

This is a great example of open innovation in action. Thanks LG for sharing this information with us.

P.S. The winners of the competition are Rishi Bhalerao, Eric Staffin, and Susanne. Susanne was closest on the first two questions. Eric was closest on the last. And Rishi gets a prize for the most fascinating analysis.

If you found this article useful or interesting, please press the "Like" button and post a Facebook comment below.

Win A Prize! Guess the Open Innovation Numbers

July 23, 2009

In a previous blog entry, I talked about the LG Electronics open innovation design competition.

LGE was kind enough to share some insights with me from that competition, and I will post some of them here on Monday.

But before I do, I want to give you a chance to win some prizes (no, sadly you will not get an LG phone).

Here are the questions I want you to answer…

  1. How many submissions were made?
  2. How many individuals made those submissions? (i.e., if each individual only submitted 1 design, then the number of submissions would equal the number of individuals. However, some individuals submitted more than one entry, so the number of individuals is lower than the number of submissions.)
  3. Of the submissions, how many were selected by evaluators to move forward to the second round (FYI, there were 3 rounds of evaluation in total)?

A prize will be awarded to the first person who is closest to the correct number for each question.

For fun (no prize awarded, since it is subjective): According to my contact at LGE, what % of the ideas submitted were considered “good?”

Good luck!


  • You have a choice between a copy of my “Goal-Free Living” book, my “Little Book of BIG Innovation Ideas,” or a deck of Personality Poker cards (including the quick start guide and an online streaming video).
  • Due to shipping costs, only entries from the United States will be awarded physical prizes.  International winners will be sent a digital prize to be determined.
  • Submit your answers as comments to this blog entry.  Be sure to include your email address (in the area where it is requested so that it is published on the site).
  • You can submit multiple entries, but only your most recent entry will be accepted.
  • In the case of a tie, the first entry will be awarded the prize.
  • This competition will end at 5PM EDT Sunday July 26th.
  • As a point of clarification, there will be 3 prizes awarded, one for each question.  There will be, at most, 3 winners.  Individuals can win more than one prize if they are closest on more than one question.
  • Anyone associated with the LGE competition, or someone who already received this information from LGE or anyone else associated with the competition can not participate.  I just want to be fair :-)

If you found this article useful or interesting, please press the "Like" button and post a Facebook comment below.

It’s Official!

July 15, 2009
Innovative Logo

Innovative Logo

Two weeks ago, I jumped the gun a bit when I announced my new role with InnoCentive.  Now it is official.  Amongst other things, the press release issued today says…

With its new strategic consulting offering, InnoCentive will help organizations benchmark their open innovation readiness, develop custom innovation approaches and advise them on how best to engage in the cultural and process changes necessary to be successful. This in turn will give organizations the tools to make open innovation repeatable and predictable, like other capabilities in their business.

Timed with the announcement is a blog entry I wrote for the InnoCentive website.  Given my work on Innovation Personality Poker, I focused my thoughts on the different innovation styles.  Here are some of the thoughts from that article…

As I followed InnoCentive and observed the success of the Solver community, it got me thinking about how problems are solved.

From my research on innovation personality styles, I have observed that there are two broad ways of solving challenges: relational/creative and rational/analytical.

Steve Jobs, President of Apple Inc, once said, “Creativity is just having enough dots to connect . . . connect experiences and synthesize new things. The reason creative people are able to do that is that they’ve had more experiences or have thought more about their experiences than other people.”  This beautifully articulates how creative/relational problem solving works.

To date, most InnoCentive Challenges have been more technical in nature and are typically solved through a more rational/analytical approach.

When looking at these two styles, I realized that InnoCentive Solvers are perfectly positioned to use BOTH innovation styles.  To explain why I believe this, let’s explore what happens in the brain when solving Challenges…

You can read the rest of the article on the InnoCentive blog.

I am thrilled to (now officially) be working with an amazing company that is changing the world one challenge at a time.

P.S. My other title is “Chief Innovation Evangelist.” This links to my quote in the press release where I say, “InnoCentive already has a strong reputation for open innovation. My role will be to expand this reputation to encompass innovation in general.”  UPDATE: The official name for my group is “InnoCentive Consulting: The Architects of Challenge-Based Innovation.”

P.P.S.  As mentioned in my earlier blog entry, I will continue to give speeches on innovation and write books.

If you found this article useful or interesting, please press the "Like" button and post a Facebook comment below.

Crowdsourcing = Millionaire Lifelines?

July 8, 2009

During dinner last night, I compared crowdsourcing to the lifelines on “Who Wants to be a Millionaire?”

Imagine you are sitting in the hot seat. The show’s host asks you a question. You are nervous and can’t think straight. You believe you know the answer to the question, but $64,000 is on the line. You are no longer that sure of yourself. You have all of your lifelines. What do you do?

A. Answer the question on your own.
B. Phone a Friend
C. Use the Fifty-Fifty
D. Ask the Audience

Let’s explore each option…

You could of course answer the question on your own (A). You answered the previous 10 questions correctly doing it this way. But the stakes are higher now. Maybe it’s time to get some help.

You could “Phone-A-Friend” (B). Based on the few times I watched the show (admittedly the last time I watched WWTBAM was back in 2001 when was living in England), this option rarely proved reliable. Of course, if your friend has access to Google and can search quickly, then it might be the best option. But if relying solely on brain power, this option is typically a dud. One extra brain helps little.

You could go for the “Fifty-Fifty” (C). Here, two incorrect answers are eliminated, leaving you only two choices. This is a great option. Unfortunately, according to the all-knowing Wikipedia, “the answers eliminated were not random but were pre-selected as the ones the contestant was least likely to pick.” Well, that makes things a bit harder.

You could “Ask the Audience” (D). According to Wikipedia, “This is a popular lifeline, known for its near-perfect accuracy. (Regis) Philbin once said that the audience’s answer is statistically 95% of the time correct.” However, my research shows that for more difficult questions, the audience is often clueless. The problem is, everyone answers rather than only those who know the answer. This creates a lot of invalid noise.

In the current version of the show, there are other lifelines, including “Ask the Expert.” You can guess what this is.

Ok, now, imagine you are in the hot seat at work. You are working on a pressing challenge. You think you know the answer. Or maybe you haven’t a clue how to solve it. Regardless, how do you want to bet the company’s money?

Lifelines in Business

You could Phone-A-Friend. That is, you could ask the people around your desk at work. This would most likely improve your chances. (and no, your friend can’t use Google to solve real world problems)

You could do the equivalent of Asking the Audience by requesting, for example, customer feedback. MyStarbucksIdea simulates this concept. Unfortunately, as with the WWTBAM audience, you get a lot of “noise” in the responses because everyone wants to be heard.

You could Ask the Expert. To do this you might partner with a University or hire a consultant. Or maybe you outsource the challenge to a 3rd party who takes responsibility for solving the challenge.

Or maybe there is a hybrid solution. Call it the “Ask the Audience of Experts.” This involves posting a challenge to a group of highly skilled experts; people who have a higher likelihood of solving your problems. You get the viewpoint of many people (much better than Phone-A-Friend), but you also eliminate the “noise” associated with a generic audience. This gives you the added bonus of using a “Fifty-Fifty” lifeline because you end up with fewer “wrong” answers.

Solving challenging problems requires a wide variety of techniques. No one technique is universally correct.

As far as I know, no one has ever won the million dollars on the TV show without using their lifelines. Why should your organization try to win its millions without using the best lifelines available to you?

So, what’s the correct answer? E – All of the above. And yes, that is my final answer.

Follow me on Twitter

If you found this article useful or interesting, please press the "Like" button and post a Facebook comment below.

How To Create a Culture of Innovation

July 6, 2009

I define innovation as an “organization’s ability to adapt and evolve repeatedly and rapidly to stay one step ahead of the competition.” A culture of innovation, when done right, gives you a competitive edge because it makes you more nimble with an increased ability to sense and respond to change.

A culture of innovation has less to do specifically with new products, new processes, or new ideas. There are of course discrete innovations such as the iPhone or a battery that is powered by viruses (MIT has developed this). These are valuable and necessary in order to create a culture of innovation.

But a culture of innovation is more than new ideas. It needs to be repeatable, predictable, and sustainable.  This only happens when you treat innovation like you treat all other capabilities in your business.  This means having, amongst other things, a defined process.

An organization’s innovation process must achieve three things.  It must:

  • focus on the “right” challenges
  • find appropriate solutions to those challenges, and
  • implement the best solutions.

These translate into three “portfolios” an organization must create:

  • A portfolio of challenges
  • A portfolio of solutions
  • A portfolio of projects

Let’s take each one at a time.

A Portfolio of Challenges

All companies have challenges. They can be technical challenges on how to create a particular chemical compound. They can be marketing challenges on how to best describe your product to increase market share. They can be HR challenges around improving employee engagement.

An organization’s ability to change (i.e., innovate) hinges on its ability to identify and solve challenges. Challenges are sometimes referred to as problems, issues, or opportunities. But at the end of the day, they are all just various forms of challenges. I will use these terms interchangeably here.

Where do you find these challenges? You can find them anywhere – from customers, employees, shareholders, consultants, vendors, competitors, and the list goes on.

Let’s face it, companies have no shortage of challenges.

And guess what, some of the most important challenges to solve are hidden due to organizational blind spots and assumption-making.

The “meta-challenge” for all organizations is to find which challenges, if solved and implemented, will create the greatest value. Given that organizations have limited resources and money, prioritization is critical.

My favorite quote (used many times in this blog) comes from Albert Einstein – “If I had an hour to save the world, I would spend 59 minutes defining the problem and one minute finding solutions.” Most companies spend 60 minutes of their time finding solutions to problems that just don’t matter.

Therefore, the first step in creating a culture of innovation is to surface, identify, and codify challenges. And then you must become masterful at valuing, prioritizing, and framing these challenges.

Think of your innovation portfolio much like you would handle a financial investment portfolio.  You want some safe bets (incremental innovation) and some riskier investments (radical innovation).  You also want a variety of innovations ranging from technical challenges to marketing challenges, and service challanges to performance improvement challenges.

Once you have the right challenges to solve, the next step is to find solutions.

A Portfolio of Solutions

[Read more]

If you found this article useful or interesting, please press the "Like" button and post a Facebook comment below.