Webinar: Effizienz und Performance in der Produktentwicklung messen

Good afternoon, everyone. Today, I'd like to talk with you about several important subjects in the measurement of R&D.

Five sections to today's talk.

1. I'd like to step back for a little bit, and look at the big picture of what's happened over the last 30 or so years, and where we are now;
2. Then to move on and select a couple of the most prominent metrics that have occurred in our profession in the last 30 years.
3. Move on to looking at how metrics should be aligned with R&D strategy, and those result in very, very different metrics depending on whether you're an innovator, you run a balanced portfolio, you run a platform derivative environment, or you have a strategy of being a fast follower.
4. Without regard to which strategy you have, there's usually more in the pipeline than one can properly execute, so we're gonna spend a couple of minutes talking about hurdle rates, decisions, and risk,
5. And then pull it all together in looking at an R&D metrics portfolio that emphasizes the majority of the places that people spend money.

The Big picture of the past 35 years

In the 1970s, there was the beginning of the Asian competition: Toyota, Japan in general in the automotive industry, and that caused us to need to take a serious look at how we develop products. Lower cost, higher quality products took over the consumer electronics industry, the automotive industry, the machine tools industry, the robotics industry. And since that period of time, we've been investigating how to be better at executing and innovating in R&D.

Building an end-to-end process

The very first place that we focused, in the 80s, was going over the wall from engineering to manufacturing: the hand-off the exchange, the camaraderie between R&D and product development. As we began to get into this science, we understood, through some work at UC Berkley and elsewhere, Hewlett Packard, IBM, Siemens, Philips all collaborated, and we began to understand the importance of product definition. Some 70% of everything that goes wrong in product development, can be traced back to product definition.

With that new phase on the front-end of the stage-gate process, we realized that there were these ideas that were floating around in companies, for a long period of time, ala the 3M Post-It notepad, where a great idea languished for the lack of a process, for almost seven years before it got put in front of decision-makers. And so we then went and formalized the very front end of the process: the product concepting phase.

With the front end intact and the middle taken care of, over the wall, we now moved on to talk about commercialization. Products were being thrown over the wall to sales, and the term "hockey-stick forecast", where there were supposed to be significant revenues shortly after launch, we realized there were these long, flat periods before sales actually began. Enter post-launch reviews, and an overlap of the sales function and marketing organizations in the back half of the product development process.

Around the early 2000s, we now had, in most of Western civilization, an end-to-end process. And since everybody had that, we needed to look for the next basis to compete. The next basis to compete was: those companies that had a stronger portfolio of goods that they could push through that process, began to win.

Moving to competition on technologies

Five, six years pass, everybody's up to speed on portfolio, all competing on basket-of-goods, and now, what was the next great competitive advantage? It was technology push, where those that could invent the new-to-the-world technologies and/or adapt the leading technologies into their product line, began to get advantage.

Once the competition on technologies began to take place, of course intellectual property became important to protect it. And so, all of a sudden, where IP had been a separate department in other organizations, we began to see the integration of IP into the product development process, and people working side-by-side, and the importance of locking-up and protecting your IP in advance of launch.

The Age of R&D metrics

Fast-forward, the internet of things is coming of age, and that's where we sit at the current time. All of a sudden now, we go forward to today, and there's two, three, four, 500 metrics that are out there, that are all viable measurements, KPIs, indicators of performance, and it's become very much more challenging to choose the ones that are going to optimize your specific strategy.

It's interesting though, that to this day, there is not one single metric in the product development arena that one can say has one-to-one correlation with an outcome. And so, as a result of having metrics that are not accurate predictors in R&D, a much more probabilistic place to be earning a living than in manufacturing or logistics, or any of the transaction processing functions; you can see that, in spite of the plethora of metrics, that there are only a handful, five, six, seven, that have penetrated more than 40% of industry consistently. So when one looks at today's R&D organizations, or tries to benchmark across companies, very, very few metrics are in common.

As big data and analytics grow our accuracy, and our ability to correlate, we will find that there will be a more select few that, once shown to have correlations with outcomes, will start to permeate industry on a wide basis, in R&D. This, in contrast to manufacturing, where if you looked at the metrics, most folks in operations in manufacturing companies, you'd find that 80% of those are in common; it's more, for R&D, like 20% of those are in common.

So our science is still very young, even though we have been talking about it for a long period of time, and that all has to do with the probabilisticness of R&D. So metrics are going to continue to grow. With correlation, we'll then be able to cut back, 20 to 30 years from now.

One of the things that's happened in the last 15 or 20 years, is that companies used to spend a lot of their R&D monies on new-to-the-world, new-to-the-industry, new-to-the-market products. This is some work from Robert Cooper, who is the creator of Stage-Gate, and also one of Moon-on-a-Stick's presenters, and you can see that Bob indicates, research indicates almost 70% of R&D budgets 15 years ago, and even into the mid-2000s, were on "new-to". In contrast today, where we have less than half of that amount dedicated to "new-to"; we're doing lots of baked-over, incrementals, extensions, without really reaching very far. As the Great Depression fades into the background, companies will be getting additional courage to up the riskiness of their portfolio, and that will bring an increased emphasis on metrics that capture innovation, capture newness, or capture the protection of intellectual property, of revenue and profit streams.

Looking at some of the most prominent metrics

And so, this is really a good time for companies to take a good, hard look at measurements they have been using to get through the last eight or nine years, in anticipation of more positive and bigger growth, in the years to come.

Boston Consulting Group began the big picture, did some research, a 10-year look at the three and 10-year annualized shareholder returns for innovative companies. And I think that one of the reasons that Sean has invited me to come over and speak with you all is the recognition that European innovators, right now, are under-performing all global competitors. And so hopefully, at least in our small way, with the folks that would join us, we can help to bring about some positive change and competitive advantage there. Another major thing that's important for measurement is the difference in perceptions of the people that are on the point. 

Managers of R&D are only 30 and 40% satisfied with the indicators that they have, whereas the chief executive officer and top management is almost twice as satisfied. So with a good set of measures, you can really bring people's perceptions together more closely than they are, historically.

Now let's take a look at a couple of metrics. As we mentioned, there's been a growth in metrics, and I would like to start with the fastest-rising metric in my lifetime, in research and development; it did not exist in 1988.

The Vitality Index

Most people are familiar with 3M's renowned innovative ability, and so 3M was trying to find a metric that could talk about the newness of its portfolio, the vibrancy of its portfolio, and it invented a metric called the vitality index, in 1988. This metric caught fire; it went from not existing, to now being the number-three metric in use, in R&D, in the world, or at least in Western civilization, some 27 years later. 

The vitality index measures the revenues from your new products, and divides it by the expenditures to create, pardon me, divides it by the total company revenue. So what you get is, you get a measure of the percentage of your sales that come from new products. It goes by lots of different names: new product sales, new revenues; vitality is the source of all of those permutations of the wording. 

The big choice is, how many years make a new product? Is a new product new one year into the marketplace? Most would say yes. Two years into the marketplace? Yes. Three years? Yes. And after about three years, there are not a whole lot of companies that use four or five years, but there are a few. Those that need to get spec'd into environments, or need to wait for the next budget year, and like government, and defense contracts and such. And so, 3M is one of those companies.

This is an internal memo that I have permission from 3M on, and this is the work of George Buckley, who came in to resurrect 3M's innovative ability. And so what you have, is you have his forecast, in 2011 and before were actuals, and the projection 2012 and forward, and I can't show you anything newer than this. But there was a clear effort on 3M's behalf, to attempt to grow the percentage of its total sales that are due to new products. And you can see, in the second line from the top, under the blue bar, within the past five years. So N equals five, at 3M. More than five years, it's not a new product.

Return on Innovation

During this period of economic challenge, another new metric came up, and it is kind of a wave that is currently happening: productivity and efficiency metrics.

And so, unlike the vitality index, where you're not dividing by expenditures, output over input; the return on innovation metric is a productivity or efficiency measure, classical industrial engineering output over input. And so, the cumulative profits, the return that you get from the investment is your return on innovation.

This metric was created around 2001 or 2002, and next to the vitality index, it is the second-fastest rising metric in industry. It, right now, has penetrated per my own company's research, and we'll show you that at the end of the presentation, it has now penetrated 32% of companies. Given the stability of the way that R&D measures itself over the years, these are meteoric rises for the vitality index and for return on innovation.

In fact, they're such good indicators that companies are now putting them in their annual reports. Actually, in more prominent places than places that they are supposed to be putting their generally accepted accounting principle metrics, which are required by law. The situation has gotten so bad, that the Securities and Exchange Commission in America has issued guidelines now, to keep companies from reporting these non-GAAP metrics more prominently, because they're thought to be misleading investors, and if they're not misleading investors, they're certainly not providing apples-to-apples comparisons such as the GAAP measures do. So, this is going to be an ongoing tussle, and most folks, I did a job with the London Stock Exchange Company a couple years ago, and the folks there told me that the primary interest of all of the analysts was non-GAAP metrics, and so they really wanted to beef-up their ability to communicate them. We'll go through all of this in October.

Effectiveness vs. efficiency

Closing for selected metrics: getting back to this effectiveness versus efficiency.

This is our New England Patriots, American football team, the year that they won one of their Super Bowls, back in the early 2000s, and I cannot get USA Today to do another rendition of this diagram, but if you look in the upper-right, and at the two axes, you can see that the New England Patriots, the year that they won, did the best overall job, upper right-hand corner, of scoring touchdowns when they were in the red zone, and in preventing touchdowns from the other team in our own red zone. And so, they were ultimately the most effective team.

What would be efficiency? Efficiency would be to take the payroll of the team and divide it into the result. Most companies would rather have effectiveness than efficiency, because effectiveness brings with it brand value, recognition, market leadership, increase in market cap[italisation], price premiums on products.

And so the current direction during the recession of moving away from effectiveness metrics towards productivity and efficiency metrics, where you're constantly trying to decrease the denominator, is probably not going to be the winning set of measures, as the economy improves, and GDPs grow at an increased rate again.

How metrics should be aligned with R&D strategy

Every company has its own strategy; some are much more to the research side, some are much more to the development side, and everything else in between.

Your measures, you get what you measure, so your measures should closely reflect your strategy. So in most exercises that we do with groups and with companies, they need to decide what their actual strategy is. And you'd be surprised how difficult a question that is, to get started.

We generally group them into the four different groups: Innovator = you prepared to fail big time, and have a few successes that are very big-selling products. Platform derivative companies: platform derivative continued to increase in popularity, became the number-one strategy in R&D, up until the beginning of the Great Depression, and then, or Great Recession, and then folks started backing off to a more balanced portfolio; they could not afford the period of time where the new platform was launched but the derivatives were not out yet. And so, they needed more quick, faster ROI, and people reverted back to being balanced portfolio companies.

So with that as the lay of the land for strategy, we find that innovation, and this is my own company's research, we do a study every three or four years of practices in R&D. We can see that there's been a great increase over what might have existed 20 and 30 years ago, where only pharmaceutical companies, large material companies, etc. would be doing a lot of applied research or basic research, and very little advanced development. Everything used to be done in the product development process.

Now, innovation is spread out much more along a continuum, and in addition to that, there are side groups, where folks are creating innovation skunkworks organizations for the purpose of producing one-off new-to-the-world, and then the organization just ends.

And so, measurement now is not just about product development; one needs sets of measures that get at the uniquenesses and the refinements of each of these very different R&D goals.

Failure rates: the great opportunity for companies

No matter what we've been doing, failure rates have not improved in 50 years. If you looked at the work from Arthur D. Little, in the 70s, work by John Trudell in Electronic Design Magazine in the 80s, work by Booz Allen in the 90s, work by Cooper in the 90s and 2000s, basically, 44% on average across industry, of products, fail to meet profit objectives. And so it doesn't matter whether we post slides from the 70s, 80s, 90s, or 00s, we're still failing at about the same rate.

And therein lies the great opportunity for companies to break out, if they can somehow manage their pipeline more effectively, to put more winning products in, and execute them more accurately, and to raise the yield of their pipeline.

Consumer and high-tech products, as recently affirmed by Nielsen just last June and in a survey the prior June, the consumer and high-tech companies fail at 85%, 81%, 86%, and so, no matter where you look, and you would expect that those "new-to", the more innovative products would fail at a higher rate. But the whole idea of yield of the product development pipeline, is a great opportunity, THE great opportunity for companies, going forward.

Doing the math with failure rates

This is linear equations of two different reports that looked at how many concepts get into the pipeline. You can see a combination, at the bottom, 21 ideas yield 12 approved products, four products die in the middle of the pipeline without getting launched, so it's that eight launch, and of those eight, only three achieved success. So basically, out of all of the work that companies do, it's those three products that create 100% of your new product revenues.

Now, getting the acumen to manage and make tougher decisions, improve the hurdle rates, somehow get a more winning basket of goods in; if we could just tweak, since the three make 100% of your revenues in new product revenues and profits, if we could just get that number to be a four instead of a three, that would be a 25% increase for your company, in new product revenues, just by better management.

So the opportunity's here, there's very few places you can get a 25% increase in new product revenues, this is one of 'em, and it is a focus of measures. And in order to potentially get an improved yield, and this makes the point that it's the 25% that make the 100% of revenue and profit.

Finding the right predictive metrics

We've created a series of metrics that are unique to the stage-gate pipeline process, that start to get at the different things that can be improved, as one moves earlier in the process. The whole idea would be to get indicators that somehow better-correlate in mid-project with the outcomes that are being intended. 

And so, we've created a framework of planning, proactive, predictive, and then once you go to physical form of a product, and rapid prototyping is graying this a little bit, but once you've got a physical form of the product, you're really asking the question: did the as-built model conform to the spec? And if it's no, you're really in a fix mode instead of a create mode.

Again, rapid prototyping is graying the lines a little bit, but come this October, we'll spend a good amount of time looking at those indicators, four or five in each of these areas, that can potentially give you a better ability to predict what's happening to your project, and take earlier corrective action as opposed to waiting until it launches and find out that it's not quite what you thought it would be.

Reinvestigating hurdle rates

Another thing that can be done is to reinvestigate hurdle rates. Right now, this is some work out of the Economist group in the UK. If one goes down two-thirds of the way, many CFOs are keeping increasingly high hurdle rates on things in anticipation of the economy coming back. And the Economist makes the point that there's potentially a chance to take a look at the weighted average cost of capital, and the hurdle rates that many CFOs are using these days. I don't disagree with that. Companies kind of, as they grow, get into a sort of complacency, where the product, when they were a smaller company a couple of years ago, that was success back then is no longer got enough revenues and profits in order to make it a success in the new, larger company.

And so, by managing your hurdle rates and the filtering of that at the front-end of the process, you can actually force certain of the proposals to come up with an increased bang for your R&D dollar. So we'll be talking about how one can spend time doing a little bit additional market research, customer research, in order to boost the potential sales of any new development effort, improving your business case at the beginning.

Trade-off analysis

When you get in the middle of the process, you now need to understand whether getting to market on time is your most important thing such that you would sacrifice a bit of budget in order to get to market on time, whether you would decide to launch at a higher than targeted product cost, maybe squeezing your margin a little bit at launch, but at least you're in the market.

And so, the trade-off analysis, six fundamental trade-offs are another tool that is very helpful in helping to assure an increased success rate out of your pipeline.

All of this comes back to risk. Risk is, as everybody knows, is a very complex subject and so, most companies don't do it well and there's an opportunity through metrics to improve risk mastery, which will also result in increased yield in your pipeline.

An R&D metrics portfolio

Since 1988, we've been researching, my company, the metrics that have the highest penetration in industry. Our last effort, the year before last, researched 101 metrics, and companies responded. And so, you can see here that there's a pareto order. When we get to the workshop in October, we will be talking about all 100 metrics. 

You can see, the number-three metric from the top is that vitality index, that came from nowhere to now be the number-three metric. In reality, it's actually the number-one performance metric. There are metrics and there are metrics. The top two metrics are more like the accounting metrics, just status-based, they don't tell you anything about your performance. So the vitality index is now the number-one performance metric in R&D. You have number of patents filed; number of new products released, you can call that a performance metric. And then look at number 10; out of nowhere, in the last 15 years, return on innovation. And so we'll be going through the hundred metrics.

During the Great Recession, the reorientation of these metrics was the most pronounced at any point in my professional career. R&D used to enjoy a fairly protected sandbox, where it didn't really have to report a whole lot on its actual performance, and we'll be going through a color analysis here, to show where the business performance metrics have really come to the forefront unlike any time in history, in R&D, in the last eight years. And that certainly is going to continue going forward.

How many metrics should you have?

It's very important that you define a set of metrics. If you do not define a set of metrics, and this is a half an hour of what we'll be talking about in October, you're going to end up with twice as many measures as you, and the expense to monitor them and report them, and discuss them, and meeting time, and so it's very important for companies to choose a clearly-defined set of metrics. Now, that doesn't mean you have to stay with it forever, but unless you have a set of metrics, you have twice as many metrics. 

The uniqueness of R&D

After examining the balanced scorecard, I had the privilege of introducing Robert Caplan when he first announced that in the 1990s. I was also on the agenda in that conference. I got to looking at the methodologies out there, the spider diagrams, the balanced scorecards, several other frameworks, and they really don't get at the uniqueness of R&D. R&D is not a transaction processing environment; it's a project-based environment. 70% of all monies go into projects, you make a bet, a little investment and a time on each project, and so the focus on project measures is absolutely essential for research and development.

It's also important because products do not get invented by equipment; it is people that invent products, and the expertise and competencies that they have, and the freedoms or not that they're given to experiment and innovate. And so, the focus of measures on people, and functions, and departments, and competencies, is extremely important.

Introducing the Linked Metrics Portfolio

So, I created this linked metrics portfolio, we have over a hundred implementations of this.

This is a platform derivative company. You can see just underneath the blue bar, innovator extender in an electromechanical environment. And after looking at this methodology, and working in the workshop, this company decided that in order to really be able to get five levels deep into the organization, and then pass up a handful of metrics to the CEO to look at R&D in aggregate, that it was 44 metrics that they needed. 44 may seem like a lot, but when you do not have any one metric that correlates one-to-one with an outcome, it really is a very reasonable number.

And so, as we work through the October two-day session, we will be breaking into teams and each team will be picking its desired strategy; Innovator, Platform, Derivative, Balance, the Fast Follower; and we will construct a unique set of metrics for each of the teams, or the teams will construct a unique set of metrics for each of their strategies. And we'll have a report out at the end of the summit, where each team presents the metrics that it's chosen and explains how those metrics tie to strategy.

So with that, the linked metrics portfolio will underlie two days. We'll be going through almost four or 500 metrics in two days, including NPD, efficiency, R&D productivity, return on capital which is shown to have the highest correlation with shareholder returns out of R&D, economic value-added, break-even time, time to profit, the risk mitigation index, all the TRL, MRL, system readiness level, people readiness level, things like DARPA's hard test for innovation. We will make sense of a large number of metrics and put them so that they're very easy to understand and choose from.

And with that, I thank you for your time, and let's go to Q&A. 

Q&A

- [Sean] Brad, that was great, thank you very much indeed. You mentioned that obviously there's a selection that you make at the beginning, which is what sort of company do you want to be. Do you want to be a market leader, a fast follower, a technology-led, or et cetera, et cetera? Then that defines the metrics that you then follow. Do I then assume that that then correlates all the way down into your phase-gate process? So, the questions that you would ask at the different stages of each gate of your phase-gate would be different, that they then correlate back up to your original strategy?

- Part yes and part no. There are certain metrics that are fairly standardized, that one needs to oversee any investment, and those metrics will not change significantly with your strategy. So, certain project execution metrics and certain people performance metrics, will be fairly consistent across strategies. Recall that approximately 30 to 40% of the metrics, the more innovative your company is,  the smaller that percentage will be. Then there are metrics that are truly, at the overall level, relating to your portfolio, the choice of products, the newness of them, your ability to protect them with intellectual property. IP is expensive, and so you do not always decide that you wanna invest that money. The ones with the innovator strategies will be more interested in doing that. And so, there are a base set of metrics to run a department, plus or minus five or 10% on those. And the rest really do come from what you want your company to be and how it needs to fit into your corporate business strategy.

- [Sean] Do we have a view on metrics that you require for regulated environments such as a utility company, where the rules on product quality are roughly constant, and your innovation focus is on how you better process those raw materials, rather than a sort of more straight-forward product, an R&D to launch a product type process?

- Regulatory metrics pervade just about every company, from worrying about lead and other things, in electronic assemblies, to chemical industries and material safety data sheets, the people that run water supplies and what type of equipment will result in keeping the water quality at where it's supposed to be. That would be my first point, that that's not usually a strategy; that's more of like a good hygiene for some of the standard project management metrics, that you need them just to block-and-tackle and conform. The interesting thing, now thinking pharmaceutical industry, and the 510medical device exemptions, medical products, diagnostic medical instruments, where you have a period of development and then you go into a period, pardon me for phrasing it this way, I've worked in many of these industries, but it is my view, where you're now in a validation mode, in clinicals, or 510's or investigational device exemptions, and such. And so, there still is some development work that, especially in the IDEs, that goes on. But mostly, you're trying to say, with the regulatory metrics: does the model conform to the requirements that we, the product conform to the requirements we have to conform to? So certainly, regulatory metrics gonna be a part of most company's metrics portfolios. In some cases, they will be strategic; in most cases they will be compliant.

- [Sean] Okay, that's great. But I'm assuming you can take some of the standard ones and maybe re-work them a little if you don't have, for instance, sales as a key thing. Maybe you could add in, you know, time to develop, or reduction in penalties, or something like that, to actually potentially re-work one of the existing metrics.

- Not quite sure I'm catching the intent. If you are an R&D company, and you are not producing products for sale, but you are producing enabling technologies that can then be adopted by others, or sold into larger, existing systems that would augment capabilities of those systems. Certainly, we don't wanna be talking about revenues where revenues are not relevant. We would attempt to come up with metrics that would get that efficacy, the ability of the product to improve the environment of the customer.

- [Sean] Brad, that's great. Thank you very much indeed. Thank you very much for your time, thanks very much to Brad.