The Discipline of One Metric That Matters chapter 6
Founders are magpies, chasing the shiniest new thing they see. They often
use the pivot as an enabler for chronic ADD, rather than as a way to iterate
through ideas in a methodical fashion.
But one of the keys to startup success is achieving real focus and having the discipline to maintain it. You may succeed if you’re unfocused, but it’ll be by accident. You’ll spend a lot more time wandering aimlessly, and the lessons learned are more painful and harder-won. If there’s any secret to success for a startup, it’s focus.
Focus doesn’t mean myopia. We’re not saying that there’s only one metric you care about from the day you wake up with an idea to the day you sell your company. We are, however, saying that at any given time, there’s one metric you should care about above all else. Boiled down to its very essence, Lean Startup is really about getting you to focus on the right thing, at the right time, with the right mindset.
As noted in Chapter 5, Eric Ries talks about three engines that drive company growth: the sticky engine, the viral engine, and the paid engine. But he cautions that while all successful companies will ultimately use all three engines, it’s better to focus on one engine at a time. For example, you might make your product sticky for its core users, then use that to grow virally, and then use the user base to grow revenue. That’s focus.
In the world of analytics and data, this means picking a single metric that’s incredibly important for the step you’re currently working through in your startup. We call this the One Metric That Matters (OMTM).
The OMTM is the one number you’re completely focused on above everything else for your current stage. Looking at CLV (customer lifetime value) isn’t meaningful when you’re validating a problem, but it might be the right metric to focus on as you’re approaching product/market fit.
You’ll always track and review multiple numbers. Some will be important: these are your key performance indicators (KPIs), which you’ll track and report every day. Others will be stored away for future use, such as when it’s time to tell the company history to an investor or to make an infographic. Setting up and managing instrumentation is fairly easy these days with tools like Geckoboard, Mixpanel, Kissmetrics, Totango, Chartbeat, and others. But don’t let your ability to track so many things distract you. Capture everything, but focus on what’s important.
case study
Moz Tracks Fewer KPIs to Increase Focus
Moz (previously known as SEOmoz) is a successful Software as a Service (SaaS) vendor that helps companies monitor and improve their websites’ search engine rankings. In May 2012, the company raised $18 million. Its CEO, Rand Fishkin, published a detailed post about the company’s progress up to that point.* Rand’s update did include a number of vanity metrics—when you have roughly 15 million visitors on your site each year, you have the right to a bit of vanity—but he also shared some very specific and interesting numbers related to conversions from free trials to paid subscriptions and churn.
We spoke with Joanna Lord, Vice President of Growth Marketing at Moz, to learn more about how the company handles metrics. “We are very metrics-driven,” she says. “Every team reports to the entire company weekly on KPIs, movement, and summaries. We also have a huge screen up in the office pumping out customer counts and free trial counts. We believe that having company-wide transparency into the metrics keeps us all informed, and is a great reminder of the progress (as well as the challenges) we are seeing as a company.”
For a company that’s found product/market fit and is now focused on scaling, it becomes more challenging to focus on a single metric. This isn’t surprising; there are multiple departments all growing quickly, and the business can tackle several different things simultaneously. But even with all these concurrent efforts, Joanna says that one metric stands above the rest: Net Adds. This metric is the total of new paid subscribers (either conversions from free trials or direct paid signups) minus the total who cancelled.
“Net Adds helps us quickly see high cancel days (and troubleshoot them) and helps us get a sense of how our free trial conversion rate is doing,” Joanna says.
Moz tracks other related metrics including Total Paying, New Free Trials Yesterday, and 7-Day Net Add Average. All of these really bubble up into Net Adds per day.
Interestingly, when Moz raised its last round of financing, one of its lead investors, the Foundry Group’s Brad Feld, suggested that it track fewer KPIs. “The main reason for this is that as a company, you can’t simultaneously affect dozens of KPIs,” Joanna says. “Brad reminded us that ‘too much data’ can be counterproductive. You can get lost in strange trends on numbers that aren’t as big-picture as others. You can also lose a lot of time reporting and communicating about numbers that might not lead to action. By stripping our daily KPI reporting down to just a few metrics, it’s clear what we’re focused on as a company and how we’re doing.”
Summary
• Moz is metrics-driven—but that doesn’t mean it’s swimming in data. It relies on one metric above all others: Net Adds.
• One of its investors actually suggested reducing the number of metrics the company tracks to stay focused on the big picture.
Analytics Lessons Learned
While it’s great to track many metrics, it’s also a sure way to lose focus. Picking a minimal set of KPIs on which your business assumptions rely is the best way to get the entire organization moving in the same direction.
Four Reasons to Use the One Metric That Matters
The OMTM is of most importance early on. Later, as your startup scales, you will want to focus on more metrics, and you’ll have the resources and experience to do so. Importantly, you’ll also have a team to whom you can delegate metrics. Your operations person might care about uptime or latency, your call center might worry about average time on hold, and so on.
At Year One Labs, one of the litmus tests for us as advisors and investors was the clarity with which a team understood, and tracked, their OMTM. If it was on the tip of their tongues, and aligned with their current stage, that was a good thing. If they didn’t know what it was, if it was the wrong metric for their stage, if they had several metrics, or if they didn’t know what the current value was, we knew something was wrong.
Picking the OMTM lets you run more controlled experiments quickly and compare the results more effectively. Remember: the One Metric That Matters changes over time. When you’re focused on acquiring users (and converting them into customers), your OMTM may be tied to which acquisition channels are working best or the conversion rate from signup to active user. When you’re focused on retention, you may be looking at churn, and experimenting with pricing, features, improving customer support, and so on. The OMTM changes depending on your current stage, and in some cases it will change quickly.
Let’s look at four reasons why you should use the One Metric That Matters.
• It answers the most important question you have. At any given time, you’ll be trying to answer a hundred different questions and juggling a million things. You need to identify the riskiest areas of your business as quickly as possible, and that’s where the most important question lies. When you know what the right question is, you’ll know what metric to track in order to answer that question. That’s the OMTM.
• It forces you to draw a line in the sand and have clear goals. After you’ve identified the key problem on which you want to focus, you need to set goals. You need a way of defining success.
• It focuses the entire company. Avinash Kaushik has a name for trying to report too many things: data puking.* Nobody likes puke. Use the OMTM as a way of focusing your entire company. Display your OMTM prominently through web dashboards, on TV screens, or in regular emails.
• It inspires a culture of experimentation. By now you should appreciate the importance of experimentation. It’s critical to move through the build→measure→learn cycle as quickly and as frequently as possible. To succeed at that, you need to actively encourage experimentation. It will lead to small-f failures, but you can’t punish that. Quite the opposite: failure that comes from planned, methodical testing is simply how you learn. It moves things forward in the end. It’s how you avoid big-F Failure. Everyone in your organization should be inspired and encouraged to experiment. When everyone rallies around the OMTM and is given the opportunity to experiment independently to improve it, it’s a powerful force.
case study
Solare Focuses on a Few Key Metrics
Solare Ristorante is an Italian restaurant in San Diego owned by serial entrepreneur Randy Smerik. Randy has a background in technology and data, once served as the general manager for business intelligence firm Teradata, and has five technology exits under his belt. It’s no surprise that he’s brought his data-driven mindset to the way he runs the business.
One evening at the restaurant, Randy’s son Tommy—who manages the bar—yelled out, “24!” Since we’re always looking for stories about business metrics, we asked him what the number meant. “Every day, my staff tells me the ratio of staff costs to gross revenues for the previous day,” he explained. “This is a fairly well-known number in the restaurant industry. It’s useful because it combines two things you have a degree of control over—per-diner revenues and staffing costs.”
Randy explained when staffing costs exceed 30% of gross revenues, that’s bad, because it means that you’re either spending too much on staff or not deriving enough revenue per customer. A Michelin-starred restaurant can afford to have more staff, and pay them more, because it sells customers expensive wines and enjoys good per-customer revenue. At the other end of the spectrum, a low-margin casual dining restaurant has to keep staff costs down.
The ratio works because it’s:
• Simple: It’s a single number.
• Immediate: You can generate it every night.
• Actionable: You can change staffing, or encourage upselling, the very next day, whereas ingredient costs, menus, or leasing take longer to modify.
• Comparable: You can track it over time, and compare it to other restaurants in your category.
• Fundamental: It reflects two basic facets of the restaurant business model.
As it turns out, 24% is about right. Below 20%, there’s a chance that you’re under-serving customers and that their dining experience might suffer (Randy could experiment with different staffing levels and measure the tips diners leave, or comments on Yelp, if he wanted to be really analytical).
Randy also uses a second metric to predict how many customers he’ll have. At 5 p.m. every day, his staff sends him the number of reservations that have currently been made for the evening. “If I get 50 reservations at 5 p.m., I know I’ll have around 250 covers that night,” he says. “We’ve learned that a 5-to-1 ratio is normal for Solare.”
This number doesn’t work across all restaurants—the in-demand Michelin-starred restaurant has a 1-to-1 ratio, since it’s sold out, and a fast food restaurant that doesn’t take reservations obviously can’t use the metric. But for Solare reservations at 5 p.m., plus some experience, provides a good leading indicator of what the night will be like. It also allows the Solare team to make small adjustments to staffing or buy additional produce in time to ensure that the restaurant can handle the traffic.
Summary
• Restaurants know from experience that demand is tied to reservations, and what the right ratio of staffing to revenue should be.
• Good metrics help predict the future, giving you an opportunity to anticipate problems and correct them.
Analytics Lessons Learned
Even non-technical businesses need to find a few, simple metrics that relate to their core business model, then track them over time to predict what’s going to happen and identify patterns or trends.
Drawing Lines in the Sand
Knowing which metric to focus on isn’t enough. You need to draw a line in the sand as well. Let’s say that you’ve decided “New Customers Per Week” is the right metric to focus on because you’re testing out new ways of acquiring customers. That’s fair, but it doesn’t answer the real question: How many new customers per week do you need? Or more specifically: How many new customers per week (per acquisition channel) do you think defines a level of success that enables you to double down on user acquisition and move to the next step in the process?
You need to pick a number, set it as the target, and have enough confidence that if you hit it, you consider it success. And if you don’t hit the target, you need to go back to the drawing board and try again.
Picking the target number for any given metric is extremely hard. We’ve seen many startups struggle with this. Often, they avoid picking a number altogether. Unfortunately, this means it’s difficult to know what to do once an experiment is completed. If, in our example, the user acquisition experiment is a dismal failure, any number you had picked beforehand is probably immaterial; you’ll know it’s a failure. And if your efforts are insanely successful, you’re going to know that as well. It’ll be obvious. But most of the time, experiments end up right in the big fat middle. There was some success, but it wasn’t out of this world. Was it enough success to keep going, or do you have to go back and run some new experiments? That’s the trickiest spot to be in.
There are two right answers to the question of what success looks like. The first comes from your business model, which may tell you what a metric has to be. If you know that you need 10% of your users to sign up for the paid version of your site in order to meet your business targets, then that’s your number.
In the early stages of your business, however, you’re still figuring out what your business model should look like. It won’t tell you precisely what you need. The second right answer is to look at what’s normal or ideal. Knowing an industry baseline means you know what’s likely to happen, and you can compare yourself to it. In the absence of any other information, this is a good place to start. We’ll share some industry benchmarks that may be helpful to you later in the book.
The Squeeze Toy
There’s another important aspect to the OMTM. And we can’t really explain it better than with a squeeze toy
If you optimize your business to maximize one metric, something important happens. Just like one of those bulging stress-relief squeeze toys, squeezing it in one place makes it bulge out in another. And that’s a good thing. Optimizing your OMTM not only squeezes that metric so you get the most out of it, but it also reveals the next place you need to focus your efforts, which often happens at an inflection point for your business:
• Perhaps you’ve optimized the number of enrollments in your gym, and you’ve done all you can to maximize revenues—but now you need to focus on cost per customer so you turn a profit.
• Maybe you’ve increased traffic to your site—but now you need to maximize conversion.
• Perhaps you have the foot traffic in your coffee shop you’ve always wanted—but now you need to get people to buy several coffees rather than just stealing your Wi-Fi for hours.
Whatever your current OMTM, expect it to change. And expect that change to reveal the next piece of data you need to build a better business faster.
exercise
Define Your OMTM
Can you pick the One Metric That Matters for your startup? Give it a try. If you did the exercise at the end of Chapter 2, you have a short list of good metrics you track; now pick the one you couldn’t live without.
Could your entire company work exclusively on improving that metric? What might break if you did? Could you draw a line in the sand to measure results? If not, that’s OK. For now, write down your One Metric That Matters and where it currently stands, and we’ll come back to the line later.
But one of the keys to startup success is achieving real focus and having the discipline to maintain it. You may succeed if you’re unfocused, but it’ll be by accident. You’ll spend a lot more time wandering aimlessly, and the lessons learned are more painful and harder-won. If there’s any secret to success for a startup, it’s focus.
Focus doesn’t mean myopia. We’re not saying that there’s only one metric you care about from the day you wake up with an idea to the day you sell your company. We are, however, saying that at any given time, there’s one metric you should care about above all else. Boiled down to its very essence, Lean Startup is really about getting you to focus on the right thing, at the right time, with the right mindset.
As noted in Chapter 5, Eric Ries talks about three engines that drive company growth: the sticky engine, the viral engine, and the paid engine. But he cautions that while all successful companies will ultimately use all three engines, it’s better to focus on one engine at a time. For example, you might make your product sticky for its core users, then use that to grow virally, and then use the user base to grow revenue. That’s focus.
In the world of analytics and data, this means picking a single metric that’s incredibly important for the step you’re currently working through in your startup. We call this the One Metric That Matters (OMTM).
The OMTM is the one number you’re completely focused on above everything else for your current stage. Looking at CLV (customer lifetime value) isn’t meaningful when you’re validating a problem, but it might be the right metric to focus on as you’re approaching product/market fit.
You’ll always track and review multiple numbers. Some will be important: these are your key performance indicators (KPIs), which you’ll track and report every day. Others will be stored away for future use, such as when it’s time to tell the company history to an investor or to make an infographic. Setting up and managing instrumentation is fairly easy these days with tools like Geckoboard, Mixpanel, Kissmetrics, Totango, Chartbeat, and others. But don’t let your ability to track so many things distract you. Capture everything, but focus on what’s important.
case study
Moz Tracks Fewer KPIs to Increase Focus
Moz (previously known as SEOmoz) is a successful Software as a Service (SaaS) vendor that helps companies monitor and improve their websites’ search engine rankings. In May 2012, the company raised $18 million. Its CEO, Rand Fishkin, published a detailed post about the company’s progress up to that point.* Rand’s update did include a number of vanity metrics—when you have roughly 15 million visitors on your site each year, you have the right to a bit of vanity—but he also shared some very specific and interesting numbers related to conversions from free trials to paid subscriptions and churn.
We spoke with Joanna Lord, Vice President of Growth Marketing at Moz, to learn more about how the company handles metrics. “We are very metrics-driven,” she says. “Every team reports to the entire company weekly on KPIs, movement, and summaries. We also have a huge screen up in the office pumping out customer counts and free trial counts. We believe that having company-wide transparency into the metrics keeps us all informed, and is a great reminder of the progress (as well as the challenges) we are seeing as a company.”
For a company that’s found product/market fit and is now focused on scaling, it becomes more challenging to focus on a single metric. This isn’t surprising; there are multiple departments all growing quickly, and the business can tackle several different things simultaneously. But even with all these concurrent efforts, Joanna says that one metric stands above the rest: Net Adds. This metric is the total of new paid subscribers (either conversions from free trials or direct paid signups) minus the total who cancelled.
“Net Adds helps us quickly see high cancel days (and troubleshoot them) and helps us get a sense of how our free trial conversion rate is doing,” Joanna says.
Moz tracks other related metrics including Total Paying, New Free Trials Yesterday, and 7-Day Net Add Average. All of these really bubble up into Net Adds per day.
Interestingly, when Moz raised its last round of financing, one of its lead investors, the Foundry Group’s Brad Feld, suggested that it track fewer KPIs. “The main reason for this is that as a company, you can’t simultaneously affect dozens of KPIs,” Joanna says. “Brad reminded us that ‘too much data’ can be counterproductive. You can get lost in strange trends on numbers that aren’t as big-picture as others. You can also lose a lot of time reporting and communicating about numbers that might not lead to action. By stripping our daily KPI reporting down to just a few metrics, it’s clear what we’re focused on as a company and how we’re doing.”
Summary
• Moz is metrics-driven—but that doesn’t mean it’s swimming in data. It relies on one metric above all others: Net Adds.
• One of its investors actually suggested reducing the number of metrics the company tracks to stay focused on the big picture.
Analytics Lessons Learned
While it’s great to track many metrics, it’s also a sure way to lose focus. Picking a minimal set of KPIs on which your business assumptions rely is the best way to get the entire organization moving in the same direction.
Four Reasons to Use the One Metric That Matters
The OMTM is of most importance early on. Later, as your startup scales, you will want to focus on more metrics, and you’ll have the resources and experience to do so. Importantly, you’ll also have a team to whom you can delegate metrics. Your operations person might care about uptime or latency, your call center might worry about average time on hold, and so on.
At Year One Labs, one of the litmus tests for us as advisors and investors was the clarity with which a team understood, and tracked, their OMTM. If it was on the tip of their tongues, and aligned with their current stage, that was a good thing. If they didn’t know what it was, if it was the wrong metric for their stage, if they had several metrics, or if they didn’t know what the current value was, we knew something was wrong.
Picking the OMTM lets you run more controlled experiments quickly and compare the results more effectively. Remember: the One Metric That Matters changes over time. When you’re focused on acquiring users (and converting them into customers), your OMTM may be tied to which acquisition channels are working best or the conversion rate from signup to active user. When you’re focused on retention, you may be looking at churn, and experimenting with pricing, features, improving customer support, and so on. The OMTM changes depending on your current stage, and in some cases it will change quickly.
Let’s look at four reasons why you should use the One Metric That Matters.
• It answers the most important question you have. At any given time, you’ll be trying to answer a hundred different questions and juggling a million things. You need to identify the riskiest areas of your business as quickly as possible, and that’s where the most important question lies. When you know what the right question is, you’ll know what metric to track in order to answer that question. That’s the OMTM.
• It forces you to draw a line in the sand and have clear goals. After you’ve identified the key problem on which you want to focus, you need to set goals. You need a way of defining success.
• It focuses the entire company. Avinash Kaushik has a name for trying to report too many things: data puking.* Nobody likes puke. Use the OMTM as a way of focusing your entire company. Display your OMTM prominently through web dashboards, on TV screens, or in regular emails.
• It inspires a culture of experimentation. By now you should appreciate the importance of experimentation. It’s critical to move through the build→measure→learn cycle as quickly and as frequently as possible. To succeed at that, you need to actively encourage experimentation. It will lead to small-f failures, but you can’t punish that. Quite the opposite: failure that comes from planned, methodical testing is simply how you learn. It moves things forward in the end. It’s how you avoid big-F Failure. Everyone in your organization should be inspired and encouraged to experiment. When everyone rallies around the OMTM and is given the opportunity to experiment independently to improve it, it’s a powerful force.
case study
Solare Focuses on a Few Key Metrics
Solare Ristorante is an Italian restaurant in San Diego owned by serial entrepreneur Randy Smerik. Randy has a background in technology and data, once served as the general manager for business intelligence firm Teradata, and has five technology exits under his belt. It’s no surprise that he’s brought his data-driven mindset to the way he runs the business.
One evening at the restaurant, Randy’s son Tommy—who manages the bar—yelled out, “24!” Since we’re always looking for stories about business metrics, we asked him what the number meant. “Every day, my staff tells me the ratio of staff costs to gross revenues for the previous day,” he explained. “This is a fairly well-known number in the restaurant industry. It’s useful because it combines two things you have a degree of control over—per-diner revenues and staffing costs.”
Randy explained when staffing costs exceed 30% of gross revenues, that’s bad, because it means that you’re either spending too much on staff or not deriving enough revenue per customer. A Michelin-starred restaurant can afford to have more staff, and pay them more, because it sells customers expensive wines and enjoys good per-customer revenue. At the other end of the spectrum, a low-margin casual dining restaurant has to keep staff costs down.
The ratio works because it’s:
• Simple: It’s a single number.
• Immediate: You can generate it every night.
• Actionable: You can change staffing, or encourage upselling, the very next day, whereas ingredient costs, menus, or leasing take longer to modify.
• Comparable: You can track it over time, and compare it to other restaurants in your category.
• Fundamental: It reflects two basic facets of the restaurant business model.
As it turns out, 24% is about right. Below 20%, there’s a chance that you’re under-serving customers and that their dining experience might suffer (Randy could experiment with different staffing levels and measure the tips diners leave, or comments on Yelp, if he wanted to be really analytical).
Randy also uses a second metric to predict how many customers he’ll have. At 5 p.m. every day, his staff sends him the number of reservations that have currently been made for the evening. “If I get 50 reservations at 5 p.m., I know I’ll have around 250 covers that night,” he says. “We’ve learned that a 5-to-1 ratio is normal for Solare.”
This number doesn’t work across all restaurants—the in-demand Michelin-starred restaurant has a 1-to-1 ratio, since it’s sold out, and a fast food restaurant that doesn’t take reservations obviously can’t use the metric. But for Solare reservations at 5 p.m., plus some experience, provides a good leading indicator of what the night will be like. It also allows the Solare team to make small adjustments to staffing or buy additional produce in time to ensure that the restaurant can handle the traffic.
Summary
• Restaurants know from experience that demand is tied to reservations, and what the right ratio of staffing to revenue should be.
• Good metrics help predict the future, giving you an opportunity to anticipate problems and correct them.
Analytics Lessons Learned
Even non-technical businesses need to find a few, simple metrics that relate to their core business model, then track them over time to predict what’s going to happen and identify patterns or trends.
Drawing Lines in the Sand
Knowing which metric to focus on isn’t enough. You need to draw a line in the sand as well. Let’s say that you’ve decided “New Customers Per Week” is the right metric to focus on because you’re testing out new ways of acquiring customers. That’s fair, but it doesn’t answer the real question: How many new customers per week do you need? Or more specifically: How many new customers per week (per acquisition channel) do you think defines a level of success that enables you to double down on user acquisition and move to the next step in the process?
You need to pick a number, set it as the target, and have enough confidence that if you hit it, you consider it success. And if you don’t hit the target, you need to go back to the drawing board and try again.
Picking the target number for any given metric is extremely hard. We’ve seen many startups struggle with this. Often, they avoid picking a number altogether. Unfortunately, this means it’s difficult to know what to do once an experiment is completed. If, in our example, the user acquisition experiment is a dismal failure, any number you had picked beforehand is probably immaterial; you’ll know it’s a failure. And if your efforts are insanely successful, you’re going to know that as well. It’ll be obvious. But most of the time, experiments end up right in the big fat middle. There was some success, but it wasn’t out of this world. Was it enough success to keep going, or do you have to go back and run some new experiments? That’s the trickiest spot to be in.
There are two right answers to the question of what success looks like. The first comes from your business model, which may tell you what a metric has to be. If you know that you need 10% of your users to sign up for the paid version of your site in order to meet your business targets, then that’s your number.
In the early stages of your business, however, you’re still figuring out what your business model should look like. It won’t tell you precisely what you need. The second right answer is to look at what’s normal or ideal. Knowing an industry baseline means you know what’s likely to happen, and you can compare yourself to it. In the absence of any other information, this is a good place to start. We’ll share some industry benchmarks that may be helpful to you later in the book.
The Squeeze Toy
There’s another important aspect to the OMTM. And we can’t really explain it better than with a squeeze toy
If you optimize your business to maximize one metric, something important happens. Just like one of those bulging stress-relief squeeze toys, squeezing it in one place makes it bulge out in another. And that’s a good thing. Optimizing your OMTM not only squeezes that metric so you get the most out of it, but it also reveals the next place you need to focus your efforts, which often happens at an inflection point for your business:
• Perhaps you’ve optimized the number of enrollments in your gym, and you’ve done all you can to maximize revenues—but now you need to focus on cost per customer so you turn a profit.
• Maybe you’ve increased traffic to your site—but now you need to maximize conversion.
• Perhaps you have the foot traffic in your coffee shop you’ve always wanted—but now you need to get people to buy several coffees rather than just stealing your Wi-Fi for hours.
Whatever your current OMTM, expect it to change. And expect that change to reveal the next piece of data you need to build a better business faster.
exercise
Define Your OMTM
Can you pick the One Metric That Matters for your startup? Give it a try. If you did the exercise at the end of Chapter 2, you have a short list of good metrics you track; now pick the one you couldn’t live without.
Could your entire company work exclusively on improving that metric? What might break if you did? Could you draw a line in the sand to measure results? If not, that’s OK. For now, write down your One Metric That Matters and where it currently stands, and we’ll come back to the line later.
Comments
Post a Comment