Welcome!

Java Authors: Pat Romanski, Carmen Gonzalez, Yeshim Deniz, Imran Akbar, Sharon Barkai

Related Topics: Cloud Expo, Java, SOA & WOA, Virtualization, Big Data Journal, SDN Journal

Cloud Expo: Blog Feed Post

SaaS Metrics 2.0 – A Guide to Measuring and Improving What Matters

SaaS/subscription businesses are more complex than traditional businesses

Defining a Dashboard for a SaaS Company
The following section should be most useful for readers who are interested putting together a dashboard to help them manage their SaaS business. To this, we created an excel file for an imaginary SaaS company, and laid out a traditional numeric report on one tab, and then a dashboard of graphs on a second tab (see below). These represent one view on how to do this. You may have a very different approach. But hopefully this will give you some ideas. I would recommend adding a dotted line with the plan number to all graphs. This will allow you to quickly see how you are doing versus plan.

There are two versions of the Dashboard: the one shown below, which is designed for companies using primarily monthly contracts (focused on MRR). And a second version that can be found here which is designed for companies using annual contracts, focusing on ACV (Annual Contract Value).

image

image
image
image

image

Brad Coffey, HubSpot:

At HubSpot we obsess over these metrics - and watch many of them every day. Each night we send out a ‘waterfall' chart that tracks our progress against our typical progress given the number of business days left in the month. Here is an example of what we look at to ensure we're on track to meet our net MRR goals.

image

By looking at this daily we can take action immediately if we're tracking towards a bad month or quarter. Things like services promotion (for churned MRR) or sales contests & promotions (for new & expansion MRR) are adjustments we make within a given month in order to nail our goals. (In this model we combine expansion and churned MRR into one churned MRR line).

Detailed definitions of the various metrics used

Detailed definitions for each of the various metrics used can be found in this reference document:

image

Revenue Churn vs Customer Churn - why are they different?

You might be wondering why it's necessary to track both Customer Churn and Revenue Churn. Imagine a scenario where we have 50 small accounts paying us $100 a month, and 50 large accounts paying us $1,000 a month. In total we have 100 customers, and an MRR of $55,000 at the start of the month. Now imagine that we lose 10 of them. Our Customer churn rate is 10%. But if out of the ten churned customers, 9 of them were small accounts, and only one was a large account. We would only have lost $1,900 in MRR. That represents only 3.4% Revenue Churn. So you can see that the two numbers can be quite different. But each is important to understand if we want a complete picture of what is going on in the business.

Getting paid in advance

Getting paid in advance is really smart idea if you can do it without impacting bookings, as it can provide the cash flow that you need to cover the cash problem that we described earlier in the article. It is often worth providing good financial incentives in the form of discounts to encourage this behavior. The metric that we use to track how well your sales force is doing in this area is Months up Front.

Getting paid more upfront usually also helps lower churn. This happens because the customer has made a greater commitment to your service, and is more likely to spend the time getting it up and running. You also have more time to overcome issues that might arise with the implementation in the early days. Calculating LTV and CAC

The Metric "Months up Front" has been used at both HubSpot and NetSuite in the past as a way to incent sales people to get more paid up front when a new customer is signed. However asking for more money up front may turn off certain customers, and result in fewer new customers, so be careful how you balance these two conflicting goals.

Calculating CAC and LTV

Detailed information on how to calculate LTV and CAC is provided in the supplemental document that can be accessed by clicking here.

More on Churn: Cohort Analysis

Since churn is such a critical element for success in a SaaS company, it is an area that requires deeper exploration to understand. Cohort Analysis is one of the important techniques that we use to gain insight.

As mentioned earlier, a cohort is simply a fancy name for a group. In SaaS businesses, we use cohort analysis to observe what happens to the group of customers that joined in a particular month. So we  we will have a January cohort, a February cohort, etc. We would then be able to observe how our January cohort behaves over time (see illustration below).

image

This can help answer questions such as:

  • Are we losing most of the customers in the first couple of months?
  • Does Churn stabilize after some period of time?

Then if took some actions to try to fix churn in early months, (i.e with better product features, easier on-boarding, better training, etc.) we would want to know if those changes had been successful. The cohort analysis allows us to do this by comparing how more recent cohorts (e.g., July in the table above) compared against January. The table above shows that we made a big improvement in the first month churn going from 15% to 4%.

Two ways to run Cohort Analysis

There are two ways to run Cohort Analysis: the first looks at the number of customers, and the second looks at the Revenue. Each teaches us something different and valuable. The example graph below simply looks at the number of customers in each cohort over time:

image

The example graph below looks at how MRR evolves over time for each cohort. This particular example illustrates how the graph would look if there is very strong negative churn. As you can see, the increase in revenue from the customers that are still using the service is easily outpacing the lost revenue from churned customers. It is pretty rare to see things look this good, but it is the ideal situation that we are looking for. For those wondering if this can be achieved, one company in our portfolio, Zendesk, that has numbers that are even better than those shown in the example below.

image

In the situation above, you will need a more complex formula to calculate LTV, as the value of the average customer is increasing over time. For more on that topic, you may want to check out the accompanying definitions document.

Predicting Churn: Customer Engagement Score
Since churn is so important, wouldn't it be useful if we could predict in advance which customers were most likely to churn? That way we could put our best customer service reps to work in an effort to save the situation. It turns out that we can do that by instrumenting our SaaS applications and tracking whether our users are engaged with the key sticky features of the product. Different features will deserve different scores. For example if you were Facebook, you might score someone who uploaded a picture as far more engaged (and therefore less likely to churn), than someone who simply logged in and viewed one page.

Similarly if you sold your SaaS product to a 100 person department, and only 10 people were using it, you would score that differently to 90 people using it. So the recommendation is that you create a Customer Engagement Score, based on allocating points for the particular features used. Allocate more points for the features you believe are most sticky. (Later on you can go back and look at the customers who actually churned, and validate that you picked the right features as a predictor of who would churn.) And separately score how many users are engaged with specific scores.

Over time you'll also come to discover which types of use are the best indicators of possible upsell. (HubSpot was the first company that I worked with who figured this out, and they called it their CHI score. CHI stands for Customer Happiness Index. It evolved to be a very good predictor for churn.)

Brad Coffey, HubSpot:

At HubSpot we had a lot of success looking at this metric - we called in Customer Happiness Index (CHI). First - by running the analysis we identified the parts of our application that provide the most value to customers and could invest accordingly in driving adoption in those areas. Second - we used this aggregate score as an early proxy for success as we experimented with different sales and onboarding processes. If a set of customers going through an experiment had a low CHI score we could kill the project without waiting 6 or 12 months to analyze the cohort retention.

NPS - Net Promoter Score
Since it is likely that customer satisfaction is likely to be good predictor of future churn, it would be useful to survey customer satisfaction. The recommended way to measure customer happiness is to use Net Promoter Score (NPS). The beauty of NPS is that it is a standardized number, so you can compare your company to others.  For more details on Net Promoter Score, click here.

Guidelines for Churn
If your Net Revenue Churn is high (above 2% per month) it is an indicator that there is something wrong in your business. At 2% monthly churn, you are losing about 22% of your revenue every year. That is nearly a quarter of your revenue! It's a clear indication that there is something wrong with the business. As the business gets bigger, this will become a major drag on growth.

We recommend that you work on fixing the problems that are causing this before you go on to worry about other parts of your business. Some of the possible causes of churn are:

  • You are not meeting your customers expectations.
    • The product may not provide enough value
    • Instability or bugginess
  • Your product is not sticky. It might provide some value in the first few months, and then once the customer has that value, they may feel they don't need to keep paying. To make your product sticky, try making it a key part of their monthly workflow, and/or have them store data in your product that is highly valuable to them, where the value would be lost of they cancelled.
  • You have not successfully got the customer's users to adopt the product. Or they may not be using certain of the key sticky features in the product.
  • Your sales force may have oversold the product, or sold it to a customer that is not well suited to get the benefits
  • You may be selling to SMB's where a lot of them go out of business. It isn't enough that what you're selling is sticky. Who you're selling it to must also be sticky.
  • You are not using a pricing scheme that helps drive expansion bookings

The best way to find out why customers are churning is to get on the phone with them and ask them. If churn is a significant part of your business, we recommend that the founders themselves make these calls. They need to hear first hand what the problem is, as this is so important for the success of the business. And they are likely to be the best people to design a fix for the problem.

The Importance of Customer Segmentation
In all SaaS businesses there will likely come a moment where they realize that not all customers are created equal. As an example, bigger customers are harder to sell to, but usually place bigger orders, and churn less frequently. We need a way to understand which of these are most profitable, and this requires us to segment the customer base into different types, and compute the unit economics metrics for each segment separately. Common segments are things size of of customer, vertical industry, etc.

Despite the added work to produce the metrics, there is high value in understanding the different segments. This tells us which parts of the business are working well, and which are not. In addition to knowing where to focus and invest resources, we may recognize the need for different marketing messages, product features. As soon as you start doing this segmented analysis, the benefits will become immediately apparent.

For each segment, we recommend tracking the following metrics:

  • ARPA (Average Revenue per Account per month)
  • Net MRR Churn rate (including MRR expansion)
  • LTV
  • CAC
  • LTV: CAC ratio
  • Months to recover CAC
  • Customer Engagement Score

Brad Coffey, HubSpot:

At HubSpot, we started to see some of our biggest improvements in unit economics when we started segmenting our business and calculating the LTV to CAC ratio for each of our personas and go to market strategies.

As one good example - when we started this analysis, we had 12 reps selling directly into the VSB market and 4 reps selling through Value Added Resellers (VARs). When we looked at the math we realized we had a LTV:CAC ratio of 1.5 selling direct, and a LTV:CAC ratio of 5 selling through the channel. The solution was obvious. Twelve months later we had flipped our approach - keeping just 2 reps selling direct and 25 reps selling through the channel. This dramatically improved our overall economics in the segment and allowed us to continue growing.

We ended making similar investments in other high LTV:CAC segments. We went so far as to incentivize our sales managers to grow their teams - but then would only place new sales hires into the segments with the best economics. This ensured we continued to invest in the best segments and aligned incentives throughout the company on our LTV:CAC goals. It also allowed us to push innovation down to the sales manager level. Managers could experiment with org structure, and sales processes - but they knew that if they didn't hit their LTV:CAC goals they wouldn't be able to grow their teams.

Calculating LTV:CAC by segment can be challenging, especially on the CAC side. It's relatively easy at the top level to add up all the marketing and sales expense in a period and divide it by the total number of customers (to get CAC). Once you try to segment down your spend you run into questions like ‘how much marketing expense do I allocate to a given segment', ‘how much of the sales expense'?

We solved this by allocating marketing expense based on number of leads and sales expense based on headcount but it's not perfect. For us the keys are: 1) Needs to account for all costs - no free lunch, 2) It needs to be consistent over time. Progress on improving the metric is more important than the actual value.

Funnel Metrics

The metrics that matter for each sales funnel, vary from one company to the next depending on the steps involved in the funnel. However there is a common way to measure each step, and the overall funnel, regardless of your sales process. That involves measuring two things for each step:  the number of leads that went into the top of that step, and the conversion rate to the next step in the funnel (see below).

In the diagram above, (mirrored in the dashboard), we show a very simple three phase sales process, with visitors coming to a web site, and some portion of them signing up for a trial. Then some of the trials convert to purchases.  As you can see in the dashboard, we will want to track the number of visitors, trials and closed deals. Our goal should be to increase those numbers over time. And we will also want to track the conversion rates, with the goal of improving those over time.

Using Funnel Metrics in Forward Planning

Another key value of having these conversion rates is the ability to understand the implications of future forecasts. For example, lets say your company wants to do $4m in the next quarter. You can work backwards to figure out how many demos/trials that means, and given the sales productivity numbers - how many salespeople are required, and going back a stage earlier, how many leads are going to be required. These are crucial planning numbers that can change staffing levels, marketing program spend levels, etc.

Sales Capacity

In many SaaS businesses, sales reps play a key role in closing deals. In those situations, the number of productive sales people (Sales Capacity) will be a key driver of bookings. It is important to work backwards from any forecasts that are made, to ensure that there is enough sales capacity. I've seen many businesses miss their targets because they failed to hire enough productive salespeople early enough.

It's also worth noting that some percentage of new sales hires won't meet expectations, so that should be taken into consideration when setting hiring goals. Typically we have seen failure rates around 25-30% for field sales reps, but this varies by company. The failure rate is lower for inside sales reps.

When computing Sales Capacity, if a newer rep is still ramping and only expected to deliver 50% of quota, they can be counted as half of a productive rep. That is often referred to as Full Time Equivalent or FTE for short.

Another important metric to understand is the number of leads required to feed a sales rep. If you are adding sales reps, make sure you also have a clear plan of how you will drive the additional leads required.

There is much more that could be said on this topic, but since it is all very similar to managing a sales force in a traditional software company, we will leave that for other blog posts.

Understanding the ROI for different Lead Sources

Our experiences with SaaS startups indicate that they usually start with a couple of lead generation programs such as Pay Per Click Google Ad-words, radio ads, etc. What we have found is that each of these lead sources tends to saturate over time, and produce less leads for more dollars invested. As a result, SaaS companies will need to be constantly evaluating new lead sources that they can layer in on top of the old to keep growing.

Since the conversion rates and costs per lead vary quite considerably, it is important to also measure the overall ROI by lead source.

Growing leads fast enough to feed the front end of the funnel is one of the perennial challenges for any SaaS company, and is likely to be one of the greatest limiting factors to growth. If you are facing that situation, the most powerful advice we can give you is to start investing in Inbound Marketing techniques (see Get Found using Inbound Marketing). This will take time to ramp up, but if you can do it well, will lead to far lower lead costs, and greater scaling than other paid techniques. Additionally the typical SaaS buyer is clearly web-savvy, and therefore very likely to embrace inbound marketing content and touchless selling techniques.

What Levers are available to drive Growth

SaaS businesses are more numerically driven than most other kinds of business. Making a small tweak to a number like the churn rate can have a very big impact on the overall health of the business. Because of this we frequently see a "quant" (i.e. a numbers oriented, spreadsheet modeling, type of person) as a valuable hire in a SaaS business. At HubSpot, Brad Coffey played that role, and he was able to run the models to determine which growth plays made the most sense.

Understanding these SaaS metrics is a key step towards seeing how you can drive your business going forward. Let's look at some of the levers that these imply as growth drivers for your business:

Churn

  • Get Churn and customer happiness right first (if this isn't right, the business isn't viable, so no point in driving growth elsewhere. You will simply be filling a leaky bucket.)

Product

  • You're in a product business - first and foremost: fix your product.
    • If you're using a free trial, focus on getting the conversion rate for that right (ideally around 15 - 20%). If this isn't right, your value proposition isn't resonating, or you may have a market where there is not enough pain to get people to buy.
    • Win/Loss ratio should be good
    • Trial or Sales conversion rates on qualified leads should be good

Funnel metrics

  • Increase the number of raw leads coming in to the Top of your funnel
  • Identify the profitable lead sources and invest in those as much as possible. Conversely stop investing in poor lead sources until they can be tweaked to make them profitable.
  • Increase the Conversion Rates at various stages in the funnel

Sales Metrics

  • Sales productivity (focus on getting this right consistently across a broad set of sales folks before hitting the gas)
  • Add Sales Capacity. But first make sure you know how to provide them with the right number of leads. This turns out to be one of the key levers that many companies rely on for growth. We have learned from experience how important it is to meet your targets for sales capacity by hiring on time, and hiring the right quality of sales people so there are fewer failures.
  • Increase retention for your sales people. Since you have invested a lot in making them fully productive, get the maximum return on that investment by keeping them longer.
  • Look at adding Business Development Reps. These are outbound sales folks who specialize in prospecting to a targeted list of potential buyers. For more on this topic, click here.

Pricing/Upsell/Cross Sell

  • Multi-axis pricing
  • Additional product modules (easier to sell more to existing customers than it is to sell to brand new customers)

Brad Coffey, HubSpot:

Turns out the pricing your product right can have a huge impact on the unit economics. Not simply by getting the average MRR right, or by providing upsell opportunities - but also by signaling what pieces of the product are most valuable.

At HubSpot we changed our pricing in 2011 to be tiered based on the number of contacts in the system - and actually saw an increase in adoption of the contacts application after we made the change. This is counter-intuitive but makes sense given that we sell through an inside sales team. After the pricing change, sales reps now could make a lot more money by selling the contacts. And they quickly become much better at positioning that part of the product, as well as finding companies with a contacts-based use case. Product quality will remain paramount - but it's remarkable how much impact pricing, packaging and sales commission structure can have on product adoption and unit economics.

Customer Segmentation

Customer Segmentation analysis will help point out which are your most profitable segments. Two immediate actions that are suggested by this analysis are:

  • Double down on your most profitable segments
  • Look at your less profitable segments and consider changes that would make them more profitable: lower cost marketing & sales approaches, higher pricing, product changes, etc. If nothing seems to make sense, spend less effort on these segments.

International Markets

Expansion internationally is only recommended for fairly mature SaaS companies that already have honed their business practices in their primary market. It is far harder to experiment and tune a business in far off regions, with language and cultural differences.

Brad Coffey, HubSpot

  • One of the biggest challenges we face is the trade-off between growth and unit economics (specifically churn).  Many of the things that we have done to reduce churn have (potentially) come at the expense of lowering our growth rate. Those have been some of our hardest decisions:  e.g. requiring upfront payments, requiring customers buy consulting, holding sales reps accountable for churn, etc. We are always looking at things that give us growth without the tradeoff of lower growth. For example product improvement is an obvious one - a better product is easier to sell and provides more value to the customer. Services promotions actually work well too. Many of the options that SaaS companies have to adjust their business are not simply a win-win but are still worth exploring. Too many companies think that every problem is a product problem and every solution is that the product must get better.
  • The other thing that's really important is that companies don't try to spin these numbers.  There is so much pressure to dismiss a bad customer (who hurt your churn number) or exclude costs (only count marketing ‘program' spend - not headcount).  If you can get the accounting close enough to right it actually frees management from needing to make every decision.  If the accounting is right management can obsess over setting goals (growth, LTV:CAC), hold people accountable to those goals and then give autonomy to their team on how to achieve those goals.

Plan ahead

It takes time for most initiatives to have an impact. We've learned from some tough lessons that planning has to be done well in advance to drive a SaaS business. For example if you are not happy with your current growth rate, it will often take nine to twelve months from the point of decision before the growth resulting from increased investment in sales and marketing will actually be observed.

The High Level Picture: How to Run a SaaS Business

Hopefully what you will have gathered from the discussion above is that there are really three things that really matter when running a SaaS business:

  1. Acquiring customers
  2. Retaining customers
  3. Monetizing your customers

The second item should be first on your list of things to get right. If you can't keep your customers happy, and keep them using the service, there is no point in worrying acquiring more of them. You will simply be filling a leaky bucket. Rather focus your attention on plugging the leaks.

SaaS businesses are remarkably influenced by a few key numbers. Making small improvements to those numbers can dramatically improve the overall health of the business.

Once you know your SaaS business is viable using the guidelines provided for LTV:CAC, and Time to recover CAC, hit the accelerator pedal. But be prepared to raise the cash needed to fund the growth.

Although this article is long and occasionally complex, we hope that it has helped provide you with an understanding of which metrics are key, and how you can go about improving them.

Acknowledgements

I would like to thank Ron Gill, the CFO of NetSuite, and Brad Coffey & Brian Halligan of Hubspot for their help in writing this. I would like to thank the HubSpot management team without whom none of this would be possible. Most of my learnings on SaaS have come from working with them. I would also like to thank Gail Goodman, the CEO of Constant Contact who also taught us many of the key metrics in her role as board member of HubSpot.

More Stories By David Skok

David Skok joined Matrix Partners as a General Partner in May 2001. He has a wealth of experience running companies. He started his first company in 1977 at age 22. Since then he has founded a total of four separate companies and performed one turn-around. Three of these companies went public.

Skok joined Matrix from SilverStream Software, which he founded in June 1996. Prior to its July 2002 acquisition by Novell, SilverStream was a public company that had reached a revenue run rate in excess of $100M, with approximately 800 employees and offices in more than 20 countries around the world. His work as a value added investor is best known for helping JBoss take its Open Source business to a successful exit with its sale to Red Hat, and for helping AppIQ, Tabblo and Diligent Technologies, which have all had successful exits, from their inceptions to their acquisitions by HP and IBM.

He serves on the boards of Digium (makers of the very popular Asterisk Open Source PBX/telephony software), CloudSwitch, Enservio, OpenSpan, Solidworks, VideoIQ, and HubSpot. In addition to his broad focus on enterprise software, he is specifically focused on the areas of cloud computing, Open Source, Software as a Service (SaaS), marketing automation, virtualization, storage, and data center automation.