Working With Metrics is Not Straightforward

This is Part 2 of the series titled Modeling Meaningful Metrics

{{line}}

I was confused and overwhelmed when I was getting started with metric definitions at Integromat (Make). I didn’t know most of the stuff I know today and even though I had listed down a bunch of metrics complete with definitions, I wasn’t familiar with the process of creating  metrics. 

Why is that a problem? 

Well, you wouldn’t know what data to collect in the first place if you don’t understand what it takes to create certain metrics. This will sound familiar if you’ve gone through the process of answering burning questions. That said, the data needed to answer a burning question may or may not be enough to create a metric. 

Let me invoke one of my favorite burning questions: 

“We’re acquiring a ton of users every day but very few end up hitting the activation milestone; what’s preventing the rest from performing the actions leading to activation?”

Here, all I needed was to explore a set of events on Mixpanel and create funnels to see where in the onboarding journey were users getting stuck and dropping off. The activation milestone (activation criteria), in Integromat’s case, was also straightforward: a user had to create and turn on their first scenario (workflow) to be considered activated. 

However, looking at user actions in isolation wasn’t enough and we needed account-level data to calculate the activation rate. Therefore, the first step was to come up with the definition of an activated account which brought up the following questions: 

• If any one user in an account hits the activation milestone, do we consider the account activated? Or should we consider an account activated only when every user in the account hits the activation milestone?
• What happens if a user turns off their scenario soon after turning it on? What if the scenario is turned off automatically after it encounters repeated errors? 
• And how do we treat active but incomplete scenarios that only comprise a trigger module and no action module? 

I decided to keep things simple in the early stages and concluded that an account would be considered activated as soon as the first scenario in it was turned on. At the time, it was a priority to increase the activation rate, and therefore, it was desirable to keep the definition of an activated account simple. 

However, if I could go back in time, I’d make this definition a bit more thorough by including that at least 5 scenario runs took place in an account (scenario_run_count > 4) for it to be considered activated. 

Doing so would ensure that a user can test the scenario a few times and experience the magic of a successful scenario execution rather than turning it on without even testing that it works as expected. But all said and done, I don’t regret my decision because in the early stages, simple is always better. 

Now, it quickly dawned on me that is_activated_account and is_active_account were completely different metrics and that I should come up with the definition of an active account sooner rather than later. After all, even though the priority was to increase the activation rate, the goal was to drive growth by increasing the number of active accounts or workspaces, which in turn would lead to an increase in the number of paid accounts.  

Active vs. Activated 

In simple terms, an account is considered activated if it meets the activation criteria and hits the activation milestone – activation, in essence, is a one-time event. 

On the other hand, for an account to be considered active, it needs to meet certain criteria on a continuous basis. 

Moreover, “active” is a state and at any given time, an active account can become “inactive” (and vice versa). Therefore, from a metrics perspective, the active_account_count metric must always contain a time dimension such as daily, weekly, or monthly to enable one to answer a basic question like, “How many monthly active accounts do we have?”.

Answering questions like the one above is often challenging for SaaS businesses due to the nature of their products. Allow me to illustrate this by discussing what an active account looks like for two very different products: Mixpanel and Integromat. 

To derive value from a product like Mixpanel continuously, you have to keep going back to the product to create or view reports. However, you can continue to derive value from a product like Integromat as long as your workflows continue to run smoothly – you only need to log into the product to modify or fix a workflow or to create a new one. 

Let’s dig a little deeper.

Mixpanel accounts

Let’s assume that Mixpanel considers an account as activated when a user creates their first report and shares the report by inviting a colleague, resulting in the following equation:

is_activated_account = TRUE if report_created_count > 0 AND workspace_user_count > 1 

At least one report has been created in the account (report_create_count > 0) and the number of users in the workspace is at least two (workspace_user_count > 1). 

From a metrics perspective, report_created_count and workspace_user_count act as input metrics for is_activated_account, which acts as the input metric for activated_account_count, which is finally used to calculate the activation_rate. 

Creating is_activated_account is also useful because if the definition of an activated account changes, the analyst only needs to update the is_activated_account metric rather than updating all the downstream metrics. 

Now, for an account to be considered active, regular activity must take place in it. Therefore, let’s assume that Mixpanel’s definition of an active account is “an account in which at least two users view a report during a seven-day period”, represented by the following equation:

is_active_account = TRUE if report_viewer_count_weekly > 1

Two separate users must view at least one report during a seven-day period. This is very different from the definition of an activated account (where a report had to be created and a second user had to join the workspace for it to be considered activated). 

From a metrics perspective, the value of is_report_viewer in the User table changes to TRUE when a user performs the ReportViewed event; is_report_viewer acts as the input metric for report_viewer_count, and a time dimension is applied to calculate report_viewer_count_weekly.

Integromat accounts

Now let’s look at how we calculated the two metrics at Integromat: 

is_activated_account = TRUE if active_scenario_count > 0  

As long as at least one scenario is active (or turned on), an account is considered activated. 

From a metrics perspective, active_scenario_count doesn’t have any input metrics – it’s an account property fetched directly from the Accounts (or Workspace) table.

On the other hand, coming up with the definition of an active account was a continuous struggle and brought up the following considerations:

• How many scenario runs should take place over what period of time? And how many unique scenarios should be active?
• Should we consider the frequency with which scenarios are run? Or should we only care about the number of operations consumed? 
• Does it matter if a user doesn’t log in to the account as long as its scenarios continue to run smoothly?

A big challenge for an automation tool like Integromat is that happy customers don’t necessarily need to keep coming back to the product to continue deriving value from it (as long as their workflows continue to run smoothly). This is both good and bad – good because users need not spend too much time interacting with an interface to derive value, and bad because getting users to spend time in the product is difficult. AI-enabled tools, particularly the ones that perform useful tasks in the background, will certainly face this challenge. 

Anyhow, at Integromat, I kept going back and forth regarding the definition of an active account – it didn’t feel right to consider an account active only because it had a scenario turned on (which is how we reported on active accounts at the time). However, if I were to do it all over again, I’d consider an account as active if consumes at least 100 operations across a minimum of 2 active scenarios over a seven-day period; here’s the equation

is_active_account = TRUE if active_scenario_count > 1 AND operations_count_weekly > 99

From a metrics perspective, like active_scenario_count, operations_count is an account property fetched from the Accounts table, and applying a time dimension leads to operations_count_weekly. 

Number of operations or operations_count was the key consumption metric at Integromat (still is at Make) and if an account is using at least 100 operations every week, they’re certainly deriving enough value from the product. Also, 100 might not be the ideal number for Make to consider an account as active; I came up with it based on my current usage as a happy customer of Make.

The dollar is in the details

It’s useful to keep in mind that a higher number of active scenarios on Integromat doesn’t necessitate high usage – a single scenario with dozens of steps that runs frequently can consume more operations than, say, a dozen two-step scenarios that rarely run. And the same is true for Mixpanel too – an account with a dozen reports that nobody views is more likely to churn than an account with, say, one report that a dozen users view every day. 

Every SaaS business that offers a self-serve product, or in fancy terms, adopts a product-led go-to-market motion, needs to increase product usage – both in terms of features and frequency – in order to increase its free-to-paid conversion and retention rates. 

More importantly though, the criteria for an account to be considered active must include the actions that positively impact the key consumption metric (KCM). Additionally, the KCM – not to be confused with the NSM (north star metric) – must act as a monetization lever. 

At Integromat, we offered a very generous free plan, and having hundreds of thousands of “active accounts” on the “free plan” wouldn’t do us much good – each of those accounts cost us money and it was bad for our bottom line if we were unable to convert free active accounts into customers. Therefore, for an account to be considered active, it would make more sense to look at the number of operations it consumes and the number of active scenarios it has. After all, a higher number of active scenarios was likely to increase the number of operations consumed by an account.  

Pause to ponder 🤔  

Take a moment to think about an app – B2B or B2C – that you use often. Now try to go back to the moment in time when you first experienced the magic of the product or service – the moment when a voice in your head said, “Aha! This is exactly what I needed!” That’s when you hit the activation milestone as a user! 

Now think about how often you log in to the app and the key actions you perform while using it.

Conduct this exercise 🏋️  

Put yourself in the shoes of the product/growth person who is responsible for coming up with the definition of an active account (or active user in case of a single-player app) for the app you have in mind, and complete the following equation:

is_active_account = TRUE if ________

It’s a good idea to come up with multiple definitions based on your own usage and then compare them to figure out which features of the app you derive the most value from. The more time you spend thinking on these lines based on the apps and tools you use, the easier it will be for you to come up with better metric definitions for your own product. 

As a follow-up to this exercise, conduct a workshop to gather inputs from your team on what they think is a good criteria for an active account. And make sure to invite your data or engineering counterparts too – I assure you that the outcome will be pretty good!  

{{line}}

In Part 3, I'll cover what a metric is and what isn’t a metric.