Lean product analytics — Metrics That Matter Most

[Based on the book ‘The Lean Product Playbook’ by Dan Olsen]

When designing great products for target markets, it is essential to measure customer needs (identifying problem space) and product metrics (measuring solution space) to track progress.

Based on the requirements, product managers need to undertake approach research methods to retrieve the information they need. The following graph summarizes key research methods depending upon the class of problem:

Research Methods

Thus, the above graph classifies the research methods on based two axes where the X-axis depicts the type of information needed — Qualitative and Quantitative and Y-axis pertains to the category of information — Behavioral and Attitudinal.

While research methods act as a guideline towards identifying the research mechanism, it is important to use them to track customer needs and product metrics to measure progress once a product is launched. When it comes to tracking product metrics — the AARRR framework helps to measure holistic product attributes, where:

A: Acquisition
A: Activation
R: Retention
R: Referral
R: Revenue

In his book, Dan Olsen introduces his very famous product-market fit pyramid which emphasizes that a successful product maximizes the product-market fit as shown below:

Olsen mentions that the best metric to achieve product-market fit can be visualized from the following typical retention curve as depicted below:

Retention curve

From the above retention curve, three kinds of metrics can be identified as:

Initial drop-off rate: Percentage of users who immediately reject product after first use. For example, the initial drop-off rate from the above graphs is 1–60% = 40%
Drop rate: This is the rate at which active users drop before stabilizing at a terminal value
Terminal value: This is the percentage of stable active users in the long term. Thus, this is the most important metric for product managers to track the product-market fit.

Often during product development, product managers have to prioritize which metric to focus at one point of time. In order to maximize the ROI of effort and cost investment, the following order is suggested to be the best as per Olsen’s experience:

1. Optimize retention
2. Optimize conversion
3. Optimize acquisition

The above order also forms the basis of MTMM which is short for ‘Metric That Matters Most’ or a short-term goal to prioritize efforts for product teams.

Often metrics are classified into two forms :

1. Atomic: These are the rudimentary metrics that make sense when processed or when combined with other metrics. For example, number active users, days since the first install, developer weeks undertake etc.
2. Derived: These metrics are built upon or calculated from Atomic metrics

Let’s look at a few key derived metrics used commonly:

Return rate = Returning Visitors at Time T/Visitors at Time T-1

Profit per customer = CLTV — CAC where
CLTV -> Customer Lifetime Value
CAC -> Customer Acquisition Cost

CLTV = ARPU * Customer lifetime * Gross Margin
where
ARPU = Average Revenue Per User
Customer lifetime = 1/ Churn rate

CAC = Sales & Marketing Costs / New customers added

There are also marketing metrics needed to track if the product is reaching the right set of customers and creating the desired impact. For example:

CPM = Cost per thousand impressions
CTR = Clickthrough rate
ARPU = Impression per visitor * Effective CPM/1000

Impressions per Visitor = (Visits/Visitor) * (Pageviews/Visit) * (Impressions/Pageview)

Visitors = New Visitors + Returning Visitors

New Paying Users = Free Trial Users * Trial Conversion Rate + Direct Paid Signups

Viral Loop Metrics

Many times, to maximize the ROI of a digital marketing campaign, it is important to track how viral is my campaign leading to organic growth. Following metrics aid in that process:

1. Percentage of users who are active (V1)= Active users/ Registered Users
2. Percentage of users sending invites (V2)= Users who send invites/active users
3. Average number of invites sent per sender (V3)
4. Invite clickthrough rate (V4)
5. Registration conversion rate (V5)

Viral coefficient = V1*V2*V3*V4*V5

If the viral coefficient is greater than 1, the campaign is definitely viral. However, if it is above a certain threshold, it still means decent progress. For example, a viral coefficient of 0.4 means that the user base is growing at a rate of 40% per unit time for free through viral marketing.