Measuring Success: A Structured Framework

This is part 3 of the product analytics series. So far we've covered how to choose metrics (part 1) and how to diagnose a metric that suddenly drops (part 2). In the drop scenario, we diagnosed the problem, formed a hypothesis about the cause, and used an A/B test to confirm it.

Now we turn to a different question, one every analyst hears sooner or later:

"So... how's the product doing?"

When a CEO or stakeholder asks that, they don't want a list of numbers. They want to know whether the product is delivering value, why it is or isn't, and what to do next. That's what measuring success is about. Below is a seven-step framework for answering the question properly.

Why Measure Success?

There are two reasons.

1. To judge the changes you've already made. Maybe you updated the onboarding flow, shipped a new UI, or tweaked notifications. Did those changes actually deliver what you hoped?
2. To decide what to do next. Once you see what's working and what isn't, you can pick the next areas to improve and set targets based on data instead of opinion.

It isn't enough to know that a product works. You need to know why it works and how it could work better.

What a Good Answer Needs

To answer "how's the product doing?" well, you need three things.

1. A deep understanding of the product. Before choosing any metric, be clear on:

• the product's core purpose: what problem does it solve?
• the benefits it provides to users and to the business, and
• what outcomes would show it's delivering that value.

Then look at how different users engage with it. Some care about ease of use; others want advanced features. Looking at groups separately shows which parts of the product drive engagement, where users find the most value, and what needs work.

2. A short list of metrics that matter. Not all metrics are equally useful, and tracking the wrong ones pulls the team in the wrong direction. Start from the product's purpose and pick the metrics that best reflect it. Avoid vanity metrics: total downloads or page views look nice but say little about real impact. For example:

• A subscription product should focus on retention and customer lifetime value, not just sign-ups.
• A social platform should care about how deeply and how often people engage, not just total users.
• An e-commerce site should track conversion rate, average order value, and checkout completion, not just traffic.

3. A structured analysis. With the right product understanding and the right metrics, you can connect the numbers to business goals (revenue, engagement, retention) and tell a clear story that stakeholders can act on.

Five Things to Get Right

Before the step-by-step framework, here are the areas any success measurement has to cover.

1. Line up with business goals. If the company wants more recurring revenue, focus on subscriptions, renewals, and retention, not just new users. If it wants more market share, focus on acquisition, activation, and engagement.
2. Pick a top-level success metric. Common ones are Monthly Active Users (ongoing engagement), conversion rate (how well users take a desired action), and retention rate (long-term value). Whatever you choose must reflect what success really means for your organization.
3. Add supporting sub-metrics. Break the main metric into pieces that explain its movement. If conversion rate is the main metric, the supporting ones might be add-to-cart rate (are people interested?), cart abandonment (are they dropping off?), and checkout completion (are purchases finished?).
4. Set guardrail metrics. This is the one people forget. Guardrails catch the side effects of an optimization. Say you raise conversion by adding promotional pop-ups. If bounce rate spikes or satisfaction falls, a guardrail should flag it. Common guardrails are user satisfaction (CSAT, NPS), session duration, and churn rate. They stop you from improving one number at the expense of the product's overall health.
5. Dig into the why. Numbers alone don't explain themselves. Segment users (new vs. returning, mobile vs. desktop), slice by dimensions like traffic source or device, and use tests (A/B tests, cohorts, time-series comparisons) to confirm why a number moved.

The Seven-Step Framework

Step 1: Be clear about what you're measuring

This step sets the boundaries.

Define the scope. Which product, feature, or initiative are you evaluating? The whole mobile app? A new onboarding flow? A search recommendation engine? If you try to measure everything, you end up measuring nothing.

Identify the target users. Different users define success differently. A project management tool might serve both freelancers and large companies. Freelancers care about ease of use and quick task tracking; enterprise teams need collaboration features and scale. Success looks different for each.

Consider where the product is in its life.

Measuring the wrong thing at the wrong stage wastes effort. Counting sign-ups for a mature product, or squeezing revenue out of one that hasn't found its users yet, sends the team in the wrong direction.

Step 2: Understand the product, its users, and its value

If you don't understand how the product creates value, you'll pick the wrong success metrics.

State the product's purpose in one sentence.

• A mobile banking app helps people manage their money easily from their phone.
• An AI resume builder helps job seekers create professional resumes quickly.

Identify the main user groups and what each needs. In a fitness app, casual users care about daily steps and simple reminders, while serious athletes want detailed performance analytics and training insights. An average across both groups hides what each one values.

Connect user value to business goals. If the goal is revenue, focus on paid subscriptions. If it's market share, track acquisition and engagement in new regions. If it's engagement, daily active users take center stage. If you skip this link, you'll optimize metrics that don't move the business.

Step 3: Choose a North Star Metric

The North Star Metric (NSM) is the single number that best captures the core value your product delivers. It's the guiding light that product, marketing, and operations all work toward.

A good North Star:

• captures the main reason users keep coming back,
• reflects long-term success, not a short-term spike, and
• guides decisions, so teams can see which work actually moves it.

Some well-known examples:

Common mistakes:

• Picking a vanity metric. App downloads alone don't mean anyone uses the app.
• Picking something teams can't influence. If no team's work can move it, it can't guide decisions.
• Ignoring either side. A real North Star balances what's good for users with what grows the business.

Step 4: Break the North Star into drivers and levers

You can't improve a North Star directly. You improve the things that feed it. Break it into key drivers and then into actionable levers.

For Airbnb:

$$\text{Nights Booked} = \text{Active Guests} \times \text{Nights per Guest}$$

To grow nights booked, you either get more guests or get each guest to book more nights.

For an online store:

$$\text{Total Orders} = \text{Active Users} \times \text{Purchase Frequency}$$

Either acquire more users or get existing users to buy more often.

Keep going until you reach levers: specific, measurable things a team can change. An online store's conversion depends on:

• Click-through rate: are people opening product pages?
• Add-to-cart rate: are they adding items?
• Checkout completion rate: are they finishing the purchase?

Now the numbers point to the problem:

• High click-through but low add-to-cart: people see the products but aren't interested enough. Look at pricing or product selection.
• Strong add-to-cart but weak checkout completion: something in checkout is getting in the way. Maybe payment options are unclear or shipping costs are a surprise.

Instead of "we need more bookings," the team can say "we need more repeat guests" or "we need to fix checkout conversion." That's a goal you can actually work on.

Step 5: Map the user journey and set baselines

Lay out the stages a user goes through and give each stage one or two metrics:

Then record a baseline for each one: where it stands today. If your notification open rate is 30% now and 40% after a change, you have clear evidence of improvement. Without the "before" picture, you can't prove anything got better.

This map also shows where users get stuck, stage by stage, instead of leaving you to guess.

Step 6: Find the insights, and form a hypothesis if you need one

You don't always need a hypothesis. Sometimes the job is simply to track trends, check assumptions, or understand patterns. Start by understanding what drives success, using the earlier steps:

1. Start from the North Star. For example, Monthly Active Users for a social app, or total orders for a store.
2. Look at its drivers. If conversion is the focus, check add-to-cart rate, checkout completion, and average order value.
3. Segment. Compare new vs. returning users, mobile vs. desktop. Are mobile users converting less than desktop users? If so, dig into why before assuming anything.

If you plan to test a change, write a clear, testable hypothesis.

• Observation: Mobile users convert at lower rates than desktop users.
• Hypothesis: Simplifying mobile checkout (fewer steps, better payment options) will increase mobile conversion by 10%.

If you're not testing a change, keep tracking, monitoring, and refining what you've learned.

Step 7: Pick the right way to measure

A/B testing isn't the only tool, and it isn't always the right one. Choose based on the question.

In practice you combine them. Funnel analysis finds the step where people drop off, feedback suggests why, and an A/B test confirms the fix works.

A Quick Example

Suppose you're asked how a meal-kit subscription app is doing.

1. Scope: the core subscription product, in its growth stage, so engagement and retention matter most.
2. Value: busy people get easy home-cooked dinners. Users split into weekly cooks and occasional users.
3. North Star: meals cooked per active subscriber per week.
4. Drivers: active subscribers × meals per subscriber. Levers include onboarding completion, menu selection rate, and skip rate.
5. Journey and baselines: sign-up conversion 4%, first-box completion 85%, week-8 retention 55%.
6. Insight and hypothesis: occasional users skip weeks when the menu doesn't appeal. Hypothesis: personalized menu recommendations will cut the skip rate by 10%.
7. Method: an A/B test on menu personalization, with churn and support tickets as guardrails.

Now "how's the product doing?" has a structured answer, and a plan.

Key Takeaways

1. Measure the why, not just the what. The point is to explain performance and guide decisions, not to confirm things work or show off vanity numbers.
2. Start with context: scope, users, value, business goals, and lifecycle stage decide which metrics matter.
3. Anchor on a North Star Metric that reflects the long-term value users get.
4. Break the North Star down into drivers and levers that teams can actually move.
5. Map the user journey and record baselines at every stage.
6. Protect against side effects with guardrail metrics.
7. Only form a hypothesis when you're testing a change, and make it specific and measurable.
8. Pick the method that fits the question. A/B tests are powerful, but DiD, cohorts, time series, regression, funnels, and user feedback each have their place.

Previous: Analyzing Declining Metrics · Next: Feature Impact Analysis