Analyzing Declining Metrics

This is part 2 of the product analytics series. In part 1 we covered what metrics are and how to choose them. Now we put them to work.

There are five main things you do with metrics:

This tutorial is about the first one: what to do when a key metric drops. We'll follow one example the whole way through.

Scenario: You're an analyst at Facebook. Overnight, friend requests sent per day dropped by 10%. Your manager wants to know why.

The temptation is to jump straight to guesses ("it must be the new UI!"). Resist it. A structured process gets you to the real cause faster and stops you from "fixing" the wrong thing.

Step 1: Define the Problem Clearly

Say exactly which metric dropped, by how much, and over what period.

• Vague: "Engagement is down."
• Clear: "Friend requests sent per day dropped 10% overnight." Or: "YouTube watch time dropped 12% over the past two weeks."

A precise problem statement tells everyone what you're investigating and what "fixed" will look like.

Step 2: Confirm the Drop Is Real

Before you form a single hypothesis, make sure there's actually something to explain. Skipping this step leads to three classic mistakes:

• spending days investigating a normal seasonal dip,
• treating a broken tracking script as a real trend, and
• "fixing" a problem that doesn't exist, and introducing new ones.

Ask three questions.

Has this happened before? Many drops follow a pattern. If friend requests fall every September because students go back to school, that's seasonality, not a problem to fix. Compare against the same period last year.

Is the drop bigger than normal noise? Every metric wobbles from day to day. If a metric normally moves ±3% daily, a 2% drop is nothing; a 15% drop deserves attention. Comparing the drop to the metric's usual standard deviation tells you whether it's outside the normal range.

Is the data correct? Sometimes the drop isn't real at all:

• a code deployment broke the tracking script,
• a database outage left gaps in the data, or
• the reporting pipeline is running late, so today's numbers look low.

If a logging failure stopped friend requests from being recorded, the fix is repairing tracking, not changing the product.

Step 3: List Possible Causes with MECE

Once you know the drop is real, you need a list of possible causes. The tool for this is MECE: Mutually Exclusive, Collectively Exhaustive.

• Collectively exhaustive means you cover every possible cause, so you don't miss one. If you only check whether creators uploaded less content and never look at competitors, you'll miss that TikTok just launched a feature that's pulling users away.
• Mutually exclusive means the categories don't overlap, so nobody investigates the same thing twice. "Algorithm changes" (did the recommendation logic shift?) and "content supply" (did creators upload fewer videos?) are separate questions with separate owners.

The simplest MECE split is internal (things we control), external (things we don't), and data (measurement problems). Here's the tree for our friend-request drop:

Friend requests sent dropped 10%
│
├── 1. Product (internal)
│   ├── UI/UX change: did we move or hide the "Add Friend" button?
│   ├── Bug: did a recent release break requests?
│   ├── Algorithm change: did we change how friend suggestions appear?
│   └── Engagement trade-off: did another feature grow at friend requests' expense?
│
├── 2. User behavior (external)
│   ├── Seasonality: do requests normally dip at this time of year?
│   ├── Competition: did TikTok launch a new social feature?
│   ├── Changing preferences: are people using group chats instead?
│   └── Privacy and regulation: did a new law restrict friend suggestions?
│
└── 3. Data (measurement)
    ├── Pipeline failure: is tracking broken?
    ├── Sampling bias: is the drop limited to one segment of the data?
    └── Reporting lag: is this a temporary delay?

The split also tells you how to respond. Internal causes you can fix directly (revert the change). External causes need a strategic response (a seasonal dip calls for an engagement campaign, not a product change). Data causes need debugging.

Where to look first: start with internal causes, because they're under your control and often the easiest to confirm and fix. If nothing turns up, move to external factors. (Any obvious tracking breakage should already have been caught in step 2.)

This framework works for any digital product, because every product has internal levers, lives in a competitive market, and relies on tracking that can fail. Only the specifics change:

Step 4: Find the Metrics That Drive the Main Metric

A headline metric is usually the result of several smaller ones. Metric decomposition breaks it into those parts so you can see which piece actually broke.

Take an online store where conversion rate fell from 5% to 3%:

The problem is in add-to-cart, not checkout or payments. Without decomposition, the team might have spent a week redesigning checkout for nothing.

Same idea for a streaming service where watch time fell:

People aren't leaving the platform; they're quitting episodes early. That's a content or recommendation question, not a reliability one.

Useful sub-metrics usually fall into three families:

• Engagement: session length, pages per session, funnel drop-off, feature usage.
• Conversion and retention: click-through rate, conversion rate, day-1/7/30 retention.
• Operational: load time, crash rate, latency.

If daily active users (DAU) dropped 10%, check whether retention fell (people leaving), session length fell (less engagement), or crashes rose (a technical problem).

Decomposition pays off three times: you fix the specific broken piece instead of the whole metric, your segmentation (next step) becomes more precise, and you can A/B test changes aimed at exactly that piece.

Step 5: Segment the Data

Now find out where the problem is: is it everywhere, or concentrated in one group?

Dimensions and measures

Two words you'll hear constantly:

• Dimensions describe who, what, where, and when: country, device, new vs. returning user, acquisition channel, subscription tier. They're the categories you slice by.
• Measures are the numbers: revenue, DAU, conversion rate, session time.

Measures tell you what happened. Dimensions help explain why. Slice the measure by each dimension and look for the slice where it fell hardest.

Here's an e-commerce site whose conversion rate dropped. Slicing it:

• By geography: the drop is concentrated in the EU.
• By device: only iOS users are affected.
• By channel: mostly users arriving from paid ads.

The search space just shrank from "the whole site" to "EU iOS users from paid ads."

Another example. Look at the numbers before and after a drop:

New users are hit hardest, so onboarding is a good place to look. Premium users are also down a lot, which is worth a separate question.

Internal and external dimensions

Some dimensions describe how people use the product (things you influence), and some describe the people themselves (things you don't):

If the drop lines up with an internal segment ("only users of the new Reels UI"), it points to a product cause you can fix. If it lines up with an external one ("only EU users"), it points to a market cause such as regulation, a regional outage, or a local competitor, and you adapt your strategy.

Two examples of how segments tell different stories:

• After a checkout UI change: returning users were fine but new users abandoned the flow (onboarding may be confusing), and iOS checkouts fell 30% while web was stable (likely an iOS bug).
• After a competitor's promotion: free-tier users churned more, EU users fell 12% while the US held steady, and paid-ad users converted 50% less (the competitor's offer and ads are winning those people).

Drill down

1. Start broad. Look at the major breakdowns and find the biggest gaps.
2. Combine segments. "Power users in the EU on mobile" can reveal effects that no single dimension shows.
3. Look over time. When did each segment start to diverge? That date often matches a release or an outside event.

Step 6: Analyze Statistically

By now you have suspects. Statistics help you separate the ones that matter from the ones that just happened at the same time.

Correlation checks whether two metrics move together. Did crashes rise at the same time engagement fell? Did higher ad spend line up with lower conversion? Pearson or Spearman correlation measures how strong the relationship is. But remember from part 1: correlation alone doesn't prove cause.

Regression goes further. It estimates how much each sub-metric moves the main metric while holding the others constant. For example:

$$\text{DAU} = \beta_0 + \beta_1(\text{Session Duration}) + \beta_2(\text{Crash Rate}) + \beta_3(\text{CTR}) + \varepsilon$$

• DAU is the dependent variable, the thing you're trying to explain.
• Session duration, crash rate, and CTR are the independent variables, the possible drivers.
• \(\beta_0\) is the baseline, and \(\varepsilon\) is everything the model doesn't explain.
• Each coefficient (\(\beta_1, \beta_2, \beta_3\)) says how much DAU changes when that driver changes by one unit, with the others held fixed.

When you read the output:

• Coefficients give the size and direction of each effect.
• P-values say whether an effect is distinguishable from noise. Below 0.05 is the usual cutoff.
• R² says how much of the variation in DAU the model explains. An R² of 0.8 means 80%.

A worked example

DAU dropped 10% last month. You fit the model and get:

$$\text{DAU} = 1000 + 0.8(\text{Session Duration}) - 50(\text{Crash Rate}) + 10(\text{CTR})$$

Reading it:

• Each extra minute of session time is associated with +0.8 DAU.
• Each extra percentage point of crash rate is associated with −50 DAU.
• Each extra point of CTR is associated with +10 DAU.

Now you can compare possible fixes:

Fixing crashes is clearly the priority. You can also run "what if" scenarios before committing resources. Keep refitting the model as behavior changes, because a driver that mattered last quarter may plateau.

One caution: regression on observational data still shows association. It tells you where to look and what to test. Proving cause takes an experiment, which is the next step.

Step 7: Validate Your Hypotheses

Before acting, confirm the suspected cause is actually responsible.

• Compare before and after. Does the timing of a product update line up exactly with the drop?
• Run an experiment. If you think a new UI added friction, A/B test reverting it for a share of users.
• Read qualitative feedback. Are support tickets complaining about the same thing?

For a YouTube watch-time drop, the hypothesis might be "the algorithm change reduced binge-watching." You'd validate it by comparing session lengths before and after the update, and then by testing the old algorithm against the new one.

Step 8: Prioritize and Act

Diagnosis is only useful if something changes. Rank the fixes by impact and effort:

For an e-commerce checkout drop, that might mean: revert the UI change that added friction, fix the payment bug that caused failures, and test retargeting ads to high-intent users. Then keep monitoring to confirm the metric recovers.

Putting dimensions and measures together

The final picture usually comes from crossing dimensions with measures. For an ad conversion drop:

That points to two hypotheses worth testing: a mobile bug causing checkout friction, and a competitor's evening ads pulling users away.

Key Takeaways

1. Define the drop precisely: which metric, how much, and over what period.
2. Confirm it's real: check seasonality, normal variation, and tracking before anything else.
3. Use MECE to list causes as internal, external, and data, and start with what you control.
4. Decompose the metric to find the specific piece that broke.
5. Segment by dimensions to find where it broke, then quantify with measures.
6. Analyze with correlation and regression to rank drivers, remembering that they show association, not proof.
7. Validate with before/after comparisons, experiments, and user feedback.
8. Act on high-impact fixes first, then keep monitoring.

Previous: Introduction to Product Analytics · Next: Measuring Success