How Do I Backtest Customer Health Score Logic Before Rollout?

A customer health score is a prediction: this account is more likely to leave than that one. Sales forecasts get scored against bookings. The health score ships on the strength of a meeting and stays in production because the colours look plausible.

We read 1,328 threads from r/CustomerSuccess, r/SaaS, r/sales and r/startups. 56 discuss a health score and 46 discuss predicting churn. Not one uses the word backtest, and only a handful describe checking a score against accounts that left. The threads that come closest describe the cost of skipping it.

The 3,628 public G2 reviews of the three most-reviewed customer success platforms tell the same story from the buyer's side. 347 mention a health score or churn score, and 30 of those call it inaccurate, misleading or not to be trusted. What recurs is a CSM deciding, privately, how much to believe the number.

The cost is not the churn the score missed. It is the six months of CSMs learning that red means nothing, after which a good score cannot get their attention either. Backtesting is how you avoid spending that trust on a score that had not earned it.

The test needs three things: a list of outcomes, the ability to see each input as it stood on a past date, and enough churn events to count. The middle one is where most attempts stall, so check it first.

On sample size: you need at least 15 churn or contraction events for percentages to mean anything, and 30 is comfortable. A book of 200 accounts at typical B2B retention produces 15 to 25 logo churns a year; if you have fewer, pool 24 months and include contractions above 20% as events. With under 10 events, report counts rather than percentages and treat the result as a smell test, not a verdict.

Include renewed accounts as well as churned ones. A backtest that only looks at churned accounts can tell you recall and nothing else. The false alarms live in the renewed group.

1. 1

   Fix the window and the outcomes

   Take every account with a renewal decision in the last 12 months. Outcome is churned (cancelled, or contracted by more than 20% of ARR) or renewed. Exclude accounts under 120 days old at the observation date; a new customer scored on usage volume fails for reasons the accuracy page covers, and they would distort this test.

2. 2

   Set one observation date per account

   90 days before the renewal date for annual contracts, or 90 days before the cancellation notice for accounts that gave one. For monthly contracts use 60 days before the last paid invoice. The observation date is when the score would have had to fire for a CSM to have done anything useful.

3. 3

   Rebuild every input as it stood on that date

   Using the table above. Put each account on one row with the raw inputs at T minus 90, then apply the proposed weights and thresholds to produce the score and colour. If your platform can show a historical score, use it, but check three accounts by hand against raw data first. Historical scores are often recomputed with today's rules.

4. 4

   Decide what counts as flagged

   Usually red, or red plus yellow. Run both. A score is a ranking, and the cut-off is a separate decision from the score itself. Write down which cut-off you are testing.

5. 5

   Build the 2 by 2

   Four cells: churned and flagged (hits), churned and not flagged (misses), renewed and flagged (false alarms), renewed and not flagged (correct passes). Every account lands in exactly one cell.

6. 6

   Compute recall, precision and the false alarm rate

   Recall is hits divided by all churned. Precision is hits divided by all flagged. False alarm rate is false alarms divided by all renewed. Write all three next to the cut-off you tested.

7. 7

   Run one naive rule through the same table

   Pick the single simplest signal you have, usually "logins in the last 30 days fell 30% against the prior 30", and score every account on that alone. If the composite score does not beat the naive rule by at least 10 points of recall at similar precision, the weights are adding noise, and you should ship the rule instead until the score can beat it.

8. 8

   Repeat with change instead of level

   Recompute each usage and ticket input as change against the account's own prior 90 days and run steps 4 to 7 again. In every backtest we have seen described, the change version separates churned from renewed accounts better than the level version. Keep whichever wins on your data.

The figures below are illustrative, built to show the mechanics on a sample small enough to read. Eight of the 20 accounts churned or contracted; that is a deliberate oversample of churn, which is fine for testing recall and precision as long as you do not read the false alarm rate as a book-wide rate.

Recall answers the CSM's question: of the customers who left, how many did the score warn me about? 75% recall means three in four churns had a red flag 90 days out, and one in four blindsided you.

Precision answers the manager's question: when the score says red, how often is it right? 75% precision means three of every four rescue calls go to an account that would have churned. The other call is a false alarm, which is not wasted (the customer still gets attention) but has a cost when the team has four proactive hours a week.

The two pull against each other. Flag more accounts and recall rises while precision falls; that is what the red-or-yellow rows show. The cut-off is a capacity decision. A team that can make 20 rescue calls a quarter should set the cut-off where about 20 accounts are flagged and read precision at that point.

Do not report accuracy. If 10% of the book churns, a score that flags nobody is 90% accurate. Google's machine learning course makes the same point with the same arithmetic. Keep the base rate in view: with median gross revenue retention at 88% in the Benchmarkit 2025 data for calendar 2024, most B2B SaaS books lose 10% to 15% of revenue a year, so the score is looking for roughly one account in eight, and any test that ignores that ratio will flatter a useless score.

There is no published benchmark for health score recall that we would put our name to, so this is our rule, and the reasoning is the capacity arithmetic above. Ship a score when, at the cut-off you intend to use, it reaches 60% recall and 33% precision at 90 days out, and beats the naive rule by 10 points of recall. 60% recall means the score sees most churn with a quarter's notice. 33% precision means at least one in three rescue calls lands, which is the level at which CSMs keep acting on the flag.

One more check before you ship: look at recall by ARR band. A score that catches 70% of churn overall but 30% of churn in the top band is failing where it matters. In the worked example, the level score missed both accounts over $60,000. That pattern, where large accounts keep their usage up while the decision is being made elsewhere, is the most common way a score passes on paper and fails on revenue.

The instinct after a bad result is to turn the weights until the numbers improve. Resist it. With 20 accounts you can always find weights that fit last year, and they will fit last year only. A score that fails a backtest almost always fails for a structural reason, and customer health score accuracy: why the score says green and they churn lists the nine of them with the fix for each. The backtest tells you which ones you have.

• Misses that were green with steady usage: the score measures level, and needs change against the account's own baseline. This is the most common failure and the cheapest fix.
• Misses where the champion or admin changed in the window: contact-level signals are absent. Add role change and reply latency as inputs.
• False alarms that are all under six months old: new customers are being scored on volume. Give them a separate onboarding score.
• False alarms with high ticket counts: ticket volume is being read as risk when for many products it is engagement. Test tickets as change, and split by priority.
• Misses concentrated in one product or one parent account: one score is averaging over things that need their own score.

If the score still fails after the inputs are fixed, the honest position is that a rules-based score cannot separate churn in this book, and the choice is between shipping the naive rule with a weekly sweep of the signals in early warning signs of churn when your data is scattered, or letting a model learn the weights from your own churn history. Either is better than a colour nobody believes.

A score that passed in January drifts: the product changes, the customer mix changes, and the CSMs change how they use the score. Rerunning is a two-hour job once the first sheet exists.

1. Keep a rolling window of the last four quarters of renewal decisions. Each quarter, drop the oldest and add the newest.
2. Freeze the score at the observation date automatically. If the platform cannot store a point-in-time score, export the score for every account on the first of each month to a sheet; that export is the backtest data for next year.
3. Recompute the confusion matrix at the shipped cut-off, plus the naive rule.
4. Print the result where the score is shown: "This score flagged 11 of 14 churned accounts at 90 days in the last four quarters, with 6 false alarms."
5. Read the new misses one by one, as in the checklist above. New misses are how you learn what changed.
6. Change weights or inputs only after a rerun, never between them, and record what changed and why on the same sheet.

The printed hit rate tells a CSM exactly how much to believe the flag. One thread put the point plainly.

Ownership here means someone reruns the test, someone reads the misses, and the number on the dashboard is the one from last quarter's test. If you are building the first version rather than validating a second one, customer health score template in a spreadsheet starts from a change-based design that tends to pass this test first time.

GainTrace does not ask you to set weights and test them once a year. It connects billing, CRM, product usage and support, stores every account's signals as they stood on every day, and learns from your own renewals and churn which changes predicted which outcomes, so the test on this page runs on every renewal as it happens. Churn prediction shows each flagged account with the signals behind the call and the model's own hit rate on your book, and product signals show the change against baseline that drove it, with the refresh time of every source visible.