---
title: "How Do I Backtest Customer Health Score Rules Against Real Churn?"
description: "Backtest customer health score rules before rollout: score last year's churn and renewals as they looked 90 days out, build the 2 by 2, pass at 60% recall."
topic: "Metrics"
author: "Jay Bheda, Co-founder, GainTrace"
audience: "CS Operations, Head of Customer Success"
published: 2026-09-04
modified: 2026-09-04
source: https://gaintrace.com/explore/metrics/backtest-customer-health-score-against-churn
---

# How Do I Backtest Customer Health Score Logic Before Rollout?

*Score the accounts as they looked 90 days out, then count the misses*

**Short answer:** Backtest customer health score logic against the last 12 months of churn before you roll it out. List every account that churned or renewed, rebuild each score as it looked 90 days before the decision, and count hits and misses in a 2 by 2 table. Pass at 60% recall and 33% precision. If it fails, fix the inputs, not the weights.

**Key takeaways**

- Score accounts as they looked 90 days before the renewal date or cancellation notice, never as they look today. Today's data already knows the answer.
- Report three numbers, never one: recall (share of churned accounts that were flagged), precision (share of flagged accounts that churned) and the false alarm rate. Accuracy on its own is meaningless when 9 in 10 accounts renew.
- The pass line is 60% recall at 33% precision at 90 days. Below that, the score sends CSMs to the wrong accounts more often than the right ones.
- Always compare the score with one naive rule, such as "flag any account whose logins fell 30%". If the score does not beat the rule by 10 points of recall, it adds nothing to the rule.
- Rerun the same test every quarter on a rolling four-quarter window and print the result next to the score. A score that shows its own hit rate gets trusted for what it is.

You need to backtest customer health score version two before it goes to the team, and nobody in the company has done this before. The first score was login frequency and a sentiment field, and it went green on accounts that cancelled. This one has usage breadth, ticket volume, NPS and a champion flag, the weights came out of a two-hour meeting, and the CSMs have already said they will not act on another colour they do not trust. Someone senior has asked a fair question: how do we know this one works?
This page is the answer in the form of a method you can run in a spreadsheet in a day. It sets out what data you need, the step-by-step test, a worked example with 20 accounts, precision and recall in plain words, a pass threshold you can defend, and what to do when the score fails, which it usually does the first time.

## Why does almost nobody backtest a health score, and what does that cost?

A [customer health score](https://gaintrace.com/blog/customer-health-score) is a prediction: this account is more likely to leave than that one. Sales forecasts get scored against bookings. The health score ships on the strength of a meeting and stays in production because the colours look plausible.

We read 1,328 threads from r/CustomerSuccess, r/SaaS, r/sales and r/startups. 56 discuss a health score and 46 discuss predicting churn. Not one uses the word backtest, and only a handful describe checking a score against accounts that left. The threads that come closest describe the cost of skipping it.

> "Right now our "risk detection" is a health score I built in a spreadsheet. Red when logins drop. Problem is, by the time logins drop, the decision to leave already happened weeks earlier. I'm flagging accounts I should have called a month ago."
>
> — r/CustomerSuccess, 2026

> "I tried building a health score in a spreadsheet. It flags accounts that already stopped logging in, which is like a smoke detector that goes off after the house burns down."
>
> — r/SaaS, 2026

The 3,628 public G2 reviews of the three most-reviewed customer success platforms tell the same story from the buyer's side. 347 mention a health score or churn score, and 30 of those call it inaccurate, misleading or not to be trusted. What recurs is a CSM deciding, privately, how much to believe the number.

> "I usually take this score with a grain of salt because it is not always 100% accurate and churn risk is usually too nuanced to be able to be described with a single number."
>
> — Client Success Manager, enterprise SaaS, public G2 review

The cost is not the churn the score missed. It is the six months of CSMs learning that red means nothing, after which a good score cannot get their attention either. Backtesting is how you avoid spending that trust on a score that had not earned it.

## What data do I need before I start?

The test needs three things: a list of outcomes, the ability to see each input as it stood on a past date, and enough churn events to count. The middle one is where most attempts stall, so check it first.

| Input | Where the history lives | How to reconstruct it at T minus 90 | Trap |
| --- | --- | --- | --- |
| Product usage (logins, active users, feature breadth) | Event warehouse, product analytics export, or the daily aggregate table your platform keeps | Sum events in the 30 days before the observation date. Compare with the 30 days before that for the change version. | Using a usage total that includes the final months. The account looks dead because it churned, which proves nothing. |
| Support tickets | Helpdesk export with created dates | Count tickets opened in the 90 days before the observation date. Note priority. | Counting tickets closed rather than opened; closed dates cluster around offboarding. |
| Billing (overdue invoices, downgrades, seat changes) | Billing system invoice and subscription history | State of the subscription and any invoice over 14 days late on the observation date. | Reading current plan rather than plan at the time. Contractions before churn get missed. |
| Champion and sponsor | CRM contact history, email bounces, calendar attendance | Was the signing champion still in role and still replying on the observation date. | CRM contact records overwritten on role change, so history is gone. If so, use last email reply date. |
| NPS or CSAT | Survey tool export with response dates | Most recent response before the observation date, and whether there was one at all. | Treating no response as neutral. Non-response is itself a signal and belongs in its own column. |
| CSM sentiment or manual override | CS platform field history, if tracked | Value on the observation date, if you can get it. Otherwise exclude. | The field usually has no history and was edited after the churn was known. Exclude rather than guess. |

On sample size: you need at least 15 churn or contraction events for percentages to mean anything, and 30 is comfortable. A book of 200 accounts at typical B2B retention produces 15 to 25 logo churns a year; if you have fewer, pool 24 months and include contractions above 20% as events. With under 10 events, report counts rather than percentages and treat the result as a smell test, not a verdict.

Include renewed accounts as well as churned ones. A backtest that only looks at churned accounts can tell you recall and nothing else. The false alarms live in the renewed group.

## How do I backtest customer health score rules, step by step?

> **The T-90 rebuild:** The T-90 rebuild reconstructs what each account's inputs looked like 90 days before it churned, then scores it with today's rules. It is the only way to test a score you changed last quarter against churn from last year, and it takes an afternoon with exports rather than a data project.

1. **Fix the window and the outcomes.** Take every account with a renewal decision in the last 12 months. Outcome is churned (cancelled, or contracted by more than 20% of ARR) or renewed. Exclude accounts under 120 days old at the observation date; a new customer scored on usage volume fails for reasons the [accuracy page](https://gaintrace.com/explore/metrics/customer-health-score-accuracy-why-its-wrong) covers, and they would distort this test.
2. **Set one observation date per account.** 90 days before the renewal date for annual contracts, or 90 days before the cancellation notice for accounts that gave one. For monthly contracts use 60 days before the last paid invoice. The observation date is when the score would have had to fire for a CSM to have done anything useful.
3. **Rebuild every input as it stood on that date.** Using the table above. Put each account on one row with the raw inputs at T minus 90, then apply the proposed weights and thresholds to produce the score and colour. If your platform can show a historical score, use it, but check three accounts by hand against raw data first. Historical scores are often recomputed with today's rules.
4. **Decide what counts as flagged.** Usually red, or red plus yellow. Run both. A score is a ranking, and the cut-off is a separate decision from the score itself. Write down which cut-off you are testing.
5. **Build the 2 by 2.** Four cells: churned and flagged (hits), churned and not flagged (misses), renewed and flagged (false alarms), renewed and not flagged (correct passes). Every account lands in exactly one cell.
6. **Compute recall, precision and the false alarm rate.** Recall is hits divided by all churned. Precision is hits divided by all flagged. False alarm rate is false alarms divided by all renewed. Write all three next to the cut-off you tested.
7. **Run one naive rule through the same table.** Pick the single simplest signal you have, usually "logins in the last 30 days fell 30% against the prior 30", and score every account on that alone. If the composite score does not beat the naive rule by at least 10 points of recall at similar precision, the weights are adding noise, and you should ship the rule instead until the score can beat it.
8. **Repeat with change instead of level.** Recompute each usage and ticket input as change against the account's own prior 90 days and run steps 4 to 7 again. In every backtest we have seen described, the change version separates churned from renewed accounts better than the level version. Keep whichever wins on your data.

## What does a backtest of 20 accounts look like?

The figures below are illustrative, built to show the mechanics on a sample small enough to read. Eight of the 20 accounts churned or contracted; that is a deliberate oversample of churn, which is fine for testing recall and precision as long as you do not read the false alarm rate as a book-wide rate.

| Account | ARR | Level score at T minus 90 | Level colour | Change score at T minus 90 | Change colour | Outcome |
| --- | --- | --- | --- | --- | --- | --- |
| A1 | $48,000 | 31 | Red | 22 | Red | Churned |
| A2 | $12,000 | 38 | Red | 35 | Red | Churned |
| A3 | $21,000 | 54 | Yellow | 41 | Red | Contracted 40% |
| A4 | $66,000 | 72 | Green | 39 | Red | Churned |
| A5 | $9,000 | 69 | Green | 58 | Yellow | Churned |
| A6 | $30,000 | 29 | Red | 24 | Red | Churned |
| A7 | $15,000 | 52 | Yellow | 44 | Red | Churned |
| A8 | $84,000 | 78 | Green | 71 | Green | Churned |
| B1 | $24,000 | 81 | Green | 80 | Green | Renewed |
| B2 | $18,000 | 63 | Yellow | 74 | Green | Renewed |
| B3 | $40,000 | 37 | Red | 62 | Yellow | Renewed |
| B4 | $11,000 | 76 | Green | 79 | Green | Renewed |
| B5 | $55,000 | 58 | Yellow | 66 | Green | Renewed |
| B6 | $27,000 | 88 | Green | 85 | Green | Renewed |
| B7 | $14,000 | 70 | Green | 68 | Green | Renewed |
| B8 | $33,000 | 55 | Yellow | 43 | Red | Renewed |
| B9 | $8,000 | 34 | Red | 57 | Yellow | Renewed |
| B10 | $61,000 | 83 | Green | 88 | Green | Renewed |
| B11 | $19,000 | 74 | Green | 77 | Green | Renewed |
| B12 | $45,000 | 67 | Green | 38 | Red | Renewed |

| Score and cut-off | Hits | Misses | False alarms | Correct passes | Recall | Precision | False alarm rate |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Level, red only | 3 | 5 | 2 | 10 | 38% | 60% | 17% |
| Level, red or yellow | 5 | 3 | 5 | 7 | 63% | 50% | 42% |
| Change, red only | 6 | 2 | 2 | 10 | 75% | 75% | 17% |
| Change, red or yellow | 7 | 1 | 4 | 8 | 88% | 64% | 33% |
| Naive rule: logins down 30% | 4 | 4 | 3 | 9 | 50% | 57% | 25% |

> **Reading the table:** The level score at red only, which is how most teams would have shipped it, catches 3 of 8 churned accounts. Two of the three biggest losses (A4 at $66,000 and A8 at $84,000) were green. Widening the cut-off to yellow lifts recall to 63% but flags 5 of 12 renewing accounts, so almost half of the CSM's rescue calls would go to accounts that were fine. The naive login rule, with no weights at all, beats the level score outright. The change score at red only catches 6 of 8 with only 2 false alarms, and is the version to ship. A8 stays green in every version: the champion left and usage held up, which is a missing input, not a weighting problem.

## What do precision, recall and false alarms mean in plain words?

Recall answers the CSM's question: of the customers who left, how many did the score warn me about? 75% recall means three in four churns had a red flag 90 days out, and one in four blindsided you.

Precision answers the manager's question: when the score says red, how often is it right? 75% precision means three of every four rescue calls go to an account that would have churned. The other call is a false alarm, which is not wasted (the customer still gets attention) but has a cost when the team has four proactive hours a week.

The two pull against each other. Flag more accounts and recall rises while precision falls; that is what the red-or-yellow rows show. The cut-off is a capacity decision. A team that can make 20 rescue calls a quarter should set the cut-off where about 20 accounts are flagged and read precision at that point.

Do not report accuracy. If 10% of the book churns, a score that flags nobody is 90% accurate. Google's machine learning course makes the same point with the same arithmetic. Keep the base rate in view: with median gross revenue retention at 88% in the Benchmarkit 2025 data for calendar 2024, most B2B SaaS books lose 10% to 15% of revenue a year, so the score is looking for roughly one account in eight, and any test that ignores that ratio will flatter a useless score.

**Recall (the number that matters)**

```
Recall = Churned accounts the score flagged red ÷ All churned accounts × 100
```

Where:
- Churned accounts the score flagged red: accounts that left and were red at least 90 days before they gave notice
- All churned accounts: every account that left in the backtest window, flagged or not
- Read it as: the share of churn the score would have warned you about in time to act

**Precision (the false alarm cost)**

```
Precision = Churned accounts the score flagged red ÷ All accounts the score flagged red × 100
```

Where:
- All accounts the score flagged red: every red flag raised in the window, including the accounts that renewed
- Read it as: the share of red flags worth a CSM's time. At 20% precision a CSM chases four healthy accounts for every real one

## What is a passing score for a health score backtest?

There is no published benchmark for health score recall that we would put our name to, so this is our rule, and the reasoning is the capacity arithmetic above. Ship a score when, at the cut-off you intend to use, it reaches 60% recall and 33% precision at 90 days out, and beats the naive rule by 10 points of recall. 60% recall means the score sees most churn with a quarter's notice. 33% precision means at least one in three rescue calls lands, which is the level at which CSMs keep acting on the flag.

| Result | What it means | Do next |
| --- | --- | --- |
| Recall 60% or more, precision 33% or more | The score sees most churn early and is right often enough to act on | Ship it. Print the numbers on the dashboard. Rerun in a quarter. |
| Recall 60% or more, precision under 33% | The score sees churn but flags too much of the book, so CSMs will learn to ignore it | Raise the cut-off, or add a trend input so accounts are flagged on deterioration rather than level. Retest. |
| Recall under 60%, precision 33% or more | When it fires it is right, but most churn never trips it. An input is missing, usually the champion or billing movement | Look at the misses one by one. What was true of them at T minus 90 that the score did not measure? Add it. Retest. |
| Both under the line | The score is not separating the two groups. Weights cannot fix this | Ship the naive rule for now. Rebuild the inputs from the failure-mode list on the accuracy page before testing again. |

One more check before you ship: look at recall by ARR band. A score that catches 70% of churn overall but 30% of churn in the top band is failing where it matters. In the worked example, the level score missed both accounts over $60,000. That pattern, where large accounts keep their usage up while the decision is being made elsewhere, is the most common way a score passes on paper and fails on revenue.

## What do I fix first when the backtest fails?

The instinct after a bad result is to turn the weights until the numbers improve. Resist it. With 20 accounts you can always find weights that fit last year, and they will fit last year only. A score that fails a backtest almost always fails for a structural reason, and [customer health score accuracy: why the score says green and they churn](https://gaintrace.com/explore/metrics/customer-health-score-accuracy-why-its-wrong) lists the nine of them with the fix for each. The backtest tells you which ones you have.

- Misses that were green with steady usage: the score measures level, and needs change against the account's own baseline. This is the most common failure and the cheapest fix.
- Misses where the champion or admin changed in the window: contact-level signals are absent. Add role change and reply latency as inputs.
- False alarms that are all under six months old: new customers are being scored on volume. Give them a separate onboarding score.
- False alarms with high ticket counts: ticket volume is being read as risk when for many products it is engagement. Test tickets as change, and split by priority.
- Misses concentrated in one product or one parent account: one score is averaging over things that need their own score.

**Before you touch a weight**
- [ ] Every miss has been read individually and tagged with the reason the score did not see it.
- [ ] Every input has been tested on its own across the churned and renewed groups, and any input that looks the same in both has been removed.
- [ ] The change version of each usage input has been tried and compared.
- [ ] Champion status and billing movement are in the score, or you have written down why they cannot be.
- [ ] The naive rule's numbers are on the same sheet as the score's.
- [ ] New customers and parent accounts have been excluded or scored separately.

If the score still fails after the inputs are fixed, the honest position is that a rules-based score cannot separate churn in this book, and the choice is between shipping the naive rule with a weekly sweep of the signals in [early warning signs of churn when your data is scattered](https://gaintrace.com/explore/retention/early-warning-signs-of-churn-scattered-data), or letting a model learn the weights from your own churn history. Either is better than a colour nobody believes.

## How do I rerun the backtest every quarter?

A score that passed in January drifts: the product changes, the customer mix changes, and the CSMs change how they use the score. Rerunning is a two-hour job once the first sheet exists.

1. Keep a rolling window of the last four quarters of renewal decisions. Each quarter, drop the oldest and add the newest.
2. Freeze the score at the observation date automatically. If the platform cannot store a point-in-time score, export the score for every account on the first of each month to a sheet; that export is the backtest data for next year.
3. Recompute the confusion matrix at the shipped cut-off, plus the naive rule.
4. Print the result where the score is shown: "This score flagged 11 of 14 churned accounts at 90 days in the last four quarters, with 6 false alarms."
5. Read the new misses one by one, as in the checklist above. New misses are how you learn what changed.
6. Change weights or inputs only after a rerun, never between them, and record what changed and why on the same sheet.

The printed hit rate tells a CSM exactly how much to believe the flag. One thread put the point plainly.

> "Health scores can become theater if they only label accounts. Green, yellow, red. Useful, but incomplete. The important question is what action changes because of the score. [...] A score without ownership is just a label on the account."
>
> — r/CustomerSuccess, 2026

Ownership here means someone reruns the test, someone reads the misses, and the number on the dashboard is the one from last quarter's test. If you are building the first version rather than validating a second one, [customer health score template in a spreadsheet](https://gaintrace.com/explore/metrics/customer-health-score-template-spreadsheet) starts from a change-based design that tends to pass this test first time.

## How does GainTrace backtest continuously?

GainTrace does not ask you to set weights and test them once a year. It connects billing, CRM, product usage and support, stores every account's signals as they stood on every day, and learns from your own renewals and churn which changes predicted which outcomes, so the test on this page runs on every renewal as it happens. [Churn prediction](https://gaintrace.com/solutions/churn-prediction) shows each flagged account with the signals behind the call and the model's own hit rate on your book, and [product signals](https://gaintrace.com/platform/product-signals) show the change against baseline that drove it, with the refresh time of every source visible.

## Frequently asked questions

### How do I backtest a customer health score in a spreadsheet?

One row per account with a renewal decision in the last 12 months. Columns for each input as it stood 90 days before the decision, the resulting score and colour, and the outcome. Count hits (churned and red), misses (churned and not red), false alarms (renewed and red) and correct passes. Recall is hits over churned; precision is hits over flagged. Compare with one naive rule on the same rows.

### What is a good recall and precision for a health score?

Our rule is 60% recall and 33% precision at the intended cut-off, 90 days before the decision, and the score must beat a single naive rule such as logins down 30% by 10 points of recall. There is no published industry benchmark we trust for this; the thresholds come from what a CSM team can act on without learning to ignore the flag.

### How many churned accounts do I need to test a health score?

At least 15 churn or contraction events for percentages to mean anything; 30 is comfortable. A 200-account book usually produces that in 12 to 24 months if you count contractions over 20% of ARR as events. With fewer than 10, report counts, read each miss individually, and treat the test as a smell test rather than a pass or fail.

### Why should I score accounts 90 days before renewal rather than at renewal?

Because the score only matters if it fires while a CSM can still do something. At the renewal date the score is describing a decision already made, and usage data from the final weeks will make almost any score look predictive. 90 days is the window in which a call, an exec conversation or a fix can change the outcome. For monthly contracts use 60 days.

### Our health score passed the backtest but CSMs still don't trust it. What now?

Print the backtest result next to the score: how many of last year's churns it flagged and how many false alarms it raised. Then tie one specific action to each flag and let CSMs see which flagged accounts renewed after the action. Trust follows a visible hit rate and a visible consequence, not a colour.

## How this was researched

We read 3,628 public G2 reviews of the three most-reviewed customer success platforms, isolated the 347 that mention a health score or churn score, and counted the 30 that call the score inaccurate, misleading or untrustworthy in those words. We read 1,328 threads from r/CustomerSuccess, r/SaaS, r/sales and r/startups, of which 56 discuss a health score and 46 discuss predicting churn, and found none that uses the word backtest. The method, the 90-day observation rule and the pass threshold are ours; the 20-account example uses illustrative figures. We have no customer data to report.

## Sources

- [Google Machine Learning Crash Course: Classification: accuracy, recall, precision and related metrics](https://developers.google.com/machine-learning/crash-course/classification/accuracy-precision-recall)
- [Benchmarkit 2025 B2B SaaS Performance Metrics Benchmarks (calendar 2024): median GRR 88%](https://www.hibob.com/wp-content/uploads/2025-SaaS-Performance-Metrics-Benchmarks.pdf)
- [r/CustomerSuccess: How do you predict churn without a data scientist?](https://reddit.com/r/CustomerSuccess/comments/1vhis9n/how_do_you_predict_churn_without_a_data_scientist/)
- [r/CustomerSuccess: Why isn't our health score actually predicting churn?](https://reddit.com/r/CustomerSuccess/comments/1vv37fw/why_isnt_our_health_score_actually_predicting/)
- [r/CustomerSuccess: What customer health signal has actually predicted risk for you?](https://reddit.com/r/CustomerSuccess/comments/1ubjgtd/what_customer_health_signal_has_actually/)
- [r/SaaS: We only find out customers churned after the cancellation email](https://reddit.com/r/SaaS/comments/1w56ced/we_only_find_out_customers_churned_after_the/)

## Next steps

Run the 20-row test on your own book this week, then let the score learn from every renewal instead of a yearly spreadsheet. [Start free](https://app.gaintrace.com/auth/login) or [book a demo](https://gaintrace.com/booking).