Why Is My Customer Health Score Accuracy So Poor?

Most health scores we have seen are a weighted average of whatever data was easy to get, tuned until the colours looked plausible on a dashboard, and never once checked against the accounts that left. A customer health score is a prediction. A prediction that has never been scored against outcomes is a guess with a colour.

We read 3,628 public G2 reviews of the three most-reviewed customer success platforms. 347 of them mention a health score or churn score. Reviewers who bought the platform for a score they could trust describe the same handful of failures: someone gamed it, new customers came out red, the parent account averaged its children into a meaningless yellow, the data was a day old, or the score was one number for three products. None of these is a weighting problem. Reweighting a score with a structural fault produces a different wrong answer.

Underneath all nine failure modes are two design mistakes. The score measures level (how much usage, how many tickets) when churn shows up as change (less than before, different from before). And the score has no feedback loop, so nothing about a churn ever changes the score that missed it.

Each row is a pattern that appears in the review corpus, the Reddit threads, or both. The symptom column is how you notice it in your own book. Find your rows first; most scores have three or four.

Four of the nine appear so often in the evidence, and do so much harm, that they deserve their own explanation.

Score last year's churned accounts as they looked 90 days before they left, then count how many the score called red. The test needs a churn list, a way to see historical scores (or a rebuild of the score from historical inputs), and a spreadsheet. Almost nobody runs it, and it is the only step that tells you whether the score is worth fixing.

1. 1

   List every account that churned or contracted in the last four quarters

   Include downgrades and seat reductions above 20%. Add the same number of accounts that renewed flat, chosen at random, as the control group.

2. 2

   Look up each account's score 90 days before the event

   If your platform cannot show a past date, rebuild the score from the inputs as they stood then. If you cannot do either, that is failure mode one and the rest of the test is moot until you can.

3. 3

   Count the misses

   How many churned accounts were green at 90 days. That share is your miss rate. How many renewed accounts were red. That is your false alarm rate.

4. 4

   Test each input on its own

   For every input in the score, compare its 90-day value across the churned group and the control group. An input that looks the same in both groups predicts nothing and should leave the score, whatever its weight.

5. 5

   Repeat with change instead of level

   Recompute each surviving input as change against the account's own prior 90 days and run the comparison again. Expect the change version to separate the groups better; the practitioners in our corpus who ran this comparison report the same thing, with login decay against the customer's own baseline as the usual example.

6. 6

   Write the result on the dashboard

   "This score flagged 6 of 14 churned accounts at 90 days last year." A score that shows its own accuracy gets trusted for what it is and ignored for what it is not.

Change the inputs before the weights, in the order below, and rerun the backtest after each round. The backtest leaves you with a list of inputs that predicted something and a list that did not. Changing everything at once tells you nothing about which change moved the miss rate.

1. Remove every input a person types. This is the fastest accuracy gain and the one with the most internal resistance.
2. Convert each remaining input from level to change against the account's own baseline, and add a trend field.
3. Add the champion and sponsor as inputs: role change, reply latency, meeting attendance.
4. Split new customers into their own onboarding score.
5. Split by product and by child account, and roll up as ARR at risk.
6. Smooth single events over 30 days so a call or a ticket cannot move the colour on its own.

The seventh item matters more than it looks. Practitioners in our Reddit corpus keep arriving at the same conclusion from different directions: an absolute score answers "is this customer healthy", when the question that changes behaviour is "what has changed in this account, and does the change matter".

Sometimes the backtest comes back bad after two rounds of changes. The decision rule we would use: if the miss rate at 90 days is still above 50% after you have removed typed inputs and converted to change, the problem is the inputs you have, not the weights. Either you are missing a source (usually the champion, the calendar or billing movements), or the book is too varied for one set of rules.

Missing a source is fixable without software. Early warning signs of churn when your data is scattered shows how to pull the earlier signals from the tools you already have. If the book is varied (three products, five segments, SMB and enterprise on one score) the honest answer is several small scores, and at that point you are choosing between an ops person who maintains them and a model that learns the weights from your own churn history. If you have no score at all yet and are starting from a spreadsheet, health score without a platform is the page to begin with, because the change-based design there avoids most of the nine modes from the start.

GainTrace does not ask you to pick weights. It connects billing, CRM, product usage and support, scores each account on change against its own baseline, and learns from your own renewals and churn which signals mattered, so the backtest above runs continuously rather than once a year. Churn prediction shows the accounts most likely to move, with the signals that drove the call, and product signals show the champion, usage and support changes behind each one, with the refresh interval visible.