Why we killed our own lead scoring model

We built a lead scoring model into our own product, marketed it as a headline feature, and turned it off for most customers eight months later after our own usage data made a case we could not argue with.

What the model did

It assigned every inbound lead a 0-100 score based on firmographic fit and engagement signals, trained on each customer's own historical closed-won and closed-lost data. On paper, and in our own demos, it looked genuinely useful.

What actually happened in customer accounts

Usage data showed reps overwhelmingly used the score to justify decisions they had already made from instinct, rather than to change their behavior. When the score agreed with a rep's gut feeling, they cited it. When it disagreed, they ignored it and rarely mentioned it again. Net effect on close rate across our customer base: statistically indistinguishable from zero.

The harder problem we found underneath

For most of our customers, the training data itself was the actual constraint: too few historical deals, inconsistent CRM data entry, and sales cycles too long to have accumulated a large enough labeled dataset to train anything reliable on. The model was not badly built; the data it needed largely did not exist yet for companies our size.

What replaced it

• A much simpler rules-based fit score that reps could see the exact logic behind, which they trusted and used far more than the opaque model.

• Explicit intent signals (pricing page visits, repeat site visits from the same account) weighted more heavily than firmographic data alone.

• We now recommend machine-learned scoring only past roughly 500 closed deals in a customer's own history — below that, we tell customers honestly that a rules-based score will serve them better.