The job sounds simple: read an incoming support ticket, decide which team should handle it, and send it there. We have shipped this feature three times. The first two versions were technically more accurate than the one we run today, and both made support worse.
Version one: a single classifier
We trained a model to predict one of nine team labels from the ticket text. Offline accuracy was 89%, which felt good enough to ship. In production, agents started ignoring the routing entirely within two weeks, because the 11% of misroutes were concentrated in the highest-value accounts, not spread evenly.
Version two: confidence thresholds
We added a threshold: anything under 80% confidence went to a human triage queue instead of a team. This fixed the misrouting but created a new bottleneck. The triage queue grew faster than one person could clear it, and low-confidence tickets are disproportionately the confusing, high-effort ones.
Version three: route to a team, not a person, and make the model explain itself
The change that actually worked was smaller than either previous rewrite. We kept the same classifier, but instead of a hard threshold we show the receiving team the model's top two guesses and its stated reason, and let a team lead reassign in one click if it's wrong.
Misroutes now get fixed in minutes by someone already on the team, not queued for a specialist.
We track reassignment rate per team weekly; anything above 15% triggers a model review.
The explanation text turned out to matter more than the label — agents trust a visible reason.
What we would tell someone starting this
Optimize for how wrong decisions get corrected, not just how often the model is right. A system that is wrong 15% of the time but recovers in one click beats one that is wrong 11% of the time and silently erodes trust.
