A lot of evaluation advice assumes you have a labeled test set with one right answer per example. Half of what we build does not work that way: summarizing a ticket thread, extracting an ambiguous shipping address, or deciding whether a message needs escalation. Two answers can both be reasonable.
What did not work
Exact-match scoring against human-written references penalized correct answers that were phrased differently, so our reported accuracy was consistently lower than the actual quality of the output. A second attempt using a second LLM as judge was faster to build but drifted whenever we changed the judge prompt, which meant our scores were not comparable month to month.
What we use now: rubric scoring with a fixed judge prompt and human spot checks
For each task we write a short rubric of 3-5 yes/no criteria specific to that task (for extraction: did it find the field, is the format valid, did it hallucinate a value not present in the source). A fixed judge prompt scores every output against the rubric, and the judge prompt is versioned in the same repo as the model prompts, so changing one is a visible diff.
We freeze the judge prompt for a full evaluation cycle, never mid-comparison.
10% of judged outputs get a human review every week; disagreement over 8% triggers a rubric rewrite.
Rubric criteria are binary, never a 1-10 scale — reviewers do not agree on what a 7 means.
Where this leaves us
We still do not have a single accuracy number we are fully confident in, and we have stopped expecting one. What we have is a rubric score that moves in the same direction as human judgment closely enough to make prompt and model decisions with, which is the actual job.
