Featured2 min read
How we cut LLM inference costs by 61% without touching model quality
Prompt caching, request batching and a routing layer did more than any model swap. Here is what we measured, in order.
2 articles
Topic
Large Language Models
Working with LLMs in production: cost, quality, evaluation and prompting.
Most read
Guide1 min read
Evaluating LLM output when there is no ground truth
Most of our extraction tasks have no single correct answer to grade against. This is the scoring approach we landed on after two that failed.