This is a composite case study drawn from the kinds of latency problems that recur across teams shipping AI features. The names and exact figures are illustrative, but the arc — diagnosis, decision, execution, outcome — mirrors what actually happens when a team gets serious about inference latency. We use a single thread so the lessons connect rather than float as bullet points.
The setup: a mid-sized SaaS company launched an AI support assistant. Internally it worked beautifully. In production, usage was cratering. Sessions started, the user typed a question, and then a large fraction of them left before the bot answered. The team assumed the model was "too slow" and prepared to swap it. They were about to make a classic mistake.
What follows is what they did instead.
The Situation
The assistant used a large, capable model behind a hosted API. Quality was excellent — when users waited for an answer, they liked it. The problem was that fewer and fewer users were waiting.
The symptom
The product dashboard showed a 220-millisecond average response time, which looked fine, yet session abandonment within the first message sat near 30 percent. The average and the experience disagreed violently. That contradiction was the first real clue.
The Decision: Measure Before Changing
The engineering lead resisted the swap-the-model instinct and instead insisted on instrumentation first. The team had been logging one number: total request time. They split it into network, queue, time to first token, and inter-token latency, and they switched from averages to percentiles.
The data was damning. The p50 TTFT was 280 ms, but the p95 TTFT was 3.9 seconds. The "220 ms average" was a fiction created by mixing a fast median with a brutal tail and mislabeling the metric. Nearly a third of users were hitting that tail — exactly matching the abandonment rate. This is the percentile trap described in 7 Common Mistakes with AI Inference and Latency.
The Execution
With the real bottleneck identified — tail TTFT, not the model's raw speed — the team worked the problem in order.
- Found the cause of the tail. Under load, requests queued because batching was misconfigured and a huge system prompt was reprocessed on every call. The tail was prefill plus queueing, not decode.
- Cached the prompt prefix. The 1,800-token system prompt was identical on every request. Caching it as a prefix removed most of the per-request prefill.
- Trimmed context. Conversation history was being sent in full; they capped it to recent turns and summarized the rest.
- Enabled streaming. The assistant had been waiting for the complete answer before showing anything. They switched to streaming tokens.
- Co-located and tuned batching. They moved the app closer to the inference region and tightened batch windows for the interactive path.
Crucially, they changed these one at a time and re-measured after each, following the discipline laid out in A Step-by-Step Approach to AI Inference and Latency. They never did swap the model — it was never the problem.
The Outcome
After the changes, p95 TTFT fell from 3.9 seconds to roughly 600 milliseconds, and p99 came under control for the first time. With streaming, the perceived experience improved even further: users saw text begin almost immediately rather than facing a blank pause.
The measurable result
First-message abandonment dropped from around 30 percent to single digits. Notably, the model and its quality were unchanged — the entire win came from latency engineering, not a better model. The team had nearly spent weeks swapping models to fix a problem that had nothing to do with the model.
The Lessons
This case compresses most of what matters about inference latency into one story:
- The average is a liar; the tail is the truth.
- Instrument before you change anything, or you will fix the wrong thing.
- A huge static system prompt is a latency tax you can cache away.
- Streaming converts the same underlying latency into a dramatically better experience.
- The most visible knob (the model) is often not the bottleneck.
What the Team Changed in Their Process
The technical fixes mattered, but the durable win was a process change. Before this incident, latency was something one engineer checked when someone complained. Afterward, it became a standing part of how the team shipped.
New defaults that stuck
- Percentiles on the main dashboard. The misleading average was retired. p95 and p99 TTFT now sit on the primary view, so the tail can never hide again.
- Load testing before launch. No AI feature ships without a test at expected concurrency, because the team learned the hard way that single-request numbers lie.
- Token counts in code review. Reviewers now flag prompts that balloon context, catching bloat before it reaches production.
None of these are exotic. They are the difference between fixing latency once and staying fast. The team had treated speed as tunable-later; the abandonment numbers taught them it is a design requirement, the same lesson argued in AI Inference and Latency: Best Practices That Actually Work.
What Would Have Happened Without Diagnosis
It is worth sitting with the counterfactual, because it is the more common path. Had the team trusted the instinct to swap the model, here is the likely sequence: weeks spent integrating a smaller model, a quality regression that drew its own complaints, and a tail latency that barely moved because queueing and prefill were untouched. They would have shipped a worse product and still had a slow one.
The 30 percent abandonment would have persisted, now blamed on the new model rather than the real cause. Eventually someone would have instrumented the pipeline anyway, arriving at the same diagnosis after burning the budget. The only thing the detour would have bought is delay and a lower-quality assistant. Diagnosis first is not a nicety; it is what separated a two-week win from a two-month dead end.
Frequently Asked Questions
Why didn't the average reveal the problem?
Because response times are right-skewed, a fast median pulls the average down even when a large tail of slow requests exists. The team's "220 ms average" coexisted with a 3.9-second p95. Only splitting into percentiles exposed the tail that was driving abandonment.
Was swapping the model ever justified?
No, and that is the point. The model's decode speed was fine; the latency came from queueing and prefill. Swapping it would have cost weeks, possibly hurt quality, and left the real bottleneck untouched. Diagnosis saved them from an expensive non-fix.
How much did caching the system prompt help?
It was one of the largest single wins. An 1,800-token prompt reprocessed on every request is pure waste when it never changes. Caching it as a prefix removed most of the per-request prefill cost and pulled the tail down sharply.
Did streaming change the real latency or just the feel?
Both, in effect. Streaming did not lower total generation time, but it slashed perceived latency by showing output immediately. Combined with the TTFT fixes, it transformed an experience that felt frozen into one that felt responsive.
What is the single most transferable lesson?
Measure before you change, and measure percentiles. Nearly every mistake the team almost made flowed from trusting an average and reaching for the most visible knob. Disciplined instrumentation turned a guessing game into a solvable problem.
Key Takeaways
- A flattering average hid a 3.9-second p95 that matched the 30 percent abandonment rate.
- Instrumenting per-segment percentiles revealed the real bottleneck: tail TTFT from queueing and prefill.
- Caching the static system prompt and trimming context removed most of the prefill cost.
- Streaming converted the same latency into a far better perceived experience.
- The model was never the problem — diagnosis prevented a costly, useless swap.
- Changing one thing at a time and re-measuring made every gain verifiable.