Sizing rules are useful, but they come alive only when you see them applied to real workloads. This guide walks through four concrete scenarios, each with the actual compute decision behind it: what the workload needed, what hardware it landed on, and what made the difference between a smooth deployment and an expensive scramble.
These are composite examples drawn from common patterns, not specific companies, but the numbers and reasoning reflect how these decisions genuinely play out. Read them as worked problems. For each, notice the chain from workload definition to memory math to hardware tier — the same chain laid out in our step-by-step guide.
Let's start with the most common scenario.
Example 1: A Customer Support Chatbot
A mid-size company wants an internal support assistant answering questions for about 40 concurrent agents.
The workload
- Inference only, no training.
- A 13B model, chosen for a balance of quality and cost.
- Interactive — agents wait for responses, so latency matters.
What worked
The team quantized the 13B model to 4-bit, dropping VRAM from roughly 33 GB to about 9 GB, and served it on a single 24 GB workstation card with room for batching. By batching concurrent requests, one GPU comfortably served all 40 agents.
The lesson
Quantization moved this workload down an entire hardware tier with no noticeable quality loss for support answers. The team that originally scoped a datacenter GPU saved substantially, exactly the over-provisioning trap from our common mistakes guide.
Example 2: Overnight Document Summarization
A legal team needs to summarize 200,000 documents, with results ready by morning.
The workload
- Inference, but batch — nobody is waiting in real time.
- Throughput matters far more than latency.
- A fixed deadline of roughly eight hours.
What worked
Because the work was interruptible and time-boxed, the team rented spot GPUs at a steep discount, ran maximum-batch inference overnight, and shut everything down at completion. Cost per document came out a fraction of an interactive setup.
The lesson
Matching the billing model to the workload — spot instances for interruptible batch work — was the entire win. The same job on always-on owned hardware would have wasted most of its capacity the other 16 hours a day.
Example 3: Fine-Tuning a Domain Model
A startup wants a model fluent in its niche industry vocabulary.
The workload
- Fine-tuning, not full training from scratch.
- A 7B base model.
- A one-time job, not a recurring service.
What worked
The team used parameter-efficient fine-tuning, which updates a small fraction of weights and slashes memory needs versus full fine-tuning. The job fit on a single rented 48 GB datacenter GPU for a few hours, then the instance was destroyed.
The lesson
They resisted training from scratch — the top of the cost ladder — and chose efficient fine-tuning instead. The renting-not-owning choice fit a one-time job perfectly, echoing our best practices guide.
Example 4: A High-Traffic Public API
A product team serves a public AI feature with unpredictable, spiky traffic.
The workload
- Inference, interactive, public-facing.
- Traffic swings from near-zero to large bursts.
- Reliability and latency both critical.
What worked
Rather than size for peak and idle the rest of the time, the team built an autoscaling fleet: a small baseline of reserved capacity plus rented GPUs that spin up during bursts. This met peaks without paying for them continuously.
The lesson
Elasticity reconciled "size for worst case" with "run at best case." The trade-off was added operational complexity, accepted because fixed peak provisioning would have wasted most capacity most of the time. The underlying memory math came straight from the complete guide.
Example 5: A Long-Context Document Assistant
A research team builds an assistant that answers questions over very long documents — full contracts, lengthy reports, entire transcripts.
The workload
- Inference, interactive.
- A moderately sized model, but fed extremely long inputs.
- Context windows pushing toward the model's maximum.
What worked
The team's first sizing, based on model weights alone, fit a 24 GB card. It then crashed in testing the moment users pasted long documents. The cause was the KV cache, which grows with context length and had been left out of the math. They re-sized for the longest realistic context, moved to a 48 GB card, and the crashes stopped.
The lesson
Long context is a memory cost, not just a model cost. Sizing for weights alone is a reliable way to pass testing with short inputs and then fail in production. The fix was to budget VRAM for the worst-case context, a point our checklist calls out explicitly.
What These Examples Have in Common
Five different workloads, five different hardware answers, and yet the same handful of moves appear again and again.
- Quantization moved at least two workloads down a hardware tier with no real quality cost.
- Matching the billing model to the workload — spot for batch, autoscaling for spikes — drove most of the savings.
- Sizing for worst case, whether peak traffic or longest context, prevented the production failures.
- Climbing the cost ladder kept the fine-tuning team off the expensive training rung.
The takeaway is that good compute decisions are not about memorizing which GPU to use. They are about applying the same disciplined chain — define, size, optimize, match — to whatever workload is in front of you. The numbers change; the method does not. That method is laid out step by step in our how-to guide.
How to Adapt These to Your Own Workload
These five scenarios are templates, not prescriptions. The value comes from matching your situation to the closest one and then adjusting.
If you are building anything interactive and user-facing, the chatbot and public-API examples are your starting points — think about batching first, then about whether your traffic is steady or spiky. If your work runs in bulk against a deadline, the summarization example applies, and spot capacity is almost certainly your friend. If you are customizing a model, the fine-tuning example points you toward parameter-efficient methods before anything heavier. And if you feed models long inputs, the document-assistant example is your warning to budget for the KV cache.
Identify which template fits, then walk your specific numbers through the same chain that example followed. The hardware answer will fall out of the workload, exactly as it did in each scenario above. When you want the underlying logic rather than the worked example, our complete guide and framework provide it.
Frequently Asked Questions
Why did the chatbot use one GPU for 40 users?
Because inference batches well. A single quantized 13B model on a 24 GB card can process many concurrent requests together, so aggregate throughput, not per-user hardware, served all 40 agents at once.
When are spot instances the right choice?
For interruptible, deadline-bound batch work like the summarization job. Spot capacity is heavily discounted but can be reclaimed, which is fine when work can checkpoint and resume but not for latency-critical live serving.
Is fine-tuning always cheaper than training from scratch?
Almost always, especially parameter-efficient fine-tuning, which updates only a small slice of weights. It fits jobs on a single GPU that full training could never accommodate, at a fraction of the compute.
How does autoscaling save money for spiky traffic?
It keeps only a small baseline running and adds capacity only during bursts. You avoid paying for peak-sized hardware during the long stretches of low traffic, at the cost of more complex operations.
Do these examples assume any specific cloud provider?
No. The reasoning — memory math, billing model matching, quantization, elasticity — applies across providers. The specific GPU tiers translate to comparable options wherever you run.
Key Takeaways
- Quantization let a single 24 GB card serve a 13B chatbot for 40 concurrent users.
- Spot instances made overnight batch summarization dramatically cheaper than always-on hardware.
- Parameter-efficient fine-tuning fit a domain model on one rented GPU instead of a cluster.
- Autoscaling reconciled peak provisioning with idle-time waste for spiky public traffic.
- In every case, matching the billing model and precision to the workload drove the savings.
- The same chain — define, size memory, optimize, choose tier — recurs across every scenario.