A playbook is not a tutorial. A tutorial teaches you a concept once; a playbook tells your team exactly what to do when a specific situation shows up, who does it, and what comes next. When you're operating ai model input and output modalities in production — fielding image uploads, transcribing calls, generating visuals — you need plays, not just understanding.
This is that operating model. Each play below has a trigger (the condition that calls for it), an owner (the role accountable for running it), and a sequence (what happens in order). The plays are deliberately small and composable so you can assemble them into whatever your product needs, then hand the whole thing off without losing the logic in someone's head.
We're assuming you already grasp the basics. If not, the Complete Guide covers the conceptual ground this playbook builds on. What follows is purely operational.
Play 1: Modality Intake and Routing
Trigger: A new input arrives — a file, a recording, a screenshot, a block of text.
Owner: The engineer who owns the ingestion endpoint.
The first decision in any multimodal system is where to send each input. Getting this wrong means paying multimodal prices on text that didn't need it, or starving a vision task of the encoder it required.
Sequence
- Detect the input type at the boundary — MIME type, file signature, or explicit field.
- Validate size and format against limits before anything hits a model.
- Route text to the text path, media to the appropriate encoder path.
- Reject or downscale anything outside acceptable bounds, with a clear error.
The routing layer is where cost discipline lives. A misrouted batch of high-resolution images is the single most common source of a surprise bill, a pattern the Common Mistakes breakdown returns to repeatedly.
One rule keeps this play clean: route on verified type, not on the file extension or the field name a caller claims. A file named .png may not be a PNG, and a "text" field may contain a base64-encoded image. Inspecting the actual content at the boundary is a few lines of code that prevents an entire class of mis-billing and downstream errors. Treat the caller's claim as a hint, never as the truth.
Play 2: Media Optimization Before the Model
Trigger: A media input is confirmed and routed but not yet sent.
Owner: The platform or infrastructure engineer.
Every image and audio file should pass through an optimization step before it ever reaches a model. This is non-negotiable at scale because media tokens dominate cost.
Sequence
- Downscale images to the smallest resolution that preserves task-relevant detail.
- Trim audio to the relevant window; don't transcribe silence or off-topic stretches.
- Strip metadata that adds tokens without adding signal.
- Cache the optimized artifact so repeat requests don't re-process.
This play alone often cuts media spend substantially without touching quality, because most uploads carry far more resolution than the task requires.
The owner should resist the temptation to make optimization "smart" before making it consistent. A fixed, conservative downscale applied to every image beats a clever adaptive scheme that's only wired into half the code paths. Get the play running everywhere first, measure, and only then tune the thresholds per use case. Coverage matters more than cleverness, because an unoptimized path that slips through quietly undoes the savings from every optimized one.
Play 3: Single-Model vs. Pipeline Decision
Trigger: A new feature requires a modality the system doesn't yet handle.
Owner: The technical lead or solution architect.
Before building, decide whether one multimodal model handles the feature or whether a pipeline of specialized models serves it better. This is an architecture decision, not a runtime one.
Sequence
- Identify whether the task requires reasoning across modalities or just within one.
- If cross-modal reasoning is core, default to a single multimodal model.
- If one modality demands top-tier quality, isolate it to a specialized model and feed its output downstream.
- Document the decision so the next engineer doesn't re-litigate it.
The Framework article provides the decision tree this play references; keep it linked in your runbook.
Play 4: Output Validation by Modality
Trigger: A model returns a result destined for a user or a downstream system.
Owner: The feature engineer shipping the output.
You cannot validate every modality the same way, and treating them uniformly is how bad output reaches clients.
Sequence
- Text output: Constrain to a schema where possible, validate structure programmatically, check against business rules.
- Structured extraction from media: Validate the schema, flag low-confidence fields, route those to review.
- Generated images: Run through a proven prompt template; keep human sign-off for client-facing assets until the template is trusted.
- Transcriptions: Spot-check domain vocabulary, which is where generalist models fail most.
The Best Practices guide expands each of these checks into concrete acceptance criteria.
Play 5: Cost and Latency Monitoring
Trigger: Continuous — this play runs always, not on a discrete event.
Owner: Whoever owns the production budget and SLAs.
Multimodal costs and latencies behave differently from text, so they need their own dashboards.
Sequence
- Track tokens-after-encoding per modality, not just request counts.
- Alert on per-modality cost spikes, which usually signal a routing or optimization regression.
- Watch p95 latency separately for media paths, which are slower and more variable.
- Review weekly and feed findings back into Plays 1 and 2.
The trap to avoid is monitoring request volume while ignoring tokens after encoding. Two features can issue the same number of requests yet differ tenfold in cost because one sends large images and the other sends short text. If your dashboard shows only request counts, you're blind to the metric that actually drives the bill. Instrument the encoded token count per modality from day one, or you'll be debugging cost surprises with no data to explain them.
Play 6: Graceful Degradation
Trigger: A modality-specific component fails, slows, or hits a rate limit.
Owner: The reliability engineer.
Media paths fail in ways text paths don't — encoder timeouts, oversized payloads, provider limits. Your system should degrade rather than collapse.
Sequence
- Fall back to a text-only path when a vision or audio component is unavailable.
- Queue and retry media processing rather than dropping it silently.
- Surface a clear, honest message when a modality genuinely can't be served.
Pairing this play with the Step-by-Step Approach gives newer engineers a runnable implementation path for each fallback.
Sequencing the Plays Together
Run them in dependency order, not feature order. Plays 1 and 2 are infrastructure and must exist before anything else ships. Play 3 happens once per feature, at design time. Plays 4 through 6 are per-feature runtime concerns layered on top.
A new feature therefore moves through: route it (1), optimize media (2), confirm the architecture (3), validate output (4), and only then ship — with monitoring (5) and degradation (6) already in place from the platform layer.
Frequently Asked Questions
Who should own the routing layer?
A single engineer or small platform team, not each feature team. Routing is shared infrastructure; distributing ownership leads to inconsistent cost controls and duplicated logic across features.
How often should the single-model vs. pipeline decision be revisited?
Per feature at design time, and again only when a meaningfully better model ships for one of your modalities. Don't re-evaluate on every release — document the call and move on until a real signal forces a change.
What's the most overlooked play here?
Media optimization before the model. Teams build routing and validation but skip downscaling and trimming, then absorb avoidable cost and latency on every single request.
Can these plays work with a single multimodal model?
Yes. Even if one model handles every modality, you still route inputs, optimize media, validate output, and monitor cost — the pipeline decision simply resolves to "single model" without removing the other plays.
How do I hand this playbook off?
Store each play as a runbook entry with its trigger, owner, and sequence, linked to the relevant code. A new owner should be able to read the entry and execute without tribal knowledge.
Key Takeaways
- A playbook assigns each situation a trigger, an owner, and a sequence — that's what makes multimodal operations hand-off-able.
- Routing and media optimization are platform-level plays that must exist before any feature ships.
- The single-model vs. pipeline choice is a per-feature design decision; document it so it isn't re-argued.
- Output validation differs by modality — text is checkable, generated media needs human or perceptual review.
- Monitor cost and latency per modality and pair every media path with a graceful text-only fallback.