Most teams treat data extraction as a single prompt they tweak until it looks good on a few examples. That works for a demo and falls apart in production. A real extraction operation is a sequence of plays, each with a trigger that fires it, an owner who runs it, and a defined hand-off to the next play. When you treat extraction as an operating system rather than a one-off prompt, the work becomes repeatable, auditable, and survivable when the person who built it leaves.
This playbook lays out that operating model. It is organized as a set of plays you can lift directly into your own runbook. For each play you get the trigger, the owner, the action, and the exit condition that moves work downstream. The sequencing at the end shows how the plays chain together across the lifecycle of an extraction project.
Think of it less as a tutorial and more as a reference you return to when something breaks at two in the morning.
Play One: Scope the Extraction Target
Trigger
A new request lands: someone needs structured data out of a document type the team has not handled before.
Owner and action
The data lead owns scoping. The action is to define the schema before anyone writes a prompt. List every field, its type, whether it can be null, and an example value drawn from a real document. Pull ten to twenty representative source documents, including the ugly ones, because the edge cases hide in the documents nobody wants to look at.
Exit condition
A written schema and a sample document set exist. Without these two artifacts, no prompt work begins.
Play Two: Build the Baseline Prompt
Trigger
A signed-off schema and sample set are available from Play One.
Owner and action
A prompt engineer drafts a zero-shot prompt that states the task, embeds the schema, and instructs the model to return null for absent fields and to quote supporting text for each value. The first draft should be deliberately simple. Resist the urge to add rules for cases you have not yet seen fail.
Exit condition
The baseline runs end to end on the sample set and produces parseable output, even if accuracy is not yet acceptable. For the reasoning behind starting simple, the A Framework for Prompting for Data Extraction piece lays out the underlying model.
Play Three: Establish the Evaluation Harness
Trigger
A baseline prompt produces output worth measuring.
Owner and action
The data lead labels the sample documents with ground-truth answers, then builds a script that runs the prompt across all samples and reports precision and recall per field. This harness becomes the scoreboard for every change that follows. Never tune a prompt by eyeballing a handful of outputs once the harness exists.
Exit condition
A reproducible score exists for the baseline. Every future change is judged against it.
Play Four: Tighten With Targeted Examples
Trigger
The harness shows a field or category underperforming.
Owner and action
The prompt engineer adds one or two few-shot examples chosen specifically to teach the failing pattern, then reruns the harness. The discipline here is one change at a time. If you add three examples and adjust two instructions at once, you cannot attribute the score change to any single edit.
Exit condition
The targeted metric improves without regressing others. If a change helps one field and hurts another, revert and rethink. The Best Practices That Actually Work guide covers how to choose examples that generalize rather than overfit.
Play Five: Handle Scale and Length
Trigger
Real documents exceed the context window or the volume exceeds what a single synchronous call can handle.
Owner and action
An engineer implements chunking with overlapping windows for long documents and a merge step that deduplicates on a stable key. For volume, batch requests where the platform supports it and add concurrency with rate-limit handling. Each chunk extraction is independent, so this play parallelizes cleanly.
Exit condition
The pipeline processes a full production-sized batch within the latency and cost budget.
Play Six: Add the Confidence and Review Routing
Trigger
The pipeline is accurate in aggregate but individual records still need a trust signal for high-stakes use.
Owner and action
The data lead defines a confidence rule. A common one: a record is high confidence when every required field has supporting quoted text and the output parsed on the first attempt. High-confidence records flow through automatically; the rest route to a human review queue.
Exit condition
A measurable share of records auto-approves, and the review queue is small enough for the team to clear. The Real-World Examples and Use Cases article shows a routing rule applied to invoice processing.
Play Seven: Monitor and Re-Baseline
Trigger
The pipeline is live, or a model version changes, or source documents drift in format.
Owner and action
The data lead schedules a periodic rerun of the evaluation harness against a fresh sample of recent production documents. Drift is real: vendors change invoice templates, model providers update versions, and a prompt that scored well in March can degrade by June. Treat the eval harness as a smoke detector you test on a schedule.
Exit condition
Scores stay within an agreed tolerance, or a regression triggers a return to Play Four.
Sequencing the Plays
The plays run in order for a new extraction target, but they loop. Plays One through Three are the setup phase and happen once per document type. Plays Four through Six are the build phase and iterate until the harness clears your accuracy bar. Play Seven is the steady state and runs forever in the background.
The hand-offs are the load-bearing parts. Scope hands a schema to the baseline. The baseline hands output to the harness. The harness hands a score that gates every tuning change. Skipping a hand-off is how teams end up with a prompt nobody can reproduce or improve.
Frequently Asked Questions
Who should own an extraction pipeline?
Ownership splits between a data lead, who owns the schema, evaluation, and confidence rules, and a prompt or software engineer, who owns the prompt and the surrounding code. On small teams one person wears both hats, but the responsibilities stay distinct so nothing falls through the cracks.
How long does it take to stand up a new extraction target?
For a familiar document type with an existing harness, a day or two. For a genuinely new type with messy sources, budget one to two weeks, with most of the time going into scoping, labeling, and evaluation rather than prompt writing.
What is the most commonly skipped play?
The evaluation harness. Teams under deadline pressure tune prompts by inspecting a few outputs, which feels faster but produces brittle pipelines that fail silently in production. The harness is the difference between an operation and a guess.
Do I need all seven plays for a small project?
No. A one-off extraction of a few hundred records may only need Plays One, Two, and a lightweight version of Three. The full playbook earns its keep when the pipeline runs continuously and the cost of a silent error is high.
How do I keep the playbook itself up to date?
Treat the runbook as a living document owned by the data lead. After every incident or significant change, update the relevant play. A playbook that does not reflect how the pipeline actually behaves is worse than no playbook, because it creates false confidence.
Key Takeaways
- Extraction is an operating system of plays with triggers, owners, and hand-offs, not a single prompt.
- Always scope a schema and gather messy sample documents before writing any prompt.
- The evaluation harness is the non-negotiable scoreboard; never tune by eyeballing once it exists.
- Make one change at a time so you can attribute every score movement.
- Confidence routing and scheduled re-baselining turn a working pipeline into a durable one that survives drift.