When a team first decides to extract a knowledge graph from a pile of documents, the same questions surface in roughly the same order. How specific does the schema need to be? Should extraction happen in one pass or several? How do you keep the model from inventing relationships? Each of these has a real answer, but the answer almost always depends on the shape of your text and the cost of being wrong.
This article collects the questions that come up most often and answers them with the reasoning behind each recommendation. The goal is not a checklist you follow blindly. It is enough understanding of the trade-offs that you can decide what fits your documents, your budget, and the tolerance your downstream consumers have for a noisy graph.
We have organized the answers from the questions people ask on day one toward the ones that only surface after a pipeline has been running for a while.
Getting Started Questions
What exactly does extraction produce?
Knowledge graph extraction turns unstructured text into a set of triples: a subject entity, a relationship, and an object entity. "Marie Curie discovered radium" becomes (Marie Curie) -[discovered]-> (radium). The model identifies the entities, classifies the relationship, and ideally points back to the text that supports each triple. The output loads into a graph database or feeds a reasoning layer.
How precise does my schema need to be?
More precise than feels comfortable. The single biggest determinant of extraction quality is whether the model knows exactly which entity types and relationship types it should produce. A vague instruction like "extract the relationships" invites the model to invent its own vocabulary, which changes from document to document. A closed list of relationship types, each with a one-line definition and an example, produces output you can actually aggregate.
Do I need a graph database to start?
No. You can develop and evaluate extraction prompts entirely against JSON files. The graph database matters for querying and reasoning over the finished graph, but it adds nothing to prompt development. Get extraction quality right against flat output first, then worry about storage and traversal.
Quality and Accuracy Questions
How do I stop hallucinated relationships?
Three reinforcing controls. First, give the model a closed relationship vocabulary and forbid anything outside it. Second, require that every triple include the source text span supporting it, then verify that span exists in the document. Third, set a confidence threshold below which extractions route to human review rather than straight into the graph. Each control catches a different class of error, and together they make confident fabrication rare.
Why does the same document produce different graphs on different runs?
Some variation is sampling noise, which you can reduce with lower temperature. But the more interesting cause is genuine ambiguity in the text. If "the company restructured its partnership with the vendor" can plausibly be read three ways, the model will sometimes pick each. Rather than forcing determinism, treat run-to-run disagreement as a signal that a passage is ambiguous and deserves review. The same content checks we describe in What People Get Wrong About Pulling Graphs From Text help separate noise from real ambiguity.
How do I handle entities that appear under different names?
This is entity resolution, and it is best handled close to extraction rather than far downstream. Maintain a canonical list of entities you have already seen and pass it to the model so it extends that list instead of minting a new node for every spelling. Resolving "IBM" and "International Business Machines" to one node during extraction prevents the fragmented graphs that are painful to merge later.
Scaling and Operations Questions
One pass or multiple passes?
For short, simple documents, one pass is cheaper and fine. For long documents or graphs where recall matters, decompose: extract and resolve entities first, then run a second pass that asks only about relationships among the known entities. Decomposition costs more tokens but recovers relationships that span distant paragraphs and makes the pipeline auditable. The staging is covered step by step in Building a Repeatable Workflow for Prompting for Knowledge Graph Extraction.
How do I keep cost under control at volume?
Three levers. Use the smallest model that clears your quality bar, which is often smaller than people assume once the schema is tight. Cache extraction results so reprocessing the same document is free. And reserve multi-pass extraction for documents that need it rather than applying it uniformly. A cheap classifier that routes long or complex documents to the expensive path keeps average cost low.
How do I evaluate extraction quality at scale?
Build a gold set of documents with hand-labeled triples and measure precision and recall against it on every prompt change. Precision tells you how many extracted triples are correct; recall tells you how many true triples you found. Track both, because tightening the prompt to reduce hallucination often quietly drops recall. A held-out gold set is the only way to see that trade-off honestly.
Integration and Reasoning Questions
How do I connect the extracted graph to downstream applications?
The graph becomes useful when something queries it. Most teams load validated triples into a graph database and expose traversal queries to the application: find all entities related to X by relationship type Y, or trace a path between two entities. Keep the application querying a stable interface rather than the raw extraction output, so changes to the extraction pipeline do not ripple into every consumer. The clean boundary between candidate triples and committed graph, described in Building a Repeatable Workflow for Prompting for Knowledge Graph Extraction, is what makes that interface stable.
Can the model reason over the graph it extracted?
Extraction and reasoning are separate jobs. Extraction populates the graph; reasoning answers questions over it. You can hand a relevant slice of the graph back to a model and ask it to draw conclusions, but mixing extraction and reasoning in one prompt usually degrades both. Keep extraction focused on faithfully capturing what the text says, and run reasoning as a distinct step over the trusted graph.
What happens when sources contradict each other?
Two documents may assert conflicting relationships about the same entities. The graph should record both with their provenance rather than silently picking a winner. Conflict is information: it tells the application that the underlying sources disagree, which is often more valuable than a falsely confident single answer. Resolution of contradictions is a downstream policy decision, not something extraction should quietly make.
Frequently Asked Questions
Can I extract a knowledge graph without machine learning expertise?
Yes, for prompt-based extraction. The skills that matter most are precise schema design and disciplined evaluation, both of which are closer to careful specification than to model training. You will need engineering support to run the pipeline at volume, but defining the schema and judging quality is work a domain expert can lead.
How long does it take to build a usable pipeline?
A rough baseline that extracts entities and a handful of relationship types can come together in days. Reaching production quality, with entity resolution, content validation, and an evaluation harness, typically takes weeks. Most of that time goes into edge cases and the gold set, not the core prompt.
What document types are hardest to extract from?
Documents with dense cross-references, implicit relationships, and inconsistent terminology. Legal contracts and scientific papers are hard because meaning depends on context spread across the document. Clean, templated text like product specs is much easier. Match your effort to the difficulty of your corpus.
Should I extract everything or only what I need?
Only what you need. A focused schema covering the entity and relationship types your application actually queries produces a cleaner, more accurate graph than an attempt to capture everything. You can always expand the schema later; over-broad extraction just adds noise you have to filter.
How do I keep the graph fresh as documents change?
Track which triples came from which document version. When a document changes, re-extract it and reconcile the new triples against the old ones, retiring relationships that no longer have source support. Provenance at the triple level is what makes incremental updates possible without rebuilding the whole graph.
Key Takeaways
- Schema precision, not model size, is the main driver of extraction quality; define a closed relationship vocabulary with examples.
- Prevent hallucination with three layers: closed vocabulary, span verification, and confidence-based human review.
- Resolve entities during extraction by passing a canonical list, not as a downstream cleanup, to avoid fragmented graphs.
- Decompose long documents into entity and relationship passes to protect recall; reserve the expensive path for documents that need it.
- Measure precision and recall against a hand-labeled gold set so you can see the trade-offs every prompt change introduces.