This is a working checklist, not a reading list. Run each item against your own pipeline and mark it pass or fail. Every item carries a one-line reason and a concrete pass condition, so there is no ambiguity about whether you've met it. The list is ordered from minimum viable to audit-grade, so you can stop wherever your stakes plateau.
If you fail items in the first section, fix those before touching anything later — the foundations gate everything above them. Two passing items in the audit tier mean nothing if your data snapshots are not pinned.
Foundations: The Reproducible Tuple
These are non-negotiable. Without them you do not have version control; you have backups.
- Code commit recorded — Every model version stores the exact training-code commit. Pass condition: you can `git checkout` the commit that produced any version.
- Data snapshot pinned — Every version references an immutable training set (hash or dataset version). Pass condition: you can diff the data between any two versions.
- Dependency lock captured — A locked manifest or container digest travels with each version. Pass condition: you can recreate the exact runtime environment.
- Hyperparameters and seeds logged — All training config and random seeds are recorded. Pass condition: a rerun reproduces metrics within noise.
- Artifact hashed — The output binary has a content hash stored in its version record. Pass condition: you can detect a swapped or corrupted artifact.
If any foundation item fails, start there. The first failure most teams find is the data snapshot, which is also the most expensive gap, as covered in 7 Common Mistakes with Ai Model Version Control.
Storage and Structure
How and where artifacts live determines whether the system scales.
- Recipe and artifact separated — Source control holds code and pointers; binaries live in object storage or a registry. Pass condition: cloning the repo does not download gigabytes of weights.
- No large binaries in plain Git — Checkpoints are not committed directly into the Git history. Pass condition: repo size is stable as model count grows.
- Versions are immutable — A registered version ID never gets overwritten. Pass condition: re-registering the same ID is rejected.
Promotion and Production
This section is where most velocity-first teams have the largest gaps.
Promotion gate
- Production runs a pinned version ID — Serving never points at a floating
latesttag. Pass condition: you can name the exact version live right now without checking a training log. - Promotion is gated — A candidate must match or beat production on a fixed eval set before promotion. Pass condition: a regressing candidate cannot be promoted.
- Promotion is logged — Each transition records version, approver, and timestamp. Pass condition: you can list every production change this year.
Rollback
- Last two versions deployable — Prior versions retain artifact, environment, and serving config. Pass condition: you can serve the previous version without rebuilding it.
- Rollback rehearsed — A rollback has been executed end to end in staging recently. Pass condition: the rehearsal succeeded on the first try.
These promotion and rollback items are the operational backbone described in Best Practices That Actually Work — the difference between shipping confidently and shipping blind.
Lineage and Traceability
This is what turns version control from a convenience into an auditable system of record.
- Experiment-to-release link — Every released version back-references the experiment run that created it. Pass condition: you can open any production model's originating run.
- Prediction-to-version mapping — Every served prediction is logged with the version ID that produced it. Pass condition: you can map a past decision to its model.
- Data lineage retained — Training data snapshots are kept for your full retention window. Pass condition: you can produce the exact data behind any retained version.
Audit-Grade (For Regulated or Client-Facing Work)
Add these when a regulator, client, or contract demands provability.
- Immutable promotion events — Promotion records cannot be altered after the fact. Pass condition: an attempt to edit a promotion event fails.
- Approver identity captured — Each promotion names a specific accountable human. Pass condition: "who approved this" always has an answer.
- Retention policy enforced — Versions tied to commitments are protected from deletion. Pass condition: cleanup jobs cannot delete a committed version.
For how these audit requirements play out in a real engagement, the credit-risk scenario in the examples piece shows what passing this tier actually buys you.
How to Use This Checklist
Run it quarterly, not once. Models, pipelines, and teams drift, and an item that passed in March can quietly fail by June when someone "temporarily" repoints production at a latest tag. Treat a failed foundation item as a stop-the-line event and a failed audit item as a deadline-bound remediation.
Scoring it honestly
The checklist only helps if you score it honestly, which is harder than it sounds. The temptation is to mark an item "pass" because the capability theoretically exists — you have a rollback procedure documented somewhere, so rollback passes. That's exactly the self-deception the pass conditions are designed to defeat. A pass condition is a test you actually run, not a capability you believe you have.
Apply the conditions literally. "Rollback rehearsed" passes only if you ran a rollback end to end recently and it succeeded on the first attempt. "Production runs a pinned version ID" passes only if you can name the live version right now without opening a training log. "Data snapshot pinned" passes only if you can produce a diff between two versions on demand. If you have to qualify a pass with "well, mostly" or "in theory," it's a fail, and treating it as one is what makes the checklist worth running.
Common Reasons Items Fail
A few failure patterns recur across teams, and recognizing them speeds remediation. Foundation items usually fail because recording is manual rather than automated — the discipline existed once and decayed. Storage items fail when someone commits a large checkpoint "just this once" and the pattern spreads. Promotion items fail when a deadline forced a quick repoint to a latest tag that nobody reverted.
The remediation for nearly all of these is the same: move the discipline from human memory into the pipeline or into CI enforcement. A check that fails the build when production points at a floating tag prevents that item from regressing. A pipeline step that writes the version record prevents foundation items from decaying. Automation is what converts a checklist from a periodic cleanup into a system that stays passing between reviews.
Frequently Asked Questions
Where should a brand-new project start?
With the five foundation items. They close the reproducibility gap that causes the most pain and cost the least to adopt. Add storage structure next, then promotion gates as you put models in front of users. Audit-grade items wait until a contract or regulator requires them.
How often should I re-run the checklist?
Quarterly at minimum, plus after any major pipeline change. Version control discipline decays silently, and the most common regression is production drifting back onto a floating tag. A recurring review catches drift before it becomes an incident.
What if I pass foundations but fail rollback rehearsal?
Then your rollback is a hope, not a capability. Prioritize rehearsing it in staging immediately, because the rehearsal almost always exposes a stale serving config or a changed feature schema. Fixing that before an incident is the entire point.
Is the audit tier overkill for internal models?
For purely internal, low-stakes models, you can defer it. The moment a model influences a customer-facing decision, a billing outcome, or anything a regulator might examine, the audit tier becomes mandatory rather than optional.
Key Takeaways
- The five foundation items — code, data snapshot, dependency lock, hyperparameters, artifact hash — are non-negotiable; fix failures here first.
- Keep recipes in Git and binaries in a registry; never commit large checkpoints to plain Git.
- Production must run pinned, immutable version IDs through a gated, logged promotion — never a floating
latest. - Lineage links (experiment-to-release and prediction-to-version) turn version control into an auditable system of record.
- Re-run the checklist quarterly, because discipline decays and production tends to drift back toward floating tags.