Most automation failures are not exotic. They are boring, predictable, and entirely avoidable with a short list of questions asked before launch. A workflow that looks finished in a demo can still be missing error handling, an owner, a rollback path, or a clear definition of what counts as a correct result. The gap between "it ran once" and "it runs every day without supervision" is filled with small decisions that are easy to skip when you are excited to ship.
This checklist is meant to be used, not just read. Each item is a yes-or-no question with a short justification, so you can walk a build top to bottom and find the weak joints before a client does. None of these checks require special tooling. They require honesty about what the automation actually does when the inputs get messy, the volume climbs, and nobody is watching.
Treat the list as a gate. If you cannot answer an item cleanly, that is the item worth fixing first. The order roughly follows the lifecycle of a build, from defining the job through securing it and handing it to the people who depend on it, so you can run it top to bottom in a single sitting.
Define the Job Before You Automate It
Is the manual process documented end to end?
You cannot automate a process you cannot describe. Write the steps a human takes today, including the judgment calls. If a step says "use your discretion," that is exactly the place automation will struggle. Documenting the manual flow surfaces hidden dependencies and the unwritten rules that make the work correct.
Is there a clear definition of a correct output?
Automation without an acceptance standard is just faster guessing. Decide what a good result looks like before you build, so you can test against it. A correctness definition also gives you something to monitor in production rather than waiting for complaints.
- Specify the format, the required fields, and the tolerance for error.
- Write three example outputs: one ideal, one acceptable, one that should be rejected.
Build for the Inputs You Will Actually Get
Have you tested with real, messy data?
Demo data is clean. Real data has missing fields, wrong encodings, duplicate records, and formats nobody warned you about. Run the automation against a sample of genuine inputs, not a curated set. The failures you find here are cheap; the ones you find in production are not.
Does the workflow degrade gracefully?
When something unexpected arrives, the automation should flag it, not silently produce garbage. Decide what happens to inputs that fall outside the expected shape. A quarantine queue for odd cases beats a confident wrong answer every time. The worst outcome is not a visible failure; it is a plausible-looking wrong answer that flows downstream and gets discovered three steps later by someone who has to unwind it by hand.
Have you tested the volume you actually expect?
A workflow that handles ten items cleanly can choke at ten thousand. Before launch, run a load that resembles a real busy day, including any predictable peaks like end-of-month or campaign launches. The failures that only appear under load, such as rate limits and queue backlogs, are the ones most likely to embarrass you in production.
Make Failure Visible and Recoverable
Is there logging you would actually read?
If a run fails at 2 a.m., the logs are your only witness. Capture the input, the step that failed, and the error. Logs that record only success are decoration. The goal is to reconstruct what happened without rerunning anything.
Can you roll back or reprocess?
Things will break. The question is whether you can recover without manual archaeology. A reprocess path lets you fix a bug and rerun the affected items. This is closely related to the reliability concerns covered in Building AI Workflow Automations That Actually Scale for Clients.
- Confirm that reruns are idempotent so a retry does not double-charge or double-send.
- Keep the original input so you can replay it after a fix.
Assign Ownership and Cost Limits
Does the automation have a named owner?
An unowned automation is an orphan that drifts until it breaks quietly. Assign a person responsible for its health, its alerts, and its retirement. Ownership is what separates a maintained system from a liability. The broader operational version of this is covered in How to Automate Your Own AI Agency Operations.
Are there spend and rate limits in place?
A looping automation that calls a paid model can run up a bill fast. Cap the spend per run and per day. Rate limits protect both your budget and any downstream API that could throttle or ban you.
Confirm Trust Before You Remove the Human
Is there a human-in-the-loop for high-stakes outputs?
Not everything should run fully unattended on day one. For anything that touches money, contracts, or client-facing communication, keep a person approving outputs until the error rate earns autonomy. You can remove the checkpoint later with evidence.
Have you measured the baseline you are improving?
Without a before number, you cannot prove the automation helped. Record how long the manual process took and how often it erred. That baseline is also what you will lean on when you justify the work, as outlined in Using AI Internally to Run Your AI Agency More Efficiently.
Plan for the Day It Changes
Will it survive a model or API change?
Providers update models and deprecate endpoints. Pin versions where you can and watch for change notices. A prompt that worked perfectly on one model release can drift on the next, so keep your test set ready to rerun.
Is there a documented off-switch?
Every automation needs a clean way to stop. Document how to pause it, who can do so, and what to tell stakeholders when it is down. An off-switch you have to invent during an incident is not an off-switch.
Verify Security and Data Handling
Does the automation only access what it needs?
An automation wired to a model often touches sensitive data on its way through. Scope its access to the minimum required, and confirm that no step quietly ships private data to a place it should not go. Over-broad access is a quiet liability that surfaces at the worst possible time, usually during an audit or a breach.
Are prompts and inputs sanitized against injection?
When user-supplied text flows into a model prompt, a malicious input can try to redirect the automation's behavior. Treat untrusted input as untrusted: separate instructions from data, and validate that outputs match the expected shape before acting on them. This is a failure mode that does not exist in a demo and very much exists in production.
- Grant least-privilege access to every system the automation touches.
- Keep a record of what data each run reads and writes.
Confirm It Fits How People Actually Work
Will the people affected actually use it?
An automation that technically works but disrupts how a team operates will be quietly abandoned. Walk the workflow with the people whose work it changes, and confirm the output lands where they expect, in the format they need. Adoption is part of correctness, not an afterthought.
Is the handoff between automation and human clean?
Most automations hand off to a person at some point, whether for approval, exception handling, or the next step. A messy handoff, such as an output buried in a log nobody reads, breaks the whole chain. Make sure the human in the loop receives what they need, when they need it, in a place they will see it.
Frequently Asked Questions
How long should it take to run through this checklist?
For a small automation, an hour. For something handling client money or sensitive data, plan a half-day and bring a second reviewer. The time cost is trivial next to the cost of a silent failure in production.
Do I need every item before launching?
The input, failure-handling, ownership, and cost items are non-negotiable. The human-in-the-loop and rollback items can be staged in, but only if you have a real plan to add them, not a vague intention.
What is the single most skipped check?
Testing with real messy data. Teams test the path they imagined and ship, then discover the path their users actually take. Genuine sample inputs catch the most failures for the least effort.
How often should I revisit the checklist for a live automation?
Re-run it whenever you change the model, the inputs, or the volume meaningfully, and on a fixed quarterly cadence otherwise. Automations rot slowly, so periodic review catches drift before it becomes an outage.
Can this checklist replace formal QA?
No. It is a pre-flight gate, not a substitute for testing and review. Think of it as the questions that make your QA more focused, not a reason to skip it.
Who should own the checklist itself?
Whoever owns delivery standards at your agency. Keeping one shared version means every automation is held to the same bar, and new team members inherit the standard instead of relearning it.
Key Takeaways
- Document the manual process and define a correct output before writing any automation.
- Test against real, messy inputs, and make sure odd cases are quarantined rather than silently mishandled.
- Logging, rollback, and reprocessing turn inevitable failures into recoverable events.
- Every automation needs a named owner, spend limits, and a documented off-switch.
- Measure a baseline so you can prove the automation helped and justify keeping it.
- Re-run the checklist whenever the model, inputs, or volume change.