Key takeaways
- A checkpoint carries 2 kinds of state and 5 fields: current step, plan, per-step outputs, pause payload, and the model and prompt version in force.
- Write it after every tool call, not once per phase. A crash 1 second after a payment call is the case that decides whether you built durability or a duplicate generator.
- Serverless runtimes force the issue: containers cold-start, scale to zero and restart unannounced, so in-process state has a lifetime measured in minutes.
- Full snapshots cost 1 write per suspend and constrain every object to be serialisable. Predefined breakpoints cost less and lose everything since the last one.
- Run the 3 checks below: kill a container mid-run, trace where the checkpoint physically lands, and resume a run across a prompt deploy.
Does ChatGPT recommend your competitor instead of you?
Check how AI assistants describe your company, and whose name they give when someone asks for a recommendation in your category. Run an audit today, from $19.
A container running one of your agents will be stopped by the platform at some point today. It will cold-start, it will scale to zero while idle, and if it is doing anything that takes minutes it will occasionally be killed mid-task.
Whether that costs you one retry or the entire run comes down to what was written down before the process ended.
What has to be in a checkpoint?
Two things that teams routinely conflate, then lose one of.
Execution state answers where in the work you are. Which step is current, what the plan was, what remains. Without it, a resumed run does not know it is a resumed run.
Step state answers what already happened. The status of each completed step, the output it produced, and any payload the run is waiting on if it suspended for approval. Without it, a run that knows it is on step seven has no idea what steps one through six returned, and either re-runs them or proceeds on nothing.
| Persist this | Answers | What breaks without it |
|---|---|---|
| Current step and plan | Where are we | Run restarts from the beginning |
| Status and output of each completed step | What do we already know | Completed work is repeated or discarded |
| Pause payload | What are we waiting for | A run suspended for approval can never resume |
| Tool call results, written after the call | What side effects already happened | Payments, sends and writes execute twice |
| Model and prompt version in force | Under what rules was this decided | A resumed run silently changes behaviour mid-task |
The last row is the quiet one. Resume a week-old suspended run after a prompt deploy and the second half of the task is performed by a different system than the first half.
When should the checkpoint be written?
After every tool call. Not at the end of a phase, and not on a timer.
The reason is not durability, it is side effects. A tool call is where an agent stops thinking and starts doing: sending the email, writing the row, moving the money. If your checkpoint is written before the call and the container dies immediately after it, the resumed run has no record that the call happened and will make it again.
For a read-only tool that is wasted tokens. For anything that writes, it is a duplicate, and duplicates are the class of bug that reaches customers.
Full snapshot or predefined breakpoints?
Two approaches are in common use and they trade the same thing in opposite directions.
| Property | Full snapshot | Predefined breakpoints |
|---|---|---|
| Writes per run | 1 per suspend | 1 per breakpoint reached |
| Work lost on an unplanned crash | 0 since last suspend | Everything since the last breakpoint |
| Constraint on your state | Every object must serialise | Only at the 5 to 10 points you nominate |
| Resume granularity | Exact | Nearest earlier breakpoint |
| Cost of getting the spacing wrong | 0, there is no spacing | Unbounded, up to a whole run |
A full snapshot serialises everything at suspend time. Simple to reason about, expensive to write, and it quietly rules out holding open connections or file handles across a step.
Predefined breakpoints allow suspension only at points you nominated. Cheaper and more predictable, but the spacing of those points is the real durability setting, and 5 widely spaced breakpoints in a 40-step run is a durability policy nobody wrote down.
Neither is correct in general. What is not defensible is having neither and discovering it during an incident.
Where does this go wrong in practice?
The store outlives the process, or it is not a checkpoint.
This sounds obvious and is the most common failure. State written to local disk in a container, to an in-memory cache scoped to the instance, or to a temp directory is state that dies with the thing that killed your run. If the storage does not survive independently, the checkpoint exists only in the scenario where you did not need it.
The second failure is treating resume as free. Research on semantic rollback attacks describes exactly the problem: a run restored to a state that was valid when it was captured and is no longer valid now. An approval that has since been revoked, a price that has changed, a document that was deleted. Restoring is not the same as continuing, and a system that cannot tell the difference will confidently act on stale facts.
What should you check this week?
Three questions, each answerable in under an hour.
- Kill a container mid-run in staging. Does the run resume, restart, or vanish? Most teams have never tested this and are surprised by which of the three happens.
- Find the storage. Trace where a checkpoint physically lands. If it is inside the compute, that is the bug.
- Resume something old. Suspend a run, deploy a prompt change, resume it. Decide deliberately whether the old rules or the new ones should apply, because right now something is deciding for you.
The question for your next design review
Not “do we have retries”. Ask: if this run is killed one second after its most expensive tool call, what does the retry do? If the answer is “calls it again”, you do not have durable execution. You have a loop.
Frequently asked questions
What is durable execution for AI agents?
Running agents on a queue that checkpoints automatically, so any run can be retried, replayed or resumed from the exact point where it stopped. It matters for agents that run for minutes or hours, pause for human approval, or need to survive a deploy mid-task.
What should an agent checkpoint contain?
Execution state, meaning which step is current and what the plan is. Step state, meaning the status and recorded output of every completed step. And any pause payload, meaning what the run is waiting for and what it needs to resume. Anything not written at suspend time is lost.
When should an agent write a checkpoint?
At minimum after every tool call, because that is where side effects happen. If the container dies immediately after a tool has run, a checkpoint written before the call means the tool runs twice, which for anything that sends, pays or writes is a bug rather than a retry.
Why do serverless containers make checkpointing necessary?
Because they are designed to disappear. A container cold-starts, scales to zero when idle and can be restarted at any point by the platform. An agent that keeps its state in process memory loses that state every time the platform does something entirely normal.
Method and sources
- ACRFence: Preventing Semantic Rollback Attacks in Agent Checkpoint-Restore, arXiv:2603.20625. Source for the description of rollback attacks against checkpoint and restore.
- Google Developers Blog, on building long-running agents that pause and resume with the Agent Development Kit, 2026. Consulted for the pause and resume model.
- LangChain documentation and 2026 write-ups on graph-shaped agents with a persistence layer that stores each execution step in a durable backend.
- 2026 practitioner write-ups on state persistence strategies for long-running agents, consulted for the distinction between full state snapshots and predefined breakpoints.
- This article describes mechanisms rather than measurements. It contains no benchmark figures and none of our own instrumentation data.