I’ve been building my own agent ops system. While designing the loop and the tool_calls, I kept thinking of a problem I’ve met before:
What if the tool_call you just sent is something you can’t roll back? A sent email stays sent, a payment only comes back as a refund, and deleted data may have no backup. How do you make sure it happens exactly once?
This one is familiar. When I designed financial transaction systems, I dealt with it almost every day, and it boils down to four things:
- Never charge twice: the same request may arrive several times because of network resends and retries, but it may only take effect once.
- Roll back or compensate: if something fails halfway, the steps already done must be undone, or at least someone must be able to clean them up.
- Use a unique key to look things up and to guarantee things: every operation needs an identifier, so you can ask “did this one actually go through?”
- Bring in a human when you can’t tell: some situations the system can’t judge by itself, and it should stop and hand them to a person.
Put those four next to a tool_call and it is, at heart, a high-risk operation. So my first reaction was that these operations need a human review step.
But if the system really is going to run on its own, how do you keep the damage as small as possible? That question reminded me of three common patterns from distributed systems. This post is a review of them, and a look at how they relate to agents.
First: where exactly-once is actually possible
Before starting, a common claim needs correcting. Over an unreliable network, exactly-once delivery is essentially impossible. A request can arrive while its response gets lost, and then you don’t know whether the other side did the work.
What you can actually build is this combination:
- at-least-once: make sure the operation runs at least once, and retry on failure.
- idempotent: running the same operation several times has the same effect as running it once.
Together they give you exactly-once processing: the effect happens once. In other words, the point isn’t to send a message only once, it’s to make sending it several times harmless. That idea runs through all three patterns below.
2PC: two-phase commit
2PC (Two-Phase Commit) is the most intuitive and the strongest way to get atomicity:
- Prepare phase: the coordinator asks every participant “can you do this?” A participant that agrees locks its resources, writes a prepare record to a durable log, and then answers yes. Answering yes is a promise: from then on it can only commit or abort as the coordinator decides, and it can’t back out on its own.
- Commit phase: if everyone said yes, the coordinator tells everyone to commit; if anyone said no, it tells everyone to abort.
The upside is that either all participants succeed or none do, and no intermediate state is visible from outside. The XA protocol in traditional databases is this model.

The costs are just as clear:
- When the coordinator dies, participants get stuck. They have prepared and locked their resources, they promised not to abort on their own, and they don’t know whether to commit or abort, so they wait. The locks stay held and block other transactions.
- Every participant has to support prepare. Databases can, but external HTTP APIs usually can’t.

That’s exactly where tool_calls run into trouble. The email service, the payment API, a Git push: almost none of them offer prepare. You can’t “lock the right to send from Gmail, then send everything together once it’s all ready.” So 2PC itself doesn’t really fit, but its idea is still useful: separate deciding to do something from actually doing it, so the first part can be validated and dry-run.
TCC: build your own prepare at the business level
If the external API won’t give you prepare, build one yourself at the business level. That’s TCC (Try-Confirm-Cancel), which you can think of as 2PC at the business level:
- Try: reserve the resource without making it take effect. Authorize the card (hold the amount, don’t capture), save the email as a draft or schedule it, soft-delete the data.
- Confirm: once every Try has succeeded, make it real: capture the payment, send the email, purge the data.
- Cancel: if any Try fails, release what was reserved: void the authorization, delete the draft, restore the soft-deleted rows.
Cancel looks a lot like Saga’s compensation, which comes later; the difference is timing. During Try nothing is visible from outside yet, so Cancel is close to a rollback. Void a card authorization and nothing ever shows up on the statement; once the payment has been captured, all you can do is refund it, and the statement shows two entries. The same “cancel the charge” means different things depending on when you do it.
A few things to watch when implementing it:
- Confirm and Cancel can both be resent, so both must be idempotent.
- Cancel can arrive before Try. Say Try times out, the coordinator decides to cancel, and then Try arrives late. Record the keys you have cancelled and reject any Try that shows up afterwards, or you’ll be left with a reservation nobody owns.
- Reservations usually expire. Card holds lapse and drafts may get cleaned up, so Confirm has to finish in time.
The value of TCC is that it shrinks the truly irreversible moment down to Confirm. Anything that goes wrong before that can still be undone cleanly.
Outbox: keeping local state and events consistent
Outbox solves a different, more everyday problem: writing to the DB and sending a message are two separate things, and they can’t happen in one atomic operation.
The most common mistake looks like this:
11. Write to DB: set order status to paid ✅
22. Send event: order.paid ❌ crash here
Now the DB says the order is paid, but downstream never hears about it. Send the event first and write the DB second, and you get the opposite: everyone was notified but the state never changed.
Outbox’s answer is to treat the message as data too, and write it in the same local transaction as the business data.
1BEGIN;
2 UPDATE orders SET status = 'paid' WHERE id = 42;
3 INSERT INTO outbox (id, aggregate_id, type, payload, status)
4 VALUES (gen_random_uuid(), 42, 'order.paid', '{"order_id": 42}', 'PENDING');
5COMMIT;
A separate relay process then keeps picking up PENDING rows, publishes them to the message broker, and marks them as sent once publishing succeeds. Because the business data and the outbox row share one transaction, either both are written or neither is, and the inconsistency is gone.
The relay doesn’t have to poll the table. Another common approach is CDC (Change Data Capture, for example Debezium), which reads the database’s own WAL and publishes whenever a new outbox row appears. This post uses polling, because in an agent setting it’s easier to build yourself.


The cost: the relay can die between “published” and “marked”, so messages get sent more than once. That’s the at-least-once from earlier, and it means the consumer must be idempotent.
Relays must not grab the same rows
You rarely run just one relay. If several instances all pick up PENDING rows, they grab the same batch and events go out twice. That isn’t an at-least-once resend; it’s a race you can avoid. There are two common fixes.
Option one: FOR UPDATE SKIP LOCKED, claiming, publishing and marking inside one transaction:
1BEGIN;
2 SELECT id, type, payload
3 FROM outbox
4 WHERE status = 'PENDING'
5 ORDER BY created_at
6 LIMIT 100
7 FOR UPDATE SKIP LOCKED;
8
9 -- publish to the broker; on success
10 UPDATE outbox SET status = 'SENT', sent_at = now() WHERE id = ANY($1);
11COMMIT;
Locked rows are skipped by the other runners. The downside is that the transaction stays open while publishing, so a slow broker ties up connections and locks.
Option two: a lease. “Lease” a batch of rows to one runner first, then publish and mark them, so no transaction spans a network call:
1UPDATE outbox
2SET locked_by = $1, locked_until = now() + interval '30 seconds'
3WHERE id IN (
4 SELECT id FROM outbox
5 WHERE status = 'PENDING' AND (locked_until IS NULL OR locked_until < now())
6 ORDER BY created_at
7 LIMIT 100
8 FOR UPDATE SKIP LOCKED
9)
10RETURNING id, type, payload;
If a runner dies, another one takes over once the lease expires. The cost is that the original runner may still be publishing when its lease runs out, which produces duplicates, and again consumer idempotency has to absorb them.
Also watch the ordering. SKIP LOCKED lets different runners process different rows at the same time, so events for the same aggregate can go out of order. If downstream depends on order, partition by aggregate_id, or make sure each aggregate has only one event in flight at a time.
Back to tool_calls: an intent table
When I apply Outbox to an agent, I give it another name: an “intent table”. The core idea: don’t let the agent call external tools directly. Write “I intend to do this” into the DB first, and let an executor carry it out.
1CREATE TABLE tool_intents (
2 idempotency_key TEXT PRIMARY KEY, -- hash(run_id, step_id), no args
3 run_id TEXT NOT NULL,
4 tool TEXT NOT NULL,
5 args JSONB NOT NULL,
6 args_hash TEXT NOT NULL, -- to detect changed args on a rerun
7 risk TEXT NOT NULL, -- reversible | compensable | irreversible
8 status TEXT NOT NULL, -- PLANNED | DISPATCHED | SUCCEEDED | FAILED | UNKNOWN
9 external_ref TEXT, -- identifier returned by the external system
10 created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
11 updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
12);
The key points:
idempotency_keyis unique. Rerunning the same step inserts only once (INSERT ... ON CONFLICT DO NOTHING); if the key already exists, the work is done or in progress, so just read its status.- Keep args out of the key. When an agent reruns, the model often produces slightly different args, say an email body with a few words changed. If the key is derived from the args, that’s a new key, and the same thing gets done twice. Build the key only from the stable
run_idandstep_id; if theargs_hashunder the same key doesn’t match, treat it as a conflict and stop, not as a new request. - Make the state machine explicit.
PLANNED → DISPATCHED → SUCCEEDED,FAILEDon failure, and most important of all,UNKNOWN: the request went out, it timed out, and I don’t know whether the other side executed it. - Commit
DISPATCHEDbefore making the call. If you call first and the state never lands, after a crash the DB still saysPLANNED, and the system thinks the work never happened. - Never auto-retry an
UNKNOWNwithout an idempotency guarantee. This is the most important rule in the whole design. Retrying an irreversible operation that may already have succeeded is exactly how you charge someone twice.
The same goes for executor leases. If a lease expires while the status is still DISPATCHED, the outcome of that call is unknown, so it should become UNKNOWN rather than letting another executor send it again.
Handling UNKNOWN depends on one precondition: the idempotency_key has to be sent to the external system on the very first call. External systems support idempotency in roughly two ways:
- The provider dedupes: for example Stripe’s
Idempotency-Keyheader. Resend with the same key and the provider returns the first result instead of doing it again. For these APIs anUNKNOWNcan simply be retried; that’s what the mechanism is for. - You can look it up by key: the request carries a client token or request id, and you can later query the status of that operation with it.
For the second kind, the order is:
- Ask the external system using the
idempotency_key. - If it succeeded, update the status to
SUCCEEDEDand record theexternal_ref. - Resend only if you get a clear failure or “not found” and you’re sure the original request can no longer take effect (for example, it’s past the provider’s processing window). “I can’t find it right now” isn’t enough: the original request may still be processing, or the provider’s lookup may lag.
- If you can’t be sure, or the external system supports neither, stop and hand it to a human.
Step four is the “bring in a human when you can’t tell” rule from financial systems.
Saga: a chain of compensations instead of a rollback
Saga comes from the 1987 paper “Sagas” by Hector Garcia-Molina and Kenneth Salem. The core idea: split a long transaction into a chain of local transactions T1, T2, …, Tn, where each Ti has a matching compensating action Ci. If Tk fails, run C(k-1), C(k-2), …, C1 in order to undo what has already been done.
One thing to be careful about: compensation is not rollback. A rollback returns the system to its earlier state as if nothing happened; a compensation is a new action that cancels out the earlier effect. In between, others may already have seen the intermediate state. For example:
- The charge succeeds, a later step fails, and the compensation is a refund. But the money really was taken before the refund, and the statement shows two entries.
- A PR gets created, a later step fails, and the compensation is closing the PR. But the notification may already have gone out to the reviewers.
There are two ways to coordinate:
- Orchestration: a central orchestrator decides what happens next and how to compensate on failure.
- Choreography: services trigger each other through events, with no central control.

For an agent, orchestration is the natural fit. The agent’s planner already is the central role: it knows the whole plan and what each step’s compensation is. With choreography it’s hard to tell which step an agent has actually reached, and debugging becomes painful.
The pivot transaction: the step you can’t come back from
After the original paper, practitioners commonly split saga steps into three kinds: compensable, pivot, and retriable; Chris Richardson’s Microservices Patterns uses this split. The most important of them is the pivot transaction, the dividing line in the chain:
- Every step before the pivot can be compensated. If the pivot itself fails, it didn’t happen, so you compensate backward as well.
- Once the pivot succeeds, there’s no going back. The steps after it can only retry forward until they succeed.

For tool_calls, the pivot is usually the irreversible operation: actually sending the email, actually transferring the money, actually deleting production data. So in design terms I’d suggest:
- Put irreversible operations at the end of the chain, or as late as you can. If an operation can be split TCC-style, split it: Try goes earlier as a compensable step, and Confirm becomes the pivot.
- Put a human gate before the pivot. Once past the gate, the system can only move forward.
- Make the steps after the pivot retry-safe, because they can only keep retrying until they succeed.
Classify tools by risk
All of these patterns require knowing what kind of operation each step is. What I do now is put tools into three classes:
| Class | Examples | On failure | Run automatically? |
|---|---|---|---|
| reversible | writing to a scratch directory, idempotent local edits | throw it away and redo | yes |
| compensable | opening a PR (can be closed), creating a resource (can be deleted), a card charge (can be refunded) | run the compensation | yes, but record the compensation |
| irreversible | sending email, bank transfers, deleting data with no backup | can’t compensate; only look it up or handle it manually | needs a human gate |
Card charges and transfers land in different classes because a card charge can usually be refunded, while a bank transfer often can’t be pulled back once it’s sent. Both are “paying money”; what matters is whether the mechanism underneath can be compensated.
The point of classifying isn’t to stop the system from doing anything. It’s to decide how much protection each class needs.
Back to the question: how to keep the damage small
Putting it all together, here’s how I’d let an agent run on its own without amplifying the risk:
- Plan first, then apply. Any action that changes external state first produces a plan you can inspect (a dry run), so a person or a rule can see what it will do.
- Limit the blast radius. Cap the resources, amounts and counts each run can touch, and stop when a cap is hit.
- Give every operation an idempotency key and a state machine. A retry only reads the state; it never does the work twice.
- Always put a gate before an irreversible operation. That decision can’t be left to the model.
- Never auto-retry an
UNKNOWNwithout an idempotency guarantee. Look it up first, and stop if you can’t find out. Better slow than twice. - Write down every intent and every result. Which brings us to the last idea.
Going further: WAL and the session log
WAL (Write-Ahead Log) is the most basic design in databases: before any state change, write “what I’m about to do” to the log. After a crash, the system replays the log to bring its state back to consistency. The intent table above follows the same idea: write PLANNED first, then execute. Outbox is a bit different. It’s written in the same transaction as the business data, so it isn’t write-ahead, but it shares the principle of persisting the intent first and letting something else carry it out.
Applied to an agent, this becomes the session log:
- Append-only, recording every turn and every tool_call’s request and response.
- After the agent crashes or times out, rebuild its context from the log so it knows where it left off.
- The key rule: on replay, only read recorded results; never re-execute a tool_call with side effects. If replaying the log actually sends the email again, the WAL design has failed.

That way the session log solves three problems at once: you can see the full decision trail when debugging, you can recover after a crash, and on retry you can tell which steps were actually already done.
Wrapping up
Looking back, the four things from financial transaction systems map almost one-to-one onto an agent’s tool_calls. Each of these patterns covers part of it:
- 2PC shows that atomicity is precious, but external APIs usually can’t give you prepare, so separate deciding from executing.
- TCC builds its own prepare at the business level and shrinks the irreversible moment down to Confirm.
- Outbox keeps local state and intent consistent; with an idempotency key and an
UNKNOWNstate, you can tell what is safe to retry and what needs a lookup first. - Saga replaces rollback with compensation, and uses the pivot to mark the step you can’t come back from, with the human gate in front of it.
If you’re building tool_calls for an agent too, my suggestion is: classify your tools first, then decide the gate, retry and compensation strategy for each class. Automation itself isn’t the problem. The problem is whether a person or a rule is guarding the irreversible step before it happens.
