Agents should outlive their processes

libfx is the agent kernel behind fx, packaged for JavaScript. One agent is one in-memory conversation: you prompt it, it calls your tools, you checkpoint it, you close it.

In memory is the catch. Kill the process and your agent goes with it, along with whatever it was halfway through. Hope it wasn’t a deploy.

So my agent and I built fx-durable: a thin layer that persists the agent, journals everything it does, and drags it back to life after a crash. It’s on npm as fx-durable, because of course it is.

I started with a spec. Most of it was the usual. One line wasn’t:

Unknown means unknown.

Four words. They turned out to be the whole project.

The case that matters

Saving a conversation is easy. libfx hands you opaque checkpoint bytes, you put them in SQLite, you go to lunch. The hard case is the one that ruins lunch:

The agent decides to do a deploy. The deploy tool sends the request, and the deploy occurs. Then the process stops before fx-durable records the result.

“Deploy” is an example: any operation that changes something outside the agent and is not safe to do two times, such as sending an email or charging a card. The demo does not deploy anything. Its deploy tool writes a file, and its check_deployment tool reads that file. No production servers were harmed.

When the process starts again, fx-durable cannot tell the agent that the deploy was successful, because it does not know this. It cannot tell the agent that the deploy failed. It must not do the deploy again, because then two deploys occur.

The only correct result is “unknown”. fx-durable records this result as outcome_unknown and gives this message to the model:

The previous `deploy` operation was interrupted.
It may or may not have completed successfully.
Do not repeat it blindly.
Inspect the current system state before deciding what to do.

Everything else in the design exists to make that answer possible, and to make sure we only give it when it’s true. Anyone can shrug. Shrugging honestly takes a journal.

Write it down before you do it

The rule is old, boring, and ignored often enough to keep incident reviews in business. For each operation, do these steps in this sequence:

Record the intent in the journal.
Commit.
Do the operation.
Record the result in the journal.
Commit.

sequenceDiagram
    participant M as Model
    participant X as Tool executor
    participant J as Journal (SQLite)
    participant W as The world
    M->>X: call deploy {version: abc123}
    X->>J: task = running (commit)
    X->>W: deploy()
    W-->>X: ok
    X->>J: task = completed (commit)
    X-->>M: result

If the process stops between step 2 and step 5, the journal shows a task with the status “running”. That is the only data that recovery has. It is enough.

The next step depends on the replay policy of the tool. You set the policy when you define the tool:

const deploy = defineDurableTool({
  name: "deploy",
  replay: "unsafe",
  inputSchema: z.object({ version: z.string() }),
  execute: ({ version }, { signal }) => deployVersion(version, { signal })
})

Policy	After a crash during the operation
`"safe"`	fx-durable does the operation again. For example, to read a file two times causes no damage.
`{ strategy: "idempotent" }`	fx-durable does the operation again with the same idempotency key. The downstream system uses the key to find the duplicate.
`"unsafe"`	fx-durable records `outcome_unknown`. It never does the operation again automatically.

There’s a subtlety in “never automatically”, and models will find it. After recovery, the model knows that the result of the deploy is unknown, but it can still call deploy again. fx-durable refuses the first repeat of the call and shows the same message. If the model calls deploy a second time, fx-durable lets the call continue.

The guarantee is that the system never replays an unsafe effect on its own. A deliberate second call is the agent’s decision, and the journal records that it was refused first. Receipts matter.

What libfx made us do

Two facts about libfx shaped the whole recovery design. We learned both the traditional way: by poking it until it complained.

It can only checkpoint when idle. If you call checkpoint() during a turn, libfx stops with the error cannot checkpoint while a prompt is active. Thus a checkpoint always shows the state between two turns. It never shows the state during a turn.

That sounds like a limitation. It turned out to be a gift. fx-durable writes the checkpoint in the same SQLite transaction that sets the turn to “completed”. Thus a checkpoint for an incomplete turn cannot exist. If the process stops during a turn, recovery restores the checkpoint from before the turn. Then it sends the original request again, with a summary of the work in the journal.

Tool callbacks don’t get a call id. libfx calls the execute(input, { signal }) function of your tool, but it gives no identifier for the call. After a crash, the recovered model calls the tools again. fx-durable must know if a new read_file call is the same call that completed before.

fx-durable compares three values: the tool name, a hash of the input, and the position of the call among identical calls. If the values agree with completed work, the model gets the recorded result. The tool does not run again.

And the tests needed a model with no opinions. Real models improvise, and a crash test that improvises is a coin toss. libfx accepts a replacement fetch function. We connected it to a test server and recorded the requests from libfx. Then we found the stream format from the responses that libfx expected. The result is a scripted model that uses the gateway protocol. Every test drives the real libfx kernel through it, offline, with the same result each time.

The crash suite is the product

The spec said the crash suite mattered more than the feature count. We took that personally.

Each boundary that is important for recovery has a crash point with a name. Some crash points are before the model request, during the stream, and after the tool records its intent. Others are after the external operation, in the checkpoint transaction, and at each step of recovery. A crash test does these steps:

Start a real application process.
Stop the process at one crash point with SIGKILL.
Start the process again.
Examine the database and the log of external operations.

SIGKILL is important. The process cannot run cleanup code or write its buffers. For the runtime, it is the same as a power failure. No finally, no goodbyes, no last words.

After each test, these conditions must be true:

the unsafe operation occurred one time or less,
each request ID has one submission,
no task has the status “running”,
each unknown result has its event,
each completed turn has one checkpoint,
the event sequences have no gaps.

Twenty of the twenty-one cases passed on the first run, which is suspicious all by itself. The one that failed was the best bug of the project.

If a turn stops many times, fx-durable must stop the recovery and ask a person for help. This step tried to change the agent from failed to needs_input. The state machine of the agent does not permit this change. The transition check stopped the operation, SQLite cancelled the transaction, and the database did not change. The bug was mine. The design caught it anyway, which is the entire point of having one.

Then I asked whether Effect was worth it

The spec required Effect v4 as the runtime model, so the first version used it everywhere, storage included.

When I asked my agent whether Effect had paid for itself, it gave the answer nobody puts on a conference slide: partly. I agreed.

It paid in execution. Cancellation is the hardest part of this system, and Effect handles it in this sequence:

You cancel a submission. Effect stops the fiber of the turn.
The fiber stops the libfx turn.
The libfx turn stops the AbortSignal of each tool.
The effect of each tool stops.
The finalizer of each tool records cancelled for a safe tool, or outcome_unknown for an unsafe tool.

Effect gave us that chain nearly for free. Writing it by hand with AbortControllers and try/finally, I’d have got the corners wrong, and found out at 3 a.m. During a deploy. Naturally.

It cost us in storage. node:sqlite is synchronous. In plain TypeScript, a transaction is a function without await, and no other code can run during it. In Effect, a fiber can pause between two steps. Thus we needed a semaphore and a fiber-local transaction marker to get the same guarantee. That’s two clever mechanisms to rebuild something plain JavaScript gives you for free, which is the kind of clever you want to delete.

So we pulled the durable core out of Effect.

            IMPURE / ASYNC

                 libfx            agent loop, model calls, tools
                   │
                Effect            supervisor, turns, tools, cancellation
                   │
               Database           run(journal => …) · read(storage => …)
                   │
       ── durability boundary ──
                   │
                Journal           every durable transition
                   │
                Storage           synchronous transactions
                   │
                SQLite

             SYNC / DURABLE

Storage is now a synchronous interface. A transaction is a function:

journal.transaction(() => {
  journal.transitionTask(task.id, "completed", { output })
  journal.transitionTurn(turn.id, "completed")
})

fx-durable refuses a transaction body that returns a Promise. Thus no code can put an await in a transaction.

One Journal class now holds every durable transition the system can make. The idea came from outside review: for durability software, being able to open one file and answer “what can this system write?” matters more than abstraction purity.

The refactor deleted more lines than it added (the best kind), and the journal tests now run without Effect at all: 169 of them, every allowed and rejected transition of every state machine, in about a second.

The compiler found eight violations

The reviewer’s main point was that “runtime code goes through the boundary” was only a convention. Runtime code could still call the journal directly. A convention is a rule that hasn’t been tested yet.

So we changed the structure. Runtime modules do not get a Journal. They get a Database service. The read side of this service gives a read-only view of storage, and this view has no write methods.

The moment that type existed, the compiler found eight locations where runtime code wrote to storage directly. These were cancellation flags, executor heartbeats, and a transaction for turn completion in the supervisor. Each location became a named journal method.

That was the most valuable change of the whole refactor, and it only happened because the rule became something a tool could check. A lint rule now prevents imports of the durable core into runtime code.

Then I found a hole in it myself, which is a humbling thing to admit in a section about rules. Journal kept its storage in a public field. Thus database.run((j) => j.storage.updateTask(…)) could go around all named transitions. The compiler and the lint rule did not find this. The fix has two parts:

The storage is now an ES private field, #storage.
The reader is a separate object that has only read methods. A cast cannot change it into an object that writes.

The rule we ended up with fits in a sentence: all durable writes are named domain transitions; durable reads may use a read-only view.

Two bugs we would have shipped

A stricter decoder broke every crash test. Mid-project we installed an anti-slop lint plugin. It found 149 problems, a rude number for code I thought was fine, mostly unknown flowing through places that should have had a real type. We replaced them with JSON schemas and decoded data where it enters the system.

That included the fake gateway’s request parser. Every in-process test still passed. Twenty crash tests failed at once.

The cause: the runtime in the crash test had instructions. In this condition, libfx sends the content of the system message as a string, not as a list of parts. The new decoder refused the string. The in-process tests did not set instructions, so only the tests with a real application found the error.

The published types were broken. libfx does not include TypeScript declarations, so we wrote our own. They worked in the repository. Before we published, we did these steps:

Make the package file.
Install the package in an empty project.
Examine the types with strict settings.

The result was error TS7016: could not find a declaration file for module libfx. Each user with skipLibCheck: false would get this error when they install the package.

The fix was shipping the declarations inside the package. The lesson: test the artifact, not the source. Your users install the tarball, not your repo.

The job is walking away

Nobody hires a coding agent to chat. Strip the hype off and the job looks like this:

When I hand a task to an agent, I want to walk away and come back to finished work, or to an honest account of what happened, so I can trust it with my machine.

Read it twice and you’ll notice that “walk away” is doing all the heavy lifting. Walking away is when the laptop sleeps, the dev server restarts, and the process gets a SIGKILL from something you didn’t know was running. A coding agent that only works while you watch it is a pair-programming session with extra steps.

That job is fx-durable’s job. So it needed a coding agent to do it for.

Vercel had just shipped an fx adapter for the AI SDK harness: new HarnessAgent({ harness: fx }), and fx becomes one more agent your app can drive. It runs the fx CLI in a remote sandbox and talks to it over ACP. Very cloud. Very not-surviving-kill -9.

So we implemented the same contract with fx-durable underneath. The harness spec, HarnessV1, is a small interface: start a session, run a turn, emit stream parts, read history.

Harness	fx-durable
Session	A durable agent. The session ID is the agent ID, so a new process attaches to the same journal.
Prompt turn	A submission. The adapter changes each journal event into a stream part.
Tool calls	Durable tools. fx-durable executes them and records them, so the harness sees them as executed by the provider.
History	The journal. The adapter makes the conversation again from the event log.

On top of that went ordinary AI SDK code and AI Elements, and out came a Claude Code-shaped agent in a browser tab: file and shell tools, diffs, a terminal, permission prompts, plan mode, todos, sessions, Esc to interrupt. Plus a red Crash server (kill -9) button, because a durability demo without a self-destruct button is just a chat app.

The plan was simple. The UI was the product, the library was the engine, and the scripted model had already proved the engine: two hundred and fifty-odd tests, all green.

The customer was the library

Then I gave it a real model and a real task: “Fix the failing tests.”

First run. A bug gave the session an empty folder as its workspace. The agent, an eager new hire with nothing to do, used cd ../.. and found the fx-durable repository. It ran the test suite, which includes the live tests. The live tests used the gateway key that the shell gave them. My test script had just clicked “Yes, and don’t ask again for bash”, so technically everyone followed the rules. The workspace bug is fixed, and the shell tool now removes credentials from its environment.

Second run. The agent did the right things in the right order. Then it wrote a todo list, and the turn failed with Unexpected end of JSON input. The scripted model always sends tool input as one compact line of JSON. Claude, a model of taste, pretty-prints JSON that contains an array of objects, and libfx cannot parse tool input that has line breaks. fx-durable already wraps each model response to record it, so the fix went into that wrapper: before libfx reads a tool-call part, fx-durable writes its input again as one line. Same JSON, fewer feelings.

Third run. The agent added a function, and I pushed the button halfway through. The server started again, fx-durable recovered the turn, and the agent read the file again. It got the recorded result, from before its own edit, and announced that the edit had never happened.

That one was the rule from earlier in this post working exactly as designed. If a recovered call agrees with completed work, the model gets the recorded result. That rule protects a deploy. For a read_file it serves a snapshot of the past. Tools can now set reuse: false: such a tool runs again after a crash and reads the world as it is now. Only replay-safe tools can do this, because it is safe to run them again.

A fourth bug had shown up even earlier, while I was wiring the adapter. fx-durable recorded a model call when its response stream closed, but libfx stops reading at the finish part and does not wait for the close. Thus a turn could complete before its own last model call was recorded. Now fx-durable records the call when it sees finish. The scripted model closed its streams at once, which is why the crash suite never noticed.

Here’s the twist. I built the coding agent to show off the library. The library turned out to be its first real customer. Its job was never “be a nice UI”. Its job was to be the first user that behaved like a user, and it did that job better than the whole suite: four bugs in an afternoon, each one invisible to a scripted model that never pretty-prints, never wanders, and always hangs up politely.

The scripted model was hired to make tests repeatable. The real model was hired to find out whether any of it works. Those are different jobs, and I’d been asking one employee to do both.

The third run, by the way, finished. The function landed, the new tests passed, and nobody had to watch.

Shopping at the neighbours’

Then someone pointed me at pi-durable, which does the same job for the Pi agent. Comparing notes with a competitor is humbling in the best way. You find out which of your design decisions were wisdom and which were just the first idea that compiled.

The big difference is where each library sits. pi-durable owns the agent loop. It saves a checkpoint at every task step and commits partial answers every 100 ms, so it continues in the middle of a turn. fx-durable wraps libfx, and libfx checkpoints only between turns, so fx-durable recovers whole turns. That is where the stale-read twist came from. pi-durable cannot have that bug. In return, it has no idempotency-key policy, no outcome_unknown with a refused first repeat, and I didn’t find a crash suite that SIGKILLs real processes. Each library had something the other one wanted. So I went shopping.

Steering. The job: “let me correct the agent without stopping it.” agent.steer(text) records the guidance in the journal as turn.steered. Then fx-durable gives it to libfx with turn.steer(). The model receives it at the next safe boundary, wrapped in <user_steering>. If the process stops, the recovery prompt includes the guidance. If no turn is running, or the turn completes before the guidance arrives, the guidance becomes a new submission. Nothing you type is lost.

Hooks. beforeTool and afterTool go on a runtime. The important detail is when beforeTool runs: before fx-durable records the intent. Thus a crash during a permission prompt leaves no task and no outcome_unknown. The recovered turn asks again. A crash test stops the process inside the hook for deploy and examines the result: the deploy occurs one time, with no unknown outcome. The tests also found a bug in my first version. A synchronous throw in afterTool failed the whole call. It now keeps the original result, because the effect already occurred.

Progress. A tool can call context.progress(chunk), and fx-durable records the partial text of the model. Both go into the running task’s metadata.progress, at most every 100 ms. A viewer that connects during a call can read them. Progress is state, not history: it makes no events, and the result replaces it.

Reject when busy. Sometimes the job is “do this only if you’re free.” submit(text, { whenBusy: "reject" }) submits nothing and rejects with AgentBusyError if the agent has work running or queued. The check and the insert are one transaction, so two callers cannot both get through.

Subagents. defineSubagent() makes a tool that gives a task to a child durable agent and returns its answer. The name of the child and the request ID of its submission both come from the idempotency key of the call. The crash test stops the process inside the child’s work. After the restart there is one child agent and one child submission, and the parent gets the answer. One task, one subagent, however many times the process dies.

It also found the best bug of this round. Recovery replays interrupted calls in sequence, before it starts any worker. A replayed subagent call waits for its child. The child’s worker starts only after recovery finishes. Recovery waits for the call, the call waits for the child, and the child waits for recovery. A deadlock, delivered by the feature meant to show off durability. The fix is a new tool option, resumeOnCall, for idempotent tools only. Recovery does not replay such a call. When the recovered turn calls it again, fx-durable retries it with the original key, and the call attaches to the same child.

Forks. agent.fork(id, { after }) starts a new agent from the conversation as it was after a given request. The test tells the agent “my name is Ada”, then “I like tea”, then forks after the first message. The fork remembers Ada and has never heard of tea. Forks start only at turn boundaries, because libfx checkpoints only between turns. pi-durable can fork at any entry. We can’t, and the post would be lying if it said otherwise.

What I didn’t take: compaction. pi-durable summarizes old context to make room. fx-durable can’t. libfx’s createFxAgent has no compact(), and the conversation is opaque checkpoint bytes, so there is nothing for fx-durable to summarize. The honest answer is “not yet”, and an honest “not yet” beats a clever hack that corrupts someone’s agent.

The scorecard: six features in, one bug in my own code, one deadlock caught before release, and one feature politely declined. Shopping with a crash suite is the only kind of shopping I enjoy.

Then the coding agent got all of it. Permission prompts moved into beforeTool. Typing while the agent works now steers it, and an explore subagent does the read-only digging. /fork starts a new session from the latest checkpoint. The real test went like this: I asked Claude to fix the failing tests, and while it worked I typed “also add one test for an empty cart”. It fixed the bug, wrote the extra test, and reported five of five passing. It never stopped to ask what I meant, which is more than I can say for most people I’ve pair-programmed with.

Run the coding agent

Clone the repository and build it:

git clone https://github.com/jagenaujagenau/fx-durable && cd fx-durable
pnpm install && pnpm build

Start the app:

cd examples/web-ui && pnpm install && pnpm dev

Open http://localhost:3100.
Ask the agent to “Fix the failing tests”. Each new session gets a copy of a small project with two failing tests.
While the agent works, push Crash server (kill -9).
Make sure that the agent continues and the tests pass at the end.

NOTE: Without AI_GATEWAY_API_KEY, a scripted model makes the decisions and everything else is real. To use a real model, put the key in examples/web-ui/.env.local.

Try it

You can watch a deploy die and come back in about a minute.

CAUTION: USE AN EMPTY DIRECTORY. THE DEMO WRITES FILES IN THE CURRENT DIRECTORY.

Make sure that Node.js 22.13 or a later version is installed.
Open a terminal in an empty directory.
Start the demo with this command:
```
npx fx-durable demo --crash-at tool.after-execute:deploy
```
NOTE: This command stops the process after the deploy operation. At that time, the journal does not have the result of the operation.
Make sure that the terminal shows crash injected at tool.after-execute.
Start the demo again with this command:
```
npx fx-durable demo
```
Make sure that the terminal shows Outcome unknown.
Make sure that the terminal shows Operation will NOT be replayed automatically.
Make sure that the terminal shows inspect production. The agent examines the deployment. It does not deploy again.

Examine the journal with this command:

npx fx-durable trace engineer --db .fxd-demo/demo.db

Make sure that the deploy line shows outcome_unknown.

What I’d keep

The crash suite. Not as a test suite, as the definition of done. Writing the crash points first made every later design decision cheaper, because any change that broke recovery failed loudly within twenty seconds.

outcome_unknown as a first-class state. It’s tempting to round uncertainty to success or failure because those are the states your code already handles. Every system I’ve worked on that did that eventually lied to someone.

And the split between execution and durability. Effect is very good at the asynchronous, interruptible, resource-shaped half of this problem. The other half wants to be boring, synchronous and auditable. The project got better when I stopped making one tool do both.

fx-durable is at github.com/jagenaujagenau/fx-durable and on npm as fx-durable. MIT. Run npx fx-durable demo, kill it with kill -9, and run it again. It doesn’t hold grudges.