Operations

A durable runtime accepts obligations, so someone has to be able to see them, explain them, and unblock them. This guide covers the administrative operations, obligation monitoring, and backup and restore on Node/SQLite (DN) and Cloudflare Durable Objects (DC).

Administrative operations

Five operations are members of DurableAgentRuntime, implemented over the SubmissionLedger and ConversationStore ports only, so they behave identically on DN and DC:

• explain(submissionId) / explainConversation(conversationId) returns a read-only recovery explanation: the classifier decision, its operator meaning, and the disposition a recovery pass would report. It writes nothing.
• verify(conversationId) runs read-only integrity checks with typed per-check results. It never repairs. The digest-chain check reports skipped with the reason unless per-batch producer identity is supplied out-of-band.
• retry(submissionId, { author, reason }) records an audit entry and re-drives exactly the classifier's decision, with typed refusals for settled work and lanes owned by the resolveUnknown/resolveApproval paths. Author and reason are mandatory.
• wake(conversationId) sends a droppable liveness notification.
• scanObligations(thresholds) returns the obligation report described below.

On DN, NodeDurableHost re-exposes all five, and bun run admin:durable -- <explain|verify|retry|wake|obligations> --database <file> is the CLI. On DC, the Conversation Object exposes Schema-encoded entry points (explainEncoded, verifyEncoded, retryEncoded, obligationsEncoded, wake); deployments reach them through their own Worker. Every operation, including observe, resolveUnknown, and resolveApproval, consults the OperationAuthorizer fail-closed: the default Layer preserves service-possession behavior, and a host-supplied authorizer turns denials into the typed OperationDenied before any read or write.

Obligation monitoring

scanObligations is scan-based, never a daemon. It folds the ledger's nonterminal scan into rows {submissionId, conversationId, state, blockedOn, ageSeconds, severity}, where blockedOn is one of unknown, approval, waitingForChild, ready-aged, running-aged and severity is classified against your {agingSeconds, overdueSeconds} thresholds.

Hosts own the alert loop. The framework deliberately does not schedule the scan or deliver alerts: run scanObligations periodically from your host (cron, alarm, monitoring agent), export the rows as logs or metrics, and alert on:

• any row with blockedOn: "unknown", because an Unknown Outcome needs an authorized resolveUnknown decision;
• any row with severity overdue, which marks accepted work without timely settlement;
• a growing approval backlog.

Abort unknown work

Call DurableAgentRuntime.abort(AbortCommand.make({ submissionId, author, reason })) after the host authorizes stopping that Submission. On Cloudflare, use CloudflareConversationClient.abort(receipt.conversationId, command). Abort authority remains service possession plus the host's authenticated boundary; it does not grant callers broader Tool-resolution authority.

Normal recovery/maintenance now claims an unknown head with that durable intent, cleans up and joins attached children, and records the aborted settlement. It releases queued followers without replaying uncertain ordinary Tools. The original abort audit and unknown evidence remain; abort does not claim that an external action was rolled back. Repeated commands preserve the first intent. A SettlementConflict reports an outcome that already won, including an aborted outcome whose acknowledgement was lost.

Remove consumer loops that follow abort with ResolutionAbortSubmission for every unresolved call, manually clean up children after an accepted parent abort, edit ledger state, or repeatedly wake a blocked lane. An unknown or approval-waiting head with ready followers quiesces until an authorized mutation restores its maintenance alarm. Keep host decisions about whether and when to abort; there is no automatic inactivity deadline.

A parent suspended in WaitingForChild does not acquire abort authority because its child is unknown. The host must explicitly decide to abort the parent, or apply a separately configured authorized policy. The fix owns child abort propagation, joining, and reservation cleanup only after that parent abort is durably accepted. It does not choose the parent's outcome beforehand.

Observe a Submission outcome

Persist the admission Receipt: its submissionId, receiptId, conversationId, and queueSequence identify the same obligation across retries and replacement Attempts. The APIs below supply the evidence inside a host with the real SubmissionLedger and DurableAgentRuntime services. They are not all remotely available through CloudflareConversationClient.

Need	Public API and evidence
Admitted or queued	SubmissionLedger.lookup(SubmissionLookupById.make({ submissionId })): admitted or ready; the Receipt proves admission.
Execution stage	The same lookup returns running or input-applied. This is durable operational state, not proof that a worker is alive.
Intentional wait	loadRecoverySnapshot(RecoverySnapshotRequest.make({ submissionId })) exposes suspension and joined host linkage. explain(submissionId) supplies the existing classifier decision.
Unknown outcome	Lookup returns unknown; explain adds open calls, uncertainty audits, accepted resolutions, and abort intent.
Completed, failed, aborted	awaitSettlement(receipt) returns the durable terminal identity/outcome and bounded failure diagnostic. Interrupting the wait detaches the caller; it does not abort the Submission.
Terminal stop/budget details	The canonical SubmissionSettled record carries finishReason, exhausted, and policyLimit where applicable. These fields are not currently on the returned Settlement; read the record through observation.

For live progress use DurableAgentRuntime.observe(receipt, { after }). It is a Conversation Stream, so filter by submissionId or runIdForSubmission(submissionId) as appropriate. Joined input shares its host's Run; use the snapshot's hostSubmissionId for Run evidence, and the original Submission ID for its own terminal record. For example, this Stream yields the exact terminal envelope, including budget metadata and its resumable cursor:

import { DurableAgentRuntime, type ObservationOffset, type Receipt } from "@effect-agent/session";
import { Effect, Stream } from "effect";

const observeOutcome = (receipt: Receipt, after?: ObservationOffset) =>
  Stream.unwrap(
    Effect.gen(function* () {
      const runtime = yield* DurableAgentRuntime;
      return runtime.observe(receipt, { after }).pipe(
        Stream.filter(
          ({ record }) =>
            record.payload._tag === "SubmissionSettled" &&
            record.payload.submissionId === receipt.submissionId,
        ),
        Stream.take(1),
      );
    }),
  );

On Cloudflare, consume bounded CloudflareConversationClient.readPage pages and retain the last sequence; after an empty page, awaitProgress(conversationId, sequence) waits for a hint or already committed progress before another read. Do not call readAll or explain in a polling loop: explain is a diagnostic read of conversation history. For a first run-scoped read inside an authorized host, the public recovery snapshot's inputApplied.sequence provides a lower bound for ConversationStore.read (use sequence - 1 to include the input); a ready follower has no Run history yet. Keep subsequent reads incremental and retain only the application's projection. Cloudflare already exposes explainEncoded for occasional remote diagnosis.

An independent Worker outside the Conversation Object cannot obtain the complete operational snapshot through that client. The public portCall protocol supports ledger lookup, which returns the admitted input and Submission state, but has no recovery-snapshot or nonterminal-scan request. The routed ledger keeps loadRecoverySnapshot local-only and scanNonterminal local to its owning Object. Canonical paging and progress waits do not expose the authoritative suspension, inputApplied marker, or FIFO blocker. There is no public composition that obtains all of these remotely without an additional host boundary; repeated explainEncoded calls are not a bounded snapshot substitute.

For an external Worker that needs those fields, retain an authorized, Schema-backed, read-only snapshot RPC inside the owning Conversation Object and its external client adapter. That RPC validates the Receipt against local lookup, uses the real ledger's loadRecoverySnapshot for input application, suspension, abort intent, and joined host linkage, and scans the local ordered nonterminal rows for an earlier FIFO blocker. It can expose the relevant child references for host policy. It need not copy the recovery classifier or construct dummy local services outside the Object. Any application-specific activity or Tool-result projection can use bounded canonical reads from the input marker and an incremental cursor. This remains a host adapter, not an API added by the abort fix.

Inside the owning Object, the ordered public SubmissionLedger.scanNonterminal identifies the earlier unsettled head in its Conversation; explain diagnoses that head without reimplementing recovery. Do not infer the follower's state from the Conversation's latest Run.

Canonical records replay from the saved cursor. Consumers own cursor persistence and idempotent projection or delivery; checkpointing after a side effect can redeliver it after a crash. Neither the Stream, a notification, a callback, nor a process-local finalizer guarantees durable external delivery. No exactly-once external delivery is promised. Hosts must authorize public ledger reads and observation, and handle typed storage, protocol, and authorization errors without interpreting them as a terminal outcome.

Replace only inspection/classification and outcome-polling logic that the APIs available at the caller's boundary actually cover. Keep the host snapshot RPC when an external Worker needs the operational evidence above. Keep application policy for visible responses, whether an acknowledgement is sufficient, destinations and authorization, inactivity thresholds, and delivery retries. Tool success or an application result tag is not a library-defined answer obligation; project that policy from canonical records without adding it to the runtime.

Backup and restore on DN

Any file-consistent snapshot works as a backup: copy the .sqlite, -wal, and -shm files while no process holds the database, or use VACUUM INTO or SQLite's backup API online. The claimed DN deployment shape is one process owner per database file.

Restoring a backup means accepting four semantics (the repository proves them with an executable restore drill):

1. Pre-backup history survives intact. The restored store passes the same integrity checks (verify) as the original.
2. Post-backup epochs are fenced. An ownership token minted on the original timeline after the backup point is rejected typed by the restored store. Before serving traffic from a restore, fence or terminate every producer that ever ran against the original store: a divergent producer must never be left assuming it still owns anything.
3. Post-backup external effects enter the Unknown regime. An external call whose outcome was only recorded after the backup re-enters recovery as an open call and is marked ToolCallUnknown. It is never assumed rolled back and never automatically replayed: resolve each one through resolveUnknown from external truth (the supplier's records).
4. Post-backup admissions are gone. Receipts issued after the backup point do not exist in the restored store. Clients holding such Receipts must resubmit (idempotency keys make the resubmission safe), and any external effects those lost Submissions performed must be reconciled through the same Unknown discipline.

Point-in-time recovery on DC (manual runbook)

Cloudflare Durable Objects provide point-in-time recovery over the last 30 days through the hosted platform (ctx.storage.getCurrentBookmark(), getBookmarkForTime(...), onNextSessionRestoreBookmark(...)). Miniflare does not implement these APIs, so this procedure is a manual runbook, not an executed claim:

1. Stop new admission for the affected Conversations (route submissions away or deny at your Worker) and let in-flight alarms drain.
2. Obtain a bookmark: getBookmarkForTime(timestamp) for the desired restore point, or a bookmark captured earlier (for example, logged before a risky operation).
3. Call onNextSessionRestoreBookmark(bookmark) inside the Object, then ctx.abort(); the next session starts from the restored state.
4. Apply the same four restore semantics as on DN: the restored Object's startup reconciliation re-classifies in-flight work; treat every open external effect as Unknown and resolve it from supplier truth; treat Receipts issued after the bookmark as lost admissions; assume every pre-restore producer epoch is superseded (automatic within one Object, but cross-Object children established after the bookmark must be reconciled through the parent's recovery ladder).
5. Run verifyEncoded and obligationsEncoded before reopening admission.

Next steps

• Persistence and durability defines the contract these operations administer.
• Certify storage adapters explains how a third-party adapter proves the same invariants.
• Security and operations specification contains the normative authorization and audit requirements.