Agent recovery article cover

By Alexander Akkarie. Content reviewed on 26 October 2026.

When an AI agent loses the response to a record-changing request, treat the outcome as unknown until the authoritative system resolves it. Keep one identity for the business operation, bind that identity to the exact request payload and return the recorded result on a valid retry. Test concurrent requests, changed payloads and result-check outages before granting write access. A fresh request ID can turn uncertainty into a duplicate.

The difficult moment is quite ordinary. The agent has prepared an exception, pressed the equivalent of “save” and heard nothing back. If the integration cannot establish what happened, a better prompt will not settle the question.

This guide proposes an acceptance contract for a holdings-reconciliation agent. The cases, IDs, time limits and outcomes below are hypothetical design examples. They are intended for operations and engineering to replace with their actual API behaviour.

Define the write before granting access

Give the first agent one job: compare approved holdings snapshots and create a pending exception for human review. It may read the snapshots and prepare the record. It cannot close the break, alter holdings, approve a valuation or release client output.

Creating a pending record is a write. It changes the exception system even if someone else makes the final business decision. Enforce that boundary in the API role, rather than relying on the agent to remember it.

The FINBOURNE account of OMBA's deployment describes agents working through existing APIs and entitlements with logged calls across daily investment operations. That is a useful architecture reference, reported by the supplier. It does not establish the retry, retention or concurrency behaviour of your particular integration. Nor does inheriting a human role show that the role is sufficiently narrow for this task.

For job R-207, rule 1.3 requires internal and broker holdings for the same portfolio, instrument and previous business-day close. Pending trades are excluded from both sides. The internal position is 1,000 units and the broker position is 990, leaving a 10-unit break. These inputs belong in the request record so that the exception can be checked later without asking the model to reconstruct its reasoning.

Make retry behaviour an API contract

AWS's Builders Library explains caller-supplied request identities, equivalent retry responses and atomic recording of the token with the mutation. The EC2 documentation provides a concrete service example in which reusing a token with changed parameters produces a mismatch. These are design references; the investment API needs its own explicit contract.

My analysis turns those references into the following contract for R-207.

Contract item

Requirement for the example

Business identity

Tenant, portfolio, instrument, valuation date, rule version and operation type identify one proposed exception. The caller persists key R-207-EX-01 before sending.

Payload binding

Store those fields, the two snapshot IDs, the 10-unit difference and requested status in an agreed, consistent order and format, or store a digest of that exact copy. A digest is a digital fingerprint calculated from the stored content. Ignore only transport fields explicitly excluded by the contract.

Duplicate request

Same caller scope, key and payload returns the original record ID and recorded result, with no second exception.

Changed payload

Same key with different business fields is rejected. A corrected calculation needs an explicitly authorised revision, not a disguised retry.

Concurrent arrivals

A unique operation constraint and transaction admit one creator. Other requests wait for or retrieve that operation's result. A check followed by an unprotected insert is insufficient.

Atomic result

Commit the exception and operation/result record in one transaction. If an external destination is involved, require destination deduplication and a durable delivery/outcome record; a local transaction alone cannot prove an external effect.

Retention

For this teaching design, retain the operation binding for 90 days; allow automated retries for at most 24 hours. After that, route to recovery. The example assumes these service rules; a real service needs its own approved retention and retry periods.

Late delivery

Keep a durable business-operation uniqueness check beyond the retry window. Expiry of the cache cannot authorise a second business record.

Unknown result

Mark the operation UNKNOWN and block dependent writes until an authorised recovery process establishes its outcome.

The key needs to distinguish genuine new work from repeated delivery of the same work. Hashing a payload alone can accidentally merge two intentional operations with identical values. Generating a fresh random key on every retry causes the opposite problem. Decide the business identity first, then persist it across transport attempts.

A completed lost-response test

At 09:00, supply a broker snapshot two business days old. R-207 rejects it because it cannot be compared with yesterday's internal close. No exception is created. Replace it with the correct dated snapshot and record both the rejection and accepted input IDs.

At 09:03, the agent sends R-207-EX-01. The server commits EX-184, difference 10, status PENDING_REVIEW, and the operation result together. The network drops the reply after commit.

At 09:04, an authorised result read using the original operation key returns EX-184. A retry at 09:05 uses exactly the same payload and returns the same record. The database count for this business operation remains one. At 09:06, the agent attempts approval; the service denies it and EX-184 remains pending.

Retain the initial snapshots, snapshot rejection, rule version, persisted request, payload digest, server transaction/result, retry response, denied approval and final database read. The last read establishes that holdings are unchanged and one pending exception exists. A successful HTTP response alone would establish much less.

Test the cases that defeat a simple request ID

The following are completed hypothetical test outcomes for an initial candidate and its corrected version.

Test

Initial finding

Required correction and retest result

Same key, changed difference from 10 to 12

Candidate accepts the altered payload and returns EX-184 without explaining the mismatch

Add payload comparison. Retest rejects the changed request; EX-184 still records 10.

Two simultaneous identical requests

A pre-insert lookup allows two workers to see “not found”; two exceptions appear

Enforce the business-key constraint and transactional creation. Retest records one exception and both callers resolve to it.

Crash between mutation and result storage

Candidate writes the exception, then crashes before recording the result

Commit both together. Injected pre-commit crash leaves neither; injected post-commit reply loss leaves both.

Result check uses a stale replica

“Not found” is read while the committed operation exists

Read the authoritative operation store or return an explicit uncertain/in-progress result. Retest does not authorise a fresh creation.

Key arrives after the retry window

Candidate assumes an expired cache means a new operation

Reject automatic replay and use recovery plus durable business identity. Retest creates no duplicate.

The corrected candidate earns only the permission exercised here: preparing pending exceptions under the tested conditions. These illustrative retest results are not a claim that a real system passed. Engineering must run fault injection against the actual transaction boundaries and operations must inspect the business records.

Recover an unknown result without inventing a new operation

Suppose the reply is lost and the result endpoint is also unavailable. The agent stores UNKNOWN against R-207-EX-01, stops automatic writes for that job and sends the operations owner the request, payload, last confirmed state and uncertainty. It may continue approved read-only preparation on unrelated work if the incident policy permits it. It cannot mark R-207 complete.

In this model the investment-operations owner owns the unresolved job. The integration on-call engineer restores access and investigates transport or storage failures. The owner authorises resumption only after the server record, business record and input versions agree. If no committed operation exists, recovery may permit replay of the original key within the documented contract. If an operation exists, recover its result. If the records conflict, keep the route suspended and resolve the discrepancy before another write.

At 10:20 in the completed outage scenario, the service is restored. The authoritative lookup finds EX-184 and its matching payload digest. Operations confirms one pending exception, records the recovery and resumes the route. The lost hour remains part of the job's operating cost.

For Luxembourg and EU firms, establish the applicable service and authority boundary separately. ESMA's AI statement concerns retail investment services under MiFID II and retains management responsibility. It does not decide the requirements for every AIFM back-office task.

Measure the job after review and recovery

The useful build consists of the narrow API role, operation store, transactional write path, result lookup, exception route and tests. AI performs comparison and preparation; deterministic code protects the business action. Give the operating owner visibility into unknown operations, ageing, recoveries and rejected payload changes.

For a simple hypothetical effort comparison, manual preparation and completion take 60 staff minutes. Agent-assisted preparation takes 15, review 20 and exception handling 10: 45 minutes total, saving 15 minutes or 25%. That excludes build cost and allocated recurring support, which still belong in the investment case. Measure comparable completed jobs and include recovery effort across the period, rather than selecting only uneventful days.

The accompanying guide turns this contract into one joint test session. Bring the operations owner and the engineer responsible for the write API. Stop the reply after a real test commit, and make the system show which single record it created before anyone presses save again.

Companion guide

Test uncertain exception writes, duplicate requests and recovery using a completed holdings-reconciliation example.