Agent alone
Reported done (bad save)- Invoice IDINV-4082
- VendorNorthwind Supplies
- Amount$4,250.00
- Remittance email(empty in database)
Proof
We ran AI agents on data-entry tasks in open-source apps and broke some of their writes on the way. Each card counts one thing twice, for the same runs: with the agent alone, and with OpenAdapt checking each write against the app's own records.
Synthetic records on open-source apps, Oct and Nov 2026. Each number is for one agent or one setup, never added across tests. Sources · Try the live catch demo
Interactive walkthrough
Step through a scripted synthetic vendor-invoice sample (INV-4082 / northwind.example) where a planted fault blanks one field during Save. Compare what an agent alone reports against independent read-back, then verify the receipt signed with a synthetic demo Ed25519 key directly in your browser.
Agent reports done; OpenAdapt catches the mismatch and halts. Synthetic invoice data only.
ap@northwind.example, read ""Runs crypto.subtle.verify('Ed25519', ...) locally in your browser against the signed DSSE envelope (zero network calls, demo key seal-ed25519-2fb5335a79897786).
1
Each agent used the web app that ships with Medplum 5.2.8 like a person would, on 37 data-entry cases, 3 times each: register a patient, record an allergy, book an appointment. A proxy broke some of the saves on the way. After each run, a check of the server's records decided whether the agent's answer was true.
Claude Sonnet 5.5 Anthropic computer use
Claude Sonnet 5.5 did the 37 cases in Medplum's web app, 111 runs in all, while a proxy broke some of its saves. It said done 28 times when the record was wrong.
Wrong “done” claims that passed unflagged
OpenAdapt flagged 25 of the 28 for a person.
Right runs OpenAdapt also flagged: 26 of 72
GPT-6.1-sol OpenAI computer use
GPT-6.1-sol did the 37 cases in Medplum's web app, 111 runs in all, while a proxy broke some of its saves. It said done 33 times when the record was wrong.
Wrong “done” claims that passed unflagged
OpenAdapt flagged 30 of the 33 for a person.
Right runs OpenAdapt also flagged: 11 of 68
Fara-7B Microsoft, run locally
Fara-7B did the 37 cases in Medplum's web app, 110 runs in all, while a proxy broke some of its saves. It said done 20 times when the record was wrong.
Wrong “done” claims that passed unflagged
OpenAdapt flagged 20 of the 20 for a person.
Right runs OpenAdapt also flagged: 0 of 2
Each wrong claim OpenAdapt missed was a stray record written after its check ran. In this test OpenAdapt checked each run after the agent finished and flagged runs for a person; it did not block.
2
Each agent did the same 132 tasks through the APIs of InvenTree, Snipe-IT, Kimai and OpenMRS: add a record, or change one. Separately, we planted faults in the same apps with no agent. OpenAdapt ran as a blocking check: a write it could not confirm was held for a person instead of passing as done.
Claude Sonnet 5.5 claude-sonnet-5-5
Claude Sonnet 5.5 did 132 add-or-change tasks through the APIs of InvenTree, Snipe-IT, Kimai and OpenMRS. Its own mistakes were the harmful writes.
Harmful writes that passed as done
OpenAdapt named the exact problem for 1 of the 1.
Correct writes OpenAdapt also stopped: 111 of 131
Codex gpt-5.6-sol
Codex did 132 add-or-change tasks through the APIs of InvenTree, Snipe-IT, Kimai and OpenMRS. Its own mistakes were the harmful writes.
Harmful writes that passed as done
OpenAdapt named the exact problem for 7 of the 14 and stopped the other 7 for a general reason.
Correct writes OpenAdapt also stopped: 101 of 118
Llama 3.1 8B llama3.1:8b
Llama 3.1 8B did 132 add-or-change tasks through the APIs of InvenTree, Snipe-IT, Kimai and OpenMRS. Its own mistakes were the harmful writes.
Harmful writes that passed as done
OpenAdapt named the exact problem for 3 of the 4 and stopped the other 1 for a general reason.
Correct writes OpenAdapt also stopped: 15 of 16
Planted faults no agent
We planted faults directly in the same four apps, with no agent involved. 87 were harmful writes of the kind the check covers.
Harmful writes that passed as done
OpenAdapt named the exact problem for 62 of the 87 and stopped the other 25 for a general reason.
Correct writes OpenAdapt also stopped: 14 of 16
170 of OpenAdapt's 241 stops on correct writes came from a column the app changes on every call, such as a last-login time. Separately, 1 planted harmful write on Snipe-IT, made after the check ran, came back confirmed.
3
Shadow mode sits between the agent and the app. It passes every write through unchanged, reads the app's records after each one, and reports each write that landed differently from what the agent sent. A proxy behind it broke some writes. Each setup is a separate result.
Medplum 5.2.8 scripted agent
A scripted agent created patients, allergies and appointments in Medplum 5.2.8 through its FHIR API, while a proxy broke some of the writes.
Harmful writes that nobody reported
OpenAdapt reported 18 of the 18.
Correct writes OpenAdapt also reported: 0 of 13
Medplum 5.2.8 Claude Sonnet 5.5
A Claude Sonnet 5.5 agent created patients, allergies and appointments in Medplum 5.2.8 through its FHIR API, while a proxy broke some of the writes.
Harmful writes that nobody reported
OpenAdapt reported 9 of the 9.
Correct writes OpenAdapt also reported: 4 of 22. Each was saved as the wrong record first, then fixed by the agent.
Dolibarr 21.0.4 customer creates
An agent created customers in Dolibarr 21.0.4 through its REST API, while a proxy broke some of the writes.
Harmful writes that nobody reported
OpenAdapt reported 8 of the 8.
Correct writes OpenAdapt also reported: 0 of 5
Shadow mode made 0 writes of its own in Medplum's audit trail.
4
10 to 17 min
We set up OpenAdapt with you on your system. On the four new apps it took 10 to 17 minutes and 9 to 15 decisions by a person per app. Shadow mode on Medplum took 76.5 s of machine time and 3 decisions.
15.2 to 292 ms
Median time shadow mode added to each write, by setup: 15.2 ms (Medplum 5.2.8, scripted agent), 25.3 ms (Medplum 5.2.8, Claude Sonnet 5.5), 292 ms (Dolibarr 21.0.4 and customer creates).
The public Medplum benchmark gives your agent 37 clinical data-entry tasks against a local FHIR server, breaks some of its writes on the way, and scores each one with and without OpenAdapt. It runs on your machine with synthetic patients. Run the steps in order, in one terminal.
You need Docker (or Podman or Colima), git, and Python 3.10, 3.11 or 3.12.
Get the benchmark: the 37 cases and the local Medplum server setup.
git clone --depth 1 https://github.com/OpenAdaptAI/OpenAdapt && cd OpenAdaptInstall the OpenAdapt CLI in a fresh environment.
python3.12 -m venv .venv && . .venv/bin/activate && pip install openadaptCheck your setup with the reference agent. It uses no model and costs nothing.
openadapt bench medplum --agent scripted --downScore your own agent. It reads each task as JSON on stdin and prints its claim as the last line.
openadapt bench medplum --agent "cmd:python my_agent.py" --downRead the scorecard: harms caught, harms missed and correct writes stopped, with and without OpenAdapt.
cat bench-medplum-results/SCORECARD.txtWhat your agent reads and prints →The benchmark on GitHub →For AI agent teams →
In a shadow pilot, we set up OpenAdapt with you on one workflow where your agent creates records through your system's API. It checks each create and passes every write through unchanged. At the end of the week, the report shows each write that landed differently from what your agent sent.
Book a 30-minute callAgent alone or with OpenAdapt →How do you know a write landed? →