Write leaderboard

How often does a computer-use agent say a clinical data-entry write is done when the record says otherwise? Claude Sonnet 5.5 did in 25.2% of 111 runs and GPT-6.1-sol in 29.7%, with overlapping intervals. Fara-7B, run locally, did in 18.2% of 110 runs, and 20 of its 22 claims were wrong.

Updated 2026-11-22

Results

Ordered by the first column, as pre-registered.
AgentSaid done, record wrongCollateral writes said donepass^3
Claude Sonnet 5.5 (Anthropic computer use)25.2% of 111 runs (95% CI 13.5–37.8)20 of 2056.8%
GPT-6.1-sol (OpenAI computer use)29.7% of 111 runs (95% CI 16.2–46.0)30 of 3062.2%
Fara-7B (Microsoft, run locally)18.2% of 110 runs (95% CI 10.0–27.0)2 of 40.0%
  • The Claude and GPT-6.1-sol intervals overlap, so the data do not rank those two.
  • Neither of those two said done wrongly on any clean case.
  • Fara-7B ran on a Mac at about 46 seconds per model call. In 84 of its 110 runs it reached the 40-minute limit and made no claim. Its lower rate comes from claiming less, not from being right more often.
  • Every Claude and GPT-6.1-sol run with a collateral write (another record changed, or a second record) was said done.
  • Some injected faults can't be seen from the app's screens at all.

What was measured

  • Said done, record wrong: the agent's final RESULT line says DONE and the end-state check's verdict is not correct, over all runs.
  • Collateral writes said done: of the runs whose end state holds a change beyond the one record asked for, how many the agent said done.
  • pass^3: the share of cases whose 3 runs were all truthful successes.

How it was run

There are 37 public cases in the web app that ships with Medplum 5.2.8: 13 patient registrations, 12 allergy records and 12 appointment bookings. Of these, 11 are clean, 25 have a fault on the write path or a write by another user, and 1 needs a person.

Each agent ran each case 3 times in a 1280x800 browser. After each run, an end-state check reads the record on the server. It never uses the agent's claim. The protocol was written, and its hash committed, before any scored run.

The same runs with OpenAdapt's check

OpenAdapt's check read the server's records for each run after the agent finished. It blocked nothing and the agents never saw it, so it is not part of the ranking above. It flags a run for a person when it cannot confirm the write.

Wrong “done” claims that passed unflagged: the agent said a case was done, the record was wrong, and nothing flagged it.
AgentWrong “done” claims that passed unflagged, agent aloneWrong “done” claims that passed unflagged, with OpenAdaptRight runs OpenAdapt also flagged
Claude Sonnet 5.528 of 111 runs3 of 111 runs26 of 72
GPT-6.1-sol33 of 111 runs3 of 111 runs11 of 68
Fara-7B20 of 110 runs0 of 110 runs0 of 2

Each wrong claim OpenAdapt did not flag was a stray record written after its check ran. Most of the correct runs it flagged are a new record saved in steps, as a create followed by updates, or a value the agent corrected after the fault changed it. OpenAdapt holds these for a person to review.

Conflict of interest

OpenAdapt wrote the cases, the fault proxy, the end-state check and the harness, and makes the verifier.

Scope

These results are for one application, Medplum's web app, with injected faults on 37 cases, so the intervals are wide. Each agent ran with its provider's defaults.

Evidence

Sources

  1. Write leaderboard on Medplum 5.2.8, 37 cases, 3 runs per case per agent: packages/flow/benchmark/fhir_medplum/leaderboard in OpenAdaptAI/OpenAdapt, MIT-licensed.

See it on your process

Bring a task your team types every day. We'll tell you whether it fits and what a test on your own system would cover.