Claude Sonnet 5.5 vs Sonnet 5 on Fastio's Agent Benchmark

Claude Sonnet 5.5 and Claude Sonnet 5 each ran four document tasks seven times on the same 2,163 files through the Fastio connector. With each task’s highest and lowest run dropped, Claude Sonnet 5.5 averaged 99.5% across the tasks, where Claude Sonnet 5 averaged 82.5%. Claude Sonnet 5.5 was also 1.7x faster to a finished answer, and its model cost was 18% higher, counting model usage at Anthropic list rates only.

What we ran

The agent was Claude Code with no personal configuration, at high effort, with the Fastio connector (our Model Context Protocol, or MCP, server) as its only permitted tool and the same prompt for each task. Each model completed seven valid runs of each task between 29 September and 2 October 2026, against the same 2,163 synthetic business files. This compares two models on the same storage. It is not a comparison of storage providers, and it is not a claim that Fastio made either model better.

The four tasks:

• Contract renewal review. Which customer agreements renew automatically in the next two quarters, the notice deadline for each, and whether the nondisclosure agreement (NDA) is still in force.
• Contract contradictions. Every place one document contradicts another for the same customer. The answer key lists 17 conflicts.
• Customer relationship audit. One customer’s agreements, invoices and credits, scored on 12 facts and 4 traps.
• SOW revenue review. All 62 statements of work (SOWs), including three image-only scans.

Results

The table shows, for each model and task, the mean of seven runs with the highest and lowest run dropped. Accuracy is scored against our answer keys. Time is minutes and seconds to a finished answer. Cost is model usage at Anthropic list rates only; storage, ingestion and indexing fees are not included.

Averaged across the four tasks, accuracy went from 82.5% to 99.5%, a gain of 17 points. Total time fell from 42:38 to 25:01, 1.7x faster. Model cost rose 18% and tokens 25%. The medians of all seven runs tell the same story: 17 points more accurate, 1.6x faster and 23% higher cost.

Contract contradictions

The conflicts in the key include an amendment that misquotes the agreement it amends, two amendments with the same number, an amendment to an agreement that had already been restated, and a draft that differs from the executed version.

Claude Sonnet 5.5 found all 17 conflicts in six of seven runs and 16 in the other, where it said the executed version of one restated agreement was not in storage; it was, under a generic file name. Claude Sonnet 5 found 14, 4, 11, 8, 6, 10 and 13. With each model’s highest and lowest run dropped, accuracy was 56.5% for Claude Sonnet 5 and 100% for Claude Sonnet 5.5.

Contract renewal review

Several agreements had been replaced by restated versions. Claude Sonnet 5 listed one or more of those replaced agreements as renewing in 6 of 7 runs. Claude Sonnet 5.5 did so once. With each model’s highest and lowest run dropped, accuracy was 76.1% for Claude Sonnet 5 and 98% for Claude Sonnet 5.5.

Two Claude Sonnet 5.5 renewal runs scored below 100%. One scored 75% under the key as written: its deadlines were one day earlier, under a convention it stated. The other scored 90%: it listed one replaced agreement as renewing. Dropping each model’s highest and lowest run removes the 75% run and keeps the 90% run, which is why the Claude Sonnet 5.5 figure for this task is 98%.

The audit and the SOW review

On the other two tasks, Claude Sonnet 5.5 scored 100%, and Claude Sonnet 5 scored 97.5% on the customer relationship audit and 100% on the SOW review. Claude Sonnet 5.5 got there 1.7x faster on the audit (1:38 against 2:43) and 2.3x faster on the SOW review (4:09 against 9:32).

Reliability

Claude Sonnet 5 ended 5 of its 33 attempts with its agents still working and no answer to score. Claude Sonnet 5.5 finished 28 of 28. The unfinished runs were rerun and are not in the scores. One more Claude Sonnet 5 run failed on a network error and is not part of that count.

Cost

Claude Sonnet 5.5 was not cheaper. Model cost was 18% higher and tokens were 25% higher. Most of the increase came from the renewal review, where Claude Sonnet 5.5 splits the work across parallel agents and cost 56% more than Claude Sonnet 5. It cost 7% less on the contradictions task and 25% less on the SOW review. These figures are model usage at Anthropic list rates; storage, ingestion and indexing fees are not included.

Method and open scoring decisions

The two models alternated within each round of runs. The Fastio MCP server was version 2026.09.66 for the first three rounds, 2026.09.68 for the next two and 2026.09.69 for the last two, and both models ran on the same version within each round. Answers were checked by hand and scored against our answer keys. Items an answer listed that fell outside a key’s scope, such as NDAs, vendor agreements and leases, were neither credited nor penalized.

Two scoring decisions are still open, and they could move the headline:

• Customer NDA drafts in the contradictions task. If they were counted, the headline gain would be about 13.5 points rather than 17, and Claude Sonnet 5.5 would score 73.9% on that task.
• The look-alike company in the audit. If quietly avoiding it earned credit, the headline gain would be about 16.3 points rather than 17.

The files are the same 2,163 used on our benchmarks page, which reports a different study: a comparison of storage providers, run in Claude Cowork with Claude Opus 5 at medium effort. Its figures are not comparable with the ones here.

What this means for teams running agents

The largest gains came where the work means reading documents against each other, such as a misquoted amendment or an agreement that a restated version has replaced. Those are also the tasks where Claude Sonnet 5’s results varied most from run to run.

If your team runs agents on contracts or other business records, rerun your own checks when a new model ships, and track cost next to accuracy. On this benchmark the more accurate model also cost 18% more in model usage.

The connector used in these runs works with Claude Code and Claude Desktop. Setup steps for both are on the Claude MCP server page.