Why an Agent That Works 80% of the Time Finishes a Third of the Job
Eighty percent sounds like a product you could ship. Most people hear it as "works, with the occasional miss": a bit annoying, easy to live with, the kind of thing a retry button covers. Then someone uses it on a task with five steps in it.
This post runs the numbers, looks at what three agent benchmarks (standard sets of tasks that researchers use to score agents) found, and ends with what to change in your own design.
The arithmetic nobody runs
ComponentBench
The authors make the point against their own results: a task that needs five critical interactions, each succeeding 80% of the time, can only succeed from start to finish about 33% of the time at best. Their benchmark also found that 55.8% of failed runs end with the agent stuck repeating the same action, and that this happened evenly across every underlying cause. So the loop is the symptom you see, not the fault that caused it.
Five steps at 80% each is 0.8 multiplied by itself five times, which comes to about a third. Seven steps gives you 21%, and ten gives you 11%.
None of this is surprising math. Product decisions just rarely get tested against it, because reliability is usually quoted per action but felt per task, and agent integrations quietly fail in the gap between the two.
It's also the clearest argument for treating agentic experience design as its own practice. An eleven-step flow can still feel pleasant to a human, but if each step works 80% of the time, an agent finishes it about one time in twelve.
It changes what a step costs, too. On a flow an agent drives, each extra confirmation screen costs finished tasks. If the agent gets through each screen 80% of the time, adding one more takes a 33% success rate down to 26%, so about a fifth of the tasks that used to finish no longer do. Keep the screens that guard money or things that can't be undone, and cut the rest.
One try isn't the number that matters
Average success rates hide something worse. They tell you about one attempt, and real use means doing the same thing again and again.
τ-bench
A benchmark for agents working with real business policies and tools, scored by comparing the final state of the database with the result the task should have produced. GPT-4o succeeded on fewer than half of tasks. Its pass^8 score on the retail tasks, meaning it completed the same task correctly on all eight of eight attempts, fell below 25%.
Ask an agent to do the same task eight times, and it gets all eight right less than a quarter of the time.
If you're deciding whether to keep handing a job to an agent, that's the number that drives the decision. Nobody experiences your average, only one attempt after another, and one bad outcome in that run is what ends the habit, especially if it was a duplicate order or a booking that got canceled when it shouldn't have been.
A third of the wins aren't really wins
ST-WebAgentBench
222 tasks, each paired with explicit policy rules, scored on six things: user consent, staying within its limits, strict execution, following rules in order of priority, coping with tricky or hostile input, and error handling. Under Completion Under Policy, which only counts a finished task if it followed every rule that applied, three leading agents averaged less than two-thirds of the completion rate they would otherwise get credit for.
In those cases the agent finished the task, but it broke a rule along the way. It acted outside the limits it was given, skipped a consent step, or kept going past a point where it should have stopped.
Counting those as successes is how a system looks fine in testing and then generates complaints once it's live. The six areas also work as a design brief, because each one is a question about your interface. Where does this flow need explicit consent, and would an agent working through the page notice?
How compounding failure should change your design
Shorten the path before polishing any single step. At 80% a step, removing one raises the task's success rate by a quarter, which is as much as making that step perfect would. If your checkout has nine steps, the fastest way to an agent that works is seven steps, not a better date picker. The agent journey map is one way to count those steps across a whole task instead of a single screen.
Make actions safe to repeat, and say so. Retrying is the main way an agent recovers when something goes wrong. If a retry can create a second order, every failed attempt turns into a support ticket instead of a second chance. Engineers call an action that's safe to repeat idempotent (doing it twice has the same effect as doing it once), and your documentation should say which of your actions are.
Make three kinds of failure look different. Bad input, try again later, and never going to work should each get a different response. An agent that can't tell them apart keeps retrying what will never succeed, and gives up on what would have worked a moment later.
Give every task one end state that can be checked. The strongest thing you can offer is something your customer can check afterwards, such as a reference number, a receipt, or a link to the thing that changed. It's also the cheapest protection against the eight-attempts problem above, because it turns a failure nobody saw into one somebody can see.
Where the numbers are soft
These figures come from benchmarks, and a benchmark score measures the scaffolding around the agent (the code wrapped around a model to guide it through a task) as much as the agent itself. WebArena, a test of agents doing everyday jobs on realistic websites, first reported a best score of 14.41%. A different team later reached 25.4% on the same tasks, mostly by building better scaffolding around the model. Scores also move quickly. TheAgentCompany, a test of agents doing office work inside a pretend software company, reported in the first version of its paper that the best agent could finish 24% of tasks on its own, and 30% in a later version. A single benchmark number tells you what was possible on the day it was measured.
The compounding, though, has nothing to do with how benchmarks are run. It's arithmetic, and it holds whatever the per-step rate turns out to be next year, so cutting a step helps at any per-step rate short of perfect.