ComponentBench
Researchers at Duke and Amazon's AGI SF Lab tested seven models on 97 standard types of interface component across 2,910 verified tasks, built with the Ant Design, MUI, and Mantine component libraries. They kept the model and the task the same and changed only what the agent was allowed to see. GPT-5 mini scored 83.1% when it worked from the page's accessibility tree (a plain outline of every button, link, and field that browsers build for screen readers) and 48.9% when it worked from coordinates on a screenshot.
Most advice about building for AI agents starts with the model: which one to use, how good it is, and what the next version will fix. That way of looking at it holds up until someone runs an experiment where the model doesn't change.
It also leaves designers with nothing to do, which is why this result belongs in AXD (agentic experience design, which means designing how people and AI agents work together on a job) and not on a machine learning roadmap.
Change only what the agent is shown, either the accessibility tree or pixels and coordinates, and the same model on the same tasks moves by more than thirty points. Its intelligence stayed the same. What changed was whether the page explained itself through its markup (the HTML code underneath a page that says what each part is).
What the gap is made of
The ComponentBench authors sorted 8,864 failed attempts by cause, and reasoning isn't at the top of the list. The biggest cause, at 20.2%, is what they call continuous calibration error: the agent knew what it wanted to do and couldn't land on it exactly. Next come transient state loss at 19.9% (losing hold of something that's only on screen briefly), a missing final step, such as pressing Save, at 11.6%, and target acquisition at 11.2%, which means clicking the wrong copy of the right thing.
The biggest and fourth-biggest causes, about a third of failures, were about aim rather than judgment, and how easy a control is to aim at is decided by whoever designed the interface.
The results for each family of components make the same point more bluntly. Averaged across six models, here is how often agents passed. The family names are the study's own. Roughly, "discrete choice" means picking from a fixed set of options, "disclosure and progressive" covers controls that reveal more as you go, such as expandable sections, and "continuous precision" means setting a value along a scale, as with a slider.
| Component family | Pass rate |
|---|---|
| Command and navigation | 91.6% |
| Overlays and transient UI | 89.2% |
| Discrete choice | 86.3% |
| Disclosure and progressive | 72.1% |
| Date and time | 71.2% |
| Advanced editors | 61.9% |
| Continuous precision | 59.4% |
| Drag, drop and workspace | 47.7% |
A drag-to-reorder list is close to a coin flip, while a menu almost always works.
Easy for people, hard for agents
Nine component types in the study take a person two steps or fewer, yet agents pass them less than 60% of the time. A range slider takes people 1.9 steps and agents pass it 39.9% of the time. A window splitter (the bar you drag to resize two panes) takes 1.3 steps and scores 38.3%. Resizable columns take 1.7 steps and score 24.4%.
Even a native select element (the browser's own built-in dropdown), at 1.4 steps for a person, scores only 53.1%.
Nothing in the study was hard for people and easy for agents, so instincts built from watching people use your site don't carry over. The control you pick because it feels effortless may be the one that stops an agent dead.
This is where designing for agents breaks away from designing for people instead of simply adding to it. A person can usually find a way round a confusing control, but in these studies agents often failed at the same control.
WorkArena
ServiceNow Research and Mila tested agents on 33 tasks drawn from real ServiceNow instances (company installations of ServiceNow's workplace software), covering 19,912 task instances. On the six list-filtering tasks, GPT-4o, GPT-4o-V, GPT-3.5, and Llama3-70B each scored 0.0%. The authors put this down to a non-standard HTML widget, a control built from custom code instead of the browser's own parts, and note the tasks are "fairly simple for a human to complete."
One widget, every model, zero
The WorkArena result is the one to show anyone who wants a number.
Four models tried six tasks, and every one of them scored zero percent. The same agents completed 77.8% of service catalog tasks and 80% of knowledge base tasks, but they couldn't filter a list, because the filter was a custom widget instead of being built from standard elements.
A person barely notices the difference. The control looks like a filter, works like a filter, and has a little funnel icon on it. But to anything reading the markup, it's a div (a generic, unlabeled box) with some code attached, and there's nothing there to operate.
The same paper also measured page size. Even after scripts, styles, and empty elements are stripped out, the HTML of those pages runs to between 40,000 and 500,000 tokens (the chunks of text a model reads; every model can only take in so many at once). The authors used accessibility trees instead, calling the raw HTML prohibitively large. So the size of a page now matters for more than speed, because it also limits how much of the page an agent can take in.
The half agentic experience design can fix
SeeAct
Ohio State researchers found GPT-4V completed 51.1% of tasks on live websites when a human turned its written plans into actions by hand, meaning a person did the work of finding the right element on the page. Their conclusion was that "grounding still remains a major challenge." Grounding is the step of matching a plan to the exact thing on screen.
Split an agent's job in two. One half is deciding what to do, and the other is finding the thing on the page to do it to. SeeAct took the second half away from the agent by having people do it, and the agent completed 51.1% of tasks on live websites. The researchers also built their own method for letting the agent find elements itself, using both the page's code and how it looks. Even their best version left what they called "a substantial gap" below that 51.1%.
That split matters because the two halves belong to different people. The first half is the model maker's problem, and it gets better without you doing anything. The second half is your markup, and it only gets better if you improve it.
What to change in your interface
None of this is exotic. It's close to the accessibility advice that has been on this site for a year, because people using assistive technology and agents need the same things, for different reasons.
Prefer standard elements over custom ones. In WorkArena, one custom filter widget stopped every model. Where you really do need a custom control, give it the roles, states, and labels a standard one would have had. Those are the hidden labels in the code that tell software what a control is, whether it's open, and what it's called.
Count the steps in your most important flows. Each required interaction is another chance to fail, and those chances multiply. If each step works nine times in ten, five steps in a row work only about six times in ten. Our post on why reliability compounds goes into this.
Give sliders and drag controls a backup. Sliders, splitters, drag handles, and color pickers belong to the families at the bottom of the table above. Where one of them is the only way to set a value, add a second way that takes a typed number.
Look at your page the way an agent does. That means the accessibility tree, which is what the better agents read, rather than the page as it looks on screen. Your browser's developer tools will show it to you. If a control is missing from that tree, then as far as an agent is concerned it doesn't exist.
For the decisions underneath these, such as which control to use and when, our interaction patterns for AXD cover the ones you can reuse.
The caveat these numbers deserve
Benchmark figures age badly, and people in this field tend to keep quoting them long after they stop being true. A 2025 paper from Ohio State and Berkeley, pointedly titled An Illusion of Progress?, argues that earlier reported results overstate what agents can do, largely because of how the benchmarks are built.
Read the percentages here as true on the dates they were published, and no later. The direction is more likely to last than the numbers: in each study, what the page showed the agent made a large difference, and that part is yours to fix.