Learning Path

An agent is a medium, not an actor with its own interests. When an assistant fetches a page, fills a form, or calls your API, there is a specific person on the other end who asked for it — the same way a browser request has a person behind it. Adversarial automation still exists and still needs blocking, but it is defined by what it does (scraping at volume, stuffing credentials, posting spam), not by the fact that a human hand was not on the keyboard. Treating every non-human request as hostile is now a way of turning away paying customers who happen to have arrived through a different medium.

For twenty-five years, “bot” was a useful shorthand on the web. Bots were the traffic you did not ask for: search crawlers you tolerated, scrapers you tried to stop, and attack tooling you blocked outright. All of it shared a defining property — no user was waiting on the other end. That property is what made blanket bot defenses safe. Blocking automation cost you nothing because automation was never a customer.

Delegated agents break the shorthand. A person asks their assistant to compare two plans, check whether an order shipped, or book the thing they have been meaning to book. The request that arrives at your service is automated in mechanism and human in origin. If your defenses cannot tell those apart, they will treat a customer with intent to buy exactly the way they treat a credential-stuffing script.

Two populations that share a transport

The useful distinction is not human versus machine. It is whether a specific person authorized this specific request, and whether the traffic pattern serves them or exploits you.

Adversarial automation

No authorizing user for the individual request.
Volume and breadth are the point: enumerate catalogs, harvest listings, mirror content.
Often tries credentials it does not own, or posts content nobody asked to send.
Evades identification — rotating residential proxies, forged user-agents, headless fingerprint spoofing.
Blocking it costs you nothing.

Delegated agents

Every request traces to a person who asked for an outcome.
Depth over breadth: a handful of pages or calls that answer one question.
Frequently already authenticated as the user, in the user’s own session.
Usually willing to identify itself if identifying itself gets it served.
Blocking it costs you the customer, not the agent.

There is a third population worth naming separately, because it is the one that made publishers angry in the first place: training and index crawlers. Those are not delegated — no user is waiting — but they are not adversarial either. They are a commercial relationship you may or may not want, and they belong in a policy discussion about compensation and attribution rather than in your WAF. We treat them separately in agent–service relationships.

What blanket blocking actually costs

The cost is invisible in your metrics, which is why it persists. A blocked agent does not file a support ticket. The customer sees their assistant report that it could not access the site, shrugs, and asks it to try somewhere else. You record a 403 and a slightly lower conversion rate with no explanation attached.

Lost transactions you never see. When an agent cannot complete a purchase or a booking on your site but can on a competitor’s, the substitution happens inside a chat window you have no visibility into.
Support cost transferred to humans. Blocking the assistant does not remove the task. It routes an order-status check that could have been a cached API response into a phone queue.
Absence from the answers. If your product pages cannot be read, the comparison an assistant produces is built entirely from your competitors and from third-party descriptions of you.
Blocking your own logged-in users. The worst version: an authenticated customer using an agentic browser gets challenged or locked out of an account they own, on a session they started themselves.

The legal picture moved: a user-directed agent is the user acting

The question of whether a site can forbid a customer from using an agent to visit it stopped being hypothetical. On August 4, 2026, the Ninth Circuit vacated Amazon’s injunction against Perplexity’s Comet browser. The reasoning is the part that matters for anyone writing access policy: an agent operating at a user’s direction, in the user’s own authenticated session, is best understood as the user acting — not as a third party gaining unauthorized access.

One appellate decision does not settle American internet law, and the underlying dispute continued. But it establishes a direction that access-control design should account for rather than bet against.

The practical reading: a policy that says “no automated access” is increasingly a policy that says “no assistive tooling,” and it now has to survive the argument that you are restricting how your own customers use their own accounts. Rules about conduct — rate, scope, resale of extracted data, account sharing — are far more defensible than rules about mechanism.

What that implies for your terms and your rate limits

Write terms against behavior, not tooling. Prohibit bulk extraction, redistribution, circumvention of authentication, and account sharing. Do not prohibit “use of automated means to access the Service,” which now describes a large share of legitimate customer activity.
Rate-limit per authenticated principal, not per user-agent class. A signed-in user’s agent should get the user’s budget. If that budget is too small for an agent to do anything useful, the budget was probably tuned for a slower medium.
Separate the read policy from the act policy. Whether a crawler may ingest your content for training is a commercial question. Whether a customer’s assistant may check their order status is a product question. Conflating them in one robots.txt-shaped decision produces bad answers to both.
Make blocks legible. If you do refuse, return a status and a machine-readable reason the agent can relay to its user, so the person learns they need to do this step themselves instead of concluding your service is broken.

The hard part: there is often no bot signal at all

Detection advice usually assumes the automation announces itself. Declared crawlers do: GPTBot, ClaudeBot, PerplexityBot and their peers send distinctive user-agents and publish IP ranges, and you can verify them. Agentic browsers do not fit that model at all.

Comet, Claude in Chrome, and in-browser assistant modes are Chromium. They render with a real engine, execute JavaScript, carry the user’s cookies, and present an ordinary desktop browser user-agent. There is frequently nothing at the HTTP layer that distinguishes “the user clicked this” from “the user’s assistant clicked this on their behalf.” Behavioral heuristics fare little better: an agent that reads three pages and submits a form looks like an efficient human, and a human comparison-shopping across twelve tabs looks like a scraper.

Design consequences of an unobservable medium

Do not build policy that depends on detecting the browser-based case. You will be wrong in both directions, and the false positives land on logged-in customers.
Reward voluntary identification instead of punishing it. Cryptographically verified agents — via HTTP message signatures and published key directories — should receive better limits than unidentified traffic. Identification only spreads if it buys something.
Assume your public HTML is read by a model. If it renders only after client-side hydration, or hides the answer behind an interstitial, plan for that content to be absent from the summary the user receives.
Put the friction on consequence, not on suspicion. Step-up verification before payment, address changes, or destructive operations is defensible for every visitor. A CAPTCHA in front of a product page is a tax on the medium.

The reframe is the whole point. Once agents are a medium rather than a threat category, the design question changes from “how do I keep them out?” to “what does this customer need my service to expose so their agent can finish the job?” That is the subject of the delegation cycle.

The Delegation Cycle Back to Applications

Stay Updated

Analysis of AI search, crawler policy and agent standards — sent when there is something worth reading, roughly twice a month. Unsubscribe anytime.