Agent Experiences

AgentExperiencesScanner

AgentExperiencesScanner fetches a page when a signed-in person asks the Agent Analyzer to check it. It does not crawl the web on its own schedule.

User agent

AgentExperiencesScanner/1.0 (+https://www.agentexperiences.com/agent-analyzer/bot)

It never presents itself as a browser or as another company’s crawler.

What one scan requests

  • /robots.txt, to check whether the site has opted out of this scanner.
  • The page itself, following up to five redirects.
  • If the page could be read: the sitemap robots.txt declares (or /sitemap.xml),/llms.txt, /ai.txt and /.well-known/api-catalog.
  • The headers only of the page’s og:image, and of http:// on your host to check the HTTPS redirect.

Responses are capped at 5 MB for pages and 10 MB for sitemaps, and each request times out after 15 seconds. It does not run JavaScript, submit forms, or send cookies.

Rate limits

No more than 30 scans of one host per minute across every user, and each user is limited to a few scans a minute. Asking again for a page it read in the last ten minutes returns that result without fetching again.

Blocking it

Name the Agent Analyzer in robots.txt:

User-agent: AgentExperiencesScanner
Disallow: /

The Agent Analyzer reads robots.txt before every scan. When a group names AgentExperiencesScanner and disallows the page, it stops without fetching the page, and the person who asked sees that the site opted out. They are not charged.

A catch-all User-agent: * rule does not stop it. The scan is requested by a person, usually the site’s owner, and an over-broad catch-all block is one of the problems the report exists to show them. To refuse it at the network level instead, block the user agent at your edge; the AI crawler controls page covers the difference between crawler rules and edge enforcement.

Contact

Problems with the Agent Analyzer’s behaviour: alan@getprimeco.com.