Agent Experiences

Everyone Is Buying the Last Mile of AI

By Alan WeibelPublished 8 min read

In six months, most of the companies that sell AI to large businesses have built or bought the same thing: a team whose job is to make AI work inside one customer at a time.

Different companies, different models, one bet. AI only works in production when it is built and tuned for the specific customer, and the work of doing that is now worth billions.

We covered why software is drifting back towards custom builds in Software Is Going Back to Custom. This post is about the work itself: what the engineers placed inside a customer actually do, why it takes them weeks, and how a company can make it take days.

Why a demo agent is not a production agent

A demo agent works on the vendor's data, with the vendor's examples, against questions the vendor chose. It is usually impressive. A production agent has to work on a Tuesday afternoon inside one particular company, and that company is never the demo.

Take a support agent going live at a mid-sized retailer. Before it can answer a single real customer, someone has to settle:

None of that is a model problem. The same model that impresses in the demo gives wrong refunds in production if it does not know the December exception. This is why Cresta's job description asks its engineers to "fine-tune prompts and configurations" for each deployment, and why Microsoft describes its specialists' job as to "co-design, co-innovate, deploy and continuously improve AI systems". Continuously is the important word. Tuning is not a setup step. The rules, data, and edge cases keep changing after launch.

Most of the time goes on access, not models

Ask what slows a deployment down and the answer is rarely the AI. It is everything the AI has to reach:

Each of these adds days or weeks, and none is the engineer's to fix alone. They belong to the customer.

What makes a company quick to deploy into

AWS's stated aim is for its engineers to leave behind self-sufficient teams "in a matter of weeks". Whether that happens depends mostly on the company they arrive at. A company that has done the following before the engineers arrive gets their time spent on the agent, not on finding a way in:

Every action staff take has an API. List what people do in each system and check it can be done without the screen. Fill the gaps the agent will need first.

The rules are written down where an agent can read them. Policies, exceptions, and approval limits in plain text, kept in one place, with an owner who updates them.

Agents get the permissions of the person they work for. Scoped access that mirrors the user's, not a separate set of rules for software.

The public site is readable without a browser. Help content, policies, and prices in the page's HTML, with clear structure, so the agent and the customer read the same facts.

There are real examples to test against. Past tickets, orders, and cases, with the right answer, so every change to the agent can be checked before it reaches a customer.

Someone owns the agent after the engineers leave. A named person who updates the rules, reviews what the agent did, and decides when it needs retuning.

How this site helps forward deployed engineers

We built the tools on this site for the first week of exactly this kind of work: finding out what an agent can and cannot reach before anyone promises a launch date.

Start with the Agent Analyzer. Run it on the customer's help centre, policy pages, and product pages on day one. It reports which pages an agent cannot read, where content needs JavaScript, and where structure or metadata is missing, with a fix for each. A site crawl covers several pages from the sitemap at once. That list is the access work, written down before it becomes a delay.

Show stakeholders what the agent sees. The Agent Visualizer walks an agent through a page and draws where it stops. A screen showing an agent failing to find the returns policy gets a fix prioritised faster than a ticket does.

Let your own agents run the checks. The Analyzer is also an MCP server. Connect it to the coding agent working on the customer's site and it can scan a page, read what failed, apply the fix, and scan again to confirm, without copying reports between windows.

Hand the customer a plan they can own. The AXO audit checklist covers the public site, and the implementation guide for applications covers the patterns that make a product usable by agents: the APIs, permissions, and documentation that outlast the engagement.

Give them a baseline. The State of Agent Readiness shows how well-known sites score, so a customer can see where it stands against companies it knows. Re-scan at the end of the engagement and share the report's score card to show what changed.

The engineers are expensive and the engagement is short. Every day they do not spend discovering that the policies page is unreadable is a day spent on the agent.

See the journey on your own site

Before the agent goes live, see what it can reach. Watch an agent work through a customer's help or policy page and see where it stops.