In May 2026, Google released a small series of audits for agentic browsing readiness. To me, the most interesting part was that the audits doubled down on a performance metric, Cumulative Layout Shift, and an accessibility requirement, a valid accessibility tree. As it turns out, the best way to optimize your site for the robot revolution is to build the fast, stable, and highly semantic website that human users have deserved all along.
There are different ways that people are optimizing for AI currently, and I’ll talk about two of them here. First, they are optimizing for agentic web browsing, where an agent operates a web browser on behalf of the user. This is an emerging form of automation that is just now catching on. The second type I’ll discuss today is optimizing a site for a web fetch. In a web fetch, a chat-based client like Claude, Gemini, or ChatGPT retrieves information that isn’t in its training set from an external website.
In urban planning, there’s a concept known as the Curb-Cut Effect. When sidewalks were first redesigned with sloped curb cuts to accommodate wheelchair users, planners quickly noticed that everyone benefited. Parents pushing strollers, travelers rolling suitcases, skateboarders, and delivery workers all used the curb cuts.
By examining what it takes to make a site AI-ready through the lens of universal design, we can see this effort as a digital “curb-cut” that’s an opportunity to benefit machines and humans alike.
Past curb cuts
The emphasis on AI-web readiness shares similarities with parts of web history. For example, smartphones were major market disruptors, and when Google told site owners to develop a mobile-friendly site, more people started implementing responsive design. This emphasis on new device types allowed more people to access information in new ways, regardless of the device they were using and whether they had a Wi-Fi connection.
Later, when Google introduced the Core Web Vitals metrics and disclosed that it uses them as a ranking signal, more people started paying attention to the performance of their sites. As businesses chased better placement on a search engine results page, users often got faster sites and a better overall experience as a result.
Agentic browsing
Agentic browsing is when an AI agent uses a web browser on a person’s behalf. Instead of simply retrieving and summarizing a page, the agent can inspect the rendered interface, follow links, fill out forms, and interact with controls to complete a task.
Side-effects
Taking a look at the audits Google rolled out for agentic browsing readiness, Google repurposed other metrics and gave them added importance. Cumulative Layout Shift was already part of the performance audits in Lighthouse and is a stable Core Web Vitals metric, and a well-formed accessibility tree aligns with the broader accessibility audit category of Lighthouse.
From Google’s own documentation:
Ensure a sound a11y tree: Prioritize semantic HTML and proper ARIA labeling, as these are the “machine-eye view” of your page.
Optimize for stability: Reduce layout shifts to ensure that agents can reliably interact with your UI without elements moving unexpectedly.
These criteria benefit humans and agents alike.
Web fetch in LLMs
The other type of AI-readiness optimization I’ll discuss today is a web fetch. A web fetch is typically more direct. A tool or service retrieves a page’s content and supplies the response as context, often without exposing a rendered, interactive page to the model.
As products like Gemini, ChatGPT, and Claude carve out more market share, it’s important to think about how these systems interact with the web. These systems are all capable of fetching web content, and I gave each of them the following prompt: “When you do a web fetch, tell me what tools you use and what your performance expectations are.” That query yielded the following responses:
| Feature / Category | Claude | ChatGPT | Gemini |
|---|---|---|---|
| TTFB & Latency | Crucial; TTFB is nearly the entire latency budget. | TTFB < 200ms (via HTTP/2/3, avoid redirects). | TTFB < 500ms to prevent timeouts. |
| Render Mode | Server-returned HTML (client-side mounting yields a spinner). | Complete, Server-Rendered HTML (no heavy client-side JavaScript rendering). | Raw HTML or Markdown (client-side JS dependencies lead to missing content). |
| Markup & DOM Bloat | Avoid page clutter competing for context window. | Lightweight Markup (small payload size, limited hidden DOM nodes, and JS bundles). | Payload Size < 1-2 MB (excessive HTML/DOM bloat exhausts context window). |
| Structural Fidelity | Crucial; relies on semantic markup and proper document outlines. | Clear Hierarchy (clean, semantic elements rather than deeply nested <div> wrappers). |
Not explicitly mentioned in this list. |
| Accessibility & Context | Alt text & captions are the only way to see images or scanned PDFs. | Accessible Markup (ARIA landmarks, clear tables, descriptive anchor text). | Not explicitly mentioned in this list. |
| Crawl, URLs & Metadata | Not explicitly mentioned in this list. | Crawl Efficiency (canonical URLs, sitemaps, robots.txt, semantic links) & Structured Data (JSON-LD) & Stable, Readable URLs. | Not explicitly mentioned in this list. |
| Security & Bot Gateways | Hard failures on bot challenges, consent interstitials, redirect chains, and 403s. | Not explicitly mentioned in this list. | No CAPTCHAs/Turnstiles or Bot Gateways (automated agents cannot solve interactive challenges). |
Building for machines and benefiting humans
If you look closely at the list of LLM requirements, you’ll notice none of these requirements are unique to AI. All of them are things that engineers like me have been recommending for years, and they’re all things that make your site better for humans. This is the curb-cut I was talking about.
Let’s take a look at how aligning your site with AI expectations results in a better experience for people.
1. Fast TTFB and low latency
When Gemini or ChatGPT demands a TTFB under 200-500ms to prevent a timeout, they are asserting their own compute constraints. But for a human visitor, a slow TTFB is the difference between staying on a site and bouncing to a competitor. A fast TTFB impacts everything else downstream.
The faster you can provide a response, the quicker a user can see the content that’s important to them. This is true regardless of whether the user is a human or AI.
2. Lightweight server-rendered HTML
LLMs don’t want to wait around for a complex single-page application (SPA) to mount, and they hate looking at a spinner just as much as you do. This is a massive win for users. The front-end became massively complex in the last decade, and an AI emphasis might be the thing that helps us dial back the complexity.
Large JavaScript bundles cause lots of activity on the main thread. As a result, they have the potential to make an interface feel laggy, consume battery power, heat up mobile devices, and completely fall apart on slow, spotty mobile networks.
Agents also want a smaller DOM because a large DOM bloats the context window, and your browser wants a small DOM because it can construct the render tree more quickly and provide your users with meaningful content sooner.
For years, performance experts have advised making sure the important content is delivered in the form of HTML in the first payload, specifically within the first 14 kB. This recommendation is due to TCP slow start, and your agents are subject to the same TCP limitations as human users. Fixing it for a user fixes it for AI and vice versa.
3. Clear structure and sound accessibility
Claude, ChatGPT, and Gemini rely heavily on semantic HTML, ARIA landmarks, clear table layouts, and descriptive alt text because they cannot “look” at a webpage and intuitively understand layout hierarchy without it.
When you structure your site with semantic markup instead of nesting a dozen <div> elements, you enable users of assistive technology, such as screen readers, to navigate your page with relative ease.
If you’ve ever worked with tools like Playwright or Puppeteer, you know they make heavy use of the accessibility tree and rely on element roles for selectors. This advice not only makes it easier for agents, but also improves the experience for users and makes it easier for engineers to test. My accessibility friends hate when you use <div>s for everything, and so do agents.
4. Eradicating CAPTCHAs and bot gateways
Automated agents cannot solve CAPTCHAs, and these types of bot mitigation strategies cause hard failures for web fetches. If your business depends on LLMs being able to pull and summarize your content, you are highly incentivized to get rid of these interactive challenges.
Do you know who hates CAPTCHAs more than LLMs? Humans. Real, living, breathing people. Moving away from invasive, interactive gatekeeping toward invisible, passive backend security measures is a blessing for every human user who just wants to read an article without identifying traffic lights first.
Universal design wins
For years, performance advocates and accessibility experts have been pleading with organizations to prioritize semantic markup, fast server responses, and lean payloads. Often, these pleas were met with pushback: “Accessibility only affects a small portion of our users,” “our users have fast enough phones to handle our JavaScript,” and my favorite: “it works fine for me, and we really need to get this next set of features out.”
Now, the business case has completely shifted. If your site is invisible to AI agents, it is functionally invisible to the millions of users who rely on these tools to search, summarize, and browse the web. I don’t love the fact that robots are getting more attention than people, but I’ll take the win any way I can get it.
The beauty of universal design is that we don’t need to choose between building for humans or building for machines. By optimizing for AI-web readiness, we aren’t creating a separate, sterile web for robots. We are simply returning the web to its core strengths: semantic clarity, outstanding performance, and universal accessibility.