Is AI replacing SEO?

Micah Boswell · Lead Experience Design, AT&T · designing for AI · September 2026

No. I log every request to four sites I run. Over three days they took 5,073 of them. People made 2,593. AI agents made 977. Search crawlers made 853. Every one of those AI agents came in through robots.txt and sitemap.xml, the same two doors Googlebot has used since the nineties. Nothing was replaced. A second reader showed up and let itself in through the door that was already open.

I put server-side logging at the edge of those sites because I was tired of reading other people's opinions about AI traffic. The middleware classifies the user agent on every request and writes a row. No sampling. No JavaScript, which matters, because the readers in question never run any.

Here is who came through between September 12 and September 15.

The AI agents outnumbered the search engines. That is the headline everyone wants, and taken alone it is misleading, because it says nothing about how they got in.

Who is actually crawling your site?

Google is fourth. Yandex, which I have never optimized for in my life, outcrawled Google five to one. ClaudeBot outcrawled Googlebot three to one.

That is a change in volume. It is not a change in mechanism.

Every agent on that list found the pages the same way: robots.txt first, then sitemap.xml, then the HTML. A site Googlebot cannot index is a site ClaudeBot cannot read, for identical reasons. Broken canonicals, pages that only exist after JavaScript runs, a sitemap that lies about what is there. The 2015 checklist did not expire. It got a bigger audience.

Does llms.txt do anything?

This is the part of the field that is loudest right now, so I checked it against the same logs. Requests to each machine-readable file, four sites, three days:

I wrote the llms.txt on this site. I stand behind every line of it. It is eight requests. The two files carrying the entire load are a convention from 1994 and a standard from 2005.

Write the llms.txt anyway. It costs an afternoon and it is the correct place to put a plain-language summary of who you are and what you do. Just file it under cheap insurance, not under channel. If you are choosing between a clean sitemap and a beautiful llms.txt, the numbers are not close.

What does AEO actually change?

Answer Engine Optimization changes the scoreboard, not the sport. The old question was where you rank. The new question is whether you get cited in an answer the person never leaves to verify.

I measure that one directly too. A scheduled job puts forty real questions per site to Claude and to Grok with web search turned on, then records every URL each engine cited. Same questions every run, so the number means something over time.

Six out of 320. I keep that number where I can see it, because it is the honest baseline that every vendor deck about AI visibility is priced against.

What got cited is the more useful finding. Somebody asked which to buy, a Lamy 2000 or a Pilot Custom 823. The engine answered with sixteen sources, and one of them was a page from my fountain pen almanac: the comparison view for those two pens, with the measured numbers side by side.

A comparison page. Structured, specific, one question per URL, no preamble before the answer. That is a 2012 SEO artifact. The reader is new. What the reader rewarded is not.

So what did we build?

A third reader is arriving, and this one does not read. It calls.

A crawler takes your page and hopes the number it needs survived the HTML. An agent holding an MCP endpoint asks your site the question and gets the number, typed, with the source URL attached. Model Context Protocol is the plumbing for that, and it turns a website from a document into a set of functions.

So we published three of the sites as servers.

One rule held across all three. The server reads the same data the pages render, so the two can never disagree. When a race finishes, the page and the tool change in the same deploy. An assistant that quotes the standings is quoting the site, not a copy of the site that stopped being true in July.

The second rule was smaller and mattered more. Every tool returns the page URL along with the data, so an agent that uses the numbers can send the person back to where the numbers live.

Is the MCP layer working?

Not yet, and I will say so plainly rather than let the deploy stand in for a result.

The endpoints took 34 requests in three days, all from agents that send no identifying user agent, which means I cannot tell a curious developer from an assistant doing somebody's homework. Against 5,073 requests to the pages, that is a rounding error.

The reason is discovery. There is no robots.txt for MCP and no sitemap standard for it. A registry entry and a link in your docs is the whole surface, and mostly an endpoint gets found because a person points an assistant at it. The crawl side had twenty years to standardize that problem. The call side is two years old.

It is a bet. The difference between a bet and a result is whether you say which one you are holding.

Three readers now. A search crawler that indexes you, an answer engine that cites you, and an agent that calls you. They arrived in that order, and not one of them left when the next one showed up. The work compounds instead of cycling: the structured page that ranked in 2015 is the page that gets cited in 2026, and the clean data behind it is what the tool returns.

Anyone selling you a replacement for SEO is selling you the part that was already free.

Every one of them still starts at robots.txt.

Four questions, answered short

Is SEO dead because of AI?

No. Across four sites over three days, AI agents made 977 requests and search crawlers made 853, and every one of those agents arrived through robots.txt and sitemap.xml before reading a page. AI traffic is growing faster, on the same plumbing. A site Googlebot cannot index is a site ClaudeBot cannot read.

What is AEO and how is it different from SEO?

Answer Engine Optimization is the same work judged by a different scoreboard: cited rather than ranked. In a measured run of 320 questions put to two answer engines with web search on, six answers cited one of my sites, and the page that earned the clearest citation was a structured comparison page, which is an ordinary SEO artifact.

Is llms.txt worth writing?

Write one, it is cheap. Do not expect a channel. Over those same three days robots.txt took 415 requests and sitemap.xml took 305, while llms.txt took 8 and agents.md took 1.

Should I turn my site into an MCP server?

Only if the site holds structured data an assistant would otherwise scrape out of your HTML. The endpoint should read the same source the pages render, so the two can never disagree, and every tool should return the page URL with the data. Discovery is the open problem: MCP has no sitemap standard, and an endpoint mostly gets found because a person points an assistant at it.

← back to conscious_shell