AI SEO and GEO: What a Website Actually Needs

AI SEO is not the checklist most advice describes: an llms.txt file, some extra schema, a wait for citations. It’s something lower-level and less exciting: whether a crawler can actually reach your page, and whether anything is still there once it does. Once you see those two gates, the checklist stops mattering.

Direct Answer

AI SEO is the practice of making a website legible to the crawlers that feed large language models, and GEO is what happens after that: earning the citation once a model can actually read the page. Together they come down to three things: whether the crawler can fetch the page at all, whether the page still has text once JavaScript is stripped out, and whether its claims are structured and corroborated enough to quote with confidence. An llms.txt file is not one of them.

Key Takeaways

• AI SEO and GEO are not about ranking in ten blue links: they are about whether a language model can fetch, read, and trust a claim on your page at all.
• An llms.txt file does not move this. It is a proposed convention, not something any major AI crawler is documented to read or act on.
• Google’s own developer guidance says there is no special schema you need to add for AI Overviews or AI Mode: ordinary structured data still helps, for the reasons it always did.
• If your page renders its content with client-side JavaScript, some AI crawlers may never see that content, because they work from the fetched HTML rather than rendering scripts the way a browser or Googlebot does.
• The fix is boring: keep crawlers allowed in, serve real text in the initial response, mark up entities, and back claims with sources a model can check.

What AI SEO and GEO actually mean

“AI SEO” and “GEO” get used interchangeably, and for a website owner the distinction rarely matters day to day. Both describe the same shift: search is no longer just a list of links a person clicks through. A growing share of queries get answered inside a chat window, an AI Overview, or an agent’s own research step, and none of that requires a click to your site at all. An earlier piece on this site covers the citation-versus-ranking distinction in more depth. What it doesn’t cover, and what actually decides whether any of this works for a given page, is the plumbing underneath.

Two systems have to succeed before a citation is possible. First, a crawler has to be allowed onto the page and has to be able to extract something from it. Second, whatever it extracts has to be confident enough, structured enough, and corroborated enough elsewhere that a model is willing to attribute a claim to your domain instead of a competitor’s. Most public advice about AI SEO addresses neither of these. It addresses a third thing (a file, a markup snippet) that doesn’t determine either outcome. And the checklist fails in the same three places every time: a crawler that’s blocked, a page that’s empty until JavaScript runs, and a claim with no structure or corroboration behind it.

Does an AI crawler even reach your page?

This is the first gate, and it’s the one operators skip past because it sounds like a solved problem. It isn’t, for two reasons.

The first is that the volume of AI-related crawling has grown enough that plenty of infrastructure teams now default to blocking it. Cloudflare reported that automated, non-human requests crossed 50% of the web traffic it measures during 2026, a threshold it had not expected to reach for another year. A meaningful share of that is AI crawlers and AI agents, and the instinct at a lot of companies has been to treat that traffic the way you’d treat a scraper: rate-limit it, block it at the CDN, or leave a restrictive robots.txt rule in place from a security review that had nothing to do with search. Any of those, aimed at GPTBot, ClaudeBot, or PerplexityBot specifically, makes the rest of this article irrelevant for your site. A page that can’t be fetched can’t be cited.

The second reason is less about policy and more about not knowing your own defaults. Plenty of CDN and WAF configurations ship with “AI bot” blocking enabled, or bundled into a generic bot-management ruleset a team turned on for an unrelated reason. If citation and AI-referred traffic are goals, that default needs a deliberate decision, not an accident. Check what your edge network is doing with GPTBot, ClaudeBot, Google-Extended, and PerplexityBot specifically: “we block bad bots” is not an answer to that question.

The part almost nobody checks: does the page still have text?

Assume the crawler is let in. The next question is what it actually receives, and this is where a page that ranks perfectly well in Google can be functionally invisible to an AI answer engine.

Search engines and AI crawlers do not all handle JavaScript the same way. Googlebot (and by extension Google’s AI Overviews, via the Google-Extended crawler that inherits Googlebot’s rendering pipeline) executes client-side JavaScript before indexing a page. That’s been true for years, and it’s why a modern single-page app can still rank in Google. A large-scale analysis of AI crawler behavior by Vercel and MERJ, built on real fetch logs rather than a lab test, found the opposite pattern for the crawlers behind the major chat assistants: GPTBot and ClaudeBot work from the initial HTML response, with no evidence of executing the JavaScript on the page, including on requests where they did download a page’s script files.

The practical consequence is specific. Take a storefront built as a client-rendered app (platform choice matters here: see best AI website builders for small business ecommerce for how that plays out across specific tools): the product price comes from an API call that fires after the page loads, and a “use client” component paints it into the DOM once that call resolves. A browser sees the price. Googlebot sees the price, because it waits for the render. GPTBot and ClaudeBot, on this evidence, see an empty price field, because nothing in the initial HTML response ever contained it: the same gap applies to a spec table, an FAQ answer, or the paragraph that actually explains your product. It sees the shell. (The same logic touches plain page speed, for human visitors and crawl budget alike: see how to speed up a small business website for that side of it.) Your on-page SEO score can be excellent and your GEO exposure can still be close to zero, for a reason no amount of schema markup fixes.

Structured data is not irrelevant here, but it isn’t the lever most vendors sell it as either. Google’s own developer documentation on optimizing for AI features states plainly that there is no special or new structured data required for AI Overviews or AI Mode: the same Organization, Article, and FAQPage markup that already helps disambiguate a page for ordinary search does the same job here, for the same reason: it removes ambiguity about what a piece of content is and who published it. Still worth doing. Just not the sentence.

That table is the actual AI SEO checklist. Notice that none of its rows are a file you download once and forget.

Summary

None of this is a new discipline bolted onto SEO. It’s the same discipline (crawlable, fast, unambiguous pages) pointed at a newer and, by Cloudflare’s numbers, now-larger audience than the human one. The sites that already treated crawl access and server-rendered content as non-negotiable, for ordinary search reasons, are the same ones an AI crawler can already read without anyone doing anything new for it. The gap belongs to sites that outsourced their content to the client and assumed a good performance score meant a machine could read them. It doesn’t, and no llms.txt file closes that gap for you.