llms.txt and AI Crawlers: What Actually Reads Your Site
Somebody sent you a checklist. Item four was “add an llms.txt file so AI can find you.” Your developer added it in ten minutes, you waited a month, and nothing happened — no new citations in ChatGPT, no change in AI Overviews, no traffic. You are now wondering whether you did it wrong.
You did not do it wrong. The file does very little for AI search, and the evidence for that is now reasonably strong. But the underlying question — what actually reads my site, and can I control it — is a real and important one, and almost every business gets the answer backwards. The most common failure we see in audits is not a missing llms.txt. It is a site that has quietly blocked the crawlers that send it traffic while leaving wide open the ones that do not.
This guide separates the two. First, what llms.txt is, where it came from, and what the data says about it. Then the crawler landscape that genuinely determines whether you can be cited, the robots.txt rules that follow from it, and the Singapore legal position, which is more permissive towards AI training than most business owners assume. If you want the wider strategic picture rather than the plumbing, start with our guide to AI SEO in Singapore and come back here.
The short answer, from the people who run the crawlers
Google’s own documentation on AI features is unusually blunt. To appear in AI Overviews and AI Mode, “a page must be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements.” Beyond that, Google says: “There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.” And, directly on the point: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.”
That is about as clear as vendor documentation gets. Google’s Gary Illyes has separately confirmed the company does not support llms.txt and has no plans to, and John Mueller has compared it to the long-dead keywords meta tag — a file publishers fill in about themselves, which is precisely why search engines learned to ignore that class of signal.
None of this makes llms.txt harmful. It costs nothing, it breaks nothing, and there is a narrow case where it genuinely helps, covered below. It is simply not the lever, and treating it as one displaces work that would actually move the number.
What llms.txt actually is, and what it was designed for
llms.txt was proposed on 3 September 2024 by Jeremy Howard of Answer.AI, published on answer.ai and llmstxt.org. The format is deliberately minimal: a Markdown file at your site root, at /llms.txt, containing a curated index of your most important pages with a one-line description of each. An optional companion, llms-full.txt, inlines the actual Markdown text of those pages so a model can pull your whole documentation set into a single context window.
The problem Howard described had nothing to do with search rankings. It was that language model context windows are too small to hold an entire website, and that converting production HTML — navigation, cookie banners, advertising, client-side JavaScript — into clean text is lossy and error-prone. The use case he had in mind was a developer asking a coding assistant a question about an API, and the assistant needing to find the right reference page fast without burning tokens on chrome.
Read that back and the mismatch is obvious. A proposal aimed at making technical documentation cheap for coding agents to read got picked up by the SEO industry and resold as a citation mechanism. It was never that. Adoption followed the original intent: Answer.AI itself, Anthropic’s documentation, Vercel and Stripe are among the recognisable implementers, and they are all developer-facing.
The evidence that AI search engines are not reading it
Three independent lines of evidence point the same way.
Adoption without effect. SE Ranking’s study of 300,000 domains found roughly 10.13% carrying an llms.txt file, and — the more interesting finding — their predictive model of AI visibility got more accurate when the llms.txt variable was removed. It was adding noise, not signal. Search Engine Land’s smaller follow-up reported that eight of nine sites saw no measurable traffic change after implementation.
Nobody is fetching it. Crawler-log monitoring across more than 500 million AI bot visits over a 90-day window found only 408 requests that targeted /llms.txt directly. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Google-Extended overwhelmingly skip the file and crawl HTML like every other crawler. This is the single most damning number in the debate, because it is a direct observation rather than a correlation.
No operator has committed to it. As of Q1 2026, no major AI company — OpenAI, Google, Anthropic, Meta or Mistral — has publicly committed to reading or acting on llms.txt in production systems. John Mueller said in mid-2025 that no AI system he was aware of used it, and Google’s documentation quoted above still says you do not need AI text files.
The honest caveat: this is evidence of current behaviour, not proof of permanent uselessness. Standards sometimes get adopted late. But you should size the bet accordingly — ten minutes of a developer’s time, once, with no expectation attached.
Where llms.txt genuinely earns its keep
There is a real use case, and it is the original one. AI coding assistants — Cursor, GitHub Copilot, Windsurf, Cline, Aider — retrieve documentation live while a developer works. Anthropic recommends the format in its guidance on writing for agents, and OpenAI uses it for the Agents SDK docs. In that context the file does exactly what it was designed to do: it saves tokens and points the assistant at the right page.
So the test is simple. Do developers integrate with your product? If you sell an API, ship an SDK, or maintain technical documentation that engineers read while building something, add llms.txt — it will make your docs cheaper and more accurate to consume, and that is worth having. If you run a clinic in Novena, a renovation firm, an F&B group or a professional services practice, the file will sit there being fetched by almost nobody. Add it if it makes you feel tidy. Do not expect it to do anything.
The split that actually matters: three kinds of AI crawler
Here is the distinction that decides whether you can be cited, and the one most “block the AI bots” advice gets catastrophically wrong. AI crawlers are not one thing. They fall into three functional groups, and they have opposite consequences for your traffic.
- Training crawlers collect content to train or fine-tune models. They send you nothing back, ever. There is no referral mechanism because there is no product surface with links in it.
- Retrieval or search crawlers index your pages so they can be surfaced and cited in live answers. These are the ones that generate referral traffic. Block them and you remove yourself from the answers you are trying to appear in.
- User-action fetchers retrieve a page because a human asked the assistant to look at it, right now. They also produce visits, and some of them explicitly may not follow robots.txt, because the operator treats a user-initiated fetch as closer to a browser request than a crawl.
The table below is built entirely from the operators’ own published documentation, not from third-party lists.
| User agent | Operator | Stated purpose | Sends referral traffic? |
|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used in training generative AI foundation models | No |
| OAI-SearchBot | OpenAI | Surfaces websites in search results in ChatGPT’s search features | Yes |
| ChatGPT-User | OpenAI | Handles certain user actions in ChatGPT and custom GPTs | Yes (may not follow robots.txt) |
| OAI-AdsBot | OpenAI | Validates the safety of web pages submitted as ads on ChatGPT | n/a |
| ClaudeBot | Anthropic | Collects web content that could contribute to model training | No |
| Claude-SearchBot | Anthropic | Analyses content to improve relevance and accuracy of search responses | Yes |
| Claude-User | Anthropic | Accesses sites when a Claude user asks a question | Yes |
| Googlebot | Search indexing — the same index that feeds AI Overviews and AI Mode | Yes | |
| Google-Extended | Controls whether crawled content trains future Gemini models and grounds Gemini apps | No | |
| PerplexityBot | Perplexity | Indexes pages for answers | Yes |
| Perplexity-User | Perplexity | User-initiated page fetch | Yes |
Two rows deserve emphasis. Google-Extended is not a Google Search control. Google’s documentation states plainly that it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” You can block it, refuse to feed Gemini training, and keep every bit of your Search and AI Overviews visibility. That is the cleanest trade available to a Singapore business that dislikes the training bargain.
And there is no AI-Overviews-only opt-out. AI Overviews draw on the ordinary Search index, so the controls Google offers — nosnippet, data-nosnippet, max-snippet, noindex — restrict what appears in Search generally. Using them to stay out of AI Overviews also shrinks or removes your normal blue-link snippet. For nearly every business that is a bad trade, but if you want the mechanics of ranking in these surfaces instead, we cover them in how to show up in AI Overviews.
Crawl-to-refer: what the exchange is actually worth
The fairest way to judge an AI crawler is the ratio between how many pages it takes and how many visitors it sends back. Cloudflare Radar publishes the data, and it is uncomfortable reading for publishers.
For a rolling 28-day window ending 21 July 2026, the reported ratios were roughly: Mistral 3,389 pages crawled per referral, Anthropic 2,237:1, Perplexity 225:1, OpenAI 217:1, Microsoft 35:1, Google 4.6:1 and DuckDuckGo 2.5:1.
| Operator | Crawl-to-refer, 28 days to 21 Jul 2026 | What that reflects |
|---|---|---|
| Mistral | ~3,389 : 1 | Primarily training; almost no linking surface |
| Anthropic | ~2,237 : 1 | Improving fast from a far worse base |
| Perplexity | ~225 : 1 | Answer engine that does cite and link |
| OpenAI | ~217 : 1 | ChatGPT search sends real, measurable traffic |
| Microsoft | ~35 : 1 | Copilot sits on top of a search index |
| ~4.6 : 1 | A search engine first, still by far the best exchange | |
| DuckDuckGo | ~2.5 : 1 | Near parity — what a pure search crawler looks like |
Two caveats, and they matter. First, these numbers move violently with the measurement window. Earlier 2026 readings put Anthropic near 24,000:1 and OpenAI above 1,200:1. Quoting any single figure as “the” ratio is misleading; quote the window or do not quote it. Second, referral attribution undercounts. A large share of AI-driven visits arrive with no referrer header and land in Direct, so the true return is better than the ratio implies — though not by an order of magnitude.
The strategic reading is the same either way. Crawl purpose determines the ratio. Search and user-action crawlers have structurally decent numbers because referrals are part of how the product works. Training crawlers have no mechanism to send anything back and never will. That is your decision framework in one sentence.
The robots.txt to actually write
Most Singapore SME sites we audit have one of two problems: an empty robots.txt that expresses no preference at all, or a copy-pasted “block the AI bots” block that took out the retrieval crawlers along with the training ones. Here is a defensible default for a business that wants AI citations but does not want to donate to model training.
# Search and AI answer engines - allowed
User-agent: Googlebot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
# Model training - declined
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Sitemap: https://example.com.sg/sitemap_index.xmlThree notes on using it. Replace the sitemap line with your own. Keep the training block deliberate rather than reflexive — if you would happily have a model learn from your content, delete that half and move on. And test the file after publishing: Search Console’s robots.txt report will tell you whether Googlebot reads it the way you intended, which matters because a stray directive at the top of the file can change how the rest is parsed. Our technical SEO basics guide covers the parsing rules in more detail.
Content Signals, and whether any of this is enforceable
robots.txt expresses a preference. It has never been a fence. Cloudflare’s Content Signals Policy, launched on 24 September 2025 and rolled out across more than 3.8 million domains using its managed robots.txt, tries to make the preference more precise by splitting usage into three declared signals:
- search — may be used to build a traditional search index with links and short excerpts, excluding AI-generated summaries
- ai-input — may be used as real-time input to a generative answer, i.e. retrieval-augmented generation
- ai-train — may be used to train or fine-tune a model
The syntax is a human-readable line such as Content-Signal: search=yes, ai-input=yes, ai-train=no, placed alongside your existing directives. “Yes” means allowed, “no” means not allowed, and the absence of a signal means no preference has been expressed — not consent, and not refusal.
It is worth adding. It is not a standard. It has not been ratified, the IETF’s AI Preferences (AIPREF) working group is still doing the standardisation work, and no crawler operator is obliged to honour it.
How much compliance can you assume? The honest answer is: some, from the reputable operators, and it is contested at the edges. In August 2025 Cloudflare published research alleging that Perplexity was using undeclared crawlers, rotating user agents and source ASNs to reach content on sites that had blocked its declared bots, and de-listed Perplexity as a verified bot in response. Perplexity denied the allegation, characterised the report as a publicity stunt, said the bot named was not theirs and argued the evidence showed no content actually being accessed. The dispute was never definitively resolved in public, which is itself the lesson: if you need enforcement rather than a request, that is a WAF and bot-management problem, not a text file.
The Singapore layer: the law is more permissive than you think
This part surprises most Singapore business owners, and it changes what your terms of use are worth.
Singapore’s Copyright Act 2021, in force since 21 November 2021, contains a computational data analysis exception at sections 243 and 244. It permits copying a work for computational data analysis — expressly including text and data mining and machine learning — without the copyright owner’s permission, subject to conditions. The main ones: the copy must be made for that purpose and not another; it must not be supplied onward except to verify the results or for collaborative research; and the user must have had lawful access to the source material in the first place.
The provision that catches people out is section 187. It makes the exception non-excludable by contract: a term in your website’s terms of use purporting to prohibit computational data analysis is void and unenforceable to that extent. So the “no AI training permitted” clause your lawyer added to the footer does not, on its own, create a right you can enforce in Singapore against someone who lawfully accessed the page.
Three qualifications worth stating plainly. Lawful access still binds — circumventing a paywall or a login is not covered, which is why gating genuinely valuable material behind authentication remains the one control that has teeth. The exception is under review: the Singapore Academy of Law published a reform paper on the computational data analysis exception in 2025, so treat the current position as the live one rather than the settled one. And this is general commentary on a published statute, not legal advice — if the training question is commercially material to you, take proper advice on your specific content.
The practical consequence for a Singapore SME: robots.txt and Content Signals are how you express a preference, authentication is how you enforce one, and your terms of use are doing less work than you think.
A 45-minute audit you can run this week
You do not need a tool for this. You need server logs and one cup of coffee.
| # | Step | What good looks like | Common failure |
|---|---|---|---|
| 1 | Export 30 days of server or CDN logs; filter by the user-agent tokens in the table above | You can see which crawlers visit and how often | Logs only retained 7 days, or bot traffic stripped before logging |
| 2 | Count requests per crawler; compare against AI referral sessions in GA4 | Your own crawl-to-refer figure, by operator | Comparing against total sessions instead of AI-attributed ones |
| 3 | Fetch your own /robots.txt in a browser and read every line | Retrieval crawlers explicitly allowed; training decision deliberate | A legacy Disallow: / from a staging site, or a wildcard AI block |
| 4 | Check CDN and WAF bot rules separately from robots.txt | The one-click “block AI bots” toggle is understood before it is used | Cloudflare or similar silently blocking OAI-SearchBot at the edge |
| 5 | Fetch three money pages with JavaScript disabled | Headline, body copy and key facts present in the raw HTML | Client-side rendered content invisible to every crawler in the table |
| 6 | Write down your training decision and the date | An explicit, reviewable position | Nobody remembers who added the block or why |
Step 5 is the one that most often produces a real finding. A retrieval crawler that cannot see your content in the server-rendered HTML cannot cite it, and no file at your site root will change that. If that audit turns up a rendering problem, the fix belongs in the same workstream as your on-page SEO checklist rather than in an AI-specific project.
What if you genuinely want to block AI training?
It is a legitimate position, particularly for publishers, course creators and consultancies whose written output is the product. Go in with clear eyes about what blocking buys you.
What it costs: essentially nothing in referral traffic, because training crawlers do not send any. This is the rare case where the cautious option is close to free.
What it does not achieve: it is partial and retrospective at best. Content already absorbed into existing training sets does not come back. Crawlers that ignore robots.txt are unaffected. Third parties who quote or summarise you elsewhere are unaffected. And under Singapore’s section 243/244 exception, lawful crawling for computational data analysis is permitted regardless of what your terms of use say.
The realistic posture for most Singapore businesses is the one in the robots.txt above: allow the retrieval crawlers so you can be cited, decline the training crawlers because it costs you nothing, add the Content Signals line, and put anything genuinely proprietary behind a login. Then spend the time you saved on the work that actually drives citations — which, per Google’s own documentation, is the same work that drives rankings. We set out what that looks like in SEO for AI search and in building an AI SEO strategy.
Frequently asked questions
Should I add llms.txt to my Singapore business website?
Only if developers consume your documentation. If you sell an API, ship an SDK or maintain technical docs, add it — coding assistants genuinely use it. For a clinic, a retailer, a renovation firm or a professional services practice, it will be fetched by almost nobody and will not affect AI Overviews or chatbot citations. It costs ten minutes and does no harm, so add it if you want the box ticked, but do not let it displace the content and technical work that does move the number.
If I block GPTBot, do I disappear from ChatGPT?
No. GPTBot collects content for training OpenAI’s foundation models and has no mechanism to send you traffic. The crawler that decides whether you can be surfaced in ChatGPT’s search features is OAI-SearchBot, and user-initiated fetches come from ChatGPT-User. Block GPTBot and leave those two allowed and you keep your ChatGPT visibility while declining the training bargain.
Can I appear in Google Search but stay out of AI Overviews?
Not cleanly. AI Overviews and AI Mode draw on the ordinary Search index, and the controls Google offers — nosnippet, data-nosnippet, max-snippet and noindex — limit what is shown from your pages in Search as a whole. Using them to suppress AI Overviews also suppresses or truncates your normal search snippet. Google-Extended is different: it governs Gemini training and grounding only, and Google states it does not affect inclusion or ranking in Search.
Does robots.txt legally stop an AI company from using my content in Singapore?
robots.txt is a request, not a legal instrument. Under Singapore’s Copyright Act 2021, sections 243 and 244 permit copying for computational data analysis, including machine learning, where the user had lawful access and meets the other conditions — and section 187 makes that exception non-excludable by contract, so terms of use purporting to forbid it are void to that extent. Authentication and paywalls are the controls with real force, because they remove lawful access. This is general commentary, not legal advice.
Is llms-full.txt worth creating?
Only in the documentation case, and even there, weigh the cost. llms-full.txt inlines the entire Markdown text of your listed pages so a model can load everything in one context window — useful for a developer asking a coding assistant about your API, pointless for a marketing site, and a maintenance burden that goes stale quickly. If your llms.txt is not being fetched, its bigger sibling will not be either.
How do I find out which AI crawlers are actually visiting my site?
Filter 30 days of server or CDN access logs by user-agent string using the tokens in the table above, then count requests per crawler. Cloudflare, Fastly and most managed hosts expose this in their bot analytics without any log wrangling. Compare the crawl counts against AI-attributed sessions in GA4 to get your own crawl-to-refer ratio, which is far more useful than any published average because it reflects your content and your market.
Where this leaves you
llms.txt is a good idea aimed at a real problem that turned out not to be the SEO problem. The people selling it as an AI-visibility lever are, at best, working from a hopeful reading of a proposal that its own author framed as a documentation convenience. Add it if developers read your docs. Otherwise, spend the hour on the audit above.
What genuinely determines whether an AI answer engine can cite you is duller and more durable: your pages are indexed, they render server-side, the retrieval crawlers are allowed through at both robots.txt and CDN level, and the content is worth quoting. That is the same list that determined whether you ranked in 2019, which is exactly what Google’s documentation says and exactly why nobody can sell it as a separate discipline with separate fees.
If you want that checked properly on your own site — logs read, crawler access verified end to end, rendering tested, and a training position you can actually defend to a board — that is the kind of work our AI SEO service in Singapore covers. You can see the sort of results that follow from getting the fundamentals right in our Singapore case studies, or read the wider cluster starting with GEO vs AEO vs LLM SEO if the terminology is still doing your head in.
