Does llms.txt Actually Matter? Evidence, Use Cases, and SEO Impact

Brief verdict

For most publishers, e-commerce websites, and content projects, llms.txt is currently more of a nice-to-have experiment than a priority ahead of technical accessibility, robots.txt, indexability, and genuinely citable content.

• As an AI search / GEO ranking signal, it is still supported by very little evidence.
• The strongest evidence currently supports the conclusion that the hype is greater than the demonstrated production use for third-party website crawling.

When it makes sense

• API documentation and developer docs.
• Product knowledge bases that should be easy for an agent to consume.
• Situations where implementation is inexpensive and maintenance is close to zero.

When it is not a priority

• When the website is struggling with basic indexability.
• When important content is not easy to read in HTML.
• When you expect it to increase AI citations or restore traffic.

1. What llms.txt actually is

According to the specification at llmstxt.org, llms.txt is a simple text or Markdown hub linking to important parts of a website. Its purpose is to make content easier for LLMs to process, especially where HTML is complex, disrupted by navigation, or burdened with JavaScript.

The specification itself, however, does not promise a ranking boost or a higher position in AI Overviews, an automatic increase in citations, or preferential crawling by major LLM providers. At the design level, it is therefore more of a data-delivery extension than a real visibility signal.

2. Three questions that are often confused

The debate around llms.txt is often confused because it jumps between three different levels of evidence.

1. The file is published. The domain has a /llms.txt or /llms-full.txt file. This is the lowest bar.
2. The file is downloaded. A specific crawler or agent actually visits this path. That is a stronger signal, but it still may mean nothing.
3. The file is actually used. AI systems use it to decide what they find, what information they download, what they select as a source, or how the entire content-processing workflow proceeds. This is the highest bar, and it is precisely where the least evidence exists today.

The public internet is full of articles that jump straight from the first level to the third. But the fact that someone publishes the file does not yet imply that major AI crawlers read it regularly. And the fact that someone occasionally downloads it does not yet imply a measurable impact on citations or traffic.

3. What Google, OpenAI, Anthropic, and Perplexity say

Google: explicitly says it is not needed

In its guide to generative AI Search, Google Search Central states that standard SEO remains essential for AI Overviews and AI Mode: crawlability, indexability, high-quality content, and eligibility to appear in Search with a snippet. In the myth-busting section, it also explicitly says that there is no need to create llms.txt, special AI markup, or Markdown files for visibility (Google AI optimization guide).

Practical conclusion: for Google Search today, llms.txt is not a recommended layer of the stack. This is not an absolute claim that Google never sees the file anywhere, but for website decision-making it is a strong argument against prioritizing it.

OpenAI: yes for docs, not for the publisher workflow

OpenAI publicly hosts developers.openai.com/llms.txt. This is good evidence that the format has value for documentation and agentic use. It is not, however, evidence that OpenAI uses llms.txt as a general signal for third-party websites.

When OpenAI explains to publishers how to appear in ChatGPT search, its guidance is built around OAI-SearchBot, robots.txt, and optionally noindex, not llms.txt (OpenAI Publishers and Developers FAQ, OpenAI crawler docs).

Anthropic and Perplexity: the same pattern

Anthropic has a section in Claude Docs for AI ingestion that lists llms.txt and llms-full.txt (Claude Docs resources). At the same time, its public crawler documentation is based on specific bots and blocking through robots.txt (Anthropic crawler guidance).

Perplexity similarly hosts its own docs llms.txt, but its crawler documentation for website visibility covers PerplexityBot, Perplexity-User, robots.txt, and IP addresses (Perplexity Crawlers).

Interpretation

A provider can use llms.txt on its own documentation website while not using it as the basis of its public crawler-control or search-visibility workflow for third-party websites. This is currently the most important nuance in the entire debate.

4. What server log data shows

Server log data is not a perfect picture of the entire internet, but it is a very useful reality check against marketing claims.

The WISLR study monitored bot traffic for 48 days and recorded more than 12,000 bot requests. According to the authors, none of the monitored AI bots requested /llms.txt or /llm.txt; the only recorded access came from an analytics service, not a major AI platform (WISLR log analysis).

Seekio analyzed approximately 900 domains over 191 days. It found only 1,227 requests for llms.txt, llms-full.txt, and related paths, while recording almost 45 million requests from AI bots overall during the same period (Seekio log study).

I have data from many websites, and it confirms these findings. For example, on my blog, the /llms.txt file was downloaded a total of nine times over the past year by Dataprovider.com and once by AI-Security-Scanner—and that was all. No Google, OpenAI, Anthropic, etc.

For amusement, I will also add an example from a larger website… llms.txt is most often downloaded by bots that check whether it is available.

That is the key contrast: AI bots really do crawl websites, but the available log evidence does not currently show that they typically do so through llms.txt. A secondary synthesis by Ahrefs describes the same situation as a proposed standard whose hype exceeds its confirmed adoption (Ahrefs).

What follows from this

The absence of file downloads is not absolute proof that nobody ever uses the file. It is, however, a strong practical signal that it is not a primary production discoverability layer for an ordinary website. And even if someone occasionally downloads the file, the download itself does not imply a citation uplift, referral uplift, or ranking signal.

5. Practical recommendation

For most websites, the following order of priorities is reasonable:

1. First address robots.txt, indexability, clean HTML, server-side content availability, and preview controls.
2. Then address content quality, entity clarity, information architecture, and citation usability.
3. Only then consider adding llms.txt as a low-cost experiment.

The best way to put it is this: llms.txt may help when you want to provide an LLM or agent with an inexpensive map of your documentation. But that is a different problem from appearing in an AI search answer, and a different problem again from controlling crawler access.

Create it when

• you have a strong documentation layer, API, or product knowledge base
• you can generate Markdown versions without manual maintenance
• you want to be more agent-friendly and support internal tooling
• you are bored

Do not prioritize it when

• the website has technical SEO or indexing debt
• the content is not easy to read in basic HTML
• the expectation is a direct increase in AI citations, rankings, or referral traffic

FAQ

Is llms.txt the new robots.txt for AI?

No. robots.txt remains the primary publicly documented crawler-governance layer for Google, OpenAI, Anthropic, and Perplexity. llms.txt is more of a content map for easier ingestion.

Can llms.txt increase my citations in ChatGPT or Perplexity?

It is theoretically possible in narrow use cases, but today there is no good public evidence for it. When measuring AI visibility, it is more important to track citations, referrals, bot traffic, and query clusters than to assume an effect from the file’s mere existence.

If OpenAI, Anthropic, and Perplexity have one, should I have one too?

Perhaps, if you have a similar documentation use case. But the fact that an AI platform publishes its own documentation in an LLM-friendly format does not mean its crawler uses the same mechanism as its main way of working with third-party websites.

How will I know when the situation has changed?

A strong signal would be Google, OpenAI, Anthropic, or Perplexity beginning to recommend llms.txt for third-party discoverability; independent log studies showing regular fetching by major crawlers; or a credible experiment demonstrating a measurable citation or referral uplift.

What should I do when someone tries to sell me “GEO” and claims that llms.txt is essential?

Please refer them to this article.

Main sources

• llmstxt.org specification
• Google AI optimization guide
• OpenAI Publishers and Developers FAQ
• OpenAI developers llms.txt
• Claude Docs resources
• Perplexity crawler documentation
• WISLR AI bot behavior log analysis
• Seekio: AI bots ignore llms.txt
• Ahrefs: What is llms.txt

Update May 20, 2026

So that we do not get bored: on May 15, 2026, Google stated in its “GEO” guide that llms.txt has no value from an SEO perspective.

Only to add it on May 20, 2026, at Google I/O 2026 among the things being tested in Google Chrome Lighthouse. Here I would focus on the wording: “llms.txt: Checks for the presence of a machine-readable summary at the domain root.” They are testing whether the file is present, but this says nothing about whether they actually use it for anything. I would probably see this as the position of two groups of people. The first group consists of people from Google Search… for them, this file has no value. And based on my conversations with search-engine developers, until a signal is reliable, it is ignored completely. My view is that llms.txt is usually full of nonsense, so it is ignored. The second group consists of the creators of AI agents and WebMCP, who check whether this file exists and may have some agents that use it—perhaps not Google’s agents, but agents made by other people, for whom this check is intended.

Update: Martin Splitt from Google explained at the Google Search Central Prague event that they added it to Chrome Lighthouse because people are interested in it and talking about it—not because anyone at Google uses it. 😉 .