HomeBlogStructured ContextDoes llms.txt Matter in 2026?

    Does llms.txt Matter in 2026?

    6 min

    🤖Summarize with AI

    Get instant summaries and insights from this article using AI tools

    Opens with: Summarise this article for me: "Does llms.txt Matter in 2026?" https://cite-met.com/blog/post/what-is-llms-txt

    Google ignores llms.txt and 97% of the files Ahrefs studied got no requests in May 2026. See when one still helps and how to keep it current.

    The short answer

    Mostly, no. Google says Search ignores llms.txt files, and no major AI search engine documents reading them. Ahrefs studied 137,210 domains and found that 97% of the llms.txt files on them received zero requests in May 2026. A file is still worth publishing if you run developer documentation that coding tools read. (Sources checked 15 October 2026.)

    | Question | Answer | Source | |:---|:---|:---| | Does Google Search use llms.txt? | No. It neither helps nor harms your rankings. | [Google's AI optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), 10 Jul 2026 | | Do you need special files to appear in Google's AI features? | No. There is no extra file or markup to add. | [Google: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features), 10 Dec 2025 | | How many llms.txt files got any request in May 2026? | 3%. The other 97% got none. | [Ahrefs study](https://ahrefs.com/blog/llmstxt-study/), 15 Jun 2026 | | Do AI bots go looking for the file? | No. The study found zero AI bot requests for llms.txt files that don't exist. | [Ahrefs study](https://ahrefs.com/blog/llmstxt-study/) |

    That doesn't make the file useless. It means it is the wrong thing to spend your week on if you want to be cited in ChatGPT, Perplexity, Claude or Google's AI answers.

    What llms.txt is

    llms.txt is a Markdown file placed at the root of a site, at /llms.txt. It was proposed in 2024 by Jeremy Howard, co-founder of Answer.AI and fast.ai, and the current version of the [proposal](https://llmstxt.org) lives at llmstxt.org. The idea: give an AI agent a short, curated map of your site, so it can find the right page without crawling everything.

    The format is plain. The only required part is an H1 with the name of your site or project. After that you can add a one-paragraph summary in a blockquote, then sections of links, each with a short note about what is behind the link.

    ``` # Acme API > Acme is a payments API. These pages cover authentication, webhooks and errors. ## Guides - [Authentication](https://example.com/docs/auth.md): API keys and OAuth, with examples - [Webhooks](https://example.com/docs/webhooks.md): events, retries and signature checks ## Optional - [Changelog](https://example.com/docs/changelog.md): release notes ```

    Two things it is not. It isn't a robots.txt-style rule: it controls nothing and blocks nothing. And it isn't a sitemap. A sitemap lists every page for search engines. An llms.txt is a short list of the pages you most want an agent to read. The proposal also suggests serving clean Markdown copies of those pages, for example at page.md.

    What Google says about llms.txt

    Google's guide to optimizing for generative AI search lists llms.txt among the things you can ignore for Google Search. Google Search doesn't use these files, so publishing one will neither harm nor help your visibility or rankings. If you want one for other services that use it, that's fine.

    Google's page on [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) says the same thing from the other side. To appear in AI Overviews and AI Mode, a page must be indexed and eligible to be shown with a snippet. There are no additional technical requirements, and you don't need new machine-readable files, AI text files or markup.

    So for Google, the work that counts is the ordinary work: pages that can be crawled and indexed, a title and snippet that aren't blocked, and content worth quoting.

    What the data shows

    Ahrefs looked at the server logs of 137,210 domains that use Ahrefs Web Analytics. About 28% of them publish an llms.txt file. Of those files, 97% received no request at all in May 2026, from bots or people. Ahrefs notes that its customers skew technical and SEO-aware, so treat 28% as an upper bound for the wider web.

    The files that did get requests tell their own story. Ninety-six percent of those requests came from bots, and about 19.5% came from named AI tools. GPTBot, one of OpenAI's training crawlers, fetched the most. Claude Code, Anthropic's coding agent, came second. The search crawlers behind ChatGPT search, Perplexity and Claude (OAI-SearchBot, PerplexityBot and Claude-SearchBot) made only a couple of hundred fetches between them across thousands of sites.

    Ahrefs adds a caution that applies to every number here: a fetch is not the same as a read. A bot can request the file and never act on it. So these figures are a ceiling on real use, not a measure of it.

    When an llms.txt file is still worth having

    The people who use llms.txt are mostly not search engines. They are coding agents. If you publish developer documentation, an API reference or an SDK, your readers increasingly ask a coding tool to read it for them. A short index that points the tool at clean Markdown pages saves it from parsing navigation and scripts. The llmstxt.org proposal notes that the files are used most heavily for software documentation, and that OpenAI, Anthropic and Gemini each publish one for their own developer docs.

    John Mueller of Google has described the file as a stopgap for AI coding tools reading developer documentation, according to the Ahrefs study. That matches what the logs show.

    If you run a marketing site, a blog or a local business site, an llms.txt file is optional. It does no harm. It also gives you no documented route to being cited, and that is the main thing to remember before you spend time on it.

    How to keep one current

    A stale llms.txt is worse than none, because it sends an agent to pages that have moved or no longer exist. If you publish one, build it so it can't drift.

    **Generate it from the same source as your pages.** Create it in your build step from the same list that produces your sitemap and navigation, so a new page appears and a deleted page disappears without anyone remembering to edit a file.

    **Keep it short.** List the pages you would point a new colleague to first. For documentation, that is the quick start, the main guides and the reference. Put secondary links under an Optional heading.

    **Check it loads as text.** Open /llms.txt in a browser. You should see raw Markdown. If you see your site's HTML error page, the file isn't there. The free scan on cite-met.com runs this check, and it reports the result as information, with no effect on your score.

    **Skip the extras.** You don't need Markdown copies of every blog post, a version number in every line or a long list of tags. Add Markdown copies only for your docs. Serve them at stable addresses and link to those.

    What to do instead

    Put the same effort into the things that are documented. Make sure your robots.txt doesn't block the crawlers behind the assistants you care about. With no matching rule, a crawler is allowed, so you don't need a line for each one. Make sure your words are in the HTML your server sends. Google's crawler renders JavaScript, but the vendors don't say their search crawlers do. We cover that in [Do AI crawlers run JavaScript?](/blog/post/blank-page-problem-react-invisible-ai).

    Then measure what you can. Server logs show which crawlers fetched your pages. Bing Webmaster Tools and Google Search Console show citations and impressions in their own AI features. Our guide to [measuring CiteMET results](/blog/post/beyond-clicks-measure-citemet-impact) explains what each one can and can't tell you.

    Questions people ask about llms.txt

    **Will adding llms.txt hurt my SEO?** No. Google says Search ignores the file, so it has no effect on rankings in either direction.

    **Is llms.txt the same as robots.txt?** No. robots.txt tells crawlers which URLs they may fetch. llms.txt controls nothing. It is a curated summary that an agent can read on demand.

    **Should I publish Markdown copies of every page?** Not for Google. Its guide says Markdown versions aren't needed to appear in Search. For developer docs, clean Markdown pages linked from llms.txt are useful to coding tools.

    **Is there a standard I should follow?** Follow the format at llmstxt.org: an H1, an optional blockquote summary, then H2 sections of links. It is a proposal from one author, not a standard backed by a search engine.

    Scanner checks this post covers

    • CM-905 llms.txt: llms.txt is a proposed Markdown file at /llms.txt that summarises a site for language models.
    CYY

    Cho Yin Yong

    Co-Founder & Chief Architect at cite-met. AI engineering leader and University of Toronto lecturer working across AI, web architecture, and user experience. Co-authored two patents in AI applications.

    Artificial Intelligence Engineering
    Answer Engine Optimization (AEO)
    Web Architecture
    User Experience Design

    Related Articles

    technical

    Beyond the Sitemap: Using Dynamic JSON-LD to Speak "Machine Native"

    Hard-coded JSON-LD drifts out of date. Generate it from the same data as your pages, and keep only the schema types Google still uses.

    Cho Yin Yong•8 min
    guide

    What AI Visibility Tracking Costs in 2026, per Answer

    AI visibility tracking costs $2 to $82 per 1,000 answers. We priced the trackers, our own included, per answer and at a sample size you can trust.

    Mahmoud Halat•10 min

    Ranking isn't enough. Get cited.

    Scan your site now. Talk to us when you want it fixed for you.

    Talk to us

    Free scan. No signup.