llms.txt file, proposed in 2024, is the newest and most debated lever: a plain markdown file at your domain root that tells AI systems what your site contains and which pages matter most. This guide covers what llms.txt actually does (and does not do), how the major AI crawlers behave in 2026, and the complete setup we implement at Digimau for sites that want to be cited rather than ignored.
—
Table of Contents
What llms.txt Actually Is
The specification is deliberately simple. You place a file namedllms.txt at the root of your domain, written in markdown. It contains a one-line summary of your site, a short paragraph of context, and a curated list of your most valuable pages with brief descriptions. The idea: LLM inference happens with limited context windows, so instead of forcing a model to wade through navigation, ads, and boilerplate, you hand it a pre-digested index of what matters.
Anatomy of a minimal file:
# Digimau > Performance marketing, SEO, and web development agency helping businesses grow through data-driven digital strategy. ## Guides - [Technical SEO Audit Checklist](https://www.digimau.com/technical-seo-audit-checklist-2026/): Full audit framework - [Keyword Research Guide](https://www.digimau.com/keyword-research-guide-2026/): 2026 process and tools ## Services - [SEO Services](https://www.digimau.com/): Overview and engagement modelsThere is a related convention,
llms-full.txt, which concatenates the full content of your key pages into one large document for crawlers that prefer bulk retrieval. Most implementations start with the curated file and add the full variant only if server resources allow.
Do AI Systems Actually Use It in 2026?
Here is the honest picture. Adoption is real but uneven. A meaningful ecosystem of AI tools and agentic systems reads llms.txt when present, and several site platforms (including major CMS plugins and static-site generators) now generate it natively. The large consumer assistants have made fewer explicit commitments, and Google has stated its systems do not currently use the file — while Anthropic added support for retrieving llms.txt during crawl. Perplexity and a long tail of RAG-based tools respect it inconsistently. The pragmatic framing: llms.txt is a low-cost, standards-track bet. The file costs almost nothing to produce and maintain, it demonstrably helps the systems that do read it, and if a major engine adopts it later, early implementations inherit the benefit. It is not a ranking factor, and no vendor should present it as one — it is a retrieval and citation aid.| System | Reads llms.txt? | Crawler | 2026 Status |
|---|---|---|---|
| Anthropic (Claude) | Yes | ClaudeBot | Documented support for retrieval at crawl time |
| OpenAI (ChatGPT) | Unconfirmed | GPTBot, OAI-SearchBot, ChatGPT-User | Crawls actively; llms.txt use not officially stated |
| Perplexity | Partial | PerplexityBot | Independent tools report mixed handling |
| Google (AI Overviews) | No | Googlebot, Google-Extended | Stated non-use; relies on classic indexing |
| RAG/agentic tools | Often yes | Various | Strongest adoption segment |
Managing AI Crawler Access
Before you write a single line of llms.txt, decide who you actually want in. AI crawlers are controlled throughrobots.txt, and each vendor runs separate bots for training versus live search. GPTBot (training) is different from OAI-SearchBot (search citations), which is different from ChatGPT-User (real-time user fetches). Blocking GPTBot does not remove you from ChatGPT search, and vice versa. The strategic question is whether you want your content training future models, being cited in current answers, or both — many publishers now allow search-related bots while restricting training-only crawlers, or vice versa depending on their content monetization model.
Verify your current state before changing anything: check server logs for ClaudeBot, GPTBot, and PerplexityBot hits, and test whether /llms.txt already exists on your domain (some platforms add it automatically during updates).
How to Build Your llms.txt Step by Step
The build is a curation exercise, not a writing exercise. Follow this sequence:1. Inventory and select
List the 10-25 pages that best represent your expertise and answer the questions your customers actually ask. Prioritize evergreen guides, definitive service pages, and original data or research. Exclude thin pages, conversion-only landing pages, and anything you would not want quoted.2. Write the header
One H1 with your brand name, a blockquote of one or two sentences describing what you do, and nothing else. The models that parse this file weight the top heavily — think of it as your elevator pitch in the citation pipeline.3. Group and describe
Organize links into logical sections (Guides, Services, Research) and give every link a short, factual description. Descriptions should state what the page contains, not market it — factual density is what survives into generated answers.4. Deploy and validate
Upload to your domain root, confirm it returns HTTP 200 withtext/markdown content type, and check that it renders as plain text in a browser. Add llms-full.txt only if you can generate and maintain it automatically.
5. Maintain on a schedule
Every time you publish a major guide, add it. Quarterly, prune anything stale. A llms.txt full of dead links is worse than none at all.Beyond llms.txt: The Full AI Citation Stack
The file alone will not make you citable. What actually earns citations is the same machinery that earns classic rankings, tuned for extraction. Structure content so answers can be lifted cleanly: definition-first paragraphs, question-formatted headings, tables for comparisons, and statistics with dates. Keep AI crawlers unblocked for the answer engines you want to appear in. And track your visibility — asking the major assistants which brands they recommend for your category, monthly, is currently the most direct measurement available. For the broader strategy this fits into, see our complete guide to generative engine optimization, and for the classic foundation underneath it all, the technical SEO audit checklist — crawlability and clean structure serve both kinds of engines. There is also a measurement layer: server-log analysis tells you which AI bots visit and how often, and it pairs naturally with the workflow in our log file analysis guide. If citations are the goal, our AI Overviews optimization guide covers the Google-specific playbook in depth.Common Mistakes
The recurring failures are predictable. Marketing fluff in the header (“the leading provider of innovative solutions”) gives a model nothing to cite. Link lists without descriptions waste the format — the descriptions are the payload. Stuffing every URL on the site dilutes the curation signal. Letting the file go stale with 404s actively damages trust in your domain. And the biggest misconception of all: treating llms.txt as a substitute for the unglamorous work of publishing genuinely citable content. The file is a signpost, not a destination.Frequently Asked Questions
What is llms.txt in simple terms?
It is a plain markdown file at your domain root that lists your most important pages with short descriptions, written so AI systems can quickly understand what your site contains and retrieve your best content without parsing your full HTML.
Does Google use llms.txt for AI Overviews?
No. Google has stated its systems do not currently use llms.txt. AI Overviews draw on Google’s normal index, so classic SEO remains the path to appearing there. llms.txt matters for the other systems that do read it.
Does llms.txt affect my SEO rankings?
Not directly. It is not a search ranking factor. Its value is retrieval and citation in AI answer engines, and it costs little enough that most sites implement it as cheap optionality alongside standard SEO.
Which AI crawlers should I allow in robots.txt?
Decide per bot: OAI-SearchBot and other search-retrieval bots if you want citations in live answers, GPTBot and ClaudeBot if you accept training use, PerplexityBot for Perplexity answers. Many publishers allow search bots while restricting training-only crawlers.
How is llms.txt different from robots.txt?
robots.txt grants or denies crawler access. llms.txt assumes access and curates what the crawler should read first. You generally need both: robots.txt to set the rules, llms.txt to set the priorities.
How long should my llms.txt file be?
Keep the curated file tight: 10-25 links with one-line descriptions, plus the header. Exhaustive link dumps defeat the purpose, which is pre-digesting your site into a context-window-friendly index.
Where do I put the file and how do I verify it?
At your domain root (yourdomain.com/llms.txt). Verify it returns HTTP 200, serves as plain text or markdown, and renders correctly in an incognito browser. Some CMS platforms and SEO plugins can generate and host it for you.
What is llms-full.txt and do I need it?
It is a companion file concatenating the full text of your key pages into one document. It helps bulk-retrieval systems but can be large; implement it only if you can generate it automatically and keep it in sync with your content.
How do I know if AI systems are citing my site?
Run a monthly prompt panel: ask each major assistant the questions your customers would ask and log which brands and URLs appear. Server logs showing AI crawler hits tell you retrieval is happening; the prompt panel tells you citation is happening.
Is llms.txt just a fad?
Possibly, and the cost asymmetry answers the question: the file takes an hour to build and minutes per month to maintain. If the standard stalls, you lose nothing; if a major engine adopts it, early sites benefit. Treat it as cheap optionality, not a strategy.
Related Articles
Continue building the growth stack:
