llms.txt Explained: What It Is and How to Set It Up Correctly
A single text file in your root directory can decide whether AI models understand your website at all. Here's where llms.txt comes from, how it's structured, and how to create your own in 10 minutes.
What is llms.txt?
llms.txt is a markdown file in a website's root directory - reachable at yourdomain.com/llms.txt, similar to robots.txt or sitemap.xml. The difference: robots.txt tells crawlers what they're NOT allowed to visit. llms.txt tells AI models WHAT your website is and WHERE the most important content lives - a structured summary instead of an HTML page full of navigation, ads, and JavaScript.
The standard was proposed in September 2024 by Jeremy Howard and the team at Answer.AI - in response to a concrete problem: AI models' context windows are too small for entire websites, and regular HTML pages are unnecessarily hard for language models to process.
Does Google actually use llms.txt? The current state
No major AI provider has officially confirmed that it actually reads llms.txt. Google has been the most explicit: the file isn't necessary for Google Search or AI Overviews, because Google already crawls and processes full page content through its existing indexing pipeline - a separate summary file doesn't add anything there. There's no official confirmation from ChatGPT, Claude, or Perplexity that their crawlers specifically fetch /llms.txt either, and log-file analyses from several SEO tools have so far found no evidence that the known AI crawlers actually request the file.
In practice, that means llms.txt is currently not a proven GEO signal with a demonstrated effect on citations or rankings - it's a voluntary, community-driven proposal without official backing from the major AI providers. It can still be worthwhile as clean product documentation for yourself and for tools that do explicitly look for it (some RAG systems or AI coding agents, for example) - just don't treat it as your main AI-visibility strategy. Signals with a demonstrated effect, like schema markup, crawler access, and citable content, are covered in our article on What is GEO?, along with Schema Markup for AI Citations.
How is an llms.txt structured?
# Project nameAn H1 with the name of your website or product - required by the specification.> Short descriptionA blockquote paragraph: what the website is, in one or two sentences, with no marketing fluff.## Sections with linksFreely chosen H2 headings (e.g. "Docs", "Pricing", "API") with markdown links to the most important subpages.## OptionalA section for less important links that AI models can skip if needed.# MyProduct > MyProduct is a tool for X that solves Y in under 60 seconds. ## Docs - [Getting started](https://example.com/docs/start): Quick start guide - [API reference](https://example.com/docs/api): Full API documentation ## Pricing - [Pricing & plans](https://example.com/pricing): All plans at a glance
Full specification: llmstxt.org ↗
llms.txt vs. llms-full.txt: what's the difference?
llms.txt stays deliberately short - an overview with links. llms-full.txt (optional, at /llms-full.txt) covers the same topics in more depth: detailed pricing, FAQs, technical specs. For AI models that can load more context at once, the expanded version delivers more direct, citable answers - without making users click through several linked subpages first.
Common mistakes when setting it up
Marketing copy instead of facts
llms.txt should be precise and factual - "X is a Y for Z that does A" instead of marketing language. AI models extract direct statements from it.
File blocked or behind a login
llms.txt needs to be publicly accessible without authentication - otherwise AI crawlers can't read it at all.
Never updated
New features, changed pricing, or renamed products go stale in the file if it isn't maintained alongside those changes.
Wrong format
llms.txt has to be valid markdown, not an HTML page with a .txt extension - otherwise many parsers can't process it cleanly.
Frequently asked questions about llms.txt
What is llms.txt?
llms.txt is a markdown text file in a website's root directory (similar to robots.txt) that explains, in a structured, compact form, what a website is, what it offers, and where the relevant content lives - specifically for AI models to read. It was proposed as an open standard in September 2024 by Jeremy Howard (Answer.AI).
Is llms.txt required for AI visibility?
No. It's not an official standard from Google, OpenAI, or Anthropic, and none of the major AI providers have confirmed actually using llms.txt. Google has explicitly stated the file isn't necessary for Search or AI Overviews. It remains a voluntary, community-driven proposal with no proven effect on AI citations.
Does Google use llms.txt for Search or AI Overviews?
No. Google has stated that llms.txt is unnecessary for Google Search and AI Overviews, because Google already crawls and processes full website content through its existing indexing pipeline. A separate summary file doesn't change that.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a short overview - product description, key points, links to further resources. llms-full.txt (optional) contains the same information in expanded form, often including pricing, FAQs, and technical details, for AI models that can load more context.
Where does llms.txt need to live, and how do I find it?
In the domain's root directory, reachable at yourdomain.com/llms.txt - just like robots.txt or sitemap.xml. It has to be publicly accessible without a login so AI crawlers can read it.
Do you already have an llms.txt?
AuditAI automatically checks whether llms.txt and llms-full.txt exist and are correctly formatted - as part of 19 GEO signals in under 60 seconds. Start without registration, sign up free for the full report with all scores.
Check your GEO score nowWhat is GEO? How to Get Recommended by ChatGPT and Claude
llms.txt is one of 19 GEO signals - the full overview of what else counts.