Quick answer
llms.txt is a Markdown text file placed at the root of a website, for example yourdomain.com/llms.txt, that gives AI systems a short, curated summary of the site and links to its most useful pages. It was proposed by Jeremy Howard in September 2024.
It is not a web standard, and no major search engine or AI company has documented that it reads the file to decide what to show or cite. Google says plainly that Google Search does not use it.
So: publishing one is cheap and harmless, and it can help in narrow cases such as developer documentation. It is not a way to rank in Google or get cited by ChatGPT, and it is not worth paying for as an “AI visibility” service.
What llms.txt is meant to do
The proposal’s own rationale is that “Agents are best served by concise, expert-level information gathered in a single, accessible location.”
The problem it tries to solve is real. Web pages carry navigation, scripts, cookie banners and layout code. A language model reading a page has to dig the content out of all that, and it has limited room to hold text at once. A clean Markdown summary with links to the important pages is easier for a model to use.
The idea borrows its name from robots.txt, but the two work very differently, as the comparison below shows.
The llms.txt format
The specification at llmstxt.org defines a simple structure. Only the first element is required.
|
Element |
Required? |
What it contains |
|
H1 heading (# Site name) |
Yes, the only mandatory part |
The name of the site or project |
|
Blockquote (> …) |
Recommended |
A short summary with the key information |
|
Markdown sections |
Optional |
Extra detail in normal paragraphs or lists |
|
H2 “file lists” (## Section) |
Optional |
Lists of Markdown links, each with an optional note |
|
## Optional section |
Optional |
By convention, secondary links “an agent can skip when a shorter context is needed” |
The proposal also suggests offering clean Markdown copies of pages, either at the same URL with .md appended or with the extension replaced, and linking to them with rel=”alternate” type=”text/markdown”.
Example for a Singapore service business
This is an illustrative example for a fictional company, following the format:
# Example Aircon Services
> Example Aircon Services Pte Ltd is a Singapore aircon servicing,
> repair and installation company serving HDB, condo and commercial
> customers since 2015. Bookings are made online or by WhatsApp.
## Services
– [Aircon servicing](https://example.sg/aircon-servicing/): General and chemical servicing, prices by unit type
– [Aircon repair](https://example.sg/aircon-repair/): Common faults, diagnosis fee, same-day slots
– [Aircon installation](https://example.sg/aircon-installation/): Brands supplied, HDB and condo packages
## Company
– [About us](https://example.sg/about/): Licences, team and service areas
– [Contact and booking](https://example.sg/contact/): Phone, WhatsApp, hours
## Optional
– [Blog](https://example.sg/blog/): Maintenance tips and guides
A useful file is short, factual and consistent with the website. It is not a place for marketing claims.
llms.txt vs robots.txt vs sitemap.xml
|
robots.txt |
sitemap.xml |
llms.txt |
|
|
Purpose |
Tell crawlers what they may and may not fetch |
List URLs for search engines to discover |
Give AI systems a curated summary and key links |
|
Format |
Plain-text directives |
XML |
Markdown |
|
Standard |
Yes, widely supported by major crawlers |
Yes, supported by major search engines |
No, a community proposal |
|
Documented use by Google Search |
Yes |
Yes |
No. Google says Search does not use it |
|
Controls AI crawler access |
Yes, for crawlers that honour it |
No |
No |
|
Affects rankings |
Indirectly, through what is crawled |
Helps discovery |
No evidence it does |
If your goal is to control what AI crawlers can access, llms.txt does nothing. That job belongs to robots.txt and your CDN or firewall settings.
What Google and AI companies have actually said
This is where most articles selling llms.txt go wrong, so the sources matter.
Google’s guide to optimising for generative AI search, last updated 10 July 2026, is direct:
- “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search.”
- “Google Search itself doesn’t use them.”
- Creating them “will neither harm nor help your site’s visibility or rankings in Google Search.”
At Google’s Search Central Deep Dive event in Asia Pacific in July 2025, Gary Illyes was reported as saying Google does not support llms.txt and is not planning to, and that normal SEO is what gets content into AI Overviews.
OpenAI, Anthropic and other AI companies
We could find no documentation from OpenAI, Anthropic or Perplexity stating that their crawlers or assistants read llms.txt when deciding what to retrieve or cite. Several AI companies publish llms.txt files for their own developer documentation. Publishing a file is not the same as consuming other sites’ files.
What these companies do document is their crawlers and how to control them through robots.txt, for example OpenAI’s OAI-SearchBot for ChatGPT search and Anthropic’s Claude-SearchBot.
Google Chrome’s Lighthouse
The one mainstream tool that checks for llms.txt is Chrome’s Lighthouse, which added an llms.txt audit under an “Agentic Browsing” category. It describes llms.txt as “an emerging convention”, and when the file is missing, the audit is marked Not Applicable rather than failed, “as providing the file is optional at the moment.”
So Google Search says it does not use the file, while a Google developer tool checks for it without penalising its absence. That mixed signal is exactly the gap that “llms.txt optimisation” services are sold into.
Adoption
The specification’s August 2026 update notes that thousands of sites publish llms.txt, that documentation platforms generate it automatically, and that plugins including Yoast SEO and AIOSEO, and platforms such as Wix, can create one. Adoption figures from third-party studies vary widely by method, so treat any single percentage with caution.
When llms.txt is genuinely useful
- Developer and API documentation. People working with AI coding assistants often point the assistant at a documentation site. A clean index of Markdown pages is easier to use than crawling a docs site.
- Product knowledge for your own AI tools. If you build an internal assistant or a customer support chatbot, a curated Markdown summary is a sensible starting input.
- Users who paste your file into a chat. Someone researching your company can give an assistant the file directly.
- Large documentation or knowledge-base sites where navigation makes the important pages hard to find.
For a typical Singapore SME brochure or service website, none of these is a strong reason, but the cost of publishing a short, accurate file is close to zero.
Should your business website have one?
|
Your situation |
Recommendation |
|
Your SEO plugin can generate one automatically |
Fine to turn on, then check the content is accurate |
|
You run developer docs, an API or a SaaS knowledge base |
Worth publishing and maintaining |
|
You have a standard service or ecommerce website |
Optional, low priority |
|
An agency is charging you for “llms.txt optimisation” to get into ChatGPT or AI Overviews |
Do not pay for this. There is no documented mechanism behind it |
|
You want to stop AI companies training on your content |
llms.txt will not do this. Use robots.txt rules and your CDN’s bot controls |
|
Your AI visibility is poor |
Fix crawler access, rendering and content first. See below |
If you publish one: a checklist
- Place it at the root: yourdomain.com/llms.txt, served as plain text with a 200 status.
- Start with an H1 containing the business name, then a short blockquote summary.
- Use facts, not claims. Founding year, services, locations, prices or price ranges, contact details. Avoid “leading”, “best” and “number one”; they add nothing an AI system can verify.
- Match your website exactly. If the file says one founding year and the About page says another, you have created a contradiction, not clarity.
- Link only to live, indexable pages with a one-line note on what each contains.
- Keep it short. A few dozen links at most for a business site.
- Put it on your update checklist. Out-of-date prices, team sizes, grant information or services are worse than no file.
- Do not block it in robots.txt, and do not expect it to change rankings.
- Check your server logs after a few weeks to see which user agents, if any, actually request it.
What actually improves AI visibility
If the real goal is appearing in AI Overviews, ChatGPT search, Copilot or Perplexity answers, these have far more impact than llms.txt:
- Crawler access. Allow search-oriented AI crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot in robots.txt, and check your CDN is not blocking them.
- Content in the HTML. Network-level testing published by Vercel and MERJ in December 2024 found the major AI crawlers from OpenAI, Anthropic, Meta, ByteDance and Perplexity did not execute JavaScript. Content that only appears after JavaScript runs may be invisible to them.
- Specific, citable facts on the page: prices, timelines, specifications, locations and dates.
- Consistent business information across your site, Google Business Profile and directories.
- Mentions on other trusted sites, because AI answers draw on multiple sources.
- Measurement. Use Google Search Console’s Generative AI performance report and Bing Webmaster Tools’ AI Performance report to see where you appear.
Our step-by-step AI search readiness audit covers each of these, and GEO vs SEO explains how AI visibility relates to traditional SEO.
Frequently asked questions
What is llms.txt?
A Markdown file at a website’s root that summarises the site and links to its most important pages, intended to help AI systems understand it. It was proposed by Jeremy Howard in September 2024 and is not an official web standard.
Does Google use llms.txt?
Google says Google Search does not. Its AI optimisation guide states that such files “will neither harm nor help your site’s visibility or rankings in Google Search.”
Does ChatGPT read llms.txt?
OpenAI has not documented that ChatGPT or its crawlers use llms.txt when choosing sources. OpenAI does document its crawlers, such as OAI-SearchBot, which you control through robots.txt.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers what they may fetch and is honoured by major crawlers. llms.txt is a curated summary with no access control and no documented consumption by major search or AI companies.
Can llms.txt block AI from training on my content?
No. Use robots.txt rules for training crawlers such as GPTBot, ClaudeBot and Google-Extended, and your CDN or firewall’s bot controls.
What is llms-full.txt?
A companion file containing the full content of a site’s key documentation in a single Markdown file, rather than just links. It was popularised by the documentation platform Mintlify, which developed it with Anthropic for Anthropic’s developer docs, and is mostly used for developer documentation.
How do I create an llms.txt file?
Write it by hand following the format above, or use your platform’s generator. Yoast SEO, AIOSEO and Wix can generate one. Always review the generated content for accuracy.
Will llms.txt help my SEO?
There is no evidence it helps rankings in Google. It is a low-cost optional file, not an SEO tactic.
Should I pay an agency to create one?
Not as a visibility service. It takes minutes to write. Spend the budget on crawler access, content quality and technical fixes instead.
Get your site ready for AI search properly
MediaPlus Digital runs AI search readiness audits and AI SEO for Singapore businesses, with 13 years in market, a team of 60 and more than 1,800 projects delivered. We will tell you when a tactic has no evidence behind it, and focus on the fixes that do.
See our AI SEO services in Singapore or learn more about answer engine optimisation.
Specification details and statements from Google verified 17 September 2026 from llmstxt.org, Google Search Central and Chrome for Developers documentation.
