People are talking about LLMs.txt, a lot when it comes to generative engine optimization. Debates center around whether the plain-text file and proposed web standard helps increase website visibility from AI search platforms.

It is true that LLMs.txt can surface in AI and even classic search, but that is not the full picture when it comes to these files. If you're only focusing on potential traffic you’re missing the point.

The truth is that the architects of our AI future—the likes of OpenAI, Google, Amazon, Microsoft, and Anthropic—are all actively building with LLMs.txt because…

• It’s a means of creating and delivering content specifically for AI chat bots and agents 

• The file is flexible, so it’s easily adapted to a dynamic AI landscape

• It’s the most efficient web file format for LLM and AI agents 

I reviewed over 1,400 LLMs.txt files and found that these benefits deliver business value beyond traffic. They set websites up for a range of AI-first engagement.

This is in line with the protocol documentation, which says that LLMs.txt files serve several purposes.

The versatility of llms.txt files means they can serve many purposes - from helping developers find their way around software documentation, to giving businesses a way to outline their structure, or even breaking down complex legislation for stakeholders.

LLMs.txt are being used in singular implementation but also programmatically, and in multiple formats. As was intended.

How are LLMs.txt being implemented

For my research, I crawled and assessed over 1,400 publicly listed LLMs.txt files dozens of unindexed files for sites like YouTube and Gemini.

My analysis in Google’s index, Screaming Frog, Ahrefs, and a dedicated python script, revealed the following trends about LLMs.txt implementation across the web.

• Google indexes between 30,000 and 60,000 LLMs.txt globally. Testing Google Advanced Search over a week (using site:*/llms.txt) showed that the number of URLs fluctuates within this range.

  
  

• The average LLMs.txt file is 9.8kb, around 275 times smaller than the average page weight on the web (2600kb in 2024), confirming that these pages are significantly more crawl efficient for LLMs and other crawlers.

  
  

• The average LLMs.txt file contains 428 URLs, suggesting that it’s being used to facilitate content discovery for AI.

  
  

• Only 62% of LLMs.txt files are located at the root of a domain, 10% are in subfolders, and 28% are in subdomains.

Drilling down, five core use cases emerged when I looked at these files. The biggest players in AI are using LLMs.txt to…

1. Signal the most contextually important information about a website

2. Act as a sitemap for LLM-friendly markdown content

3. Inform agent-to-agent (A2A) architecture

4. Provide direct instructions to reduce LLM hallucinations

5. Serve as a robust source of grounding context for large language models

01. Signal the most contextually important information about a website

The most common use of LLMs.txt is to make core content “LLM friendly.” A LLMs.txt file provides context about the domain and points LLMs to where they can find the most contextually relevant information about the website. This is built into the format of the document: it includes the name, site summary, and then additional information, including links. 

For example, the LLMs.txt on the Wix website, Izzy’s Wheels, summarizes the core of the business offering, before describing key business information like offerings, contact details and product catalog. This file is automatically generated for Wix Premium eCommerce users.

The goal here is to not to replace robots.txt with directives for LLMs but to send positive signals and provide context for the site for LLM users, in a highly crawl efficient manner.

As users become more agentic in their use of AI—for example, creating tools that access websites to complete tasks for them—maintaining these files could deliver customer value and give your business a significant competitive advantage.

02. Act as a sitemap of LLM-friendly markdown content

One of the most significant benefits of LLMs.txt is that the formatting removes style elements built for humans. LLMs are not particularly interested in your design elements or visual identity. 

This functionality is not exclusive to LLMs.txt. But much like a sitemap.xml or robots.txt is typically found at a standard location, the proposed protocol gives LLMs a consistent destination to find files of this nature. This means that users who have created the other markdown files for LLM consumption can list them on the LLMs.txt in order to facilitate LLM discovery.

Google uses LLMs.txt for markdown directories

Google has created and actively maintains, multiple LLMs.txt files for many of their APIs and other products. Files for Gemini Developer API Docs and API Reference, Chrome Developer Documentation, Firebase, and Flutter are online and accessible to humans and LLMs. In each of the files they point to a number of text-only markdown (md.txt) files that can offer additional context with increased crawl efficiency for LLMs. Until December 2025, LLMs.txt files were also attached to many developer.google.com folders including the YouTube API reference, Google Maps API, and Google Search Central pages. In December 2025, these were removed. 

For example, at https://ai.google.dev/gemini-api/docs/llms.txt, there's a list of over 50 markdown pages, providing contextual information for LLM use.

LLMs.txt is core to Cloudflare’s AI discoverability 

The team at Cloudflare LLMs.txt have taken a similar approach to their documentation, using their LLMs.txt to add links to markdown versions of content on over 30 topics.

In Cloudflare’s AI Consumability statement, the team explains that the tactic is employed to make “content visible to AI as well as making sure it's easily consumed in a plain-text format.” For this team, the LLMs.txt file is just the start; they’ve invested in options to create markdown versions of all of their documentation. 

As a marketer, this tactic seems most applicable to enterprise teams with millions of pages and/or significant potential for agentic use. An LLMs.txt like this one has potential to focus a crawl to the pages that provide the most context for information retrieval on technical topics, but also for agentic users like developers and the AI agents themselves.

03. Inform agent-to-agent (A2A) architecture

Anthropic, AWS, and Google have built LLMs.txt into the agent-to-agent architecture for their products, advising developers to use the files to inform the knowledge base for MCPs and AI agents.

In Google’s Agent Development Kit, the Read Me documentation states that for those developing agents that “the llms.txt and the llms-full.txt can be used as context.”

The team behind Anthropic’s documentation were early adopters of LLMs.txt. They originally published their file in 2024 at https://docs.anthropic.com/llms.txt. They have since migrated it to https://docs.claude.com/llms.txt and update it regularly, suggesting that the file has enduring value.

Along with using the file themselves, in a support article providing advice on how developers can boost performance of their tools, Claude’s team references the fact that “LLM-friendly documentation can commonly be found in flat llms.txt files on official documentation sites.” This statement signals that agentic and other users should be actively looking for files of this nature, as standard.

AWS Powertools for Lambda, a key component for those building on Amazon’s AI ecosystem, has a link to the respective LLMs.txt for each of their utilities. The files are easily accessible via the Resources tab in the menu on more than 100 documentation pages. 

In a blog for the AgentCore, LLMs.txt is recommended as a substitute for MCP with developers who are building layered information architecture, being advised to “install the MCP or use the llms.txt for the Agent Framework SDK and Bedrock AgentCore SDK for a combined documentation layer.”

04. To instruct LLMs and potentially reduce hallucinations

Within an LLMs.txt, it's possible to provide direct instructions and recommendations to LLMs. This has the potential to help reduce hallucinations and to improve response accuracy in LLMs.

For example, the Weather.com LLMs.txt includes instructions on the URL structure for their forecast pages in the opening paragraph. When I asked a similar question on Perplexity, it returned the LLMs.txt as the top source with a citation; it’s also the first page reviewed under “Steps.”

And this occurs despite the fact that a verbatim search for this phrase returns zero website results, meaning that content on the LLMs.txt is likely being accessed independently.

Other LLMs.txt files, like that for Zapier’s documentation llms.txt, include instructions on how to make API calls.

This use case isn't applicable to every business. But for teams delivering SaaS solutions, putting an LLMs.txt to work in this manner could facilitate more accurate responses to technical queries or make it easier to identify use cases for a tool stack.

05. Provide robust grounding context for LLMs

LLMs.txt can be long and extensive used as context for LLMs and agents. LLMs.txt is being used as a robust tool for grounding LLMs with context about how to carry out tasks, where to find information, and step-by-step instructions on how to complete tasks.

Google's Agent Developer Kit explains, "If you are to develop agent via vibe coding the llms.txt and the llms-full.txt can be used as context to LLM. While the former one is a summarized one and the later one has the full information in case your LLM has big enough context window."

In the case of Vercel’s AI SDK documentation LLMs.txt, the team has taken a unique approach, creating a beast of a document that’s over 200,000 words and contains internal and 3rd party external links.

For example, here’s how the file describes troubleshooting a migration error:

While I wouldn’t generally recommend including all of your documentation in a single webpage, I believe there’s value in the intentionality. This file is designed to be queried by agentic users, and the structure of the file reflects that. If you have a website with a large database of knowledge, and evidence of agentic usership, you could create a grounding document using a similar technique.

Finally, Microsoft has LLMs.txt files for multiple projects, in their GitHub repos, but in the LLMs.txt for GenAIScript, they have a section on "Guidance for Code Generation" that tells LLMs what to do when "generating code" and gives directions to the LLMs like " you add TODOs where you are unsure so that the user can review them". This is explicitly intended for LLMs and can add brand value by improving agentic user experience.

Various use cases for an emerging protocol

We’ll continue to see a range of implementations as LLMs.txt evolves. The protocol was designed to be flexible in order to serve a wide range of use cases and websites.

Supporting AI visibility is an important and valuable potential use for these files, but as you can see here, many of the leaders in the AI space are adopting LLMs.txt to inform AI infrastructure. Your team can learn from these tactics to potentially drive growth and better prepare your website for agentic uses.