- Ray Martinez
- Dec 18, 2025
- 4 min read
In early 2025, the team at Archer Education, an enrollment marketing and education technology company, developed an LLMs.txt file for several niche sites in the education space.
LLMs.txt isn’t a universally adopted protocol, but we (at Archer) are committed to testing unrecognized standards. We approach these new tools with wonder, a willingness to learn, and a desire to fail quickly.
As part of this process, we added LLMs.txt files to seven sites and analyzed the bot pings over the course of a three-month period. I’ll share those findings in this article, plus what it all means for the future of LLMs.txt implementation.
Our goal with LLMs.txt implementation
Our goal was to drive bot pings and see how LLMs interact with the file. We deployed these files via an alternate link relationship in the <head> section of our homepage, which indicated that there was an alternate document for LLMs to consider.
We automated the creation of the LLMs.txt files and incentivized user agents by adding the last modification date directly into the file. We did this to signal freshness to LLM bots. We believed LLM user agents would prioritize the file due to the need for fresh content: LLMs often retrieve fresh information to avoid serving stale or outdated answers.
To gather and collect log file data, we developed a backend widget for our multisite CRM instance. Through some nifty scripting on our server side, we were able to pull bot pings into the backend of our multisite instance. Our widget allowed us to filter bot pings to the file and view pings to other URLs, enabling us to measure differences in crawl behavior and activity. We expected a few bot pings.
The results of our LLMs.txt test
During the initial days of the test, the text files were sparsely pinged. On May 30th 2025, the staging file on our dev sites began receiving bot pings from OpenAI and Google. It took another week to see a few pings, and there was limited activity over the next few weeks.
OpenAI bot pings
Then, we saw a surge from OpenAI. Suddenly, at the end of June, we saw both pings to the file rise from a handful to well over 200 per day. That activity continued in the 300+ range per day and eventually led to our files receiving pings every few seconds apart.
As of the last file pull, OpenAI’s Searchbot has crawled the file over 5,000 times. Most of these bot pings occurred between July 16th and July 24th.
Meanwhile, Google crawled Archer Education’s LLMs.txt files 300+ times in the same period. While Google initially dismissed the file, they seem to have relaxed their stance as of December 2025. At Google Search Central Live in Zurich, representatives said they would treat LLMs.txt as a text file and that adding one would not harm your website. This is consistent with our bot logs.
Plus, as you can see below, LLMs.txt files are indexed in Google at a comparable rate as ads.txt. Not to mention, Google is tapping into some notable LLMs.txt use cases for its own content.
Why did the file peak?
Here are some theories why the file may have taken off the way it did.
The release of ChatGPT 5
The peak coincided with the release of ChatGPT 5, which was officially released on August 7th. ArcherEdu’s LLMs.txt file went live on July 23rd, and we published a write-up on LLMs.txt on July 24th. Our LLMs.txt received a visit from GPTBot on July 24th, one day after we published the article. If you prompt ChatGPT 5 without selecting a modality, my write-up is cited as a source. I believe we caught lightning in a bottle as ChatGPT 5 was searching for fresh content.
Social media could have played a role, too
On July 22nd, we published Linkedin posts that garnered over 400 engagements within a day, leading to our quick citation. LLMs use social signals as a sign of the popularity and relevancy of a document.
The future of LLMs.txt
After the initial spike led to crawls from Searchbot, GPTBot, and ClaudeBot in late July, we observed a sharp decrease in bot traffic to our files, with the rate of LLMs.txt dropping to 95% of its peak.
But I believe the file has staying power. Googlebot pings the file weekly. And in September 2025, Anthropic announced further implementation of this file in their internal docs, which discusses the file’s utility for agent building. Perplexity and other AI and tech leaders have developed LLMs.txt files for use within their own internal documentation.
Given the limitations around LLM crawlers’ inability to render JavaScript and parse unstructured data, LLMs.txt offers a real solution that satisfies their need to find fresh content while reducing page weight to markdown-sized files.
Final thoughts on LLMs.txt
Like most things in SEO, the potential power of LLMs.txt isn’t without debate. LLMs.txt isn’t a magic bullet for generative engine optimization, but it’s beginning to show promise in helping LLMs find fresh content and train models and AI agents.
I believe this file will be important going forward, especially as models and pre-training periods are updated. Couple this with Anthropic and Perplexity implementing LLMs-full.txt for internal documentation, and we may be seeing a new standard emerge in real time.
