Our honest take on llms.txt

Every week another AI agent shows up at our door: Perplexity, Claude, some coding assistant a developer pointed at our site and whatnot. It’s a new type of bot and we have to prepare for it.

That is the problem llms.txt tries to solve. We looked into the spec, implemented it on marsbased.com, and formed an opinion. Here it is, without the hype.

What llms.txt actually is

Websites were built for humans and search crawlers, not for language models. When an LLM tries to read a page, it hits four walls:

• Limited context. Models have finite context windows and cannot ingest a whole website effectively.
• Noisy HTML. A standard page is buried under nav bars, ads, JavaScript, animations, and cookie banners.
• Manual clean-up. Turning a site into clean text an LLM can use is a manual, error-prone job.
• Documentation gaps. AI coding assistants want the raw documentation, not the web wrapper around it.

llms.txt is one plain Markdown file, placed at the root of your domain: yourdomain.com/llms.txt. That is the whole idea.

It sits at the root on purpose, mimicking the conventions we already know from robots.txt and sitemap.xml. Inside, it holds the essentials: a title, a short summary, and a curated list of links, each with a one-line description.

There is also an optional companion file, llms-full.txt, which contains the fully expanded content. That one is meant to be pasted whole into a context window when an agent needs the complete picture.

Ours is roughly like this:

# MarsBased
> Ruby on Rails and web development studio based in Barcelona.

## Docs
- [Onboarding guide](/docs/onboarding.md): How new clients get set up
- [API reference](/docs/api.md): Full REST API reference

## Optional
- [Blog](/blog): Engineering and product writing
- [Changelog](/changelog.md): Release history

Or you wan check it out here. Conceptually, it’s similar to other files sitting at root level of your web projects. I’m talking about these:

• robots.txt tells crawlers what they may or may not crawl.
• sitemap.xml lists every URL, for indexing completeness.

llms.txt is a curated summary of what matters, for models working with limited context.

The key word is curated. A sitemap dumps every page automatically. llms.txt lets you choose. You can point agents at your services and documentation while quietly leaving out the half-finished section and the outdated newsletter archive.

If this reminds you of humans.txt, the old file that listed the people behind a project, you are not wrong. However, that one never caught on, so it’s worth keeping it in mind before we get too excited about llms.txt.

Who actually uses it

Adoption is growing but still small. Measured across the top 10,000 sites, it went from about 1 percent in July 2025 to roughly 5.6 percent by June 2026, around 5.4x growth in a year (see Casey Burridge's HTTP Archive analysis). Broader crawls put it higher, up to 8 to 10 percent depending on the sample, but the direction is the same: real growth off a tiny base.

Most of that jump has one cause: Shopify silently pushed a default llms.txt to all 7 million of its storefronts in late April and early May 2026, no announcement, no admin banner. Shopify alone sits near 78 percent adoption, which is not organic developer enthusiasm, but it counts. I can only imagine what’d happen if Wordpress did the same: over 40% of the internet would suddenly have an llms.txt file.

Beyond Shopify, the adopters are mostly infrastructure and platform companies: Cloudflare, Vercel, Netlify, WordPress. Meanwhile Reddit, the BBC, X, Facebook, and the WSJ are not using it, just to name a few tier-one web domains and big corporations.

A few signals worth weighing:

• Google does not support it. Its AI optimization guide, updated June 2026, tells site owners plainly that they do not need machine-readable files or Markdown to appear in Google Search, because Search does not use them. Gary Illyes confirmed the same in July 2025, and John Mueller earlier compared llms.txt to the old keywords meta tag: a file that claims what a site is about, when a crawler could just read the site.
• No major LLM provider has committed to using it as a ranking or citation signal. OpenAI and Anthropic publish their own llms.txt files, but that is documentation for their tools, not a promise to consume anyone else's.
• The bots barely touch it. An Ahrefs study of 137,000 domains found that of the roughly 38.000 sites with a valid file, 97 percent got zero requests for it in May 2026. No bots, no humans, nothing.

So if your hope is "add llms.txt and the robots will flock to you", or you’ve been sold that this is going to boost your SEO, the data is against you.

Where it genuinely helps

So far, we think that llms.txt is just useful in specific places:

• Coding agents: Documentation sites read directly during developer sessions with Cursor, Claude Code or Copilot. The moment a developer points an agent at a URL, a well-placed llms.txt goes a long way.
• Documentation ecosystems: Internal or partner docs built for secure integrations, where you want strict control over what a proprietary system ingests and which tool fetches the context.
• Human-in-the-loop workflows: The "read my llms.txt first" pattern, where a person explicitly tells the agent where to look, gives you accurate context before any processing happens.

In shrot: llms.txt shines when a human or a tool deliberately reaches for it, not when you are hoping a crawler stumbles onto it.

What we learned shipping it

We added llms.txt to our main site, marsbased.com. We selected the high-relevance sections for potential clients, such as services and portfolio, and left out the low-value ones like the newsletter. That curation is the real work, and it is where the value is.

Two lessons for y’all:

First, maintenance is the catch. The file has to stay in sync with the site, and it drifts the moment content changes. Any change to the site’s content or structure must be reflected in llms.txt.

The fix is to automate regeneration whenever the site updates, so the version bots read matches the version humans read using skills or automated routines that fire before every commit.

Second, there is no standard way to advertise it. Neither robots.txt nor the sitemap spec includes a convention for referencing an llms.txt file, so bots have to know the path directly. And if you go the verbose route with llms-full.txt, split context across smaller files the way you would with sitemaps, rather than dumping everything into one file and overloading the agent.

One honest measurement problem: thanks to zero-click behavior, agents often consume your content without ever sending a visitor your way. We have not seen a meaningful bump in human traffic. What llms.txt does do is steer bots toward the content that matters and stop them wasting crawl budget on the parts that do not.

Should you integrate it?

Our verdict, in four lines:

• Low cost, low risk. It is a static Markdown file, takes five minutes to write, can be automated, and needs no build step.
• Worth adding to documentation platforms, client-facing docs, and automated API reference systems.
• Not a SEO enhancement. Do not sell it as one, and do not expect it to move your rankings. When SE Ranking modeled it, the presence of an llms.txt file did not correlate with being cited more often by AI systems.
• Keep watching. Adoption could tip quickly if any major LLM provider changes its position on supporting these standards.

Add it where it helps, automate the maintenance, and do not oversell it. That is the entire play for now.

If you want a hand implementing llms.txt on your own site or documentation, that is exactly the kind of thing we do. Get in touch.