Is llms.txt Dead? The Current State of Adoption in 2025

Just over a year ago, Jeremy Howard from Answer.AI proposed a simple yet ambitious standard: llms.txt, a way for websites to provide AI-friendly content to large language models. The response was immediate and enthusiastic, with major tech companies like Anthropic, Cloudflare, and Stripe implementing the standard within weeks.

But as we approach the end of 2025, a critical question looms: Is llms.txt actually being used by the AI systems it was designed for?

The answer is complicated. While over 784 websites have implemented llms.txt files, creating a thriving grassroots movement among developer documentation sites, zero major LLM providers have officially committed to using them. Google has explicitly rejected the standard, comparing it to the discredited keywords meta tag. And perhaps most tellingly, server logs from publishers paint a contradictory picture of whether any AI crawlers are actually fetching these files.

This isn't a simple story of success or failure. It's the story of a well-intentioned solution caught between enthusiastic early adopters betting on an AI-first future and skeptical platform operators unconvinced of its necessity. Let me walk you through what's really happening with llms.txt in 2025, backed by data, expert opinions, and real-world evidence.

What llms.txt Was Supposed to Solve

When Jeremy Howard introduced llms.txt in September 2024, the problem was clear and relatable. He had just released FastHTML, a Python library for building web applications, and the most common complaint from developers was that AI coding assistants couldn't help them use it. The library was too new—created after the models' knowledge cutoff dates.

The solution Howard proposed was elegantly simple: a Markdown file placed at your website's root (/llms.txt) that provides a curated, LLM-friendly overview of your most important content. Think of it as a navigation structure specifically designed for AI systems, complemented by llms-full.txt, which contains your complete documentation in a single consumable file.

The core insight was practical: LLMs struggle with messy HTML filled with navigation menus, ads, JavaScript, and CSS. Converting complex web pages into clean text wastes valuable tokens on non-content elements. By providing pre-curated Markdown content, site owners could eliminate this inefficiency and direct AI systems to the most relevant information.

Unlike robots.txt, which tells crawlers what not to access, or sitemap.xml, which provides URLs without context, llms.txt offers a curated guide with descriptions—a human-readable, AI-friendly way to say "here's what matters on our site."

The Explosive Early Growth That Got Everyone Excited

Initial reception in September 2024 was cautiously interested but skeptical. Hacker News discussions questioned whether LLMs really needed this help and whether major platforms would ever use it. Some critics worried about content manipulation, while others saw it as making life easier for companies scraping content without permission.

But everything changed in November 2024. On November 14th, Mintlify announced automatic llms.txt support for all documentation sites hosted on their platform, bringing thousands of developer documentation sites into the ecosystem overnight. The very next day, Alex Albert, Head of Claude Relations at Anthropic, announced that all Claude documentation was now available as concatenated plain text files—a post that received over 371,000 views.

This triggered a cascade of community development.

• Multiple directory sites emerged to track implementations (directory.llmstxt.cloud, llmstxthub.com, llmstxt.site)
• Generator tools were created by Firecrawl and dotenvx
• Integration plugins appeared for VitePress, Docusaurus, and Drupal
• WordPress plugins from Yoast and Rank Math added automatic generation features

The list of early adopters read like a who's who of developer infrastructure: Anthropic, Cloudflare, Vercel, Stripe, Zapier, Supabase, Cursor, ElevenLabs, Shopify, Hugging Face, Pinecone, and NVIDIA all implemented llms.txt files.

Cloudflare's implementation was particularly extensive, with 3.7 million tokens in their llms-full.txt file. Vercel's file resembled a 400,000-word novel.

By mid-2025, community-maintained directories were listing over 784 websites with llms.txt implementations, with growth continuing steadily month over month.

The Adoption Numbers That Tell Two Very Different Stories

Understanding llms.txt adoption in 2025 requires looking at data from multiple sources that paint dramatically different pictures.

When Rankability scanned the top 1,000 most-visited websites globally, they found:

• January 2025: 0 sites with llms.txt
• March 2025: 0 sites with llms.txt
• June 2025: 3 sites (0.3%) with llms.txt

Zero major consumer platforms like Google, Facebook, Amazon, or mainstream news sites have adopted it.

But community-maintained directories tell a completely different story, as seen here:

• llms-text.com: 784+ websites
• llmstxthub.com: Over 500 sites
• directory.llmstxt.cloud: Approximately 684 implementations
• NerdyData: 951 domains as of July 2025

An independent crawl of the Majestic Million dataset found just 15 sites in February 2025 growing to 105 sites by May—a 600% increase from a near-zero base.

These seemingly contradictory statistics are both accurate—they're just measuring different populations. The top-1000 scan measures mainstream web adoption (essentially zero), while community directories aggregate any site with the file (heavily concentrated in tech).

The pattern is unmistakable: llms.txt has achieved rapid penetration in developer tools, AI companies, technical documentation, and SaaS platforms, while remaining completely absent from the mainstream web. Adoption is growing explosively in percentage terms but remains negligible in absolute numbers across the broader internet.

Some companies report tangible benefits. Vercel claims 10% of their signups now come from ChatGPT, attributing this to Generative Engine Optimization efforts including llms.txt. Windsurf highlights token and time savings when AI agents don't need to parse complex HTML.

But these success stories come exclusively from the technical documentation sector, and most lack rigorous measurement frameworks to validate the attribution.

Why Google Said No (Very Publicly)

The most damaging development for llms.txt came in June 2025 when Google's John Mueller stated bluntly: "No AI system currently uses llms.txt. It's super-obvious if you look at your server logs. The consumer LLMs/chatbots will fetch your pages—for training and grounding, but none of them fetch the llms.txt file."

Mueller went further, comparing llms.txt to the discredited keywords meta tag—a feature where site owners claimed what their content was about rather than letting systems determine it objectively. His argument cuts to a fundamental trust issue: if AI systems need to verify the llms.txt content anyway, why not just parse the original HTML directly?

In July 2025, Gary Illyes made Google's position even clearer at the Google Search Central Deep Dive event: "Google doesn't support LLMs.txt and isn't planning to." He emphasized that normal SEO practices are sufficient for ranking in AI Overviews, and Google won't be crawling llms.txt files.

This matters enormously because Google isn't just the dominant search engine—they're also a major AI player with Gemini. Their rejection isn't just about one company's preferences; it's a philosophical stance on web standards and trust.

Google's position is that llms.txt:

• Creates opportunities for manipulation
• Allows content cloaking (showing different content to AI versus humans)
• Solves problems that are rapidly becoming obsolete as LLMs improve at parsing HTML directly

Multiple publishers with large domain portfolios have confirmed Mueller's observations. One hosting provider managing 20,000 domains reported that no mainstream AI agents or bots download llms.txt files—only niche crawlers like BuiltWith.

Redocly tested extensively and found that unless you explicitly paste the llms.txt URL into an LLM, "it doesn't do anything... No model we tested spontaneously 'read' or respected llms.txt on its own."

The Contradictory Evidence That Changes Everything

Yet there's conflicting evidence suggesting some experimental usage that makes this story far more complex than "nobody uses it."

Ray Martinez, founder of Archer Education, documented OpenAI's GPTBot crawler fetching his llms.txt file every 15 minutes. Mintlify cites data from Profound, a Generative Engine Optimization tracking company, claiming that Microsoft and OpenAI are actively crawling and indexing both llms.txt and llms-full.txt files, with a heavy preference for the full version.

According to their data, ChatGPT accounts for the majority of visits to these files across multiple tracked websites.

This creates a puzzling situation: server logs from some publishers show active crawling, while others see nothing. OpenAI has created llms.txt files for their own API documentation but has made no official statement about whether their crawlers use these files from other sites. Anthropic specifically requested Mintlify implement llms.txt for their documentation and publishes comprehensive files, but they've never confirmed that Claude's crawlers use llms.txt files from other websites.

The most plausible explanation is opportunistic experimentation. Some LLM systems may be testing llms.txt usage or accessing files when they're manually provided by users, but no systematic crawling or prioritization exists.

There's a vast difference between crawling files to see what's there and actually using them to improve responses or influence what content gets retrieved. No major LLM company has published official documentation about llms.txt support, included it in their crawler specifications, or provided guidelines for implementation.

This silence is deafening after 12 months of community adoption.

What the Developer Community Actually Thinks

The community reaction to llms.txt has been intensely polarized, creating one of the more contentious debates in web standards in recent years.

The Optimists

Developer tool companies and early adopters emphasize practical benefits.

• Token efficiency: Providing pre-formatted Markdown saves processing costs
• Better AI accuracy: Curated content helps AI coding assistants provide better answers
• Knowledge cutoff solution: New libraries and APIs can be documented immediately
• Competitive advantage: Being AI-ready now provides an edge before it becomes table stakes

They point to real momentum: Anthropic's specific request for the feature, inclusion in Google's Agents to Agents (A2A) protocol, active GitHub development, and measurable business impact at companies like Vercel.

The Skeptics

But SEO professionals and infrastructure companies are far more critical. Ryan Law from Ahrefs put it bluntly: "llms.txt is a solution in search of a problem. I could also propose a standard (let's call it please-send-me-traffic-robot-overlords.txt), but unless the major LLM providers agree to use it, it's pretty meaningless."

The critical camp points to:

• High maintenance overhead without clear ROI
• Trust and gaming concerns similar to those that killed meta keywords
• Redundancy with existing standards like Schema.org and sitemaps
• Temporary problem: LLMs are getting better at parsing HTML directly

In October 2025, Redocly published a particularly damning post titled "llms.txt is overhyped," stating: "We tried it. We measured it. We learned from it. And now we can say it out loud: llms.txt is overhyped." They found it to be "mostly smoke, not fire" after extensive testing.

Search Engine Journal documented a "misinformation loop" where SEO tools flag missing llms.txt as an issue, users feel anxiety about AI visibility, WordPress plugins add auto-generation to meet demand, more tools follow suit, and the perception of necessity reinforces itself—all despite no evidence that any major LLM actually uses these files.

Even Squirrly SEO, which added the feature, honestly told users: "There is currently zero proof that it helps with being promoted by AI search engines."

The Competing Approaches That Might Win Instead

llms.txt isn't operating in a vacuum. Several alternatives are gaining more traction with official platform support.

Model Context Protocol (MCP)

The Model Context Protocol, introduced by Anthropic in November 2024, has gained significantly more traction with official support from major providers. MCP creates secure, two-way connections between AI systems and external data sources, enabling dynamic interaction rather than static content snapshots.

Official adopters include: Claude Desktop, OpenAI, Google DeepMind, Cline, and Cursor AI, with SDKs available in Python, TypeScript, C#, and Java.

MCP is more complex to implement—requiring robust engineering effort, secure APIs, and client-server architecture—but it solves a broader set of problems. It transforms LLMs from passive readers into active assistants that can interact with tools and data sources in real-time.

For enterprises needing sophisticated AI integration, MCP represents the direction major providers are actually moving, not llms.txt.

Schema.org Structured Data

Schema.org structured data offers another established alternative that multiple experts argue is fundamentally superior. Microsoft has confirmed that Bing's Copilot uses Schema markup for LLM understanding.

Unlike llms.txt, Schema provides:

• Semantic context that LLMs can reason over, not just text to parse
• Integration with existing web infrastructure
• Extensive tooling and validation
• Dual purpose: Serves both traditional search and AI systems
• Single source of truth: Updates automatically with content changes

A study by Data World found that LLMs grounded in knowledge graphs achieve 300% higher accuracy versus unstructured data. Schema.org enables creating these knowledge graphs through embedded JSON-LD in HTML, providing a single source of truth rather than requiring manual markdown maintenance.

Clean, Semantic HTML

Many technical experts argue the best optimization is simply clean, semantic HTML with proper structure.

• Clear heading hierarchy
• Logical content organization
• Semantic HTML5 elements
• Server-side rendering
• Concise writing

Modern LLMs are improving rapidly at parsing HTML directly, with expanding context windows (Gemini 1.5 now handles over 1 million tokens) and better vision capabilities for understanding page layout. The core problem llms.txt addresses may be becoming obsolete as model architectures evolve.

The Technical Barriers Nobody Talks About

Beyond adoption questions, llms.txt faces genuine implementation challenges that go deeper than most advocates acknowledge.

The Maintenance Problem

Unlike sitemap.xml, which can be automatically generated by content management systems, llms.txt requires substantial manual oversight.

• Curating which pages are "LLM-worthy"
• Writing and maintaining descriptions
• Keeping Markdown files synchronized with HTML content
• No authoritative schema defining what makes content suitable for AI consumption

This burden hits smaller teams hardest.

The Synchronization Nightmare

If Markdown files fall out of sync with primary web content, LLMs may ingest outdated or misleading data, potentially leading to hallucinations when AI systems confidently provide wrong information.

• No built-in versioning system
• No mechanism to signal when content is stale
• No standardized validation tools comparable to Google's Rich Results Test for Schema markup

The Scale Problem

The llms-full.txt files can become enormous. Cloudflare's is 3.7 million tokens, potentially exceeding the context windows they were designed to fit within. There's no clear guidance on file size limits, how to structure documentation for sites with thousands of pages, or when to split content across multiple files.

The Discovery Problem

Perhaps most fundamentally, there's no discovery mechanism. Unlike robots.txt and sitemap.xml, which crawlers automatically discover and process, LLMs don't spontaneously find and use llms.txt files. They must be manually provided to AI systems, severely limiting their utility for the automatic content retrieval and ranking use cases many implementers hope for.

Three Scenarios for the Future

The future of llms.txt depends entirely on decisions being made right now at OpenAI, Anthropic, and other major providers. Here are the most likely scenarios:

Scenario 1: Gradual Official Adoption (40% probability)

One major platform announces official support in 2026, others follow incrementally, and by 2027-2028 it becomes a standard similar to Open Graph Protocol's adoption curve.

This requires someone breaking ranks first, likely Anthropic or Microsoft, and creating momentum that brings others along. Given Anthropic's early support and Microsoft's experimentation, this scenario remains plausible.

Scenario 2: Evolution and Absorption (30% probability)

llms.txt merges with or is superseded by the Model Context Protocol, becoming one component of a broader AI content framework rather than a standalone standard. A more comprehensive solution emerges that addresses llms.txt's limitations while incorporating its useful elements.

This could happen by 2026-2027 as platforms converge on interoperability standards. The technical community's preference for MCP suggests this path.

Scenario 3: Gradual Fade (20% probability)

llms.txt remains a niche tool used manually by developers but never achieves crawler support from major platforms. Companies stick with existing protocols, the standard becomes a footnote in AI history like numerous other proposed but never-adopted specifications.

Google's rejection and the year of silence from others suggest this possibility. By late 2025 or early 2026, it may become clear this experiment failed.

Scenario 4: Fragmentation (10% probability)

Different AI companies develop competing standards without universal adoption, creating a fractured landscape that benefits no one. This seems least likely given the coordination costs involved.

So Is llms.txt Dead or Alive?

No, llms.txt is not dead—but it exists in a precarious state that cannot persist indefinitely.

The standard has achieved remarkable grassroots success by any measure:

✅ 784+ documented implementations
✅ Active tool development and ecosystem
✅ Major companies like Anthropic and Cloudflare publishing comprehensive files
✅ Continuous community engagement
✅ Genuine enthusiasm from developer documentation teams

But it lacks the single thing that matters most: confirmation that any major LLM provider actually uses these files when retrieving and processing web content.

• Google has explicitly said no
• OpenAI, Anthropic, and Meta remain silent after 12 months
• Conflicting evidence of crawler activity suggests at most experimental testing, not systematic adoption

The standard solves a real problem with an elegant solution, but it may have arrived too early—proposed before platforms committed to using it—or in the wrong form—too simple compared to more sophisticated alternatives like MCP that major providers are actually adopting.

What You Should Do Right Now

Your decision depends on your specific situation.

Implement llms.txt if you're:

• Running developer documentation where AI coding assistant accuracy genuinely matters
• Willing to experiment with emerging standards
• Using platforms like Mintlify that auto-generate files
• Able to automate the creation and maintenance

The cost is low if you can auto-generate files using our free tool, and there's potential upside if adoption materializes.

Skip llms.txt if you're:

• An SEO-focused site expecting immediate traffic gains
• A small team without automated tools
• Needing proven ROI before investing resources
• Running a mainstream consumer website

Prioritize Schema.org structured data, clean semantic HTML, and traditional SEO fundamentals—all of which have confirmed effectiveness—over speculative standards with zero official platform support.

The Bigger Question This Raises

The llms.txt debate reveals broader tensions about how the web should adapt to AI.

Should site owners proactively create AI-friendly content formats in hopes platforms will use them? Or should platforms first commit to supporting standards before asking for implementation work?

Should we trust self-reported content from site owners (llms.txt) or verify everything independently (current AI approach)?

Will the future web be human-readable HTML that AI parses, or will we maintain parallel AI-optimized content streams?

These questions remain unresolved, suspended between optimistic early adopters betting on an AI-first future and skeptical operators waiting for concrete evidence that their efforts will matter.

The Verdict

llms.txt isn't dead—but without official platform commitment soon, it may never truly live either.

The next 12 months will be decisive. Either a major platform breaks the silence and commits, creating momentum for wider adoption, or the standard gradually fades as developers realize their server logs show no meaningful crawler activity.

For now, it exists in limbo: a grassroots success without the institutional support needed for long-term viability. A promising experiment that solved a real problem but may have been proposed at the wrong time or in the wrong form.

The community watches and waits. And the question "Is llms.txt dead?" remains impossible to answer with certainty—because it's neither fully alive nor completely gone.