The llms.txt Standard

llms.txt is a proposed standard for providing AI systems with a concise, markdown-formatted overview of your site, like a README for AI crawlers.

Proposed Standard

llms.txt is not yet widely adopted by AI systems. While implementing it is a forward-looking best practice, most current AI crawlers (GPTBot, ClaudeBot, etc.) do not specifically look for or use this file. Consider it part of a comprehensive AI optimization strategy rather than a guaranteed solution.

In this guide

What llms.txt is and why it matters
How to structure your llms.txt file
Content recommendations
Integration with other AI signals

8 min read Prerequisite: RSS/Atom Feeds

What is llms.txt?

llms.txt is a plain-text file at your site's root that provides AI language models with essential information about your site, products, and content. Think of it as a combination of robots.txt and a site overview, specifically designed for AI understanding.

Key Characteristics

• Located at /llms.txt (root of domain)
• Written in Markdown format
• Human-readable but optimized for AI consumption
• Concise overview of site purpose and content

Why llms.txt Matters

Efficient Context

AI can quickly understand your site without crawling every page. One file provides essential context.

Accurate Representation

You control how AI understands and describes your business, products, and services.

Token Efficiency

AI systems with limited context windows can get essential information without processing large HTML pages.

llms.txt Structure

A well-structured llms.txt includes these sections:

# Acme Corp

> Acme Corp builds CRM software for small B2B sales teams.

## About

Acme Corp was founded in 2019 in Austin, TX. We help small
businesses manage customer relationships without enterprise
complexity.

## Products

- **Acme CRM** - Contact management and deal tracking ($29/month)
- **Acme Insights** - Sales analytics and reporting ($49/month)

## Key Facts

- Founded: 2019
- Headquarters: Austin, TX
- Customers: 5,000+ small businesses
- Team size: 45 employees

## Important Links

- Homepage: https://acme.com
- Pricing: https://acme.com/pricing
- Documentation: https://docs.acme.com
- Blog: https://acme.com/blog
- Support: https://acme.com/support

## Contact

- Email: [email protected]
- Phone: (512) 555-0100
- Twitter: @acmecorp

Content Guidelines

Be Concise

Keep the file under 2,000 words. AI systems benefit from density over length. Focus on essential information.

Include Facts AI Can Cite

Include specific, citable facts: pricing, founding date, location, customer count, product names. These help AI give accurate answers.

Link to Key Pages

Include URLs to your most important pages. This helps AI and AI-powered tools find the right content quickly.

Keep It Updated

Update llms.txt when key information changes: new products, pricing updates, company milestones.

Extended Version: llms-full.txt

For more comprehensive information, you can also provide /llms-full.txt:

llms.txt

Essential overview (~500-2000 words). Quick context for any AI interaction.

llms-full.txt

Comprehensive details. Product specs, FAQs, detailed descriptions for deep research.

Implementation

Add llms.txt to your site root and reference it in robots.txt:

# robots.txt
User-agent: *
Allow: /

# Point to llms.txt
# LLMs-txt: /llms.txt

Relationship to Other Standards

Standard	Purpose	Relationship to llms.txt
robots.txt	Crawl permissions	Complementary, controls access
sitemap.xml	Page discovery	Lists pages; llms.txt summarizes content
Schema.org	Structured data	Per-page detail; llms.txt is site-wide
RSS/Atom	Content updates	Dynamic content; llms.txt is static overview

Key Takeaway

llms.txt is a forward-looking investment.

While not yet widely supported by AI crawlers, llms.txt is simple to implement and positions your site for future AI systems that may adopt this standard. Combine it with proven methods like structured data and RSS feeds for comprehensive AI visibility.

Sources

llms.txt Specification: Proposed standard specification and examples
llms.txt Examples: Real-world implementation examples