llm.energy

Extract llms.txt documentation and install.md instructions from any website for AI agents, LLMs, and automation workflows.

📖 Overview

llm.energy is a web application and MCP server that fetches, parses, and organizes documentation from websites implementing the llms.txt and install.md standards. It transforms raw documentation into structured, agent-ready formats optimized for large language models, AI assistants, and developer tooling.

Standard	Description	Learn More
llms.txt	Machine-readable documentation for AI systems	llmstxt.org
install.md	LLM-executable installation instructions	installmd.org

✨ Key Features

Feature	Description
🔍 Smart Detection	Automatically finds `llms.txt`, `llms-full.txt`, and `install.md`
📄 Organized Output	Splits content into individual markdown files by section
🤖 Agent-Ready	Includes `AGENT-GUIDE.md` optimized for AI assistants
📦 Multiple Formats	Export as Markdown, JSON, YAML, or ZIP archive
🔌 MCP Server	Integrate with Claude Desktop, Cursor, and more
⚡ Batch Processing	Process multiple URLs simultaneously
📚 Site Directory	Browse 19+ curated llms.txt-enabled websites
✏️ llms.txt Generator	Create your own llms.txt files with a guided wizard
🚀 install.md Generator	AI-powered: generate from GitHub repos, docs URLs, or manually

🎯 Use Cases

─────────────────────────────────────────────────────────────────
                                                                 
   📝 Feed docs to AI coding assistants (Cursor, Windsurf)       
   🤖 Build context-aware AI agents with up-to-date docs         
   🔄 Create documentation pipelines for RAG systems            
   📦 Aggregate docs from multiple sources automatically        
   ✏️  Generate llms.txt/install.md for your own projects       
   🚀 Auto-generate install.md from any GitHub repo             
                                                                 
─────────────────────────────────────────────────────────────────

🚀 install.md Generator

Generate LLM-executable installation instructions from any source:

Mode	Description	Use Case
From GitHub	Analyze any public GitHub repo and generate install.md	Perfect for creating install.md for existing projects
From URL	Extract from any documentation page	Convert existing docs to install.md format
Manual	Build from scratch with guided wizard	Full control over every detail

How It Works

GitHub Mode: Analyzes README, package.json/pyproject.toml/Cargo.toml, GitHub Actions, and releases
URL Mode: Scrapes documentation pages, detects platforms (Mintlify, Docusaurus, GitBook, etc.)
AI Synthesis: Uses Claude to generate a properly formatted install.md

Quick Example

# Generate install.md for any GitHub project
curl -X POST https://llm.energy/api/generate-install \
  -H "Content-Type: application/json" \
  -d '{"url": "https://github.com/anthropics/anthropic-sdk-python", "type": "github"}'

Visit llm.energy/install-generator to use the web interface.

📚 Documentation

Full documentation is available at llm.energy/docs

🚀 Installation

Web Application

Visit llm.energy to use the hosted version.

Local Development

# Clone the repository
git clone https://github.com/nirholas/extract-llms-docs.git
cd extract-llms-docs

# Install dependencies
pnpm install

# Start development server
pnpm dev

🌐 The application runs on http://localhost:3001

MCP Server

📦 Click to expand MCP Server setup

Add to your MCP client configuration (Claude Desktop, Cursor, etc.):

{
  "mcpServers": {
    "llm-energy": {
      "command": "npx",
      "args": ["-y", "@llm-energy/mcp-server"]
    }
  }
}

See MCP Server Documentation for detailed setup.

🔌 API Reference

POST /api/extract - Extract documentation from a URL

curl -X POST https://llm.energy/api/extract \
  -H "Content-Type: application/json" \
  -d '{"url": "docs.anthropic.com"}'

Response includes parsed sections, metadata, and download URLs.

POST /api/validate - Check if URL has llms.txt support

curl -X POST https://llm.energy/api/validate \
  -H "Content-Type: application/json" \
  -d '{"url": "docs.example.com"}'

POST /api/batch - Process multiple URLs

curl -X POST https://llm.energy/api/batch \
  -H "Content-Type: application/json" \
  -d '{"urls": ["docs.anthropic.com", "docs.stripe.com"]}'

GET /api/download - Download in various formats

# Formats: markdown, json, yaml, zip
curl "https://llm.energy/api/download?url=docs.anthropic.com&format=zip"

POST /api/generate-install - AI-generate install.md from GitHub or docs URL

# Generate from a GitHub repository
curl -X POST https://llm.energy/api/generate-install \
  -H "Content-Type: application/json" \
  -d '{"url": "https://github.com/anthropics/anthropic-sdk-python", "type": "github"}'

# Generate from a documentation URL
curl -X POST https://llm.energy/api/generate-install \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.anthropic.com/en/docs/quickstart", "type": "docs"}'

# Analyze only (preview mode)
curl "https://llm.energy/api/generate-install?url=https://github.com/anthropics/anthropic-sdk-python&type=github"

📖 Full API Reference →

🤖 MCP Server Tools

The MCP server exposes the following tools for AI agents:

Tool	Description
`extract_docs`	Extract documentation from a URL with llms.txt support
`validate_url`	Check if a URL has llms.txt available
`verify_llms_txt`	Verify llms.txt exists and get file info
`discover_documentation_urls`	Find documentation URLs for a domain
`list_sites`	Get directory of known llms.txt-enabled sites
`search_sites`	Search the site directory by category or keyword

🏗️ Architecture

┌──────────────────────────────────────────────────────────────────┐
│                        llm.energy                                │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐               │
│  │   Web App   │  │  REST API   │  │ MCP Server  │               │
│  │  (Next.js)  │  │  /api/*     │  │   (stdio)   │               │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘               │
│         │                │                │                      │
│         └────────────────┼────────────────┘                      │
│                          │                                       │
│                   ┌──────▼──────┐                                │
│                   │    Core     │                                │
│                   │  - Parser   │                                │
│                   │  - Extractor│                                │
│                   │  - Cache    │                                │
│                   └──────┬──────┘                                │
│                          │                                       │
│         ┌────────────────┼────────────────┐                      │
│         ▼                ▼                ▼                      │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐               │
│  │  llms.txt   │  │ install.md  │  │   Sitemap   │               │
│  └─────────────┘  └─────────────┘  └─────────────┘               │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

📁 Project Structure

extract-llms-docs/
├── src/
│   ├── app/              # Next.js pages and API routes
│   │   ├── api/          # REST API endpoints
│   │   │   ├── extract/  # Documentation extraction API
│   │   │   ├── generate-install/  # AI-powered install.md generation
│   │   │   └── ...       # Other API endpoints
│   │   ├── extract/      # Extraction interface
│   │   ├── batch/        # Batch processing page
│   │   ├── directory/    # Site directory browser
│   │   ├── generator/    # llms.txt generator wizard
│   │   └── install-generator/  # install.md generator with tabs
│   ├── components/       # React UI components
│   │   └── install-generator/  # GitHubTab, UrlTab, Preview
│   ├── lib/              # Core utilities
│   │   ├── github-analyzer.ts  # GitHub repo analysis
│   │   ├── docs-analyzer.ts    # Documentation URL scraping
│   │   └── ...           # Parser, extractor, cache
│   ├── hooks/            # React hooks
│   └── types/            # TypeScript definitions
├── packages/core/        # Shared parser and types
├── mcp-server/           # MCP server package
└── docs-site/            # MkDocs documentation source

🛠️ Technology Stack

⚙️ Configuration

Environment variables (optional):

RATE_LIMIT_REQUESTS=100     # Max requests per window
RATE_LIMIT_WINDOW_MS=60000  # Window duration in ms
CACHE_TTL=3600              # Cache time-to-live in seconds
ADMIN_KEY=your-secret-key   # Admin API key (required for cache management)

💻 Development

pnpm dev          # 🚀 Start development server
pnpm build        # 📦 Production build
pnpm start        # ▶️  Start production server
pnpm test         # 🧪 Run tests (163 tests)
pnpm test:coverage # 📊 Tests with coverage report
pnpm typecheck    # ✅ TypeScript validation
pnpm lint         # 🔍 ESLint check

🔗 Related Projects

Project	Description
llms.txt Standard	Machine-readable documentation format
install.md Standard	LLM-executable installation format
Model Context Protocol	Protocol for AI tool integration
MCP Servers Directory	Community MCP servers

🌐 Sites with llms.txt Support

Browse 19+ curated websites with verified llms.txt support at llm.energy/directory

Featured sites include:

🤖 Anthropic Documentation
⚡ Vercel Documentation
💳 Stripe API Reference
🗄️ Supabase Docs
📘 Mintlify Documentation
...and more!

🤝 Contributing

Contributions are welcome! Please follow these steps:

🍴 Fork the repository
🌿 Create a feature branch (git checkout -b feature/amazing-feature)
💾 Commit your changes (git commit -m 'Add amazing feature')
📤 Push to the branch (git push origin feature/amazing-feature)
🔀 Open a Pull Request

Adding a site to the directory? Edit src/data/sites.ts and submit a PR.

📄 License

MIT License - see LICENSE for details

🔗 Links

🌐 Website • 📚 Documentation • 💻 GitHub • 📦 NPM

👤 Author

Built by nich

_{Made with ⚡ for the AI community}

Author

Built by nich - https://x.com/nichxbt

Keywords: llms.txt, llms-full.txt, documentation extraction, AI documentation, LLM context, Model Context Protocol, MCP server, AI agents, documentation parser, markdown extraction, API documentation, developer tools, AI coding assistant, RAG, retrieval augmented generation, context injection, AI-friendly documentation, vibe coding, cursor, windsurf, claude, chatgpt, copilot

extract-llms-docs

Documentation