Status: [LIVE] Endpoints found: https://eusef.github.io/auto-llms-txt/llm https://eusef.github.io/auto-llm Source: https://github.com/eusef/auto-llms-txt # auto-llms-txt Make any personal website machine-readable for LLMs and AI agents. A daily GitHub Action traverses your sitemap, scrapes each page, and publishes a `llms.txt` + a human-friendly landing page to GitHub Pages. Zero maintenance after setup. ## How It Works ``` yoursite.com/llms.txt (301 redirect from your host) -> yourusername.github.io/auto-llms-txt/llms.txt (rebuilt daily) ``` 1. GitHub Actions runs at 6 AM UTC, fetches your sitemap, and scrapes each page 2. `llms.txt` and `llms-full.txt` are committed to `data/` (only if content changed) 3. A landing page + both text files deploy to GitHub Pages 4. Your site's `/llms.txt` 301-redirects to GitHub Pages so it stays current automatically ## Live Example This repo powers [philjohnstonii.com](https://philjohnstonii.com)'s LLM layer: | File | Description | |---|---| | [`/llms.txt`](https://eusef.github.io/auto-llms-txt/llms.txt) | Concise page index — titles, URLs, descriptions | | [`/llms-full.txt`](https://eusef.github.io/auto-llms-txt/llms-full.txt) | Full content of every page | | [`/`](https://eusef.github.io/auto-llms-txt/) | Human-friendly landing page | ## Project Structure ``` your-repo/ ├── config.json # Your site URL and metadata — edit this ├── scraper/ │ ├── scrape.py # Sitemap-driven scraper (no customization needed) │ └── requirements.txt ├── docs/ │ └── index.html # Landing page (auto-populated from llms.txt) ├── data/ # Generated output (committed by the workflow) │ ├── llms.txt │ └── llms-full.txt └── .github/workflows/ └── scrape.yml # Daily scrape + GitHub Pages deployment ``` ## Deploy Your Own ### 1. Fork this repo Fork on GitHub. You can rename it to anything. ### 2. Edit `config.json` ```json { "website_url": "https://yoursite.com", "site_name": "Your Name", "description": "One-line de