# writer.nleone.us — A Fiendish Publishing Co. # Deploy to: /robots.txt (site root) # Last reviewed: 2026-08-30 # # Rule of thumb: robots.txt controls CRAWLING, not INDEXING. To keep a page # out of results, use and leave the # page crawlable so the tag can actually be read. Do not do both. # --------------------------------------------------------------------------- # Default: everything open to everyone # --------------------------------------------------------------------------- User-agent: * Allow: / # --------------------------------------------------------------------------- # AI training and answer crawlers # # The catalog pages stay open — that is how the books get found and cited. # The vault reader and the works registry stay closed. read.html draws prose # onto a canvas, works/registry.js is the plain-text source behind it, and # neither is meant to be lifted wholesale into a training corpus. # # To close a bot off entirely, change its Disallow to a single "/". # To open one fully, delete its block so it falls back to the "*" group above. # --------------------------------------------------------------------------- User-agent: GPTBot Disallow: /read.html Disallow: /works/ User-agent: OAI-SearchBot Disallow: /read.html Disallow: /works/ User-agent: ChatGPT-User Disallow: /read.html Disallow: /works/ User-agent: ClaudeBot Disallow: /read.html Disallow: /works/ User-agent: Claude-Web Disallow: /read.html Disallow: /works/ User-agent: Google-Extended Disallow: /read.html Disallow: /works/ User-agent: PerplexityBot Disallow: /read.html Disallow: /works/ User-agent: Applebot-Extended Disallow: /read.html Disallow: /works/ User-agent: meta-externalagent Disallow: /read.html Disallow: /works/ User-agent: Amazonbot Disallow: /read.html Disallow: /works/ User-agent: CCBot Disallow: /read.html Disallow: /works/ User-agent: Bytespider Disallow: / # --------------------------------------------------------------------------- # SEO scrapers and link-graph miners # No upside for a five-page author site, real bandwidth cost on a homelab. # --------------------------------------------------------------------------- User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: DataForSeoBot Disallow: / # --------------------------------------------------------------------------- # Sitemaps # --------------------------------------------------------------------------- Sitemap: https://writer.nleone.us/sitemap.xml Sitemap: https://writer.nleone.us/feed.xml