robots.txt & Sitemap
Build a robots.txt with per-crawler rules and an AI-crawler preset (list as of October 2026), and an XML sitemap. Google ignores sitemap priority and changefreq.
User-agent rules, a block-AI-crawlers preset (as of October 2026), a sitemap URL and crawl-delay. robots.txt is advisory: well-behaved bots honour it, others ignore it.
The "Block AI Bots" list is as of October 2026. User agents were checked against the vendors' own pages: OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-User, Claude-SearchBot), Perplexity, Amazon, Apple, Google, Meta and Common Crawl. Bytespider, FacebookBot, cohere-ai, Omgilibot and the retired anthropic-ai and Claude-Web names are listed by Known Agents, not by the vendor. Vendors add and rename crawlers often, so re-check it. robots.txt is advisory: it does not stop bots that ignore it.
Presets
User-Agent Rules
Global Settings
Generated robots.txt
An XML sitemap; paste URLs in bulk. Google ignores priority and changefreq, and lastmod is only useful when it is accurate, so it starts blank.
Google ignores priority and changefreq. lastmod starts blank on purpose: only add it where it is the real modification date, since a blanket today date misleads crawlers.
Default values (applied to new URLs)
Add URLs
URL list 0 URLs
Generated XML