robots.txt for AI Bots

You realised your writing may be feeding model training and you want that refusal on record. The usual place for it is robots.txt, at the root of your domain. Toggle four targets on or off here: OpenAI's GPTBot, Anthropic's ClaudeBot along with the older anthropic-ai name, Common Crawl's CCBot, and Google-Extended. Each crawler you keep produces two directives, a User-agent line and a Disallow slash line.

robots.txt is a request, not a wall. It works on whoever chooses to honour it, and every company names its crawlers differently. Google-Extended does not remove you from Google Search: it signals only that your content should not feed Gemini training and answers. Blocking CCBot does not erase what Common Crawl archived in earlier passes. And the live fetchers that run when a user asks a question go by names other than these four.

The file lives at yoursite.com/robots.txt and every directive needs its own line: User-agent on one, Disallow on the next, a blank line between blocks. Add the snippet to the robots.txt you already have, without deleting existing rules or the Sitemap line. After deploying, open the address in a browser and read it back. Stopping crawlers that ignore the file is a server or CDN job. The assembly happens in your browser.