robots.txt Generator

How does robots.txt work?

robots.txt tells crawlers (Google, Bing and the rest) which URLs to index or skip. Don't treat it as real protection, though. It's only a hint, and malicious bots ignore it outright.

The file has to live at /robots.txt, in your domain root. Use Disallow: /admin/ to keep the crawler away from a path, and Allow: / to leave everything open.

Pointing to your sitemap.xml inside it helps indexing happen faster.

What is robots.txt

The robots.txt file is a plain-text document placed at the root of a website (for example, https://example.com/robots.txt) that tells automated crawlers which paths they may or may not request. The convention was created by Dutch engineer Martijn Koster in 1994, after a misbehaving spider overloaded his servers, and quickly became a de facto standard known as the Robots Exclusion Protocol (REP). For nearly three decades it lived as an informal agreement implemented differently by each search engine. That changed in September 2022, when the IETF published RFC 9309, which finally formalised the protocol's syntax, parsing rules, and error semantics.

Despite its simplicity, robots.txt is one of the most consequential files on a website. A single misplaced character can deindex an entire domain or, conversely, expose private staging areas to the open web. Because the file is purely advisory, it depends on the goodwill of well-behaved bots: Googlebot, Bingbot, DuckDuckBot, Yandex, and most major AI crawlers honour it, but malicious scrapers routinely ignore the rules. For that reason robots.txt should be treated as a crawl-budget and discovery tool, never as a security mechanism.

Syntax and supported directives

A robots.txt file is organised into groups. Each group starts with one or more User-agent lines that identify which crawler the rules apply to, followed by Allow and Disallow directives. The wildcard User-agent: * targets every bot that does not have its own specific group.

User-agent: *
Disallow: /admin/
Disallow: /cart
Allow: /admin/public/

User-agent: Googlebot
Allow: /

Sitemap: https://example.com/sitemap.xml

• User-agent: identifies the crawler (product token). RFC 9309 requires case-insensitive matching on this field.
• Disallow: forbids access to URLs that begin with the given path. An empty Disallow: means "nothing is disallowed".
• Allow: explicit permission that can override a broader Disallow when the path is more specific.
• Sitemap: an absolute URL pointing to a sitemap or sitemap index. It is independent of any user-agent group and may appear anywhere in the file.
• Crawl-delay: a non-standard directive that suggests seconds between requests. Bing and Yandex honour it; Googlebot ignores it.

RFC 9309 also fixes practical details: the file must be UTF-8 encoded plain text served as text/plain, Google enforces a maximum size of 500 KiB, and the cache lifetime should not exceed 24 hours unless the file becomes unreachable. When the server returns a 4xx status the crawler may treat the site as fully allowed; on 5xx errors it must assume a complete disallow until the file is reachable again.

Block crawl vs block indexing

The single most common misunderstanding about robots.txt is that "Disallow" hides a page from search results. It does not. Disallow blocks crawling, not indexing. If Google or Bing discovers the URL through external links, they may still list it in search results with no snippet, showing only the URL and an automatically generated title. Worse, because the crawler is forbidden from reading the page, it cannot see any <meta name="robots" content="noindex"> tag you placed inside it.

To remove a page from the index you must let the bot fetch it and serve a noindex directive — either as a meta tag in the HTML or as an X-Robots-Tag HTTP header (useful for PDFs and images). Only once the page has been re-crawled and dropped from the index should you add a robots.txt Disallow to save crawl budget. Reversing this order leaves URLs frozen in the index indefinitely.

Wildcards and pattern matching

Although RFC 9309 only standardises literal prefix matching, all major search engines accept two wildcards added by Google's extension:

• * matches zero or more characters of any kind. Disallow: /*?session= blocks any URL containing the ?session= parameter.
• $ anchors the pattern to the end of the URL. Disallow: /*.pdf$ blocks every PDF.

Matching is case sensitive on the path portion: /Admin/ and /admin/ are different rules. When two rules conflict, the longest (most specific) wins; if both have the same length, Google applies the least restrictive interpretation, which usually means Allow wins. Wildcard support varies among lesser-known crawlers, so avoid relying on complex patterns when targeting niche bots.

Well-known bots and AI crawlers

Some user-agents you will encounter in production:

• Googlebot, Googlebot-Image, Googlebot-News, AdsBot-Google — Google's search and ads crawlers.
• Bingbot, AdIdxBot — Microsoft Bing.
• DuckDuckBot, YandexBot, Baiduspider, Applebot — other major search engines.
• GPTBot — OpenAI training crawler.
• OAI-SearchBot, ChatGPT-User — ChatGPT browse and live search agents.
• Google-Extended — toggles Google's use of your content for Gemini and Vertex AI training (does not affect Search).
• ClaudeBot, anthropic-ai, Claude-Web, Claude-User, Claude-SearchBot — Anthropic's training and live-fetch agents.
• CCBot — Common Crawl, the dataset most LLMs train on.
• PerplexityBot, Bytespider, Amazonbot, Meta-ExternalAgent — other AI/scraper agents.

To block AI training crawlers while keeping classic search visible, target the AI bots specifically:

User-agent: GPTBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: anthropic-ai
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

Note that companies usually run several bots: blocking ClaudeBot stops training but not Claude-SearchBot or Claude-User; blocking GPTBot does not stop OAI-SearchBot. Decide per use case whether you also want to block live-fetch agents that retrieve content on behalf of a human asking a question.

Common mistakes

• Treating robots.txt as access control. The file is a polite request, not a firewall. Protect sensitive areas with HTTP authentication, IP allowlists, or app-level checks.
• Combining Disallow with noindex. If the URL is disallowed, the crawler cannot see your noindex meta tag, so the URL stays in the index. Use one or the other, in the correct order.
• Forgetting the Sitemap line. Adding Sitemap: in robots.txt is the simplest way to advertise the sitemap to every search engine without manual submission.
• Wrong encoding or BOM. A UTF-16 file or one starting with a byte-order mark may be rejected. Save as plain UTF-8 without BOM.
• Relative sitemap URLs. The Sitemap directive requires an absolute URL including scheme and host.
• Overusing wildcards. Long, complex patterns are fragile and slow. Prefer a handful of clear prefix rules.
• Disallowing the entire site by accident. A stray Disallow: / under User-agent: * can deindex everything within a week.

FAQ

Where should the robots.txt file be placed?

At the root of each host and protocol: https://example.com/robots.txt. Subdomains and HTTP versus HTTPS need their own files.

Will Google still index a disallowed URL?

Possibly. If other sites link to it, Google may show the URL with no snippet. To suppress it, allow crawling and add a noindex directive instead.

Does Googlebot honour Crawl-delay?

No. Google ignores Crawl-delay; adjust crawl rate via Search Console. Bing and Yandex do support it.

How big can robots.txt be?

Google parses the first 500 KiB and discards the rest. Keep the file lean and let your CMS handle dynamic exclusions via meta tags.

Is the order of rules important?

Within a group, the most specific match wins regardless of order. Across user-agent groups, a crawler picks the most specific group that matches its name and ignores all others.