1001Ferramentas
๐Ÿค–SEO

robots.txt Generator

Generate robots.txt files for SEO: define User-agent, Allow/Disallow per path and link to sitemap.xml.


    
  

How does robots.txt work?

robots.txt tells crawlers (Google, Bing, etc.) which URLs they can or cannot index. It is not a real protection โ€” just a hint. Malicious bots ignore it.

Place the file at /robots.txt at your domain root. Disallow: /admin/ blocks the crawler; Allow: / opens everything.

Include the sitemap.xml link to speed up indexing.

Related Tools