Robots.txt generator
robots.txt tells crawlers which paths on your site they may fetch; it does not keep a page out of search results on its own. Choose your rules, add your sitemap, decide what each AI crawler may do, and copy or download the file for your site's root.
robots.txt
User-agent: * Allow: / Sitemap: https://www.example.com/sitemap.xml
- Default for all crawlers
- Allow everything
- Crawlers blocked by name
- None
- Crawlers allowed by name
- None
- Sitemap
- https://www.example.com/sitemap.xml
- File size
- 69 bytesGoogle reads up to 500 KiB.
- Upload it as https://www.example.com/robots.txt: crawlers only read robots.txt at the root of a host, and each subdomain needs its own file.
How does a robots.txt file work?
Before crawling a site, a crawler that follows the Robots Exclusion Protocol (RFC 9309, which Google supports) fetches /robots.txt from the root of the host. It looks for the group whose User-agent line names it, and only if there is none does it use the User-agent: * group. Google's crawlers never combine the two, so a crawler with its own group ignores everything in the * group; that is why an allowed crawler's group here repeats your disallowed paths.
Inside a group, the rule with the longest matching path wins, and when an Allow and a Disallow rule match equally, the Allow rule is used. Paths are case-sensitive, * matches any run of characters and a $ at the end means the URL must end there. Disallow: / matches every URL on the site.
robots.txt is a request, not a lock. RFC 9309 says its rules are not a form of access authorization, and Google says it is up to each crawler to obey them. Well-behaved crawlers do; OpenAI says its rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page.
Blocking a URL stops crawling, not indexing. Google says a disallowed URL can still be indexed if other pages link to it, so keep a page out of search with a noindex rule on a page crawlers are allowed to fetch.
User-agent: * ← every crawler without its own group Disallow: /admin/ ← don't fetch URLs starting /admin/ User-agent: GPTBot ← this crawler reads only this group Disallow: / Sitemap: https://www.example.com/sitemap.xml
Which crawlers can a robots.txt file control?
| Token | Operator | What the operator says it does | What a robots.txt rule does |
|---|---|---|---|
| GPTBot | OpenAI | Collects content that may be used to train OpenAI's generative AI foundation models. | Blocking it is OpenAI's signal not to use your content for that training. It is set independently of OAI-SearchBot. |
| OAI-SearchBot | OpenAI | Finds sites to show in ChatGPT's search features. | Sites that block it are not shown in ChatGPT search answers, though they can still appear as navigational links. |
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT user or a Custom GPT asks for it. | OpenAI says robots.txt rules may not apply, because a user started the visit. |
| ClaudeBot | Anthropic | Collects web content that could contribute to training Anthropic's generative AI models. | Anthropic says its bots honour standard robots.txt directives. |
| Claude-SearchBot | Anthropic | Reads web content to improve the quality, relevance and accuracy of search results for users. | Anthropic says its bots honour standard robots.txt directives. |
| Claude-User | Anthropic | Fetches a page when a Claude user asks a question that needs it. | Anthropic says its bots honour standard robots.txt directives. |
| Google-Extended | Not a crawler: a robots.txt token that controls whether content Google crawls may be used to train Gemini models and for grounding. | Google says it does not affect inclusion or ranking in Google Search. Crawling is done by Google's existing user agents. | |
| Applebot-Extended | Apple | Not a crawler: a robots.txt token that controls whether Apple may use your content to train its generative AI models. | Apple says pages that disallow it can still be included in search results. |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity's search results. Perplexity says it is not used to crawl content for AI foundation models. | Perplexity recommends allowing it in robots.txt if you want to appear in its search results. |
| Perplexity-User | Perplexity | Fetches a page when a Perplexity user asks a question. | Perplexity says it generally ignores robots.txt, because a user requested the fetch. |
| CCBot | Common Crawl | Crawls the web for Common Crawl's open repository of web crawl data. | Common Crawl's documented way to stop it is a robots.txt group with Disallow: /. |
How to create a robots.txt file
- 1Choose the defaultAllow all or block all. This becomes the User-agent: * group that every crawler without its own group follows.
- 2List the paths to keep crawlers out ofType one path per line, starting with /, such as /admin/ or /*.pdf$. Blank lines are ignored.
- 3Add your sitemap and pick a rule per crawlerPaste the full sitemap URL, then set each crawler to no rule, allow or block.
- 4Copy or download the fileSave it as robots.txt at the root of your site, then open https://your-domain/robots.txt to check it loads.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT answers?
Where do I put robots.txt?
What does Disallow: / do?
Will robots.txt keep a page out of Google search results?
Does blocking Google-Extended affect my Google Search ranking?
Sources
- RFC 9309: Robots Exclusion Protocol (IETF) (checked 20 Oct 2026)
- Google: How Google interprets the robots.txt specification (checked 20 Oct 2026)
- Google: Introduction to robots.txt (checked 20 Oct 2026)
- Google: Create and submit a robots.txt file (checked 20 Oct 2026)
- Google: Block search indexing with noindex (checked 20 Oct 2026)
- Google: Google's common crawlers (Google-Extended) (checked 20 Oct 2026)
- OpenAI: Overview of OpenAI crawlers (checked 20 Oct 2026)
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (checked 20 Oct 2026)
- Apple: About Applebot (checked 20 Oct 2026)
- Perplexity: Perplexity crawlers (checked 20 Oct 2026)
- Common Crawl: CCBot (checked 20 Oct 2026)
Need a custom dev tool for your team?
We build internal tools, dashboards, APIs, and SaaS platforms. If you can describe it, we can ship it.
Built by CountryDevs · No signup · No ads · We count visits, never what you enter