Robots.txt for AI crawlers is a small text file at the root of your website that tells automated bots which pages they may crawl — and in the AI era, it is the main tool for separating search from training. The direct answer: allow AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, because they are the bots that fetch your pages so AI assistants can cite them in answers. Block AI training crawlers such as GPTBot, ClaudeBot and Google-Extended if you do not want your content used to train models. But know the limits: robots.txt cannot remove a page from the index, cannot force an AI tool to cite you, and cannot stop bots that ignore it or publish no user agent at all. This guide explains the three bot types, the exact rules that work, and how to check your changes — everything you need to set up robots.txt for AI crawlers on your own site.
The facts in this guide come from Search Engine Journal’s Robots.txt Guide – Best Practices For Search & AI Crawlers by Vahan Petrosyan (18 November 2026), and from the AI-agent findings in Search Engine Journal’s SEO Pulse (21 November 2026). The guest-posting angle — why the sites you earn links on need to stay visible to AI search crawlers — is our own.
What is robots.txt — and what can it actually do?
Robots.txt is a plain text file that lives at the root of your domain, for example at vonixco.com/robots.txt, and it tells crawlers which URLs they are allowed to fetch. It is a request, not a lock: well-behaved bots follow it, but nothing in the file physically stops a bot from visiting your site.
The file is built from just four directives. User-agent names the bot a rule applies to. Disallow tells that bot which paths it may not crawl. Allow makes an exception inside a disallowed area. And Sitemap points crawlers to your sitemap so they can find your pages.
Matching follows a small set of rules that are easy to get wrong. Only two wildcards exist: *, which matches zero or more characters, and $, which marks the end of a URL. URL path matching is case-sensitive, so /Blog/ and /blog/ are different paths, while directive names and user-agent matching are case-insensitive.
When several rules could apply to the same bot and the same URL, precedence decides. Bot-specific rules beat the catch-all User-agent: * group. Within one group, the most specific rule wins — a long, exact path beats a short, broad one. And when two rules tie, the least restrictive rule wins, so Allow beats Disallow at equal specificity.
What are the three kinds of AI bots?
The single most important idea in the AI-era robots.txt guide is that “AI crawler” is not one thing. There are three separate bot types, and they do three different jobs. Blocking the wrong one is the most common mistake: block a search crawler and your content disappears from AI answers.
Search crawlers fetch your pages so an AI assistant can cite them in its answers. OAI-SearchBot, Claude-SearchBot and PerplexityBot belong here — they retrieve content for ChatGPT search, Claude search and Perplexity answers respectively.
Training crawlers collect content to feed model training. GPTBot, ClaudeBot and Google-Extended belong here. Blocking them affects training use only, not search citations.
User-triggered fetchers visit a page because a specific user asked the assistant to look at it — for example, pasting a link into a chat and asking for a summary. ChatGPT-User is the best-known example, and several of these fetchers may ignore robots.txt entirely.
| Bot type | What it does | Example tokens | Should you block it? |
|---|---|---|---|
| Search crawlers | Fetch pages so AI assistants can cite them in answers | OAI-SearchBot, Claude-SearchBot, PerplexityBot | No — blocking removes your content from AI answers |
| Training crawlers | Collect content to train AI models | GPTBot, ClaudeBot, Google-Extended | Your choice — blocking does not affect search or citations |
| User-triggered fetchers | Visit a page when a user asks the assistant to look at it | ChatGPT-User, Claude-User, Perplexity-User | Usually no — and several may ignore robots.txt anyway |
The scale of the decision is visible in the numbers. A BuzzStream study reported by Search Engine Journal in January 2026 found that 79% of the US and UK news sites examined disallow at least one training crawler, while 71% disallow at least one search or retrieval crawler. That second figure is the worrying one: nearly three quarters of those news sites are also blocking the bots that would bring their reporting into AI answers.
Which AI bots should I allow — and which should I block?
Start from what you want the outcome to be, not from a list of bots to fear. If you want your content cited in AI answers, allow the search crawlers. If you do not want your content used for model training, block the training crawlers. Those are two separate decisions, and the file lets you make them separately.
OpenAI’s setup is the clearest example. OpenAI uses OAI-SearchBot to surface websites in ChatGPT search answers, while ChatGPT-User handles user-requested visits. You can allow OAI-SearchBot and ChatGPT-User while blocking GPTBot. The same pattern holds for Anthropic: allow Claude-SearchBot and Claude-User while blocking ClaudeBot. PerplexityBot and Perplexity-User work the same way.
Google and Apple use a different mechanism that is easy to misunderstand. Google-Extended and Applebot-Extended are not separate crawlers at all — they are tokens that control training use. Google-Extended covers Gemini training and specified grounding uses, and blocking it does not block Googlebot or affect ordinary Google Search inclusion. Applebot-Extended controls foundation-model training without blocking Applebot’s search crawl. Because neither token is a separate crawler, these rules do not reduce crawling requests by themselves: Googlebot and Applebot will still visit your site exactly as before. Our guide to AI bot management and block lists covers how these tokens fit into a wider bot-control setup.
As a default, allow the search crawlers and the user-triggered fetchers, and make a deliberate decision about each training crawler. There is no single correct answer — but blocking everything because the bots were never separated in the first place is the one clear mistake.
How does robots.txt for AI crawlers work in practice?
Writing the file is the easy part. A robots.txt file is a few lines of text, and the example below shows the pattern most sites will want: search bots allowed, training bots blocked, each named by its own token. Read it as an example to adapt, not as a rule to copy blindly.
# Example: allow AI search, block AI training — adapt to your own site User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: /
A few notes on that example before you use it. First, the user-triggered fetchers are listed as allowed, but remember that several of them may ignore robots.txt anyway — the allow lines are there for clarity and for the bots that do honour the file, not because the file controls them. Second, remember the precedence rules: these bot-specific groups beat any User-agent: * group in the same file, so a broad allow for everyone does not undo a specific block for GPTBot.
To write your own version: decide your goal first, write an allow group for each search crawler you want, a disallow group for each training crawler you want to refuse, check the case of every path, then upload the file to the domain root and validate it as the next sections describe.

Can robots.txt block Meta’s Muse?
No — and this is the sharpest limit of the whole system. Meta’s Muse, the AI agent that browses and acts on the web on a user’s behalf, publishes no user agent for a robots.txt rule to name. There is no token to write a rule against, so robots.txt cannot stop it. If your robots.txt strategy assumes every AI visitor can be named and controlled, Muse is the exception that breaks the assumption.
The scale is not small. According to HUMAN Security’s October report, covered in Search Engine Journal’s SEO Pulse on 21 November 2026, Meta’s and OpenAI’s agents made around 90% of agent requests on protected sites and apps — Muse accounted for 59% on its own, and three OpenAI agents made up around 30% combined.
Publishers are responding with contracts and terms instead of text files. United Airlines updated its terms on 11 November 2026 to require written permission for AI agents that search, book or buy, and to bar using its data to train or fine-tune models — while, as of 14 November, United’s robots.txt named no AI crawlers.
The same logic applies to the user-requested bots in general. Whether each one honours robots.txt varies: ChatGPT-User may not apply it, Perplexity-User generally ignores it, Claude-User honours it, Google-Agent does not, and meta-externalfetcher does not. Because robots.txt only tells bots which URLs they may crawl, it was never designed to protect private content — if a page must stay private, use authentication.
What does robots.txt NOT do?
Robots.txt is widely misunderstood, and the misunderstandings cause real damage. The most important one: robots.txt does not remove a page from the index. Disallowing a page stops crawlers from visiting it, but if Google already knows about the page — from a link, a sitemap, or an earlier crawl — it can still appear in search results, often as a bare URL with no description. To remove a page, use noindex, and keep the page crawlable so Google can read the noindex instruction.
The same pattern applies to files. To block PDFs from being crawled, the rule is Disallow: /*.pdf$ — the * matches any file name and the $ anchors the match to the end of the URL. But if the goal is to remove a PDF from the index, the tool is X-Robots-Tag: noindex, and the PDF must stay crawlable so the tag can be read. Blocking and removing are different jobs, and they need different tools.
Two more limits deserve a plain statement. Disallowing training bots cannot guarantee your content stays out of training datasets, because crawlers that ignore robots.txt can still collect it — the file is a voluntary protocol, and only well-behaved bots follow it. And blocking or allowing bots does not force AI tools to visit or cite your site: robots.txt controls permission, not attention. Our guide to technical SEO for AI search explains what makes a page worth citing once the crawlers are allowed in.

How do I check my robots.txt changes worked?
A robots.txt change that nobody validates is a guess. Google Search Console gives you three ways to check, and they are worth using after every critical change. First, open Settings and then the robots.txt report: it shows the robots.txt files Google has fetched, along with parsing warnings and errors, so you can see whether Google is reading the file you think you published.
Second, use URL Inspection to test whether an individual page is blocked. Enter the exact URL and the tool will tell you whether Googlebot can crawl it under your current rules — the fastest way to confirm that a specific allow or disallow is doing what you intended. After a critical change, request a recrawl from the robots.txt report so Google picks up the new version promptly. Third, for a deeper check, Google’s open-source robots.txt parser is available on GitHub, and you can run your file through it to catch matching and precedence mistakes before they go live.
Check the file itself, too. Fetch your own robots.txt in a browser and read it as a visitor would: confirm it is at the domain root and confirm the tokens are spelled correctly. A single mistyped token name means the rule applies to nobody.
What does VonixCo’s own robots.txt look like today?
Checked today, VonixCo’s own robots.txt is the WordPress default: only /wp-admin/ is blocked, with /wp-admin/admin-ajax.php allowed, plus a Sitemap line pointing to https://vonixco.com/sitemap_index.xml. There are no AI crawler rules in it at all, so the site currently allows all AI crawlers, including training crawlers.
That is the default allows-everything starting point, and it is worth stating plainly rather than treating it as a mistake. For a site that wants to be cited in AI answers, allowing the search crawlers is exactly right — OAI-SearchBot, Claude-SearchBot and PerplexityBot can all fetch VonixCo’s articles and cite them. The open question is only the training side: whether the site’s content should also be available to GPTBot, ClaudeBot and Google-Extended for model training. That is a business decision to make deliberately, not a default to drift into.
When should you hire a service like VonixCo?
Robots.txt work is easy to do yourself on one site: read the file, decide search versus training, write the rules, and validate them in Search Console. It becomes a different job when the question is the sites your links live on: a guest post only earns a citation if the publisher’s site is visible to AI search crawlers in the first place.
That is where the guest-posting angle matters. A placement on a site that blocks OAI-SearchBot or PerplexityBot will not be fetched for AI answers, however good the article is — and a placement on a site whose robots.txt is a careless block-everything file may be invisible to the very systems you are trying to reach. Choosing publishers is therefore partly a robots.txt check now: does the site allow the search crawlers, does it keep its content crawlable, and does it avoid the broad disallows that cost it citations? Our guide to what citation core means in AI search explains how citations flow from a visible page to an AI answer.
VonixCo is a niche-relevant guest posting and SaaS link building service based in Lahore, Pakistan; services are SaaS guest posts, niche guest posts, link building and guest post content writing; plans are $500 / $950 / $1,800 per month with content included and a live link report; contact info@vonixco.com and WhatsApp +92 340 0096621. A service like VonixCo fits when you want guest posting handled as an ongoing operation — finding relevant publishers, checking that their sites are genuinely visible to search and AI search crawlers, writing the content, and placing it — rather than doing the publisher checks yourself one site at a time.
Frequently asked questions about robots.txt for AI crawlers
What is robots.txt for AI crawlers?
Robots.txt for AI crawlers is the use of your site’s robots.txt file to control which AI bots may crawl your pages. It uses the same four directives as ordinary robots.txt — User-agent, Disallow, Allow and Sitemap — with one rule per bot token, so you can allow AI search crawlers while blocking AI training crawlers.
Which AI bots should I allow in robots.txt?
Allow the AI search crawlers — OAI-SearchBot, Claude-SearchBot and PerplexityBot — if you want your pages cited in AI answers, since those are the bots that fetch content for ChatGPT search, Claude search and Perplexity. Allow user-triggered fetchers such as ChatGPT-User too.
Which AI bots should I block in robots.txt?
Block the training crawlers if you do not want your content used to train models: GPTBot, ClaudeBot and Google-Extended, plus Applebot-Extended for Apple’s foundation-model training. Blocking them does not block search citations and does not affect ordinary Google Search inclusion, because training use and search use are controlled separately.
Can robots.txt block Meta’s Muse?
No. Meta’s Muse publishes no user agent for a robots.txt rule to name, so there is no token to write a rule against and robots.txt cannot stop it. Muse accounted for 59% of agent requests on protected sites and apps in HUMAN Security’s October report. Use terms of service and authentication for content that must stay out of an agent’s reach.
Does robots.txt remove a page from Google?
No. Disallowing a page stops crawlers from visiting it, but Google can still list the URL if it knows about it from a link or a sitemap. To remove a page, use noindex and keep the page crawlable so Google can read the instruction; for PDFs, use X-Robots-Tag: noindex and keep the file crawlable.
How do I check whether my robots.txt rules are working?
Use Google Search Console: the robots.txt report under Settings shows the files Google fetched and any parsing warnings or errors, and you can request a recrawl after a critical change. Use URL Inspection to test whether an individual page is blocked, and run your file through Google’s open-source robots.txt parser on GitHub to catch matching mistakes before they go live.
Conclusion
Robots.txt for AI crawlers is a small file with a large job: it is where you decide, in public and in plain text, which AI systems may read your site and for what purpose. Allow the search crawlers so your pages can be cited, make a deliberate choice about the training crawlers, and remember what the file cannot do — it cannot remove a page, guarantee a citation, or stop a bot that publishes no user agent. Read your own file today and make the search-versus-training decision on purpose instead of inheriting the default.