# How to Open Your Site to ChatGPT, Claude, and Perplexity Search Crawlers
> Names of search and training crawlers, a complete robots.txt example, sitemap, text availability, and citation limitations.
Source: https://ailoot.ai/blog/en/ai-search-crawlers-robots-txt/
Canonical: https://ailoot.ai/blog/en/ai-search-crawlers-robots-txt/
Language: en
Publisher: Ailoot (https://ailoot.ai/)
Translation: guide-144
Revision: sha256-dce5596f05ec396ec33336da23e0f1a15d2e96c495593db298323b7349504a39
You built a website or product, but when someone asks, “recommend a service for this task,” ChatGPT, Claude, and Perplexity don’t mention it. It’s worth checking whether an assistant can read your site and whether access is blocked.
There is no button to “get into an AI answer,” and no one can guarantee a citation. ChatGPT, Claude, and Perplexity have separate crawlers for search: you can allow them to access your site through `robots.txt`. Then check whether they can access the page text and whether your site’s security is getting in the way.
Practical instructions for working with Ailoot are in the [documentation](https://ailoot.ai/docs/).
## What does it mean to “be visible to AI”?
Key takeaway
AI assistants have crawlers for different tasks: some collect websites for search within the assistant, while others collect text to train models. The first type matters for search answers. You can allow a crawler access, but that doesn’t require the assistant to cite you—no one can guarantee a citation.
When someone asks ChatGPT or Perplexity “where can I read about X?” or “which service should I choose?”, the assistant sometimes searches the web and adds links to pages. For your page to make it onto this list of candidates, the assistant’s crawler must have read it and not been denied access.
There are three things to check on your site:
- the crawler can access the site: it isn’t blocked in `robots.txt`, and security measures aren’t preventing access;
- the crawler can see the page text, not an empty shell that the browser fills in;
- the page answers a specific question honestly and clearly, instead of replacing an answer with a list of keywords.
Start with access: if the crawler couldn’t read the page, it won’t get the useful text on it either.
## How is a search crawler different from a training crawler?
Key takeaway
Each major assistant has crawlers with separate tasks. A search crawler collects websites so they can appear in the assistant’s answers and search results. A training crawler collects text to train a model. These settings are independent: you can allow the search crawler and block the training crawler.
Access should be allowed or blocked separately for each crawler. A general decision to “let AI onto the site” doesn’t account for the differences between their tasks.
- **A search crawler** lets your pages appear in the assistant’s answers when a user asks something. Allow it if you want your site to be available for search.
- **A training crawler** collects text that may be used to train future versions of a model. Whether to allow it is up to you; it doesn’t directly affect whether you appear in current search answers.
OpenAI’s documentation gives an example: you can allow a search crawler to crawl your site while blocking the training crawler, so your site’s text isn’t used for training. These two settings are configured independently in the same file.
## What are the names of the ChatGPT, Claude, and Perplexity crawlers?
Key takeaway
For OpenAI, the search crawler is OAI-SearchBot, the training crawler is GPTBot, and ChatGPT-User visits at the user’s request. For Anthropic: Claude-SearchBot (search), ClaudeBot (training), and Claude-User (at the user’s request). For Perplexity: PerplexityBot (search) and Perplexity-User (at the user’s request). Google has a separate Google-Extended token that controls the use of site content for generative AI.
To configure access, you need the exact crawler names (user-agent), which you specify in `robots.txt`. The table lists the names and purposes according to each developer’s official documentation as of September 2026.
| Crawler (user-agent) or token | Belongs to | What it does | Allow for visibility? |
|----|----|----|----|
| `OAI-SearchBot` | OpenAI | Shows websites in ChatGPT search | Yes |
| `GPTBot` | OpenAI | Collects text to train models | Up to you |
| `ChatGPT-User` | OpenAI | Visits a page at the user’s request | Usually yes |
| `Claude-SearchBot` | Anthropic | Indexes websites to improve search quality | Yes |
| `ClaudeBot` | Anthropic | Collects text to train models | Up to you |
| `Claude-User` | Anthropic | Visits at a Claude user’s request | Usually yes |
| `PerplexityBot` | Perplexity | Shows and links to websites in search | Yes |
| `Perplexity-User` | Perplexity | Visits at the user’s request | Usually yes |
| `Google-Extended` | Google | A token controlling the use of site content for Google generative AI | Up to you |
The `ChatGPT-User`, `Claude-User`, and `Perplexity-User` crawlers visit a page at a person’s request in a conversation, but each developer has different access rules. OpenAI says `robots.txt` rules may not apply to `ChatGPT-User`. Perplexity says `Perplexity-User` usually ignores them because the visit was initiated by the user. Anthropic controls `Claude-User` access through `robots.txt`: if you block it there, Claude won’t retrieve pages in response to user requests. Access for search crawlers is configured separately.
`Google-Extended` doesn’t have its own user-agent line when crawling—it is only a control token in `robots.txt`. It determines whether your content can be used for Google generative AI and does not affect regular Google Search.
If you want to build a site with a clear structure and accessible text, think through how to organize the information.
## How to allow crawlers in robots.txt: step by step
Key takeaway
The robots.txt file is in the site root and sets access rules for crawlers. To allow assistants’ search crawlers, add their user-agents with permission to crawl. Each subdomain needs its own file.
`robots.txt` is a plain text file in your site’s root, available at an address like `https://example.com/robots.txt`. Crawlers read it before crawling and follow its instructions, with the exceptions for visits at a user’s request described above.
1. **Open your current file.** Enter your site’s address in a browser and add `/robots.txt`. See what’s already in the file and whether the crawlers you need have accidentally been blocked by a `Disallow: /` line. Also look for `Disallow:` lines for restricted areas—account pages, payment pages, the admin section: you’ll need them in the next step.
2. **Add a group for assistants’ search crawlers.** Each group specifies a crawler name and access rules. To allow crawling, use `Allow: /`. A crawler follows the rules in the group where its name is specified separately; it doesn’t take rules from the general `User-agent: *` group. Restrictions that appear only in `User-agent: *` don’t apply to that crawler. If you have restricted areas, repeat the same `Disallow:` lines in each new group, or they will be open to that crawler.
3. **Make a separate decision about training crawlers.** If you don’t want text used for training, set `Disallow: /` for `GPTBot` and `ClaudeBot`. This won’t affect search visibility.
4. **Save the file in your site’s root** and check that it opens at `/robots.txt`. If you have subdomains (for example, `blog.example.com`), each one needs its own file—crawlers read the file belonging to each subdomain.
Before copying the example, check the restrictions for restricted areas. If you leave `Disallow: /dashboard/` only in `User-agent: *`, it won’t apply to `OAI-SearchBot`, which has its own group. List restricted paths in each group where you allow a crawler to crawl.
In the example below, assistants’ search crawlers are allowed, but the `/dashboard/` restricted area is blocked directly in their groups; training crawlers are blocked entirely. Blocking training crawlers is optional—it’s your decision, and you should replace the list of restricted paths with your own.
```
# Allow assistants' search crawlers, but block restricted areas
User-agent: OAI-SearchBot
Allow: /
Disallow: /dashboard/
User-agent: Claude-SearchBot
Allow: /
Disallow: /dashboard/
User-agent: PerplexityBot
Allow: /
Disallow: /dashboard/
# Block training crawlers (optional)
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
# The sitemap helps crawlers find all pages
Sitemap: https://example.com/sitemap.xml
```
According to OpenAI and Perplexity documentation, their systems may need about a day to take changes to `robots.txt` into account. Don’t expect an immediate effect after saving the file.
## What does this look like on a live site?
In the described example of `robots.txt` settings, the assistant crawler names—`GPTBot`, `ChatGPT-User`, `OAI-SearchBot`, `ClaudeBot`, `PerplexityBot`, `Perplexity-User`, `Google-Extended`, and others—are gathered in one group. Below them are `Allow: /` and the same `Disallow:` lines for restricted areas as in the general `User-agent: *` group: `/dashboard/`, `/admin/`, `/login`, `/account/`, `/order/`, and others. The sitemap URLs are listed at the end of the file using the `Sitemap:` directive.
The restrictions are repeated in the assistant crawler group so restricted areas are blocked for them too. You can open any site’s `/robots.txt` and see how it is configured—the file is always public.
## What else helps crawlers understand a site?
Key takeaway
After configuring access, check whether the page text is visible without running browser scripts, whether there is a sitemap.xml file, and whether the answer itself is clear.
Even when crawling is allowed, a crawler may not get the page content. In addition to access rules, check how the site delivers text and helps people find its content.
- **The text is available immediately.** If scripts render the page content in the browser, some crawlers will see an empty shell. To check, open the page, choose “view page source,” and search for your text. If it isn’t there and appears only in the regular page view, you need to look into how the page is delivered.
- **There is a sitemap.** The `sitemap.xml` file is a list of all your pages. It helps crawlers avoid missing content. Add a link to it in `robots.txt` with a `Sitemap:` line.
- **The answer is clear to a person.** Assistants’ crawlers look for pages that answer a question. A clear heading, a short answer at the beginning, and honest terms work better than a list of keywords.
## Why might a site be invisible to AI?
Key takeaway
There are three causes worth checking: the crawler is blocked in robots.txt, the text is generated by scripts and isn’t visible without a browser, or the site’s security blocks crawlers. In the first case, check the file; in the second, check how pages are delivered; in the third, check the security settings.
If your site doesn’t appear in answers, check for these possible access barriers.
- **The crawler is blocked in `robots.txt`.** Sometimes the file contains a general restriction that blocks the crawlers you need. Check whether there is a `Disallow: /` in a group that applies to assistants’ crawlers.
- **The text isn’t visible without a browser.** If scripts generate the content, the crawler may not get it. The source code check is described above.
- **The site’s security blocks crawlers.** Many sites use protection against unwanted traffic—for example, Cloudflare. It may block access regardless of `robots.txt`. You use `robots.txt` to control where a crawler may go, while the site’s security decides whether to let its request through. Perplexity recommends manually allowing crawlers in the security settings by their user-agent names and published IP addresses. Set crawling rules through `robots.txt`: Anthropic separately warns that blocking crawlers only by IP address is unreliable. If you use a similar service, check its settings.
Check the causes one by one: start with `robots.txt`, then look at the page source code, and after that check the security settings. The absence of these barriers by itself doesn’t guarantee that your site will appear in answers.
## How can you check whether a crawler reached the page?
Key takeaway
There is no direct “I was cited” counter. But you can check the basics: whether robots.txt opens, whether the text is visible in the source code, and whether the sitemap is available. Crawler visits are visible in server logs by their user-agent.
This check helps assess your site’s accessibility and find records of crawler visits.
- Open `https://example.com/robots.txt` in a browser—the file should open and contain the required groups.
- Open the sitemap at `https://example.com/sitemap.xml`—it should open and list pages.
- View the source code of an important page and look for your text in it.
- If you have access to server logs, search them for crawler names—`OAI-SearchBot`, `PerplexityBot`, and others. Their visits are visible by user-agent.
The check won’t show whether the assistant will choose your page in particular. That depends on the user’s question and how well the page answers it.
## What the settings do not guarantee
Access to your site and its appearance in an assistant’s answer are different outcomes. These settings have limitations.
- **Citation is not guaranteed.** Open access only gives a crawler the opportunity to read your site. Whether the assistant cites you is up to the assistant, based on the question.
- **No one promises rankings or traffic.** The `robots.txt` setting opens access, but doesn’t secure a place in search results.
- **Some crawlers may ignore the file’s rules.** This applies to some crawlers that visit in response to a direct user request—for example, OpenAI’s `ChatGPT-User` and Perplexity’s `Perplexity-User`.
- **Rules change.** Developers update crawler names and conditions. It’s worth checking the documentation every few months; links are at the end.
You control access and the quality of your site, but the assistant decides whether to include a link. The goal of these settings is to make a useful page available for search and reading, without trying to “trick” the assistant.
## What next
Open `robots.txt`, add assistants’ search crawlers, and check the sitemap and text availability. If security is blocking access, check its settings too. These actions help identify and remove technical barriers. Then check the pages themselves: do they give clear answers to specific questions from your audience?
### Sources
- [OpenAI: crawler overview (OAI-SearchBot, GPTBot, ChatGPT-User)](https://platform.openai.com/docs/bots)
- [Anthropic: Claude crawlers and how to control access](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity: crawlers and site security settings](https://docs.perplexity.ai/guides/bots)
- [Google: crawler overview and the Google-Extended token](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
Continue exploring Ailoot at [ailoot.ai](https://ailoot.ai/).
## Questions about How to Open Your Site to ChatGPT, Claude, and Perplexity Search Crawlers
### Why can adding a named crawler group reopen paths I blocked under User-agent: *?
A crawler with a matching named group uses that group’s rules rather than inheriting the wildcard group. Repeat the required restricted-path Disallow rules in each named group you allow.
### Does allowing a search crawler guarantee that an assistant will cite my page?
No. Check robots.txt, page-source text, sitemap availability, and server logs for access evidence. Those checks establish accessibility or visits, not that the assistant will select the page for an answer.
## Ailoot · The company and its project portfolio
Ailoot combines established operating experience, a large client base, stable service and ongoing KPI control.
The company and its project portfolio: millions of customers; monthly sales in the billions.
Ailoot brings that operating experience to API access for AI features in customer products, agents, applications and automations.