How to Open Your Site to ChatGPT, Claude, and Perplexity Search Crawlers

You built a website or product, but when someone asks, “recommend a service for this task,” ChatGPT, Claude, and Perplexity don’t mention it. It’s worth checking whether an assistant can read your site and whether access is blocked.

There is no button to “get into an AI answer,” and no one can guarantee a citation. ChatGPT, Claude, and Perplexity have separate crawlers for search: you can allow them to access your site through robots.txt. Then check whether they can access the page text and whether your site’s security is getting in the way.

What does it mean to “be visible to AI”?#

When someone asks ChatGPT or Perplexity “where can I read about X?” or “which service should I choose?”, the assistant sometimes searches the web and adds links to pages. For your page to make it onto this list of candidates, the assistant’s crawler must have read it and not been denied access.

There are three things to check on your site:

• the crawler can access the site: it isn’t blocked in robots.txt, and security measures aren’t preventing access;
• the crawler can see the page text, not an empty shell that the browser fills in;
• the page answers a specific question honestly and clearly, instead of replacing an answer with a list of keywords.

Start with access: if the crawler couldn’t read the page, it won’t get the useful text on it either.

How is a search crawler different from a training crawler?#

Access should be allowed or blocked separately for each crawler. A general decision to “let AI onto the site” doesn’t account for the differences between their tasks.

• A search crawler lets your pages appear in the assistant’s answers when a user asks something. Allow it if you want your site to be available for search.
• A training crawler collects text that may be used to train future versions of a model. Whether to allow it is up to you; it doesn’t directly affect whether you appear in current search answers.

OpenAI’s documentation gives an example: you can allow a search crawler to crawl your site while blocking the training crawler, so your site’s text isn’t used for training. These two settings are configured independently in the same file.

What are the names of the ChatGPT, Claude, and Perplexity crawlers?#

To configure access, you need the exact crawler names (user-agent), which you specify in robots.txt. The table lists the names and purposes according to each developer’s official documentation as of September 2026.

The ChatGPT-User, Claude-User, and Perplexity-User crawlers visit a page at a person’s request in a conversation, but each developer has different access rules. OpenAI says robots.txt rules may not apply to ChatGPT-User. Perplexity says Perplexity-User usually ignores them because the visit was initiated by the user. Anthropic controls Claude-User access through robots.txt: if you block it there, Claude won’t retrieve pages in response to user requests. Access for search crawlers is configured separately.

Google-Extended doesn’t have its own user-agent line when crawling—it is only a control token in robots.txt. It determines whether your content can be used for Google generative AI and does not affect regular Google Search.

If you want to build a site with a clear structure and accessible text, think through how to organize the information.

How to allow crawlers in robots.txt: step by step#

robots.txt is a plain text file in your site’s root, available at an address like https://example.com/robots.txt. Crawlers read it before crawling and follow its instructions, with the exceptions for visits at a user’s request described above.

1. Open your current file. Enter your site’s address in a browser and add /robots.txt. See what’s already in the file and whether the crawlers you need have accidentally been blocked by a Disallow: / line. Also look for Disallow: lines for restricted areas—account pages, payment pages, the admin section: you’ll need them in the next step.
2. Add a group for assistants’ search crawlers. Each group specifies a crawler name and access rules. To allow crawling, use Allow: /. A crawler follows the rules in the group where its name is specified separately; it doesn’t take rules from the general User-agent: * group. Restrictions that appear only in User-agent: * don’t apply to that crawler. If you have restricted areas, repeat the same Disallow: lines in each new group, or they will be open to that crawler.
3. Make a separate decision about training crawlers. If you don’t want text used for training, set Disallow: / for GPTBot and ClaudeBot. This won’t affect search visibility.
4. Save the file in your site’s root and check that it opens at /robots.txt. If you have subdomains (for example, blog.example.com), each one needs its own file—crawlers read the file belonging to each subdomain.

Before copying the example, check the restrictions for restricted areas. If you leave Disallow: /dashboard/ only in User-agent: *, it won’t apply to OAI-SearchBot, which has its own group. List restricted paths in each group where you allow a crawler to crawl.

In the example below, assistants’ search crawlers are allowed, but the /dashboard/ restricted area is blocked directly in their groups; training crawlers are blocked entirely. Blocking training crawlers is optional—it’s your decision, and you should replace the list of restricted paths with your own.

# Allow assistants' search crawlers, but block restricted areas
User-agent: OAI-SearchBot
Allow: /
Disallow: /dashboard/

User-agent: Claude-SearchBot
Allow: /
Disallow: /dashboard/

User-agent: PerplexityBot
Allow: /
Disallow: /dashboard/

# Block training crawlers (optional)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# The sitemap helps crawlers find all pages
Sitemap: https://example.com/sitemap.xml

According to OpenAI and Perplexity documentation, their systems may need about a day to take changes to robots.txt into account. Don’t expect an immediate effect after saving the file.

What does this look like on a live site?#

In the described example of robots.txt settings, the assistant crawler names—GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, and others—are gathered in one group. Below them are Allow: / and the same Disallow: lines for restricted areas as in the general User-agent: * group: /dashboard/, /admin/, /login, /account/, /order/, and others. The sitemap URLs are listed at the end of the file using the Sitemap: directive.

The restrictions are repeated in the assistant crawler group so restricted areas are blocked for them too. You can open any site’s /robots.txt and see how it is configured—the file is always public.

What else helps crawlers understand a site?#

Even when crawling is allowed, a crawler may not get the page content. In addition to access rules, check how the site delivers text and helps people find its content.

• The text is available immediately. If scripts render the page content in the browser, some crawlers will see an empty shell. To check, open the page, choose “view page source,” and search for your text. If it isn’t there and appears only in the regular page view, you need to look into how the page is delivered.
• There is a sitemap. The sitemap.xml file is a list of all your pages. It helps crawlers avoid missing content. Add a link to it in robots.txt with a Sitemap: line.
• The answer is clear to a person. Assistants’ crawlers look for pages that answer a question. A clear heading, a short answer at the beginning, and honest terms work better than a list of keywords.

Why might a site be invisible to AI?#

If your site doesn’t appear in answers, check for these possible access barriers.

• The crawler is blocked in robots.txt. Sometimes the file contains a general restriction that blocks the crawlers you need. Check whether there is a Disallow: / in a group that applies to assistants’ crawlers.
• The text isn’t visible without a browser. If scripts generate the content, the crawler may not get it. The source code check is described above.
• The site’s security blocks crawlers. Many sites use protection against unwanted traffic—for example, Cloudflare. It may block access regardless of robots.txt. You use robots.txt to control where a crawler may go, while the site’s security decides whether to let its request through. Perplexity recommends manually allowing crawlers in the security settings by their user-agent names and published IP addresses. Set crawling rules through robots.txt: Anthropic separately warns that blocking crawlers only by IP address is unreliable. If you use a similar service, check its settings.

Check the causes one by one: start with robots.txt, then look at the page source code, and after that check the security settings. The absence of these barriers by itself doesn’t guarantee that your site will appear in answers.

How can you check whether a crawler reached the page?#

This check helps assess your site’s accessibility and find records of crawler visits.

• Open https://example.com/robots.txt in a browser—the file should open and contain the required groups.
• Open the sitemap at https://example.com/sitemap.xml—it should open and list pages.
• View the source code of an important page and look for your text in it.
• If you have access to server logs, search them for crawler names—OAI-SearchBot, PerplexityBot, and others. Their visits are visible by user-agent.

The check won’t show whether the assistant will choose your page in particular. That depends on the user’s question and how well the page answers it.

What the settings do not guarantee#

Access to your site and its appearance in an assistant’s answer are different outcomes. These settings have limitations.

• Citation is not guaranteed. Open access only gives a crawler the opportunity to read your site. Whether the assistant cites you is up to the assistant, based on the question.
• No one promises rankings or traffic. The robots.txt setting opens access, but doesn’t secure a place in search results.
• Some crawlers may ignore the file’s rules. This applies to some crawlers that visit in response to a direct user request—for example, OpenAI’s ChatGPT-User and Perplexity’s Perplexity-User.
• Rules change. Developers update crawler names and conditions. It’s worth checking the documentation every few months; links are at the end.

You control access and the quality of your site, but the assistant decides whether to include a link. The goal of these settings is to make a useful page available for search and reading, without trying to “trick” the assistant.

What next#

Open robots.txt, add assistants’ search crawlers, and check the sitemap and text availability. If security is blocking access, check its settings too. These actions help identify and remove technical barriers. Then check the pages themselves: do they give clear answers to specific questions from your audience?