The short answer
For most B2B SaaS companies, no, not across the board. AI crawlers do different jobs: some collect training data, some power search answers, and some fetch pages when a user asks. You can opt out of training and still allow the search crawlers that decide whether ChatGPT, Claude, and Perplexity can cite you.
Key takeaways
- AI crawlers serve distinct purposes, including training, search indexing, and real-time user-triggered page fetching.
- Blocking search crawlers may result in your content being excluded from AI-powered search answers, while blocking training crawlers only impacts how your content is used to develop models.
- SaaS companies should explicitly audit their robots.txt and CDN settings to ensure they haven't accidentally blocked search visibility.
Publishers have good reasons to limit how AI companies use their content. B2B SaaS companies usually have the opposite problem: they want to be found, quoted, and recommended.
The trouble is that many sites are blocking more than they mean to. In July 2025, Cloudflare began blocking AI bots by default on the sites it hosts, with options for customers to allow specific crawlers.1 If your site sits behind Cloudflare and nobody has checked those settings, you may be blocking search crawlers without knowing it.
What do AI crawlers actually do?
They fall into three jobs. The job decides whether blocking hurts you.
- Training crawlers collect content that may be used to train AI models. Blocking them is a signal that your content shouldn't be used for training.
- Search crawlers index pages so an AI product can find and cite them in answers. Blocking them can remove you from those answers.
- User-triggered fetchers visit a page because a person asked the AI to look at it. Some of these don't follow robots.txt, because a user started the request.
| Company | Crawler | Job | What blocking it does |
|---|---|---|---|
| OpenAI | GPTBot | Training | Signals your content shouldn't be used to train OpenAI's models2 |
| OpenAI | OAI-SearchBot | Search | Your pages won't be shown in ChatGPT search answers, though they can still appear as navigational links2 |
| OpenAI | ChatGPT-User | User fetches | OpenAI notes robots.txt rules may not apply2 |
| Anthropic | ClaudeBot | Training | Excludes your future content from Anthropic's training data3 |
| Anthropic | Claude-SearchBot | Search | May reduce your visibility and accuracy in Claude's search results3 |
| Anthropic | Claude-User | User fetches | Claude can't retrieve your pages when users ask, which may reduce visibility3 |
| Perplexity | PerplexityBot | Search | Perplexity says this bot surfaces and links sites in its results, and isn't used to train foundation models4 |
| Perplexity | Perplexity-User | User fetches | Perplexity says it generally ignores robots.txt, because a user triggered the request4 |
| Google-Extended | Gemini training and grounding | Google says it doesn't affect inclusion or ranking in Google Search5 |
What happens if you block them?
Block a training crawler, and you lose nothing in search visibility. Block a search crawler, and you're opting out of that engine's answers.
Google is the special case. Google-Extended controls whether your content is used for Gemini training and some grounding, and Google says it doesn't affect your inclusion or ranking in Google Search.5 Google's AI Overviews and AI Mode draw on pages indexed by Google Search, so blocking Googlebot itself is what would remove you, and almost no SaaS company should do that.6
What setup do most B2B SaaS companies want?
Allow search, decide on training. Here's a starting point that allows the search crawlers and opts out of training. If you're happy for your marketing pages to be used in training, simply leave out the training blocks.
User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: /
There's a fair argument for allowing training too. A model that learned your category from your own pages may describe you more accurately. That's a judgment call, not a technical one.
How do you audit your setup?
Open yoursite.com/robots.txt
List every AI user agent it mentions, and what each line allows or blocks.
Check your CDN and firewall
Look for AI bot blocking settings, especially if you use Cloudflare.
Match each crawler to its job
Use the table above to see whether you're blocking search, training, or both.
Decide on training as a business
Make it a deliberate choice, written down, not a default.
Add a robots.txt review to every website release checklist. Developers often copy a "block all bots" template from staging into production.
The goal isn't to block AI or welcome it blindly. It's to choose, crawler by crawler, which answers you want to be part of.
Read next: How does ChatGPT search choose its sources?
The bottom line
The goal isn't to block AI or welcome it blindly. It's to choose, crawler by crawler, which answers you want to be part of.
Sources
- MIT Technology Review, Cloudflare will now, by default, block AI bots from crawling its clients' websites, July 1, 2025. www.technologyreview.com/2025/07/01/1119498/cloudflare-will-now-by-default-block-ai-bots-from-crawling-its-clients-websites/
- OpenAI, Overview of OpenAI crawlers, accessed October 13, 2026. developers.openai.com/docs/bots
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?, accessed October 13, 2026. support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Perplexity, Perplexity crawlers, accessed October 13, 2026. docs.perplexity.ai/guides/bots
- Google Search Central, Google's common crawlers, accessed October 13, 2026. developers.google.com/search/docs/crawling-indexing/google-common-crawlers
- Google Search Central, AI features and your website, last updated December 10, 2025. developers.google.com/search/docs/appearance/ai-features
About the author
Aimee van der Haar
Aimee founded Bearly Creative to give great startups and tech companies the marketing they deserve. She works across strategy, design, content, and AI search, and she still does the work herself.
