A crawler probably read your website this morning, copied the text, and helped train somebody’s AI model. Blocking AI training bots used to force an ugly choice: protect your content or keep your Google traffic. That tradeoff just changed.
Your content, someone else’s model
Here’s the situation most site owners don’t realize they’re in. Big AI companies run crawlers that sweep the web and collect text to train their models. Your blog posts, your product pages, your guides. All of it is fair game unless you say otherwise.
For years, saying otherwise came with a catch. Some companies use a single crawler for two jobs: collecting training data and powering regular search. Google’s ecosystem is the famous example. Block the wrong bot and you don’t just opt out of AI training. You risk vanishing from search results, which is where most websites get their visitors.
So people faced a choice with no good answer. Let AI companies take your content for free, or risk your traffic.
The numbers show how most people felt about it. Cloudflare, which sits in front of millions of websites, reports that fewer than 1% of sites block search bots. But 17% choose to block AI training. Site owners want to be found. They just don’t want their work turned into model weights without a say.
What changed this week
Cloudflare introduced a new setting called Disallow AI Training, and it finally separates the two jobs. You stay indexed for search while refusing permission for training. According to Cloudflare’s announcement, Apple, Google, and Microsoft already honor the setting or have committed to honoring it within a set timeframe.
The company also created something called the Accountable designation for bot operators. To earn it, an operator must give site owners real controls and transparency about how content gets used. A crawler that ignores your preferences doesn’t get to claim it.
There’s more coming. Cloudflare says AI summaries are the next frontier, because a site-wide yes or no is too blunt an instrument. By early next year, the goal is letting you control how much of your content appears inside AI-generated summaries, set once instead of negotiated with every company separately.
Now let’s get practical. You have two tools, and you should probably use both.
Know your bots before you block anything
Not all crawlers are enemies. Some of them send you visitors. Blocking the wrong ones makes you invisible in AI search results, which increasingly is where people first hear about a site.
| Bot | Operator | What it does | Typical choice |
|---|---|---|---|
| GPTBot | OpenAI | Collects content for model training | Block if you don’t want training |
| ClaudeBot | Anthropic | Collects content for model training | Block if you don’t want training |
| Google-Extended | Gates Gemini training only | Block, no search cost | |
| CCBot | Common Crawl | Open dataset used by many AI firms | Block if you don’t want training |
| Bytespider | ByteDance | Collects content for training | Block if you don’t want training |
| OAI-SearchBot | OpenAI | Indexes sites for ChatGPT search | Allow if you want citations |
| ChatGPT-User | OpenAI | Fetches a page when a user asks | Allow if you want citations |
| PerplexityBot | Perplexity | Indexes sites for its answer engine | Allow if you want citations |
| Googlebot | Classic search indexing | Never block |
One detail surprises everyone: Google-Extended has nothing to do with Google Search. It’s a separate token that only controls whether your content trains Gemini. Blocking it costs you zero search ranking. Meanwhile, Googlebot is sacred. Block that one and you have a real problem.
Method 1: the five-minute robots.txt fix
Every website can do this today, for free, regardless of host. The robots.txt file sits at the root of your site and tells well-behaved crawlers what they may access. Add these lines:
# Block AI training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
Disallow: /
# Keep the rest of the web open
User-agent: Googlebot
Allow: /Where’s the file? On WordPress, most SEO plugins (Yoast, Rank Math) let you edit robots.txt from the settings. On other platforms, it’s a file you edit via FTP or your hosting panel. After saving, check yoursite.com/robots.txt in a browser to confirm the lines are live.
Honest limits, because this matters: robots.txt is a request, not a wall. Reputable companies like OpenAI and Anthropic respect it. Less scrupulous crawlers ignore it entirely, and the file has no way to identify who’s really knocking or stop someone who doesn’t care. It works for the biggest players, which covers most training traffic, but it isn’t enforcement.
Method 2: the Cloudflare enforcement layer
If your site uses Cloudflare, you get the stronger version. Their new Disallow AI Training setting works at the network level, which means it can identify who is crawling, classify why they’re crawling, and actually block the ones that ignore your preference. Then it reports what each operator did, so accountability isn’t just a promise.
Setup is genuinely simple. Log in to your Cloudflare dashboard, select your domain, and look for the AI Crawl Control (or bots) section. Toggle the blocking preference for AI training. Done. No files to edit, and the enforcement happens before a rogue crawler even loads your page.
For site owners who want proof, Cloudflare also reports what each operator actually did on its public Radar dashboard. You can see which AI training bots came, whether they respected your preference, and who didn’t. That kind of receipts-first transparency never existed in the robots.txt era.
The two methods stack. Robots.txt states your policy publicly for everyone, including the polite crawlers that check it. Cloudflare enforces it against the ones that don’t. If you’re on Cloudflare, do both. If you’re not, robots.txt still covers the major AI training bots in five minutes.
What about AI search traffic?
Before you block everything with a robot in its name, think about the other side of the table. When someone asks ChatGPT or Perplexity a question, those tools cite their sources with links. Getting cited sends real visitors, and for newer sites it can be a meaningful discovery channel.
That’s why the smart posture most publishers are landing on looks like this: block the training bots, allow the search and citation bots. You keep eligibility for AI-era referrals while refusing the wholesale copying. The table above gives you the exact names for each group.
There’s also a content angle worth mentioning. People increasingly use AI tools at work, sometimes in ways that create risk for their employers. If that’s your world, our guide to shadow AI explains how to use these tools without the fallout. And for a look at how institutions are wrestling with AI rules, we covered the NYC AI ban in schools and what it means for parents.
How to check if AI training bots already hit your site
Curious whether this actually applies to you? Look at your own logs before deciding anything. Most hosting panels include a stats section (AWStats, GoAccess, or similar) listing the user agents that visited your site. Grep that list for GPTBot, ClaudeBot, or Bytespider. Most site owners are surprised.
If you run WordPress, security plugins like Wordfence or Jetpack show bot activity in the dashboard without touching raw logs. Five minutes of looking tells you exactly which AI training bots have been reading your work, and how often.
That number usually settles the debate. Watching a crawler you never invited pull your full archive tends to move “maybe someday” to “blocked today.”
The takeaway
Pick your posture today: check yoursite.com/robots.txt, add the training bot block if it’s missing, and flip the Disallow AI Training setting in Cloudflare if you use it. Blocking AI training bots no longer means sacrificing search traffic, so the old excuse is gone. Your writing, your rules.
