By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
Logic & LayersLogic & Layers
  • Tools
  • Earn with AI
  • Productivity
  • Automation
  • Guides
Logic & LayersLogic & Layers
  • Privacy Policy
  • About
Search
  • Tools
  • Earn with AI
  • Productivity
  • Automation
  • Guides
  • About
  • Contact
  • Blog
  • Privacy Policy
  • Complaint
  • Advertise
© 2026 Logic and Layers. Ruby Design Company. All Rights Reserved.
Website dashboard showing bot traffic being filtered
Guides

Block AI training bots without losing Google traffic

Editorial Team
Last updated: September 16, 2026 3:30 am
Editorial Team
Share
Cloudflare's crawl controls separate search bots from AI training bots

A crawler probably read your website this morning, copied the text, and helped train somebody’s AI model. Blocking AI training bots used to force an ugly choice: protect your content or keep your Google traffic. That tradeoff just changed.

Contents
Your content, someone else’s modelWhat changed this weekKnow your bots before you block anythingMethod 1: the five-minute robots.txt fixMethod 2: the Cloudflare enforcement layerWhat about AI search traffic?How to check if AI training bots already hit your siteThe takeaway

Your content, someone else’s model

Here’s the situation most site owners don’t realize they’re in. Big AI companies run crawlers that sweep the web and collect text to train their models. Your blog posts, your product pages, your guides. All of it is fair game unless you say otherwise.

For years, saying otherwise came with a catch. Some companies use a single crawler for two jobs: collecting training data and powering regular search. Google’s ecosystem is the famous example. Block the wrong bot and you don’t just opt out of AI training. You risk vanishing from search results, which is where most websites get their visitors.

So people faced a choice with no good answer. Let AI companies take your content for free, or risk your traffic.

The numbers show how most people felt about it. Cloudflare, which sits in front of millions of websites, reports that fewer than 1% of sites block search bots. But 17% choose to block AI training. Site owners want to be found. They just don’t want their work turned into model weights without a say.

What changed this week

Cloudflare introduced a new setting called Disallow AI Training, and it finally separates the two jobs. You stay indexed for search while refusing permission for training. According to Cloudflare’s announcement, Apple, Google, and Microsoft already honor the setting or have committed to honoring it within a set timeframe.

The company also created something called the Accountable designation for bot operators. To earn it, an operator must give site owners real controls and transparency about how content gets used. A crawler that ignores your preferences doesn’t get to claim it.

There’s more coming. Cloudflare says AI summaries are the next frontier, because a site-wide yes or no is too blunt an instrument. By early next year, the goal is letting you control how much of your content appears inside AI-generated summaries, set once instead of negotiated with every company separately.

Now let’s get practical. You have two tools, and you should probably use both.

Know your bots before you block anything

Not all crawlers are enemies. Some of them send you visitors. Blocking the wrong ones makes you invisible in AI search results, which increasingly is where people first hear about a site.

Bot Operator What it does Typical choice
GPTBot OpenAI Collects content for model training Block if you don’t want training
ClaudeBot Anthropic Collects content for model training Block if you don’t want training
Google-Extended Google Gates Gemini training only Block, no search cost
CCBot Common Crawl Open dataset used by many AI firms Block if you don’t want training
Bytespider ByteDance Collects content for training Block if you don’t want training
OAI-SearchBot OpenAI Indexes sites for ChatGPT search Allow if you want citations
ChatGPT-User OpenAI Fetches a page when a user asks Allow if you want citations
PerplexityBot Perplexity Indexes sites for its answer engine Allow if you want citations
Googlebot Google Classic search indexing Never block

One detail surprises everyone: Google-Extended has nothing to do with Google Search. It’s a separate token that only controls whether your content trains Gemini. Blocking it costs you zero search ranking. Meanwhile, Googlebot is sacred. Block that one and you have a real problem.

Method 1: the five-minute robots.txt fix

Every website can do this today, for free, regardless of host. The robots.txt file sits at the root of your site and tells well-behaved crawlers what they may access. Add these lines:

# Block AI training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
Disallow: /

# Keep the rest of the web open
User-agent: Googlebot
Allow: /

Where’s the file? On WordPress, most SEO plugins (Yoast, Rank Math) let you edit robots.txt from the settings. On other platforms, it’s a file you edit via FTP or your hosting panel. After saving, check yoursite.com/robots.txt in a browser to confirm the lines are live.

Honest limits, because this matters: robots.txt is a request, not a wall. Reputable companies like OpenAI and Anthropic respect it. Less scrupulous crawlers ignore it entirely, and the file has no way to identify who’s really knocking or stop someone who doesn’t care. It works for the biggest players, which covers most training traffic, but it isn’t enforcement.

Method 2: the Cloudflare enforcement layer

If your site uses Cloudflare, you get the stronger version. Their new Disallow AI Training setting works at the network level, which means it can identify who is crawling, classify why they’re crawling, and actually block the ones that ignore your preference. Then it reports what each operator did, so accountability isn’t just a promise.

Setup is genuinely simple. Log in to your Cloudflare dashboard, select your domain, and look for the AI Crawl Control (or bots) section. Toggle the blocking preference for AI training. Done. No files to edit, and the enforcement happens before a rogue crawler even loads your page.

For site owners who want proof, Cloudflare also reports what each operator actually did on its public Radar dashboard. You can see which AI training bots came, whether they respected your preference, and who didn’t. That kind of receipts-first transparency never existed in the robots.txt era.

The two methods stack. Robots.txt states your policy publicly for everyone, including the polite crawlers that check it. Cloudflare enforces it against the ones that don’t. If you’re on Cloudflare, do both. If you’re not, robots.txt still covers the major AI training bots in five minutes.

What about AI search traffic?

Before you block everything with a robot in its name, think about the other side of the table. When someone asks ChatGPT or Perplexity a question, those tools cite their sources with links. Getting cited sends real visitors, and for newer sites it can be a meaningful discovery channel.

That’s why the smart posture most publishers are landing on looks like this: block the training bots, allow the search and citation bots. You keep eligibility for AI-era referrals while refusing the wholesale copying. The table above gives you the exact names for each group.

There’s also a content angle worth mentioning. People increasingly use AI tools at work, sometimes in ways that create risk for their employers. If that’s your world, our guide to shadow AI explains how to use these tools without the fallout. And for a look at how institutions are wrestling with AI rules, we covered the NYC AI ban in schools and what it means for parents.

How to check if AI training bots already hit your site

Curious whether this actually applies to you? Look at your own logs before deciding anything. Most hosting panels include a stats section (AWStats, GoAccess, or similar) listing the user agents that visited your site. Grep that list for GPTBot, ClaudeBot, or Bytespider. Most site owners are surprised.

If you run WordPress, security plugins like Wordfence or Jetpack show bot activity in the dashboard without touching raw logs. Five minutes of looking tells you exactly which AI training bots have been reading your work, and how often.

That number usually settles the debate. Watching a crawler you never invited pull your full archive tends to move “maybe someday” to “blocked today.”

The takeaway

Pick your posture today: check yoursite.com/robots.txt, add the training bot block if it’s missing, and flip the Disallow AI Training setting in Cloudflare if you use it. Blocking AI training bots no longer means sacrificing search traffic, so the old excuse is gone. Your writing, your rules.

You Might Also Like

Pixel AI dictation on any Android: the free PixelBoard hack
Google admitted it: AI Mode traffic isn’t coming back
Google phone number verification: how to turn it off for good
Google Vids: how to make free AI videos (beginner guide)
Claude Code AGENTS.md support: setup in two minutes
TAGGED:ai crawlerscloudflarecontent protectionrobots txtSEO
Share
Previous Article Laptop screen showing a chat assistant settings page ChatGPT custom instructions: better answers in 2 minutes
Next Article iPhone showing the WhatsApp app icon on a wooden desk WhatsApp Business MCP: let AI agents run your WhatsApp
1 Comment
  • Pingback: Reddit RSS is dying: 5 fixes for your automations

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

banner banner
Create an Amazing Newspaper
Discover thousands of options, easy to customize layouts, one-click to import demo and much more.
Learn More

Latest News

Gemini 4 Hydrogen key art from Google's launch announcement
Gemini 4 Hydrogen: Google’s new fast AI model explained
Tools
Coordinator chat directing parallel AI worker threads on separate branches
Claude Code Projects just learned to run AI workers for you
Tools
smart home hub device standing on a living room shelf next to decor
UGREEN HomeAgent: local AI for your whole smart home
Automation
Hands-on test of Alibaba Qwen4-Max on real user tasks
Qwen4-Max: the $4.43 AI task now costs $1.67
Tools

Recent Posts

  • Gemini 4 Hydrogen: Google’s new fast AI model explained
  • Claude Code Projects just learned to run AI workers for you
  • UGREEN HomeAgent: local AI for your whole smart home
  • Qwen4-Max: the $4.43 AI task now costs $1.67
  • Amazon World Product Graph: what it means for sellers

Recent Comments

  1. Gemini 4 Hydrogen: Google's new fast AI model explained on Gemini 4 Argon vs GPT-6: which one deserves your money?
  2. Claude Code Projects just learned to run AI workers for you on Claude Code cross-session messaging: stop copy-pasting between sessions
  3. UGREEN HomeAgent: local AI for your whole smart home on Notion AI data breach: what leaked and what to do
  4. Qwen4-Max: Alibaba's AI teammate that costs 62% less on Ollaya decision models: run Jev-style triage locally
  5. Amazon World Product Graph: what it means for sellers on Google Universal Cart: how the AI shopping cart works

You Might also Like

ESP32 deep learning library for local LLM inference
Guides

Run an LLM on ESP32: Local AI on a $5 Microcontroller

Editorial Team
Editorial Team
11 Min Read
Eight stages of AI automation pipeline from judge to physical world
Guides

The 8 Stages of AI Automation: From Helpers to Autonomous Workers

Editorial Team
Editorial Team
13 Min Read
AI agent taking actions vs AI generating content side by side
Guides

Agentic AI vs generative AI: What the difference means for you

Editorial Team
Editorial Team
7 Min Read
//

We influence 20 million users and is the number one business and technology news network on the planet

Quick Link

  • PRIVACY NOTICE
  • YOUR PRIVACY RIGHTS
  • INTEREST-BASE ADSNew
  • TERMS OF USE
  • OUR SITE MAP

Support

  • ADVERTISE
  • ONLINE BESTHot
  • CUSTOMER
  • SERVICES
  • SUBSCRIBE

Categories

  • Tools
© 2026 Logic and Layers. All Rights Reserved.