You ask your AI to book a flight or find a decent taco spot, and it comes back empty-handed. Half the internet is quietly blocking AI agents right now, and nobody sent you the memo.
Websites are quietly blocking AI agents
First, a quick definition, because “agent” is one of those words tech companies use on purpose. An AI agent is the version of ChatGPT, Gemini or Claude that does stuff for you: opens websites, fills in forms, compares prices, books the table. Not just chats. Acts.
For about a year, the pitch was simple: hand your boring errands to the bot and go live your life. Meanwhile, the websites those bots visit started pushing back. Hard. We saw the first taste of this with ChatGPT working inside Chrome tabs, and the reaction was swift.
Yelp only lets agents in if they’re part of its paid data licensing program. Delta doesn’t let them book flights. JPMorgan Chase and Charles Schwab both confirmed they block agents from touching your accounts, citing security. Amazon has been fighting shopping agents since last year, when it banned Perplexity’s shopping feature and dragged the startup into court.
Then there’s the quiet group. Some users noticed their agents just stopped working on Cars.com and Amtrak. No announcement, no error message that explains anything. The door was simply locked one morning.
All of this comes from The Verge’s recent report, which got confirmations straight from the companies involved. Worth a read if you like receipts.
Why your bank hates your AI assistant
So why is blocking AI agents suddenly the hobby of every big website? Two reasons, and they wear different outfits.
The money reason. A site like Yelp makes money when humans visit and see ads. When an agent reads the page for you, you never see the ad. The site did all the work (reviews, photos, servers, staff) and got paid exactly nothing. Multiply that by millions of agent visits a day and the “free” internet stops adding up.
Cloudflare, which sits in front of roughly a fifth of the web, published some brutal numbers on this. Over half of all internet traffic is now non-human. For every hour people spend seeking information online, only about 15 minutes ends up on the open web. The rest happens inside chat windows, where nobody’s ads get served.
The security reason. Banks aren’t losing sleep over ads. They’re losing sleep over a bot holding your login. An agent that can click around your checking account is a brand new attack surface, and fraud teams never signed up for it. So Chase and Schwab just say no, and honestly, that one’s hard to argue with.
The result: blocking AI agents feels personal, like the machine is picking on your machine. It’s really just accounting and risk management in a trench coat.
Cloudflare made blocking AI agents the default
Here’s where it stopped being a few grumpy websites and became the web’s new normal.
On September 15, Cloudflare flipped a switch. Any new domain joining its network now blocks AI agent traffic and AI training crawlers by default on pages that show ads. Free-tier sites got the same treatment. Search crawlers still get in, because search sends visitors back. Agents don’t, so agents stay outside unless the site owner opts out.
Read that again, because the direction matters. Before, a website had to opt in to keeping agents away. Now it has to opt out. Defaults are destiny on the internet, and the default changed sides.
There’s some nuance if you run a site: existing paid domains kept their old settings, and Cloudflare published the full breakdown of the new Search, Agent and Training categories in its announcement. But for every new site launching from here on, closed-to-agents is the factory setting.
Which sites block AI agents right now
Here’s the cheat sheet, based on confirmed reporting and user reports. Expect this table to get longer, not shorter.
| Site | Status for AI agents | Why |
|---|---|---|
| Yelp | Blocked unless they pay for data licensing | Sells access, doesn’t donate it |
| Delta | Can’t book flights | Keeps bookings human |
| JPMorgan Chase | Account access blocked | Security |
| Charles Schwab | Account access blocked | Security |
| Amazon | Banned, lawsuit ongoing | Wants human eyeballs on ads |
| Cars.com, Amtrak | Stopped working (unconfirmed) | Quiet blocks, no statement |
The Amazon fight is the one to watch. A judge first ordered Perplexity’s agents to stop shopping there, then an appeals court flipped that ruling in August, and Amazon filed an amended complaint. Translation: even the courts haven’t settled whether your agent counts as “you” or as a bot. Until someone decides, expect more doors to close without warning.
What still works
So is the whole agentic dream dead? No. It just has rules now, and knowing them saves you a lot of yelling at your laptop.
- Official integrations beat browsing. When ChatGPT books through a partner integration or Gemini checks out via Shopify’s agentic checkout, nobody gets blocked, because the site invited the agent in through the front door. The blocked agents are usually the ones climbing through the window.
- Search still works fine. Cloudflare’s default allows search crawlers, so asking your AI to find information and hand you links keeps working. It’s the “do the entire task on the site” part that breaks.
- Money stays manual, and that’s fine. Logging into your brokerage through an agent was always a risky idea. Some of these blocks are doing you a favor, even if they feel like a downgrade today.
- Check whether the site even wants agents. Plenty of sites publish their preferences in robots.txt, and some block training bots while still allowing user-triggered fetches. We broke down how site owners handle this in our guide to blocking AI training bots without losing Google traffic. Reading tea leaves, but useful tea leaves.
- When it fails, assume the lock, not the idiot. If your agent whiffs on a site that used to work, the odds are good it got 403’d, not that the model got dumber overnight. Retry in a week or just do it yourself. It’ll take two minutes either way.
Does your agent even know it was blocked?
One underreported annoyance: agents are terrible at admitting defeat. Ask one to book a table at a blocked restaurant and it might say the site was slow, or the booking failed, or worse, it’ll improvise an answer from old data and present it with total confidence.
That last one deserves a warning label. An agent that can’t fetch live data is guessing, and a confident guess about a flight price or a restaurant’s hours is how you end up standing in front of a closed door in the rain. Meanwhile, the site has no idea you even tried, so there’s no error to report.
The fix is on you, for now. When a task involves a site from the table above, do it the old way and let the AI handle the parts that don’t need a login: comparing options, drafting the email, remembering the details.
Paying your way in
There’s a plan to end the stand-off, and it involves your agent opening its wallet.
Cloudflare’s Pay Per Crawl turns the ancient HTTP 402 “Payment Required” status code into a toll booth. A site sets a price, and an agent either pays per fetch or walks away. The next version, Pay Per Use, charges only when content actually helps produce an answer. Machine-to-machine payments (the x402 protocol and Cloudflare Wallets) are being built so agents can settle up without a human clicking approve on every request.
Skeptical? Fair. We’ve heard “micropayments will save the internet” before, and it went nowhere. But this time there’s real enforcement behind it, and one company controls the pipes for a fifth of the web. If agents start paying for content, your AI subscription quietly gets pricier. Nothing is free, it just gets billed somewhere else.
Your move
For now, treat your AI agent like a very capable intern who some buildings won’t let past the lobby. Use official integrations where they exist, keep logins and money manual, and don’t trust an agent’s answer when the site it needed is on the blocked list.
Blocking AI agents is the web’s new default, and the next twelve months decide whether this softens into pay-to-enter or hardens into locked doors everywhere. Either way, the free errand-running era just got its first real price tag.
