By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
Logic & LayersLogic & Layers
  • Tools
  • Earn with AI
  • Productivity
  • Automation
  • Guides
Logic & LayersLogic & Layers
  • Privacy Policy
  • About
Search
  • Tools
  • Earn with AI
  • Productivity
  • Automation
  • Guides
  • About
  • Contact
  • Blog
  • Privacy Policy
  • Complaint
  • Advertise
© 2026 Logic and Layers. Ruby Design Company. All Rights Reserved.
Illustration of website content being processed by an AI browsing agent
Automation

How AI agents read websites (it’s weirder than you think)

Editorial Team
Last updated: November 9, 2026 2:17 pm
Editorial Team
Share
Claude and Zapier: an experiment in how agents read pages

Zapier gutted one of its own web pages on purpose. Deleted every useful word, kept the URL identical, then asked Claude to read it. The agent still described the page perfectly. If you sell anything online, how AI agents read websites just became your problem, because the way they browse is nothing like the way you assume.

Contents
The experimentSurprise 1: markdown, not your pageSurprise 2: a second model reads firstWhat it means if you sell onlineHow agents even find your siteThe 10-minute self-auditDon’t over-engineer it

The experiment

Zapier’s engineering team built a little contraption they call Agent Arm Runner. It runs Claude Code inside a Docker container with a proxy that can swap what’s behind a URL. Same prompt, same address, but the production page goes to the control group and an empty variant goes to the test group.

Their target was the Slack integration page, the one listing everything Zapier can do with Slack. For the variant, they stripped the body down to almost nothing. No triggers. No actions. No integration content.

The expectation was obvious. Empty page in, empty answer out.

Instead, Claude came back rattling off specifics: 25 triggers including new mentions and reactions, 27 actions like sending direct messages, 11 searches including find user by email. All of it, from a page that no longer contained any of it. So how AI agents read websites apparently has a workaround for empty pages. The logs explained everything.

Surprise 1: markdown, not your page

The proxy logs caught the culprit. The agent’s fetch tool requested the page with a header asking for text/markdown first, HTML second.

Turns out Zapier had been experimenting with markdown versions of its pages, cleaner copies that agents can digest without all the surrounding markup. The HTML variant was gutted. Its markdown twin was not. So the agent read the intact markdown and answered from that.

They even planted a canary string in the markdown only, a fake verification code with instructions to include it in any summary. The canary came back in the agent’s response. Case closed: the agent was never reading the page you’d recognize as “the page.”

Here’s the shift. For years, SEO was about one artifact: the HTML page. Now a site can have parallel versions, and understanding how AI agents read websites means auditing every version you publish, not just the pretty one. Whatever a fetcher can request, an agent can consume.

Surprise 2: a second model reads first

There was a second plot twist hiding in the logs. When the agent fetched a page, the content didn’t go straight to the main model. It went to a cheaper, separate model that summarized the page first, and only the summary reached Claude.

That middle layer has its own instructions. Quote-length limits. Paraphrasing rules. So the version of your page that the “reader” sees is already filtered, condensed, and reshaped before the conversation even starts.

Think about what that means. Your carefully worded landing page gets pre-chewed by a model you didn’t choose, under rules you can’t see, and the main agent reasons about the leftovers. Cloudflare is now offering whole browser infrastructure for agents, and the request path keeps growing more layers, not fewer.

What it means if you sell online

Agents already shop for people. ChatGPT recommends products, Perplexity summarizes services, and browsing agents fill forms and compare prices. If an agent can’t read your site, you’re invisible at the exact moment someone asks “which option should I buy?” The mechanics of how AI agents read websites decide whether your product ever enters that conversation.

Three consequences worth internalizing:

  • Your content must exist in plain, server-rendered HTML. Most agents don’t run JavaScript. Pages that build themselves in the browser look empty to them.
  • Structured data belongs in the initial HTML too. If prices and specs inject client-side, agents never see them.
  • You have less control than you think. A summarizer model sits between your words and the answer, and it compresses ruthlessly.

How agents even find your site

Before any fetching happens, a crawler has to discover you. ClaudeBot, PerplexityBot, GPTBot and friends wander the web like Google’s spider always did, and site owners can block or allow each one in robots.txt. Plenty of sites blocked them in 2024 out of AI-training fears. That choice now has a side effect: block the crawler, and the shopping agent may never learn your store exists.

Worth a quick check. If your robots.txt bans every AI bot, you’ve opted out of agent commerce entirely, which is a strategy, but should be a deliberate one. The traffic shift is already visible elsewhere: Google itself admitted AI Mode traffic patterns aren’t bouncing back, and we’ve covered when AI search beats Google for different kinds of questions. Agents are the next layer of that same shift.

The practical stance for a small site: allow the crawlers, keep the content in server-rendered HTML, and let robots.txt do its thing deliberately instead of by accident.

The 10-minute self-audit

You don’t need Docker containers or proxies to check your own exposure. Zapier’s experiment sounds exotic, but the beginner version fits in a coffee break:

Check How What good looks like
Raw HTML test View-source your key page, search for your main offer The offer text exists in the HTML itself
Agent Q&A test Ask Claude or ChatGPT “what does [yoursite.com] sell?” Answer matches reality, not a hallucination
JS blindness test Disable JavaScript in dev tools, reload Core content still visible
Title and headings Scan the raw HTML heading order One clear h1, logical h2s
Markdown mirror Search your domain for /llms.txt or .md copies You know what versions exist

That last row matters more after Zapier’s canary test. If your platform auto-generates markdown or simplified copies of pages, they’re all in play. Audit the versions, not just the homepage. Ten minutes of checking beats months of guessing how AI agents read websites behind your back. And if your traffic graphs already look wrong, our piece on Google’s AI Mode traffic admission explains part of why.

And do the agent Q&A test on your competitors too. It’s competitive intelligence that costs nothing.

Don’t over-engineer it

Tempting response: rebuild everything, add markdown mirrors, chase every agent crawler name in your logs. Resist that for now.

The durable moves are boring ones. Server-render your important pages. Keep semantic HTML and clear headings, because agents navigate pages closer to how keyboard-only users do, stepping through elements in order. Put the facts (prices, specs, availability) directly in the markup. Answer real questions in plain language.

Beyond that, assume churn. Zapier itself closed by warning that today’s behavior won’t match tomorrow’s, even for the same tools. The teams winning right now aren’t optimizing for one agent’s quirks. They’re making their content legible to anything that fetches, then verifying with the Q&A test every few weeks.

Run that audit on your site today. Worst case, you learn your pages already read fine. Best case, you find the invisible problem before your next customer’s agent does.

Sources: What we learned by watching Claude read Zapier.com (Zapier Blog), Cloudflare Browser Run, How AI agents browse websites (Crazy Egg)

You Might Also Like

Instagram’s AI-generated profile label: what creators must do now
iPhone Shortcuts with Apple Intelligence: a beginner guide
AI training cutoffs: why your chatbot feels outdated
Stop paying for video editing. AI does 80% of it now
GPT-5.6 Sol, Terra, and Luna: Which model should you use?
TAGGED:ai-agentsAutomationChatGPTSEOweb-development
Share
Previous Article Podcast cover art for an interview about building apps with AI coding agents Build an app with Codex: a playbook that actually shipped
Next Article Claude interface showing the Fable 5.2 model picker during the free access week for Pro users Claude Fable 5.2 free week: get Pro access before Nov 16
Leave a Comment

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

banner banner
Create an Amazing Newspaper
Discover thousands of options, easy to customize layouts, one-click to import demo and much more.
Learn More

Latest News

Gemini 4 Hydrogen key art from Google's launch announcement
Gemini 4 Hydrogen: Google’s new fast AI model explained
Tools
Coordinator chat directing parallel AI worker threads on separate branches
Claude Code Projects just learned to run AI workers for you
Tools
smart home hub device standing on a living room shelf next to decor
UGREEN HomeAgent: local AI for your whole smart home
Automation
Hands-on test of Alibaba Qwen4-Max on real user tasks
Qwen4-Max: the $4.43 AI task now costs $1.67
Tools

Recent Posts

  • Gemini 4 Hydrogen: Google’s new fast AI model explained
  • Claude Code Projects just learned to run AI workers for you
  • UGREEN HomeAgent: local AI for your whole smart home
  • Qwen4-Max: the $4.43 AI task now costs $1.67
  • Amazon World Product Graph: what it means for sellers

Recent Comments

  1. Gemini 4 Hydrogen: Google's new fast AI model explained on Gemini 4 Argon vs GPT-6: which one deserves your money?
  2. Claude Code Projects just learned to run AI workers for you on Claude Code cross-session messaging: stop copy-pasting between sessions
  3. UGREEN HomeAgent: local AI for your whole smart home on Notion AI data breach: what leaked and what to do
  4. Qwen4-Max: Alibaba's AI teammate that costs 62% less on Ollaya decision models: run Jev-style triage locally
  5. Amazon World Product Graph: what it means for sellers on Google Universal Cart: how the AI shopping cart works

You Might also Like

The ChatGPT Slides tool building a presentation next to Google Slides with Gemini
Tools

ChatGPT Slides: build presentations from a chat window

Editorial Team
Editorial Team
9 Min Read
Editorial illustration depicting social media content generation through AI with comparison of different platforms and engagement metrics
Automation

Can AI run your social media without sounding like a bot?

Editorial Team
Editorial Team
12 Min Read
FLUX 3 vs Gemini Omni comparison
Tools

FLUX 3 vs Gemini Omni: Which AI image generator wins in 2026

Editorial Team
Editorial Team
6 Min Read
//

We influence 20 million users and is the number one business and technology news network on the planet

Quick Link

  • PRIVACY NOTICE
  • YOUR PRIVACY RIGHTS
  • INTEREST-BASE ADSNew
  • TERMS OF USE
  • OUR SITE MAP

Support

  • ADVERTISE
  • ONLINE BESTHot
  • CUSTOMER
  • SERVICES
  • SUBSCRIBE

Categories

  • Tools
© 2026 Logic and Layers. All Rights Reserved.