Our own site was invisible to search engines for months and we did not notice, because the only check anyone ran was “open the site in a browser.”
The symptom that finally surfaced it was an AI-visibility report reading 0%. Zero brand mentions across every tracked prompt. The obvious conclusion is a content problem. It was not a content problem.
One Cloudflare setting was returning HTTP 403 to every crawler, on every URL, including robots.txt and both sitemaps. Here is how to test for it, why the usual verification can’t see it, and what to do about it.
The signature
A blocked request looks like this:
HTTP/1.1 403 Forbidden
Content-Type: text/plain
Content-Length: 25
Server: cloudflare
Your request was blocked.Twenty-five bytes of plain text. No challenge page, no cf-mitigated header, no cf-cache-status — the request never reached our origin at all. It was refused at the edge.
The three-command test
Run this against your own domain. The only variable that changes between the two requests is the User-Agent:
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36" \
https://example.com/
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
https://example.com/
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
https://example.com/robots.txtIf the first returns 200 and the second returns 403, a bot rule is keying off the User-Agent string. If the third also returns 403, it is worse than a ranking problem: a crawler cannot even read the file that tells it what it is allowed to crawl.
Our numbers, measured over repeated runs: a Chrome User-Agent passed 48 of 50 requests. A Googlebot User-Agent passed 0 of 20. Same machine, same curl binary, same network, seconds apart.
We also confirmed what it was not:
- Not rate limiting. Ten requests at exactly 20-second spacing, alternating Chrome and Googlebot, produced 200, 403, 200, 403, 200, 403 — in lockstep. A rate limiter cannot give opposite answers to alternating requests at an identical rate.
- Not missing browser headers. A Googlebot User-Agent sent with a full set of
Accept,Accept-Language,sec-ch-uaandSec-Fetch-*headers still returned 403. A Chrome User-Agent with no extra headers at all returned 200. - Not the TLS fingerprint. One curl binary was used throughout, so that variable was held constant. (More on this below — it matters.)
Twenty-two crawler User-Agents were blocked, including Googlebot, Googlebot-Smartphone, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Applebot, Amazonbot, CCBot and ia_archiver.
Two independent oracles that don’t care what we think
A site: search is a weak signal, and search tools paraphrase. Two public archives are not opinions and can be queried directly.
Common Crawl. The index API answers whether a URL has ever been captured:
curl "https://index.commoncrawl.org/CC-MAIN-2026-25-index?url=example.com/*&output=json"For our domain, every monthly index that postdated the site’s launch returned No Captures found. And CCBot was one of the 403s. That is a causal chain, not a coincidence: the crawler that builds one of the largest public web corpora was refused, and the corpus contains nothing.
The Wayback Machine. The CDX API is equally blunt:
curl "http://web.archive.org/cdx/search/cdx?url=example.com*&output=json"Two captures in the domain’s entire history, both from 2023, both bare redirects. Not one content page had ever been archived. ia_archiver was also 403.
Being absent from both, while the crawlers for both are being refused, is about as clear as web forensics gets.
Why “check it in a browser” can never catch this
Our internal security checklist had a step for enabling this protection. Its verification step, in full, was:
Open an incognito window, visit the site — confirm it still loads normally.
Incognito Chrome sends a Chrome User-Agent. A Chrome User-Agent is exactly the traffic the block allows. The check passes 100% of the time, whether the site is healthy or completely deindexed. It was structurally incapable of detecting the failure it introduced.
That is the real lesson here, and it generalizes: a verification step that uses a different client than the thing you’re worried about is not a verification step. If you care about crawlers, test with a crawler’s User-Agent. If you care about your cron job, test from your cron job.
The trap that will waste your afternoon
We first wrote the checker in Node, using fetch. It reported 403 for everything — including a Chrome User-Agent that curl proved was fine. Measured side by side:
| Client | User-Agent | Result |
|---|---|---|
Node fetch | Chrome | 403 |
Node fetch | Googlebot | 403 |
Node fetch | none set | 403 |
| curl | Chrome | 200 |
| curl | Googlebot | 403 |
Cloudflare fingerprints the TLS and HTTP/2 client, not just the User-Agent string. Node’s HTTP stack is recognizably automated no matter what UA it presents. A fetch-based diagnostic gives you a false diagnosis — it looks like everything is blocked, so you can never see the fix working, and you can’t tell a UA rule from a client rule.
Hold the client constant. That is the entire experiment. Ours shells out to curl for exactly this reason.
Corollary worth keeping: any Node-based uptime check pointed at a bot-protected domain will fail regardless of your bot policy.
What the setting actually does
Cloudflare’s free Bot Fight Mode is one toggle with no configuration surface. Two properties matter, and both are in Cloudflare’s own documentation:
- It cannot be bypassed with a WAF custom rule’s
Skipaction. It runs before the custom-rules phase, so a skip rule you deploy to exempt a crawler never gets reached. - It has no verified-bot allowlist. Allowing verified bots is a feature of Super Bot Fight Mode, on paid plans.
Those two together are the whole problem. There is no way to keep Bot Fight Mode on and let Googlebot through. The documented remedies are to turn it off or move up a plan.
The irony in our case: we were already on the paid plan. The legacy toggle had been switched on earlier and never switched off, so we were paying for the capability that would have fixed it while the free mechanism that can’t be configured kept blocking everything. Cloudflare’s docs do say to turn Bot Fight Mode off when you enable Super Bot Fight Mode. It is an easy line to miss.
The fix, and the part that isn’t a fix
On a paid plan: Bot Fight Mode off, Super Bot Fight Mode on, verified bots set to Allow. Turn off static-resource protection too — that is what was 403ing our stylesheets.
AI crawlers are a separate decision, and since July 2026 Cloudflare splits them by behaviour: Search, Agent, and Training, each independently set to allow or block. That maps onto a real business question rather than one blunt switch — you can accept crawlers that might send a reader back to you while declining ones that only take. One caveat from the docs worth reading twice: blocking Training also blocks mixed-purpose crawlers, and several major ones do both jobs. Check the per-crawler status codes a couple of weeks later instead of assuming.
Then verify with the three commands above, and with Search Console’s URL Inspection → Test live URL, which fetches from real Google infrastructure. That last one matters: every 403 we produced came from a spoofed Googlebot User-Agent on an ordinary consumer IP, which is precisely the traffic bot protection is designed to stop. Search Console is the only test that settles whether verified Googlebot was affected too.
Two things we found on the way
A sporadic block that hits real browsers. Roughly 2–4% of ordinary browser requests were also 403’d, non-stickily — 50 rapid requests failed at #16 and #19 and recovered immediately. It is not a threshold; a threshold stays tripped. If you build a monitor for any of this, retry before you alert, or you will teach yourself to ignore it.
A soft 404 the same audit exposed. Any nonexistent URL under three of our route prefixes returned HTTP 200 with a byte-identical copy of the homepage instead of a 404. Cause: those prefixes were excluded from our Worker, so they were served as static assets, and with no 404.html present the platform fell back to index.html. Adding one static 404 page fixed all three. Worth checking on any static site behind a CDN:
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/blog/this-does-not-exist/A 200 there means every fake URL under that prefix looks like a real page.
What unblocking does not fix
Turning the setting off restores access in seconds. It does not restore anything else quickly.
Indexing is days to weeks. Being included in a public crawl corpus runs on that crawler’s schedule, not yours. And none of it touches whether anyone has reason to cite you — that is content and reputation, and it is measured in months. Fixing the block only means the work can start counting.
If you take one thing from this: go run the three commands against your own domain right now. It takes ten seconds, and the failure mode is completely silent.