Withheld Knowledge.
The implicit deal of the open web — we publish, you send traffic — has broken. When agents answer from your content, you get nothing back. Creators are responding with paywalls, robots.txt blocks, lawsuits, and retreat.
This is the AI tax. Stack Overflow questions down 76.5% since ChatGPT. Cloudflare crawl-to-referral ratios up to 73,000:1 for Anthropic. Six concrete response patterns across the web. The numbers are the story.
- 01Stack Overflow questions: −76.5% since ChatGPT launch. Volume back to May 2009 levels. 15 years of growth erased.
- 02Cloudflare crawl-to-referral ratios: Google 14:1, OpenAI 1,700:1, Anthropic 73,000:1. Anthropic crawls 73,000 pages for every visitor it sends.
- 03Pew Research: only 1% of users click citations from Google AI Overviews. Zero-click search: 56% → 69% in a year.
- 04The response is six-pronged: robots.txt blocks, pay-per-crawl, licensing deals, lawsuits, paywalls/dark forest, content poisoning (Glaze/Nightshade).
- 05The corpus shrinks where signal lives. What remains crawlable tilts increasingly toward slop. (See threat 3.)
Seven platforms. The numbers do the work.
108,563 → 25,566. Back to May 2009 levels. 15 years of growth erased. Volume now still falling +90% from 2020 peak.
894M → 442M visits. 74.8% organic-search dependent. Most exposed to Google AI Overviews.
Independent music publisher. Explicitly blames Google AI Overviews. Layoffs followed.
Home-improvement blog. −65% ad revenue in same window.
Travel blog. Site shut down following the decline.
21% staff layoffs. Forbes ~-50%, HuffPost -50% — pattern is industry-wide for mid-tier news.
1.1 billion fewer monthly visits. Multimedia bandwidth from AI scrapers up +50% since Jan 2024 (WMF official).
Six ways creators are fighting back.
Every measurable signal points the same way — but the responses fall into six categories, with very different velocities and very different effects on the substrate.
robots.txt blocking
Top-1,000 sites blocking GPTBot: 5% (Aug 2023) → 35.7% (Aug 2024). GPTBot now blocked on ~5.6M websites. ClaudeBot blocked on ~5.8M — fastest-growing. But: 13.26% of AI requests ignore robots.txt (Tollbit data 2025).
Cloudflare Pay-Per-Crawl
Launched 1 July 2025. HTTP 402 Payment Required. Default block for new Cloudflare sites — first major infra player to ship this. Signed by Condé Nast, TIME, AP, Atlantic, Fortune, Adweek.
Licensing deals
Reddit-Google $60M/yr (Feb 2024). Reddit-OpenAI ~$70M/yr. NewsCorp-OpenAI ~$250M / 5 years. Axel Springer ~$25-50M / 3 years. FT $5-10M/yr. Most other publishers offered $1-5M. Total disclosed Reddit licensing revenue 2024: $203M.
Lawsuits
NYT v OpenAI/Microsoft (Dec 2023, ongoing). Judge Stein denied motion to dismiss April 2025. 20M ChatGPT logs ordered for discovery (Jan 2026). Class-cert + summary judgment scheduled April 2026. Reddit v Anthropic (Jun 2025) for ~100,000 unauthorised scrapes. Statutory damages claim: 'billions.'
Paywalls and dark forest retreat
Substack 5M paid subscribers (Mar 2025), $1.1B valuation. Top writers (Heather Cox Richardson $5M+/yr, Jim Acosta) leaving traditional outlets. 'Dark Forest' framing (Maggie Appleton, 2023) now mainstream. The 'cozy web' (Discord, Signal, paid newsletters) absorbs the signal.
Content poisoning by creators
Glaze: 6M+ downloads by Nov 2024. Nightshade: 250K downloads in first 5 days (Jan 2024), 1.6M+ total. From U. Chicago (Ben Zhao). 92% style-mimicry block success. Explicit goal: raise training cost until licensing becomes cheaper.
The cozy web absorbs the signal.
Maggie Appleton's “The Expanding Dark Forest and Generative AI” (January 2023) is now the canonical name for what is happening. Creators retreat from the open web — which has become hostile, polluted, and economically broken — into private gatekept spaces. Discord servers. Signal groups. Paid newsletters. Members-only forums. Slack workspaces. Substack archives.
Substack's 2025 numbers tell the story: 5M paid subscribers, ~$450M annualised writer revenue, $1.1B BOND-led valuation (July 2025). Heather Cox Richardson clears $5M/year. Jim Acosta got 10,000+ paid subs in weeks after leaving CNN. The platform is now explicitly positioned as a “self-contained creator universe” — off the open web by design.
Appleton's framing: “Our new challenge as little snowflake humans will be to prove we aren't language models. It's the reverse Turing test.”
The implication for the substrate: the best signal is moving where the agents cannot read it. Agents inherit a corpus that is increasingly the residue — what creators were willing to leave outside the cozy web.
The web was a public good built on an implicit deal. That deal is gone.
For thirty years, creators wrote and posted on the assumption that visibility — and the traffic and ad revenue that came with it — was the reward. Search engines were the substrate that made the deal work.
AI agents broke the deal by extracting the signal without sending visitors back. The 1% click-through rate on AI Overviews is not a bug. It is the deal's replacement with a new one — where creators are inputs, not partners.
Threat 1 (poisoning) tells agents they cannot trust what they read. Threat 2 (withholding) shrinks what they can read at all. Together they push agents toward whatever remains — which leads directly to threat 3.