SODP Dispatch - 3 September 2026

AI is eating website traffic, websites are blocking AI – and reliable information is getting harder to find

 By Dana McKay & Damiano Spina

For 30 years, the world wide web has run on a surprisingly profound social contract: most sites are free for search engines to access, but if you use their content you give credit by linking to the source.

Recently, that social contract has begun to collapse. Artificial intelligence (AI) tools are crawling sites not to link to them, but to train models and generate answers (which may or may not be accurate).

When you search for something, ChatGPT’s response or Google’s AI Overviews may still include links to sources, but they’re a kind of optional extra to the main answer.

This has triggered a bad dynamic for website owners, the public, and even AI companies themselves: as websites lose traffic (and revenue), many are beginning to block AI scraping tools, meaning AI results depend more on low-quality websites (many of which are also generated by AI). As a result, good information can be harder than ever to find.

How we got here

In the early days of the world wide web, search engines and content creators came to an agreement about crawling (the practice of technologically examining a site to index it, so it can be served up in search results). Content creators would provide access to their sites for free, and even allow search engines to reproduce small snippets of text.

💬 We want to hear from you: Has your site started blocking AI crawlers, or are you holding off? Hit reply and tell us what tipped your decision either way — we read every response, and yours might shape a future issue.

📊 Publisher Revenue Dinner in Miami this September

We are hosting a private dinner in Miami for senior revenue leaders across news, gaming, newsletter, specialist, and brand publishing. This dinner is for publishing professionals who are actively driving revenue strategy and want a candid, peer-level conversation on how to win the biggest commercial quarter of the year. If you are a Chief Revenue Officer, CEO, Programmatic Head, or Partnerships Director making decisions around yield, inventory, first-party data, and Q4 growth, this evening is designed for you.

Date: Wednesday, September 16, 2026
Time: 6:30 PM
Venue: Coral Room, Rusty Pelican Miami Restaurant, 3201 Rickenbacker Cwy, Key Biscayne, FL 33149, United States

What you will walk away with

• Practical yield, personalisation, and first-party data moves that are lifting revenue per visit right now

• A candid read on how your peers are pricing, packaging, and positioning inventory for the Q4 surge

• Insight into which new formats — video, newsletters, and creator partnerships — are actually paying in 2026

• An attendee-only summary of the evening's key insights and growth playbooks shared around the table

The dinner is high-trust, collaborative, and entirely off-the-record under Chatham House Rule. No panels, no pitches, no slides — just one focused evening on how to make every session, impression, and first-party signal work at full value in publishing's most important quarter.

Seats are strictly limited to 15 attendees.

A big thank you to Ezoic for co-hosting this evening with us.

Apply for your seat here ▸

🎯 LiveRamp × OpenAI: Measure Your ChatGPT Ad Performance

If you're running or planning ads on ChatGPT, you can now close the measurement gap. LiveRamp's Conversions API (CAPI) Hub connects directly to ChatGPT ad campaigns via secure server-to-server data connections, meaning you get reliable conversion tracking without depending on browser-based signals that often break or go missing.

In practice, this means you can:

• See which ChatGPT campaigns are actually driving conversions — not just clicks

• Optimize spend in real time based on accurate performance data

• Justify and grow your ChatGPT ad budget with measurement data you can trust

As consumer journeys shift toward AI-powered platforms, this gives marketers the infrastructure to keep up, connecting data, measuring impact, and proving ROI on one of the fastest-growing ad surfaces in the world.

Read more ▸

By Vahe Arabian

🎯Do You Survive the Fan-Out? The AI Visibility Diagnostic Built for How AI Search Actually Works

Most "AI visibility" tools answer the wrong question: "Am I mentioned?"

The better question is: do you survive the fan-out — and does your content deserve to?

Here's how we built that into our AI Visibility report — two diagnostic engines and a four-part content optimizer.

1/ Query fan-out, not single queries

When someone researches a topic in ChatGPT or Google AI Overviews, they don't ask one question. They narrow: broad → specify → narrow → decide. Google's AI literally expands and contracts the query set as the conversation deepens.

So instead of probing one query per topic, we simulate the journey:

• An LLM designs a 4-step narrowing path per topic

• We Serper-search every step in parallel, with a 15s hard cap

• We track whether your domain is cited at each step, its position, top-3 presence, and AI Overview inclusion

• We flag dropped (cited then vanished) and weakened (position slipped) steps as the conversation narrows

• A final batched LLM call writes a concrete fix per dropped step — publish this, update that, add this schema — ranked by impact

The output isn't "you're cited 40%." It's "you win the broad question, but lose the comparison step to [competitor], and here's the exact page to fix that."

2/ Citation Regressions — what you lost since last run

A single score going up or down tells you nothing actionable. So we diff probe-by-probe between audit runs: match every probe on platform + topic, surface the ones where you were cited then but aren't now, and classify why:

• Credential not corroborated off-page — you hold it, your site says so, Google's directory says so, but the AI's entity graph hasn't connected them. Off-page fix, not a content fix.

• Competitor pushed you out — names who now holds the query.

• Query intent drifted — the AI now maps the query to a different category.

• Sampling volatility — only one probe changed; noise, not signal.

• Topical authority gap — build the pillar page.

Each regression gets the actual answer text, the competitor who gained, and a recovery recommendation.

3/ The Full Content Check — does your page deserve to be cited?

Getting found is half the battle. The other half is having content the AI will actually pull. We built a four-tab optimizer that runs on any article URL:

🔸 Coverage — Enter a target query; the LLM generates the real set of sub-questions a user asks around it, then scores your page: covered / partial / missing, with the reason and a fix for each gap. This is topical completeness, not keyword density.

🔸 Clarity — Four checks AI engines penalize: vague language ("a number of" when you could say "17"), unanchored claims (statements needing proof or citation), unclear headings (H2s that don't name the answer), and dense paragraphs that should be lists or tables. Each finding comes with a suggested replacement the AI can actually quote.

🔸 Effort — Modeled on Google's leaked contentEffort signal (an Expertise dimension feeding NSR). Scores editorial depth on five signals — research evidence, structure & depth, specificity, editorial polish, topic mastery — each /10 with a verdict and fix. Flags "engaging but shallow": a clickable page that lacks depth scores low here even if it generates clicks. Crucially, it separates engagement (clicking + staying) from satisfaction (useful content).

🔸 Persona Fit — Pick a saved persona (role, goals, pain points, buying triggers, preferred formats), and the LLM evaluates whether your page actually answers that persona's needs: fit score, tone match, answered vs. unanswered needs, and content opportunities. Because "will AI cite this" and "will my reader convert" are two different questions — and a page that nails both is the goal.

Why this matters

The citation funnel is brutal: surface → mention → cite → top-3. Most publishers only see the top of it.

• The fan-out shows you where in the journey you fall out.

• The regressions panel shows you which queries you're quietly bleeding between audits.

• The full content check tells you whether the page deserves to win once the AI finds it.

Audit your AI visibility the way the engines actually work: as a narrowing conversation, not a one-shot query. Track the deltas, not just the snapshot. And optimize the content, not just the citations.

Readers understand content partnerships, younger readers want more of them | INMA

Surrounded by bits of the Sunday newspaper, a reader flicks through one of the supplements and happens upon an interesting article. It’s a celebration of the long weekend: scenic routes to city breaks, picnics in parks, lazy days in the garden knowing work is one more day away than it should be. Another is scrolling through TikTok on the way to work. They’ve been stopped in their tracks: a video full of holiday hacks to optimise their packing before their next getaway. They mentally file a few of those tips for later.

Read more ▸

Google gives every site AI search reports, minus the clicks and queries | PPC Land

A dashboard is a claim about what matters. Whatever it counts becomes the thing people optimise; whatever it leaves out stops existing in the argument, however real it remains in the business. On August 31, 2026 Google finished rolling its generative AI performance report and its AI feature opt-out control to every website in the world. The report says how often a page turned up inside an AI answer. It does not say whether anyone clicked, and it does not say what was asked.

Read more ▸

How one newsletter writer is building a subscription business when “paywalls don’t really work” | NiemanLab

A year ago, a spirited and public-records-powered newsletter needed a mascot. Sleuthy the Seal — born of a “fortuitous misunderstanding” — was created. Journalist and newsletter founder David Covucci promised his wife he’d get the plucky mascot tattooed if he could hit 500 subscribers with the solo venture. The football helmet-wearing seal quickly had a permanent place on his forearm. After being laid off in May 2025 from The Daily Dot, along with much of the news staff, Covucci decided to unleash his prodigious public records skills on the multi-billion-dollar world of college football.

Read more ▸

I Hid 15 Names in a 705,216-Word Page. Gemini Found Four. Meta AI Found All 15. | Search Engine World

Meta AI listed them like a shopping list. Everdene. Virelune. Narrowbeck. Fitchwold. Brimsel. Yestrave. Mibbelth. Varkhoon. Cassivane. Turnbold. Wumbral. Mawthorn. Yarnwick. Paltrowe. And then: “The closing note section reads: ‘The last entry in the station logbook for that year reads simply: Semberly.'” Fifteen invented names, in the exact order they appear on the page. The last one sits in the final sentence of a document that runs 705,216 words. Meta AI got every single one right, including the order.

Read more ▸

If Audiences Are Moving, Move the Talent With Them | TVNewsCheck

There is a peculiar thing happening in local television. For years, companies have told employees the audience is moving. Streaming. Mobile. Social. YouTube. Instagram. Connected TV. Pick a platform. We have a strategy deck for it. Then, in some cases, those same companies keep measuring the value of their most recognizable talent by one old question: How many viewers are showing up at 6 p.m.? That may be the wrong question. This is not an argument that stations should never eliminate anchors.

Read more ▸

The Next AI Protocol Won’t Save Your SEO Strategy | SEJ

Recently, I found myself in another conversation that has become increasingly familiar. A senior executive had received a warning from an AI visibility assessment vendor that the company was not sufficiently prepared for AI search. Among the recommendations was something I have seen appearing more frequently: the company needed an llms.txt file. Suddenly, an emerging, and still-debated publishing format had become an executive concern.

Read more ▸