Google expands Mediapartners-Google crawler scope as Cloudflare flips default on AI bots for ad pages

Google widens the Mediapartners-Google crawler definition

Google has updated the crawler documentation for Mediapartners-Google so that the user agent is no longer described as specific to AdSense. txt rules, log parsers, and ad-ops audits, the practical question is whether the same user agent now covers a wider range of Google monetisation activity, which changes how it should be allowed or throttled. SEO teams that previously treated the token as an AdSense-only signal will need to revisit allowlist assumptions and any internal reporting keyed off that scope.

txt Checker](/seo/ai-bot-robots-checker/) to confirm the current user-agent rule is still fit for purpose after the documentation change.

Cloudflare blocks AI crawlers on ad inventory by default

Cloudflare has flipped the default for AI Training and Agent bots on ad pages, blocking them unless a publisher opts in. Sites running ad inventory through Cloudflare will see AI scrapers reclassified from tolerated traffic to blocked traffic on monetised surfaces, with the operator offering Pay Per Crawl and Wallets as paths for publishers who would rather charge those bots than refuse them. The net effect is a two-tier model: monetisation crawlers continue under Google's revised scope, while AI crawlers now require an explicit commercial decision rather than passive permission.

Spam enforcement tightens around mass-produced AI content

Reports around Google's August 2026 spam update indicate that mass-generated, SEO-first AI content was among the types of content hit particularly hard. For practitioners running scaled content programmes, the takeaway is that Google's spam systems are treating AI-generated material written primarily for ranking as a distinct target class, separate from spam tactics aimed at links or cloaking. Sites that have been publishing large volumes of lightly edited AI copy should expect that recovery work is not a matter of small edits; the update is targeting the production model itself.

Putting the crawler changes into a practitioner's workflow

Read together, the three shifts point to a clearer separation of duties on a site. Google's Mediapartners-Google change expands the scope of a monetisation crawler that publishers must keep reachable, Cloudflare's new default restricts AI crawlers on monetised pages unless a publisher chooses otherwise, and the August 2026 spam update signals that AI copy written for SEO alone is being treated as a quality problem rather than a neutral tactic. txt and meta robots all need a fresh review: allowlisting decisions for monetisation bots, blocking or monetisation decisions for AI bots, and content production decisions that may have leaned on mass AI generation.

The Ads Txt Generator and the Meta Robots Generator are useful starting points for confirming the two on-site files that intersect with each of these shifts.

What to check next

Concrete follow-up for the week ahead: re-read Google's Mediapartners-Google documentation to confirm the revised scope, run a robots.txt audit for AI bot tokens using an AI Bot Robots.txt Checker, and review Cloudflare's Pay Per Crawl and Wallets settings to decide whether monetising AI crawlers is preferable to blocking them. Sites that have relied on scaled AI content should also audit the production workflow against the August 2026 spam update, since the reports in circulation describe that class of content as directly targeted.