Google plans to support a shared robots.txt preference for AI training in 2027, allowing websites to express a broad choice across participating providers. The planned Google AI training control would sit alongside existing provider-specific rules, Search Advocate John Mueller said in Google's October 30 announcement.
Barry Schwartz's October 30 Search Engine Land report identified Microsoft, OpenAI and Anthropic alongside Google as supporters. PPC Land's October 31 report separately covered the four companies' support and differing implementation timelines.
Existing Controls Cover More Than Training
Google's crawler documentation, updated October 30, says Google-Extended governs future Gemini-model training and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Grounding supplies content to a model when it answers a prompt.
The documentation also says Google-Extended does not affect inclusion or rankings in Google Search. It has no separate HTTP request user-agent string; Google uses existing crawler identities and applies the token as a content-use control.
Google-Extended Keeps Its Specific Role
Under Google's announced plan, a shared training opt-out would be treated as a Google-Extended disallow when the site has not explicitly specified Google-Extended. An explicit instruction can preserve a separate Google-specific choice.
Read together, Google's announcement and documentation imply that a shared refusal without a Google-Extended exception would also restrict Gemini grounding. Conversely, explicitly allowing Google-Extended preserves permission for both covered uses, including training. Publishers weighing an exception should therefore consider the combined permission.
Google says websites that specifically allow Google-Extended remain eligible to appear in Gemini Apps responses and receive clicks through grounding. Eligibility does not guarantee inclusion or traffic.
A Draft Signal With Different Provider Timelines
Google's documentation targets early 2027 and names the proposed token, __usage_ai-training. The October 30 IETF submission remains an individual Internet-Draft without IETF endorsement or formal standing in its standards process.
The draft separates content-use preferences from crawler access rules. Wildcard user-agent groups do not express the shared training preference; the policy token must be named explicitly. This remains a developing specification.
Microsoft's October 30 announcement separates training and grounding preferences, including page-level grounding settings. It plans to honor the new preferences beginning February 1, 2027, as pages are crawled. After March 1, 2027, NoArchive and NoCache will cease serving as training or grounding preferences.
OpenAI's October 30 announcement targets implementation in the coming months while retaining existing crawler controls. Kevin Kelley said October 30 that Anthropic had started implementation, with further work required across services.
What Publishers Can Do Today
A useful preparation document would list each provider, the intended training choice, any grounding access the publisher wants to preserve, and the person responsible for approving changes. That gives developers a concrete policy to implement when support arrives, while giving editors and business teams a way to review the consequences before configuration changes. Record the reason for each exception alongside the intended outcome, such as preserving grounding eligibility. That gives future maintainers context when a provider changes its documentation or a site's publishing priorities shift.
Keep existing provider controls under review while implementation develops. An audit should distinguish which crawlers can fetch pages from which subsequent content uses the site's settings permit. That distinction helps prevent a configuration change based on the wrong assumption.
The next useful milestone is Google's implementation guidance: when support takes effect and how the shared preference and explicit provider rules operate together. Publishers should judge adoption provider by provider rather than assume identical behavior everywhere.
For our Technology coverage, we follow source and correction standards: this report attributes implementation commitments to the companies and distinguishes them from independent coverage of the announcements.
Image: AI-generated image.
Discussion
Join the discussion
Useful notes, corrections, and field experience belong here. Keep replies specific and constructive.