Updated November 2026
In brief: AI search is going to remain a significant strategy for getting in front of new customers, so when you’re deciding whether to block AI training from your website, consider carefully. Most B2B companies should not block AI training across their whole site. Block only the content you sell. And know that at Google, training and Gemini citations share one switch: block Google's training, and Gemini stops citing those pages.
Soon, one line of code will tell the biggest AI companies to stop training on your website. Someone who works on your site, whether that's a developer, an agency, or a well-meaning plugin, will eventually suggest adding it.
With most of the AI companies, the answer doesn’t have to be a simple on or off (Google, I’m looking at you). We’ll get into that in more detail. Instead of a yes or no, the right answer for your company might be "not across the whole site." Either way, now is a good time to start considering what your strategy should be.
What did Google, Microsoft, and OpenAI announce?
On October 30, Google, Microsoft, and OpenAI said they plan to honor a shared instruction in robots.txt, the file that tells automated crawlers what they may do on your site. Anthropic, which wrote the proposal, says it is building support, too.
The instruction looks like this:
User-Agent: __usage_ai-training
Disallow: /
Those two lines communicate an important message: "Don't use anything on this site to train AI models." Today, opting out means writing a separate rule for each company's crawler. This would cover every participating company at once.
The timing varies:
→ Microsoft starts on February 1, 2027.
→ Google is aiming for early 2027.
→ OpenAI says "in the coming months."
→ Anthropic hasn't given a date.
Nothing changes today, and the proposal is still a draft standard. But this is the time to decide what you want, before a default decides for you.
Will blocking AI training hurt my visibility in Gemini?
Google says it will treat this opt-out as if you had also blocked Google-Extended, unless your robots.txt file allows Google-Extended explicitly.
What is Google-Extended? Google-Extended is a specific setting (or "user-agent") that you place inside the robots.txt file to control how Google's AI products use your data. Think of robots.txt as a building's security desk and Google-Extended as a specific rule written on the clipboard for Google's AI team.
Unlike the other AI companies’ controls, Google-Extended has one setting—on or off—that controls whether Google can use your content to train Gemini and to inform the answers Gemini gives. Block it, and your content can drop out of those answers. If a buyer asks Gemini, "What are the best options for [your category]?" you've made it harder for Gemini to name you.
Two things it does not change:
→ Your Google Search rankings (should?) stay the same. Google treats Search indexing (Googlebot) and Gemini AI usage (Google-Extended) as completely separate systems, and its documentation asserts that blocking Google-Extended will not trigger a ranking penalty or remove your pages from standard search results.
→ AI Overviews in Google Search follow separate rules, so this opt-out doesn't remove you from them. AI Overviews are generated directly from Google’s core Search index, which uses Googlebot. It simply summarizes answers on the search results page, so any page eligible to rank in Google Search can be cited in an AI Overview, unless you’ve opted out in Search Console.
Who does the AI training opt-out affect?
→ Any company that wants to show up when buyers ask AI tools about their category
→ Any company protecting proprietary content, such as paid research, original data, course material, or gated playbooks
It’s hard to imagine a founder-led business that doesn’t belong in the first group. Who doesn’t want more people to discover their company? However, some companies sit in both.
Should my company block AI training?
Things to consider:
Your buyers ask AI tools for shortlists, comparisons, and explanations.
The AI can only describe you if it knows who you are, and it learns that from clear, public content about what you do.
Blocking training across your whole site trades that visibility for protection you may not need. Your About page, service pages, and blog posts are marketing. You want them read, quoted, and remembered. The good news is that most AI tools, Google included, let you block certain sections while leaving your marketing pages open. At Google, though, any page you block from training also stops being cited in Gemini.
(If you haven't checked what AI tools already say about you, start with my 20-minute audit: Search Your Company Like a Late-Stage Buyer Would.)
Who should decide whether to block AI training?
The person who edits robots.txt is rarely the person who owns revenue. It might be your web developer, your agency, or a setting in your website builder.
Some platforms already make it a one-click choice. Squarespace, for example, has a "Block known artificial intelligence crawlers" option under Settings > Crawlers, and it includes Google-Extended. When the shared rule goes live, expect more tools, security plugins, and website templates to offer it. Some may present it as the safe or recommended choice.
It may be safe for your content. It may not be safe for your pipeline. That trade-off has to be a discussion for leadership and whoever owns marketing and growth.
How do I decide whether to block AI training?
1. Do we want AI tools to know who we are and recommend us? For most companies, yes.
2. Is there content we sell or that gives us a competitive edge? If so, block only those sections. The rule works by folder, so you can opt out /research/ or /members/ and leave the rest of the site open.
3. If we opt out of training with other AI companies, do we still want Gemini to cite us? If so, allow Google-Extended, knowing that it also lets Google train on your content.
Then write the decision down, and send one sentence to whoever manages your website: "No changes to robots.txt or AI crawler settings without my sign-off."
Your 60-second move
Type yourdomain.com/robots.txt into your browser. Look for GPTBot, ClaudeBot, Google-Extended, or CCBot followed by "Disallow: /". If you see them and never chose to block them, someone has already made this decision for you.
Sources
OpenAI: Making it easier for websites to express AI training preferences (October 30, 2026)
Search Engine Roundtable: Google's plan to support a shared AI training control (October 30, 2026)
Search Engine Roundtable: Microsoft Bing's new AI training and grounding controls (October 30, 2026)
--
Get every tip in one free weekly email. Sign up for The 60-Second CMO Weekly at www.juliannepatty.com.
Juli Anne Patty, Fractional CMO
Juli Anne Patty is a fractional CMO for founder-led companies. She has more than 20 years of marketing leadership experience in B2B SaaS, cybersecurity, and satellite and broadcast technology, and she builds AI agents and marketing automation systems.