Google, OpenAI, Anthropic, and Microsoft Agreed on One Line That Blocks AI Training on Your Website: The 15-Minute Read Before You Add It
The four biggest AI companies now back a single robots.txt line that keeps your site out of model training. Add it blind and you may also hide from the AI answers that send customers your way. Fifteen minutes tells you what your site already says and what to change.
What happened. On October 30, Google, OpenAI, Microsoft, and Anthropic said they will honor one shared signal in a website's robots.txt file that opts the site out of AI training. The signal is an IETF draft written by an Anthropic engineer. It uses the existing robots.txt format, so there is no new file to create. The two lines look like this:
User-Agent: __usage_ai-training
Disallow: /
Rollout is uneven. Anthropic says it has started honoring the signal. OpenAI says it will in the coming months. Microsoft starts February 1, 2027. Google says 2027. Until then, each company's own crawler rules still apply.
The part that matters for a business owner. This signal covers training only. It does not control whether an AI tool can read your site to answer a customer's question like "who does water heater replacement near me." That is a separate decision, and for most local and service businesses it is the one with money on it. Google and Microsoft both warn that blocking AI access broadly can reduce how often your business shows up in AI answers.
So there are two questions, and most owners have answered neither on purpose:
- Do I want my pages used to train future models? (Many owners say no. There is no revenue in it for you.)
- Do I want AI assistants to read my pages when a customer asks for a recommendation? (Most local businesses want yes.)
The trap: your web builder or host may have already answered both for you. Several platforms added blanket AI-bot blocks in the last year. If yours did, your site may be invisible to the assistants that recommend businesses, and you never chose that.
The 15-minute check.
- Read your file. Open a browser and go to yoursite.com/robots.txt. Copy everything you see. If the page is blank or missing, your site blocks nothing, and you can skip to step 3.
- Have AI translate it. Paste the file into Claude or ChatGPT with this:
Here is my website's robots.txt file. For each rule, tell me in plain words which company's bot it affects and whether that bot is used for AI training, for AI search answers, or for regular search. Then tell me: based on this file, can ChatGPT, Claude, Gemini, Perplexity, and Copilot read my site to recommend my business to a customer? List any line that blocks them. Separate that from any line that only blocks training.
- Decide the two questions. If you want to opt out of training, add the two lines above. If your file blocks the search and answer bots and you did not mean to, remove those lines or ask your web platform to turn off its AI-bot block.
- Recheck in 2027. The shared signal is a draft and the big four turn it on at different times. Put a note on the calendar for February to reread your file.
What this is worth. A customer who asks an AI for a recommendation and gets your competitor's name is a lead you never see lost. The training decision is a preference. The answer-bot decision is revenue.
Caution. Robots.txt is a request to well-behaved crawlers. It will not remove anything a model already learned, and it does not stop a bad actor. Keep private documents off your public site in the first place.