If you run a website and want to keep its content out of AI training sets, you have until now had to write a separate rule for every company's crawler and keep that list up to date. OpenAI says it will honour a single shared signal in robots.txt that lets a site state once whether its content may be used for foundational generative AI training, which is the switch publishers have been asking for.
The signal is being drafted in the Internet Engineering Task Force's AI Preferences working group, so it is an open standard rather than another company's proprietary flag. That matters: a publisher sets one preference, and every participating provider that reads it is meant to respect it, instead of the current patchwork where you name individual crawlers one by one. OpenAI says other participants in the working group are making the same commitment in the same announcement, and it plans to implement the signal in the coming months.
For site owners this is the right shape of solution. Backing a neutral IETF draft is better for the open web than each AI company inventing its own opt-out mechanism, because a shared vocabulary is something a small publisher can actually keep up with. OpenAI also stresses that the new signal does not replace its existing per-crawler controls, which continue to work, so nothing you already have in place stops functioning.
The limits are where honesty is needed, and they are significant. The signal only helps if the big crawlers actually adopt it, and the whole scheme is voluntary, with no enforcement behind it. OpenAI's post names none of the other participants, so it is not yet clear how broad the coalition is or whether the crawlers that scrape most aggressively are on board. A standard that only some companies honour leaves the same gaps the old patchwork did.
The practical advice follows from that. This is a genuine improvement in direction, and worth welcoming, but it is a promise about a draft, not a working control you can rely on today. Site owners should keep their existing per-crawler blocks in place and watch for the implementation details OpenAI says it will share, rather than switching to the shared signal alone until it is actually live and widely respected. The idea is sound; whether it bites depends entirely on who joins.