Daily · Article 01 · 4 min read
- OpenAI
OpenAI backs a common training-preference signal; search and access stay separate
Follow-up to October 30 announcement; November 1 recheck; implementation planned
Summary
OpenAI said on October 30 it will honor a common signal for training preferences, targeting implementation in coming months. Search controls and actual protection of private material remain separate.
Can a shop remain discoverable in search while asking that its articles not be used for AI training? OpenAI's October 30 announcement is a useful starting point for separating training preferences, search settings and protection of private material.
Key facts
OpenAI announced that it will honor a common signal through which website operators can communicate training preferences to multiple participating providers. Implementation is targeted for the coming months, with details to follow. Existing crawler controls remain in place. This is a follow-up checked on November 1, not a claim that implementation shipped on October 31. October 30 announcement
The linked document is an individual draft submitted to the IETF, the community that develops internet specifications. It is not an IETF-approved standard or a service already implemented by every AI provider. Draft status and text
1. The same page can be retrieved for different purposes
A crawler is software that automatically finds and reads web pages. Search collection helps people find a relevant page; training uses material to change what a model has learned. An AI visiting a specific URL at a user's request is another case.
OpenAI's current documentation separates these roles. Settings are independent, so an operator can allow search collection while disallowing the training crawler. Crawler documentation
| Current name | Role | Operator's decision |
|---|---|---|
| OAI-SearchBot | Collection to surface websites in ChatGPT search | Search visibility settings |
| GPTBot | Collection of material that may train generative foundation models | Preference about training collection |
| ChatGPT-User | Page visits triggered by a user's request | Unlike automatic crawling, robots.txt rules may not apply |
- Current name
OAI-SearchBot
- Role
- Collection to surface websites in ChatGPT search
- Operator's decision
- Search visibility settings
- Current name
GPTBot
- Role
- Collection of material that may train generative foundation models
- Operator's decision
- Preference about training collection
- Current name
ChatGPT-User
- Role
- Page visits triggered by a user's request
- Operator's decision
- Unlike automatic crawling, robots.txt rules may not apply
Even after search collection is disallowed, navigational links to the site can remain. The documentation's approximately 24-hour search adjustment is guidance, not a measurement of your site's update.
The cat asks Q&A
The researcherDoes refusing training also remove my site from search?
Pocket NotebookCurrent OpenAI controls let you choose training and search separately. Ask which bot and which paths the operator changed. The phrase “block AI” does not tell you whether search settings changed too.
2. A proposal to communicate one purpose across providers
robots.txt is a public file through which a site tells automated visitors its rules. Rather than maintaining only separate provider names, the proposal adds a common name for the purpose of AI training: __usage_ai-training.
Imagine a fictional shop that does not want its shop articles used for training but favors that use for its press material. These examples illustrate the draft's path-specific preferences; they are not settings saved to this site or a real shop.
| Fictional page path | Training preference | Separate decision |
|---|---|---|
Shop articles under /articles/ | Against training use | Search collection and public access |
Press material under /press-kit/ | In favor of training use | Individual crawler access rules |
- Fictional page path
Shop articles under
/articles/- Training preference
- Against training use
- Separate decision
- Search collection and public access
- Fictional page path
Press material under
/press-kit/- Training preference
- In favor of training use
- Separate decision
- Individual crawler access rules
%% How a preference would travel · OpenAI implementation of the common signal is planned
flowchart TD
A["Owner declares preferences by path"] --> B["Supporting provider reads them"]
B --> C["Provider reflects them in its process<br/>Verify actual scope separately"]The signal does not replace crawler access rules. The proposal treats content-use preferences and collection traffic rules separately. Individual draft design
The cat asks Q&A
- The researcher
Where do I check whether a provider supports it now?
Pocket NotebookCheck the provider's implementation announcement and support documentation. OpenAI describes implementation as a plan for the coming months, so do not treat support as completed. Keep maintaining existing provider-specific rules.
3. Current controls and planned implementation
Supporting a signal, implementing it in a product and approving a standard are different checkpoints. Their status tells an operator whether to retain current settings or look for new implementation guidance.
| Item | Status checked November 1 | Next evidence to obtain |
|---|---|---|
| Existing OpenAI bot controls | Documented current controls | Actual public file and collection records |
| Common training signal | OpenAI targets implementation in coming months | Implementation announcement and supported scope |
| Linked IETF document | Individual Internet-Draft dated October 30 | Revisions and formal standardization status |
- Item
Existing OpenAI bot controls
- Status checked November 1
- Documented current controls
- Next evidence to obtain
- Actual public file and collection records
- Item
Common training signal
- Status checked November 1
- OpenAI targets implementation in coming months
- Next evidence to obtain
- Implementation announcement and supported scope
- Item
Linked IETF document
- Status checked November 1
- Individual Internet-Draft dated October 30
- Next evidence to obtain
- Revisions and formal standardization status
Under the draft, absence of the common training token means that preference has not been expressed. That should not be rewritten as consent to every use. Existing crawler rules remain separate.
The cat asks Q&A
- The researcher
What should we do if our file has no new line?
Pocket NotebookFirst check whether existing search and training rules match your intentions. After the provider explains actual implementation, decide how the new signal should coexist with them. Copying draft examples into production is not a mandatory response to this announcement.
4. Private material needs actual access protection
A rule file tells visitors how the owner wants them to behave. It does not stop a page opening like a password does. RFC 9309, the existing robots.txt specification, explicitly separates these rules from access authorization; private material needs actual server protection such as authentication. RFC 9309
| Shop material | Example action | Evidence to check |
|---|---|---|
| Public shop information | Choose search and training preferences separately | File, paths and provider support documentation |
| Staff-only order records | Require login and appropriate permissions | Actual response shows no records to an unauthorized request |
- Shop material
Public shop information
- Example action
- Choose search and training preferences separately
- Evidence to check
- File, paths and provider support documentation
- Shop material
Staff-only order records
- Example action
- Require login and appropriate permissions
- Evidence to check
- Actual response shows no records to an unauthorized request
The cat asks Q&A
- The researcher
What evidence should I ask for after someone says it is configured?
Pocket NotebookAsk for the actual public file, affected paths and provider support scope. Private data also needs separate evidence that unauthenticated access is blocked. Saving a setting, provider implementation and protection of records should not share one completion label.
To close
Choose discovery and training use separately
The common signal aims to reduce the work of conveying the same preference to several participating providers. For now, preserve intentional search and training controls and compare them with the implementation details that are still to come.
Sources
- October 30 announcementopenai.com
- Individual draft designdatatracker.ietf.org
- Crawler documentationdevelopers.openai.com
- RFC 9309rfc-editor.org