Cloudflare Introduces AI Training Controls Without Sacrificing Search Visibility

Cloudflare Lets Websites Block AI Training While Staying Discoverable in Search

Website owners have faced a growing problem as AI crawlers become more common: how can they prevent their content from being used for AI training without disappearing from traditional search?

Cloudflare is attempting to address that problem with a new Disallow AI Training control.

Announced on September 15, 2026, the setting is designed to separate AI training from traditional search crawling when a crawler performs both functions. Cloudflare calls these mixed-use crawlers.

The company says Apple, Google and Microsoft meet its requirements for an Accountable crawler operator, combining existing controls with commitments for additional transparency and functionality.

The announcement also expands Cloudflare's approach to AI summaries, with the company planning more granular controls for how much website content can appear in AI-generated summaries.

Why AI training and search have become difficult to separate

Website owners generally want their pages to remain discoverable through search engines. Search traffic can bring users directly to websites, which is particularly important for sites supported by advertising, subscriptions or direct customer relationships.

AI training creates a different consideration.

Content collected for model training may contribute to the development or fine-tuning of AI systems without necessarily producing a direct visit to the original website.

Cloudflare says its data shows the distinction clearly. Less than 1% of Cloudflare sites choose to block Search bots, while 17% use some mechanism to block AI training.

This creates a problem when the same crawler performs both activities.

For example, blocking a mixed-use crawler can prevent AI training, but it can also prevent the crawler from supporting traditional search discovery.

Cloudflare's new control is intended to separate those two outcomes.

What is a mixed-use crawler?

Cloudflare classifies crawlers according to their behaviour.

The company identifies three major categories:

• Search: Crawling used to build a search index.

• Training: Crawling used to train or fine-tune an AI model.

• Agent: User-directed systems that retrieve webpages on behalf of a person, including chat-fetch bots and browser-use agents.

A mixed-use crawler performs both Search and Training activities.

This distinction matters because a website owner may want a crawler to continue indexing content for search while preventing that same crawler from using the content for AI training.

Cloudflare's new Disallow AI Training setting

Cloudflare says its new Disallow AI Training setting allows site owners to publish a no-training preference through robots.txt while keeping accountable mixed-use crawlers available for search.

Under this configuration, Accountable mixed-use crawlers remain allowed to support search, while other training crawlers are blocked.

Cloudflare specifically identifies training-only crawlers operated by Amazon, Anthropic, Meta and OpenAI as separate from their respective search functions, meaning blocking those training crawlers does not affect traditional search according to Cloudflare's classification.

The setting is available as a control for Training rather than Search or Agent.

This is an important distinction from simply choosing to block AI crawlers altogether.

The available controls

Cloudflare describes four relevant configurations:

Cloudflare says the Block and Block on pages with ads settings now apply to mixed-use crawlers as well.

That means selecting Block can affect both search and training.

For websites that want to preserve search visibility while refusing AI training, Cloudflare recommends using Disallow AI Training instead.

Why robots.txt alone is not enough

Cloudflare argues that robots.txt by itself cannot completely solve the AI crawling problem.

A robots.txt file can communicate a website owner's preference, but it cannot independently identify who is crawling a website, determine the crawler's purpose or prevent a crawler from ignoring the directive.

Cloudflare's network-level approach adds another layer.

The company says it can publish the preference, identify the crawler, classify its behaviour and block crawlers that do not respect the relevant controls.

Cloudflare also says its Radar platform provides visibility into how operators actually behave.

This creates a broader model:

Preference → Crawler identification → Behaviour classification → Enforcement → Reporting

Cloudflare's "Accountable" crawler designation

Cloudflare has introduced an Accountable designation for crawler operators that meet, or have committed to meeting, specific requirements.

According to the announcement, the requirements include:

1. A mechanism allowing website owners to opt out of AI training through robots.txt or a similar standard.

2. A mechanism for opting out of AI summaries.

3. URL-level visibility into which pages were made available for training.

4. Metrics showing how content appeared in search.

5. Assurance that opting out of AI training does not affect traditional search results.

Cloudflare says Apple, Google and Microsoft meet the qualifications through a combination of currently available capabilities and time-bound commitments.

Cloudflare also categorizes relevant crawlers from Amazon, Anthropic, Meta and OpenAI as Accountable because their Search and Training crawlers are separated.

The exact capabilities differ between operators.

What Apple, Google and Microsoft are doing

Applebot

Apple allows website owners to opt out of training by adding a Disallow rule for Applebot-Extended in robots.txt.

Cloudflare also says Apple currently supports AI Summary preferences through the nosnippet directive and allows content to be labelled as paywalled content to exclude it from generative output.

URL-level inspection is not yet available, according to Cloudflare. Apple has shared details with Cloudflare about an in-progress solution expected next year.

Apple has also stated that disallowing training does not affect search ranking.

Googlebot

Google allows website owners to opt out of training through the Google-Extended robots.txt control.

Cloudflare says Google also provides a webmaster portal control for excluding website content from generative search results.

Google provides metrics and reporting related to search results and AI summary results.

Cloudflare says Google is also working on additional URL-level transparency tools associated with Google-Extended, with those tools expected to launch in the weeks following the announcement.

Google has stated that disallowing Google-Extended does not affect search ranking.

Bingbot

Microsoft provides controls through Bing Webmaster Tools.

According to Cloudflare, website owners can currently express AI training preferences through Microsoft's NOARCHIVE meta tag.

Microsoft is working on extending its system to respect a no-training preference in robots.txt at the domain or site level, targeted for early 2027.

Until that support is available, Cloudflare says selecting Disallow AI Training does not automatically transmit the no-training preference to Bing through robots.txt.

Cloudflare says Bing users can instead use Microsoft's Block URLs or Content Removal tools alongside NOARCHIVE.

Microsoft has stated that using NOARCHIVE does not affect search ranking.

What changes for existing Cloudflare customers?

Cloudflare's September 15 changes affect its Bot Management and AI Crawl Control systems.

The company says:

• Block now applies to mixed-use crawlers such as Applebot, Bingbot and Googlebot.

• Block on pages with ads also applies to mixed-use crawlers.

• Block AI Bots is being deprecated in favour of separate Search, Training and Agent controls.

• Managed Robots.txt is being deprecated in favour of Bot Preference Sync.

• Certain new domains will receive Disallow AI Training as part of recommended configurations.

• Existing customer preferences will be migrated automatically in most cases.

For domains that previously used Cloudflare's granular controls, previous Training settings using Block or Block on pages with ads will migrate to Disallow AI Training.

For domains using the older Block AI setting, Cloudflare has defined migration rules based on the previous configuration.

The practical takeaway is important: website owners who want to completely remove mixed-use crawlers must now select Block, while those who want search visibility without AI training should use Disallow AI Training.

Recommended configurations for new domains

Cloudflare is also changing how it recommends settings for new domains.

Beginning September 15, new domains are offered one of two preset configurations depending on whether the site monetizes pages through advertising.

For sites without advertising monetization, the recommended configuration is:

• Preference Sync: Enabled

• Search: Allow

• Training: Allow

• Agent: Allow

For advertising-supported sites:

• Preference Sync: Enabled

• Search: Allow

• Training: Disallow AI Training

• Agent: Block on pages with ads

The distinction reflects Cloudflare's argument that advertising-supported websites depend more directly on human visitors seeing advertisements.

AI training does not require a person to visit the page, while agents can retrieve content without necessarily generating a conventional page view.

AI summaries create a separate challenge

Cloudflare's announcement goes beyond model training.

The company argues that AI summaries create a different business and distribution question.

Training determines whether website content can be used to develop AI models.

AI summaries affect how people discover information and whether they ultimately visit the original website.

A search engine or AI system may provide enough information in a generated summary that a user does not need to visit the underlying website.

But Cloudflare also points to another side of the equation.

The company says consumers referred by AI Search convert at between three times and more than five times the rate of those referred by traditional search, while consumers who read Search summaries are more than 40% more likely to end their search after reading one.

These figures are presented by Cloudflare as evidence that AI search can produce fewer visits but potentially higher-intent visitors.

The business impact therefore depends on the type of website.

An advertising-supported publisher may value traffic volume, while a retailer may place greater value on visitors who are more likely to purchase.

Cloudflare says its role is to provide visibility and controls rather than determine which outcome is preferable for a particular website.

6

Why granular controls matter

Cloudflare's announcement reflects a broader shift in how the web is managing AI crawlers.

The debate is no longer simply about whether AI bots should be allowed or blocked.

There are now several distinct questions:

• Should a crawler index content for search?

• Should content be used for AI training?

• Should content appear in AI-generated summaries?

• Should an AI agent retrieve content on behalf of a user?

• Can website owners see which pages were accessed?

• Can they verify whether their preferences were respected?

A single "Block AI" button cannot address all of these scenarios.

Cloudflare's move towards separate Search, Training and Agent controls reflects this increasing complexity.

For enterprises and publishers, this could mean more precise governance over how public web content is accessed and reused.

It also highlights the importance of interoperable standards.

Cloudflare specifically points to emerging standards such as ai-prefs and the role of organisations including the Internet Engineering Task Force (IETF) in developing common approaches.

What comes next?

Cloudflare says its next focus is giving website owners greater control over AI summaries.

The company wants to move beyond a simple site-wide allow-or-deny choice and eventually let site owners control how much of their content can be included in summaries.

Cloudflare's stated goal is to make this configuration available through its platform rather than requiring publishers to configure each AI operator separately.

The company says it plans to continue working with AI crawler operators and publicly tracking crawler controls, transparency and reporting through Cloudflare Radar.

Additional details were not disclosed in the announcement about the final implementation of the future summary controls.

What this means for website owners

Cloudflare's latest controls give website owners a more granular way to approach AI crawling.

The key distinction is straightforward:

If you want search visibility while preventing AI training, Disallow AI Training is designed for that purpose.

If you want to stop mixed-use crawlers completely, Block will also stop their search access.

The change also illustrates a larger challenge for the web: AI systems increasingly depend on publicly accessible content, while publishers and businesses want greater visibility into how that content is used.

The development reflects growing demand for clearer technical controls, transparency and interoperable standards around AI-generated search and web crawling.

As AI search and agent-based browsing continue to develop, the distinction between being discoverable, being crawled, being trained on and being summarised is likely to become increasingly important for website operators.