Cloudflare is rolling out Bot Preference Sync, a new control that keeps a site’s robots.txt file aligned with the AI-bot policies configured in Cloudflare’s dashboard.

The practical change for site operators is that crawler policy no longer has to live in two places. Cloudflare says the feature reflects zone-level settings for Search, Agent, and Training traffic by updating the corresponding preferences in robots.txt. Where a site already has a robots file, the Cloudflare-generated block is prepended while existing disallow directives are maintained.

That matters because bot policy has become operational infrastructure. A publisher may want search discovery but not model training; a commerce site may make a different choice; and both need the public preference file and edge enforcement rules to say the same thing. Cloudflare argues that mismatches between stated preferences and enforced rules can give some crawlers room to ignore or challenge a site’s intent.

The new sync option follows Cloudflare’s July split of AI traffic controls into Search, Agent, and Training categories. According to the company, Bot Preference Sync will be available to all customers, from Free to Enterprise plans, and can be turned on or off. New Cloudflare customers will have the sync mechanism enabled by default, while customers using the older managed robots.txt option will be prompted to review their preferences before moving over.

Why hosts and site owners should care

For managed WordPress hosts, agencies, publishers, and SaaS operators, this is less a marketing toggle than a governance issue. The more traffic moves through AI crawlers and agentic assistants, the more important it becomes to know whether policy is merely documented or actually enforced at the edge.

The near-term action item is simple: review current AI crawler settings, check whether hand-maintained robots.txt rules conflict with CDN or WAF policy, and document exceptions for any crawler with a commercial or distribution agreement. Cloudflare’s update reduces one source of drift, but it does not replace a content, SEO, and security decision about which bot categories should reach which parts of a site.

Source: Cloudflare Blog — “Say it once: introducing Bot Preference Sync”.