Cloudflare halves DNS build delay and cuts memory use by 85% with targeted rebuilds

Cloudflare has successfully halved the delay between a DNS change and its data being written to Quicksilver, the company's distributed key-value store, while cutting peak memory use by 85%. This achievement stems from replacing inefficient full-zone rebuilds with targeted builds that only process specific records or settings when possible. The system now handles nearly three times the volume of events per second compared to 2022, processing about 700 DNS record changes and roughly 100 other zone setting events simultaneously.

How DNS Changes Reach the Edge

A DNS zone represents the collection of records and settings for a specific domain. Cloudflare utilizes a multi-stage pipeline to transform these changes into data available across its global network. The process begins when a customer modifies a DNS record via the API or dashboard, or another service alters a zone setting. These changes are stored in a database and converted into Kafka events by the DNS Kafka Aggregator.

The Zone Builder then reads the required zone data and converts it into keys understood by authoritative DNS servers. It compares these new keys against Quicksilver, writing only what has changed before Quicksilver propagates the updates across Cloudflare's network. This simplified view omits critical steps where the system combines records with settings, applies serving rules, and organizes results for efficient server access.

Why Full Builds Stopped Scaling

The original Zone Builder relied on a property that allowed it to ignore the reason for a change; it simply rebuilt the desired state from the current database. This involved reading every DNS record in the zone, converting them into Quicksilver keys, and comparing those keys with existing ones to remove stale data or update metadata.

However, this approach became unscalable as zones grew to hold tens of millions of records. A full build must read, encode, and compare every single record while holding the entire zone in memory—a task impossible for large zones that cannot fit into available RAM. Additionally, targeting builds for zone settings proved difficult because changes like Universal SSL can alter Certification Authority Authorization (CAA) records at the zone apex, requiring complex logic to determine exactly what needs rebuilding without risking stale data.

Making Events Say What Changed

To enable targeted rebuilding, Cloudflare modified its event producers to include changed zone fields and settings, providing both previous and new values when available. The system then traced each setting through the code that generates DNS output to classify potential impacts. Not every event was equally specific; some database functions only updated timestamps to indicate a change occurred, which could be harmless bookkeeping or a signal that data elsewhere had shifted.

Until the system could identify the caller and its purpose, timestamp-only updates still triggered full builds. The planner never turned uncertainty into less work. If an event lacked details, was unknown, or might affect records anywhere in the zone, it defaulted to a safer but more expensive full build strategy.

Choosing the Smallest Safe Build

Quicksilver output consists of record keys containing DNS answers and a smaller shared metadata key that tells authoritative servers how to use the zone. Based on this distinction, the new planner chooses one of five build types: Resources only for ordinary A, AAAA, TXT, or CNAME changes; Resources and metadata for nameserver and wildcard updates; Apex and metadata for settings like Universal SSL; Purge for removing unpublished zones; and Full zone as a fallback when safety cannot be proven.

The core rule is simple: every key written by a targeted build must be byte-for-byte identical to what a full build would produce. If this equivalence cannot be proven, the system defaults to a full build. Above a fixed record-count threshold, the Zone Builder avoids escalating to a full build entirely, instead rebuilding the apex, metadata, and specific names identified by queued events. This tradeoff is accepted because a full build of massive zones cannot finish within reasonable timeframes and would block other changes on its Kafka partition while retrying.

When One Name Means More Than One Zone

Quicksilver stores record keys separately for each zone but maintains one shared metadata key for every Cloudflare zone sharing the same public domain name. This shared key acts as a directory, listing internal zone IDs that authoritative DNS servers may use to answer queries. Multiple zones can exist for the same name during customer onboarding or offboarding processes, with a priority policy selecting a small, bounded set to publish at any given time.

For instance, if four internal zones exist for example.com but only three are selected in the metadata key, a TXT record change within one of those active zones affects only its specific record keys and does not require updating the shared metadata. However, nameserver or wildcard changes necessitate updates to both the zone's records and the metadata key because that information is stored there. At execution time, the Zone Builder reads the current list again; if it writes the metadata key, it reconstructs the whole key from this fresh view rather than patching an older copy.

Planning from the Event, Verifying Before Writing

An event describes what happened in the past, but a build may wait while more events arrive or the database changes. Consequently, planning occurs twice: first to create a conservative work list based on the initial event, and immediately before execution to verify against a fresh publication snapshot. This second check ensures current zone settings, network eligibility, and same-name zone IDs are accurate.

Fresh state can alter the build outcome in four ways: converting a targeted build into a full build if new features require complete-zone work; changing a build from a rebuild to a purge if the zone should no longer be published; skipping a build entirely if the triggering zone is removed from the metadata key; or reconstructing a shared metadata key from the current candidate set. These checks rely on read-only database replicas, ensuring builds do not compete with customer writes while Kafka delivery and replication proceed independently.

Combining Changes in One Scheduler

Previously, record work and full-zone work utilized different schedulers that could not coordinate with each other. Cloudflare replaced this with a unified scheduler where the planner decides what must be built and the scheduler determines when it can run. When multiple events for a single zone arrive before a build starts—such as an A record change at www.example.com combined with a wildcard change—the scheduler merges their work into one operation, combining required names and metadata updates.

Merging only adds work; if any event requires the complete zone, the combined build becomes a full build. The scheduler tracks DNS work separately from Kafka messages waiting to be acknowledged, ensuring that retries preserve both. Even when a full build or purge no longer needs individual record names, the associated messages remain attached until the result is safely written.

Keeping Kafka Moving

A slow build can delay unrelated customers even if workers are available, as Kafka divides messages into ordered partitions with one committed offset per partition representing completed work. Advancing this offset past unfinished work risks losing customer changes after a restart. Cloudflare employs soft committing to mitigate this: a build can finish out of order and be marked complete in memory, but Kafka's committed position advances only after every earlier message has also finished.

If the Zone Builder restarts, in-memory completions disappear and Kafka replays pending messages. The system deliberately chooses repeated work over losing a change. Targeted plans reduce how long the oldest message stays unfinished, while the scheduler prevents one busy zone from monopolizing workers by enforcing per-zone rebuild intervals that vary by Cloudflare plan. Old queued work is allowed to run before blocking a Kafka partition for too long, and zone locks ensure failing builds cannot loop indefinitely.

Releasing Safely

A smaller build is only useful if it produces identical DNS data. To verify this, Cloudflare added equivalence tests for nameserver, wildcard, and apex paths. Each test runs both a full build and a targeted build from the same database state, pins time-dependent values, and compares every Quicksilver key byte-for-byte. Planner and scheduler tests cover classification, same-name zones, merging, purges, and escalation scenarios.

During rollout, a full build was run next to every targeted build for comparison until the mirror environment provided sufficient confidence. Production ran the old schedulers while a mirror environment tested the new unified scheduler on identical events over 24 hours. The two environments consumed the same Kafka events, read from the same database, and wrote equivalent data to Quicksilver, creating a controlled side-by-side comparison.

To ensure safety during release, every deployment runs through a Mirror Zone Builder that consumes production Kafka streams and reads production databases but writes to a separate environment serving no customer traffic. A comparer checks a 20% sample of zones for missing keys, extra keys, or different values between the two environments. Since both pipelines do not always build a zone at the exact same instant, temporary mismatches are ignored; investigators look specifically for new types of differences that persist once both catch up.

The critical groups in the testing dashboard were each checked and confirmed safe before release. The diff rate measures the share of compared zones where production and mirror disagreed on at least one key at the moment of comparison. Most discrepancies are timing differences that vanish as pipelines synchronize, so teams monitor whether the rate stays low and if any new kind of difference emerges. After confirming healthy results, the scheduler was rolled out by hashing the zone name to ensure every zone sharing a name used the same scheduler instance, preventing two schedulers from writing conflicting metadata keys for the same domain.

Results

Over 24 hours, measurements showed that 86.81% of builds were resources only, 2.54% were resources and metadata, 2.30% were apex and metadata, and 1.30% were purges. Collectively, 92.95% of builds avoided rebuilding the complete zone. Full builds accounted for the remaining 7.04%.

The two metadata build types represented full builds before this initiative, even for large zones, accounting for 4.84% of total builds. Combined with the 7.04% still classified as full builds, the total reduction in full builds was approximately 40%. While this percentage understates the work avoided because it includes the largest zones disproportionately, the average duration of remaining full builds is only 0.35 seconds, as these events typically involve new zones with few records.

Before this update, a planned batching system would have been necessary to optimize load. The unified scheduler eliminates this need by not waiting to form batches and merging events only when work is already queued for the same zone. Previously, long full builds held back the committed offset for their partitions; now Cloudflare consumes Kafka at the rate events arrive rather than falling behind in bursts of up to 1,550 messages per second.

Build completion delay—the time from a DNS record or zone setting change until the Zone Builder finishes writing to Quicksilver—was also measured. This metric excludes Quicksilver's distribution across the network, which varies by location and remains unaffected. The comparison was conducted with production running old schedulers and the mirror environment running the unified scheduler on identical events over the same 24-hour period. Both environments consumed the same Kafka events, read from the same database, and wrote equivalent data to Quicksilver, making it a controlled side-by-side test. These figures combine every build type and customer plan, so they do not constitute a strict latency guarantee.