Optimizing the 1.1.1.1 DNS Cache: A 100 TB Memory Recovery

Cloudflare's Big Pineapple—the engine powering 1.1.1.1, Gateway DNS, DNS Firewall, and AS112—is a beast of scale. At any given moment, it manages more than 250 billion DNS cache entries.

When operating at this magnitude, efficiency isn't just a preference; it's a necessity. To illustrate the scale: $\text{1 byte waste} \times 250,000,000,000 \text{ entries} \approx 250 \text{ GB of wasted RAM across the fleet}$

By implementing five strategic modifications to how these entries are stored, Cloudflare reduced the memory footprint per entry by over 50%. This resulted in the reclamation of approximately 100 terabytes of memory, which is the equivalent of the RAM found in 130 Gen 13 servers.

Surprisingly, this reduction in space actually improved performance:

• Insert throughput: $\uparrow 43\%$
• Lookup latency: $\downarrow 19\%$

This happened because reducing allocations and improving memory locality outweighed any potential overhead from the optimizations.

---

🔍 What Exactly is Being Cached?

The system starts with a blank slate. As queries roll in, the cache populates until it reaches its limit, triggering the eviction of stale or unpopular data.

A critical factor is EDNS Client Subnet (ECS). Because authoritative servers provide different answers based on the client's network location, Big Pineapple must store multiple versions of the same query. This multiplies the number of entries and the memory required, making these optimizations vital for ECS-heavy regions.

The Data Structures

The cache consists of a Key (the query) and a Value (the response).

// The identifier for the query
pub struct CacheKey { 
    qname : Name, 
    qtype : Rtype, 
    authenticated : bool, 
    tag : Vec<u8>, 
} 

// The stored DNS response and metadata
pub struct CacheEntry { 
    timestamp : UnixTimeStamp, 
    pub inception : Instant, 
    pub ttl : Ttl, 
    pub hits : u32, 
    pub answers : Vec<Record>, 
    pub authority : Vec<Record>, 
    pub additional : Vec<Record>, 
    pub errors : Vec<ExtendedError>, 
    ... 
}

---

🧪 Benchmarking Methodology

To ensure accuracy, Cloudflare used a custom allocator wrapping Rust's System allocator to track the size and number of allocations per entry.

Benchmark Parameters:

• Traffic Mix: 56% A records, 25% AAAA records, 19% TXT records.
• TXT Simulation: Used as a proxy for all other types, with sizes randomized between 64 and 224 bytes.
• Metrics: Tracked memory usage, insert throughput, and lookup latency.
• Validation: Verified results against resident memory on production instances during rollout.

---

🛠 The Optimizations

1. Eliminating Unnecessary Capacity

In Rust, a Vec<T> or String consists of three 8-byte fields: a pointer, the length, and the capacity.

Vec<T> / String $\rightarrow$ Box<[T]> / Box<str>

Since DNS responses are immutable once cached, the capacity field (used for growth) is useless. By switching to Box slices, they saved 8 bytes per field. Across the entry, this totaled 64 bytes saved, reclaiming over 15 terabytes of RAM.

2. Consolidating Lists and Pointers

Previously, the answers, authority, and additional sections were stored as separate lists. Each Box [T] requires a pointer and a length.

The Solution: Store all records in a single contiguous list and use small offsets to mark where each section begins.

Because the number of records per section fits within a u16, they replaced 16-byte pointers/lengths with 2-byte offsets. This removed two entire lists, saving 28 bytes per entry.

3. Fighting Padding with Bitflags

Rust aligns data in memory to optimize CPU access, which often introduces "padding" (empty space).

• Old way: Multiple bool fields scattered in the struct.
• New way: Packed booleans into a single bitflag.

This didn't just save the size of the booleans; it eliminated the surrounding padding, shrinking the struct significantly.

4. Optimizing Record Ownership

Every DNS record has an "owner" (the domain it belongs to). Usually, this is the same as the queried domain.

Example 1: Standard Query

$ dig example.com A
;; ANSWER SECTION:
example.com. 300 IN A 198.51.100.2

(Owner is the same as the query)

Example 2: CNAME Query

$ dig example.com A
;; ANSWER SECTION:
example.com. 300 IN CNAME canonical.example.com.
canonical.example.com. 300 IN A 198.51.100.2

(Owner differs from the query)

To handle this efficiently, they leveraged RFC 1035 name compression, which uses 2-byte pointers to refer back to previously mentioned domains rather than duplicating the string.

---

📊 Summary of Impact

Memory Flow Visualization

Figure 1: Conceptual representation of memory reclamation across the fleet.