Every Fast Write Moves Work Somewhere Else

Core Thesis: In storage systems, speed is never "free." Any optimization that makes a write operation faster simply shifts the associated effort—latency, risk, or cleanup—to a different part of the system or a later point in time.

Every storage engine must make a fundamental decision: At what point do we tell the client the write was successful?

The Spectrum of Durability vs. Latency

The timing of the "success" signal determines the system's resilience. We can categorize these choices by where the data resides when the acknowledgment is sent:

While a fdatasync() call on a local SSD protects against a kernel panic, it cannot protect against the physical destruction of the server. Conversely, waiting for a network volume or a quorum of database nodes ensures survival even if a whole data center zone fails, but it injects network round-trip time into every single request.

---

The Nuance of NVMe and Object Storage

It is a common misconception that NVMe always means local storage. In reality, NVMe is an interface, not a location. A remote block volume can be exposed via the NVMe interface, meaning the syscall remains the same, but the underlying latency and failure domains are entirely different.

Modern Architectural Trends

Many contemporary designs leverage a hybrid approach:

1. Object Storage: Used for immutable, sorted files. The storage service handles durability.
2. Local NVMe SSD: Used as a high-speed cache or a Write-Ahead Log (WAL).
3. LSM Tree Layout: Organizes data into a log-structured merge tree to avoid random updates to shared pages.

This separation allows compute nodes to be ephemeral while the storage service maintains the "source of truth." However, the database must still manage:

• Versioning (which record is current?).
• Indexing (how do we find data quickly?).
• Garbage Collection (removing obsolete versions).
• Crash Recovery.

Analyzing Operational Costs

To understand where work is shifted, we can look at the theoretical costs of an append-only key-value store using $\text{Big O}$ notation:

• GET: $O(\text{key bytes} + \text{returned bytes})$
  • Work: 1 index lookup $\rightarrow$ 1 value read.
• PUT: $O(\text{key bytes} + \text{payload bytes})$
  • Work: Hashing the payload $\rightarrow$ WAL append $\rightarrow$ shared fdatasync().
• DELETE: $O(\text{key bytes})$
  • Work: Appending a "tombstone" record.

Why these costs are deceptively low:

• GETs are fast because the index pre-calculates the location; we don't scan the whole history.
• PUTs are efficient because multiple writes can be batched into a single device flush.
• DELETEs are nearly instant because they don't actually delete data—they just mark it as dead, deferring the actual cleanup to a background process.

The Anatomy of a Client PUT

To distinguish between the request sent by the user and the request sent to the backend, I use two terms:

• client PUT: The KV request to the database.
• object PUT: The HTTP request to object storage.

The Write Path

When a client PUT is processed through a buffered system, the data flows as follows:

In a standard Linux environment, the write() system call typically returns as soon as the bytes hit the page cache. However, for a write to be truly durable, the system must call fdatasync(), which forces the kernel to flush those pages to the physical SSD and clears the device's internal cache.

Latency Comparison

The performance gap between local and remote writes is staggering:

• Local fdatasync(): $\approx 0.1\text{ms}$ to $1\text{ms}$.
• S3 Express One Zone: Single-digit milliseconds (e.g., $\approx 6.4\text{ms}$ to $7\text{ms}$ for 4KB).
• S3 Standard: Significantly slower than Express.

If a local write is $0.1\text{ms}$ and a remote write is $6.4\text{ms}$, the local path is 64x faster. This is a massive benchmark win, but it comes at the cost of durability (you are now vulnerable to host failure).

Summary Checklist for Evaluating "Fast" Systems

Whenever I encounter a benchmark claiming "ultra-fast" write speeds, I run through this mental checklist to find where the work was moved:

• Where is the acknowledgment point? (Memory? Local Disk? Remote Quorum?)
• What is the failure domain? (Does a power loss wipe the data?)
• What is the deferred cost? (Is there a massive compaction/cleanup task running in the background?)
• Is the latency "real" or "buffered"? (Did it actually hit the platter/cell?)

Figure 1: Conceptual representation of the inverse relationship between write speed and data safety.

Ultimately, the "magic" of a fast write is simply the art of deciding which part of the system will pay the price later.