Write Strategies

When your application calls write(), something remarkable happens—or rather, something that should happen. The data must traverse multiple layers of abstraction, crossing kernel boundaries, navigating buffer caches, and ultimately finding its way to magnetic platters or flash cells where it can survive power failures, crashes, and the entropy of time.

But here's the question that has occupied operating system designers for decades: When exactly should data be committed to persistent storage?

This isn't a theoretical concern. The answer determines whether your database can guarantee ACID properties, whether your file system survives sudden power loss without corruption, and whether your application's performance suffers under write-heavy workloads. The "write strategy" you choose—or that your operating system chooses for you—represents one of the most consequential trade-offs in systems engineering.

We begin our exploration with the most conservative approach: write-through. It's conceptually simple, provides the strongest guarantees, and pays the highest performance cost. Understanding write-through deeply will provide the foundation for appreciating why more complex strategies exist and when each is appropriate.

Write-through is a caching strategy where every write operation updates both the cache and the backing store synchronously before the write is considered complete. The operation only returns success to the caller after the data has been durably committed to persistent storage.

Let's be precise about what "synchronous" means in this context:

1. Temporal Ordering: The write to persistent storage happens during the write system call, not at some later time
2. Completion Semantics: The write() system call does not return until the backing store confirms successful persistence
3. No Buffering Aggregation: Each write operation independently triggers I/O; writes are not batched or coalesced

This creates a strict happens-before relationship: if write() returns success, the data is guaranteed to exist on the persistent medium. If the system crashes immediately after the call returns, the data survives.

The Write-Through Data Flow

Consider what happens when an application writes 4KB of data with write-through semantics:

Application: write(fd, buffer, 4096)
    │
    ▼
┌─────────────────────────────────────────────────────────┐
│  User Space → Kernel Space Transition (syscall)         │
└─────────────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────────────┐
│  VFS Layer: Validate fd, check permissions              │
└─────────────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────────────┐
│  Page Cache: Copy data to kernel buffer                 │
│  [Data now in RAM cache]                                │
└─────────────────────────────────────────────────────────┘
    │
    ▼ (Write-through: DO NOT return yet)
┌─────────────────────────────────────────────────────────┐
│  Block Layer: Translate to block device request         │
└─────────────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────────────┐
│  Device Driver: Issue I/O command to hardware           │
└─────────────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────────────┐
│  Storage Device: Write to persistent medium             │
│  [BARRIER: Wait for device confirmation]                │
└─────────────────────────────────────────────────────────┘
    │
    ▼ (Confirmation received)
┌─────────────────────────────────────────────────────────┐
│  Return path: Propagate success back to application     │
└─────────────────────────────────────────────────────────┘
    │
    ▼
Application: write() returns 4096 (success)

The critical observation is the BARRIER step. The application is blocked—suspended—until the storage device confirms the write completed. This waiting period is the source of both write-through's strength (durability guarantees) and its weakness (latency).

To truly understand write-through performance, we must descend to the hardware level. The latency penalty depends entirely on the physical characteristics of the storage medium.

Traditional Hard Disk Drives (HDDs)

For a spinning disk, a write operation involves:

1. Seek Time: Moving the read/write head to the correct track (3-15ms average)
2. Rotational Latency: Waiting for the target sector to rotate under the head (half a rotation on average)
   • 7200 RPM drive: 60,000ms ÷ 7200 ÷ 2 ≈ 4.17ms
   • 15,000 RPM drive: 60,000ms ÷ 15000 ÷ 2 ≈ 2ms
3. Transfer Time: Actually writing the data (negligible for small writes)
4. Settling Time: Head stabilization before write (~0.5-1ms)

A single write-through operation on a 7200 RPM HDD thus costs approximately 8-20ms. This yields a maximum of 50-125 random write IOPS.

Solid State Drives: A Complicated Picture

SSDs dramatically reduce write latency, but the story has nuances:

1. Program/Erase Asymmetry: NAND flash can only be written to erased blocks. If no erased blocks are available, the SSD must erase first, which takes 1-5ms per block.

2. Write Amplification: The Flash Translation Layer (FTL) may need to write more data than requested (read-modify-write for partial page writes).

3. Device Caching: Most SSDs have internal DRAM caches. A write-through policy at the OS level doesn't guarantee immediate NAND persistence unless the device is configured appropriately or Forced Unit Access (FUA) is used.

The Hidden Problem: Device-Level Write Caching

Modern storage devices have their own caches. A "write-through" policy at the OS level means:

• Data is sent to the device before write() returns
• The device may buffer the data in volatile RAM
• A power failure at this moment loses data

True durability requires either:

• Disabling device write cache (significant performance penalty)
• Using FUA (Forced Unit Access) on each write (bypasses device cache)
• Using write barriers (flushes device cache at specific points)
• Battery-backed write cache (allows device caching safely)

Operating systems provide multiple mechanisms for applications to request write-through semantics. Understanding these mechanisms is essential for building reliable systems.

Linux Implementation Mechanisms

Understanding the Semantic Differences

The distinctions between these methods matter enormously for correctness:

What "Metadata Durable" Means

File metadata includes:

• Modification timestamps (mtime, ctime)
• File size (if extended)
• Permission changes
• Link count
• Inode updates for block allocation

For pure data integrity (e.g., database files), O_DSYNC is often sufficient. For filesystem consistency (e.g., after creating a new file), full O_SYNC or fsync() may be required.

Let's develop a rigorous understanding of write-through performance through mathematical modeling. This analysis enables capacity planning and helps determine when write-through is viable.

Single-Threaded Write Throughput Model

For sequential write-through operations:

Throughput = Write_Size / (T_syscall + T_cache + T_io + T_confirm)

Where:
  T_syscall  = System call overhead (~1-5µs)
  T_cache    = Page cache copy time (~0.1µs per KB)
  T_io       = Device I/O latency (varies by device)
  T_confirm  = Confirmation propagation (~1-2µs)

For a 4KB write to various devices:

Queue Depth Impact

Write-through with queue depth 1 (one operation at a time) is the worst case. Modern storage supports concurrent operations:

Effective_Throughput = Base_Throughput × min(Queue_Depth, Device_Parallelism)

NVMe SSDs support queue depths of 64K+ across multiple queues. Even with write-through, parallelism helps:

• QD=1: 47,619 ops/s (186 MB/s)
• QD=4: 190,476 ops/s (744 MB/s)
• QD=32: 1,523,809 ops/s (5.8 GB/s) — if device supports it

However, write-through with multiple threads complicates ordering guarantees. Each thread's writes are individually ordered, but inter-thread ordering requires additional synchronization.

Latency Distribution Analysis

Real-world write latency follows a distribution, not a fixed value:

Typical NVMe SSD Write Latency Distribution:
  P50 (median):      15µs    — Half of writes faster than this
  P90:               25µs    — 90% of writes faster
  P99:               50µs    — 99% of writes faster
  P99.9:             200µs   — Occasional GC delays
  P99.99:            2ms     — Rare block erase delays

For write-through applications, tail latency matters enormously. A single slow write blocks the entire transaction. This is why write-heavy workloads often prefer write-back with periodic synchronization—the latency variance is absorbed.

Cost-Benefit Trade-off Formula

For a given workload, the viability of write-through can be expressed as:

Write_Through_Viable = (Required_Throughput × Latency_P99) < Available_IOPS

Example:
  Required: 1000 writes/second
  P99 latency: 50µs (NVMe SSD)
  
  1000 × 0.00005s = 0.05 = 5% device utilization
  
  ✓ Write-through is viable with substantial headroom

Counter-example:
  Required: 50,000 writes/second
  P99 latency: 50µs
  
  50,000 × 0.00005s = 2.5 = 250% device utilization
  
  ✗ Write-through cannot meet throughput requirements

Despite its performance costs, write-through is the correct choice—sometimes the only correct choice—in specific scenarios. Understanding these cases helps you make principled trade-offs rather than defaulting to "fastest possible."

Case Study: PostgreSQL WAL Synchronization

PostgreSQL provides a canonical example of calibrated write-through usage. Its synchronous_commit setting controls when WAL records are durable:

Each level adds latency but strengthens guarantees. Production systems choose based on their tolerance for data loss versus performance requirements.

The "Two-Second Rule"

A practical heuristic: if losing the last 2 seconds of data would be catastrophic, you need write-through or near-synchronous semantics. If losing 2 seconds is inconvenient but recoverable (e.g., user can retry), write-back with periodic sync is usually acceptable.

This rule emerges from typical power failure scenarios—a UPS can usually sustain systems long enough to flush buffers if the outage is brief, but sudden failures give no warning.

The write-through concept originated in CPU cache design, and understanding this context illuminates why the trade-offs work differently at different system layers.

CPU Cache Write-Through Architecture

In a write-through CPU cache:

1. Every store instruction updates both L1 cache and the next level (L2 or main memory)
2. The write buffer allows the CPU to continue while the write propagates
3. Cache coherency is automatic—no invalidation messages needed on writes

CPU Core 0                         CPU Core 1
    │                                   │
    ▼                                   ▼
┌─────────┐                       ┌─────────┐
│ L1 Cache│                       │ L1 Cache│
│ (8KB)   │                       │ (8KB)   │
└────┬────┘                       └────┬────┘
     │ Write propagates immediately     │
     ▼                                   ▼
┌──────────────────────────────────────────┐
│            L2 Cache (256KB)              │
│    [Always coherent with L1 writes]     │
└──────────────────────────────────────────┘
     │
     ▼
┌──────────────────────────────────────────┐
│            Main Memory (DDR)             │
└──────────────────────────────────────────┘

Why CPU Caches Favor Write-Back

Modern CPUs overwhelmingly use write-back caches because:

1. Memory Bus Saturation: Write-through generates memory traffic for every store instruction. A CPU executing 1 billion stores/second would saturate any memory bus.

2. Write Combining: Write-back allows multiple writes to the same cache line to be absorbed, sending only the final value to memory.

3. Power Consumption: Memory writes consume significantly more power than cache writes. Write-back reduces memory access frequency.

However, examining when write-through is used in CPU caches reveals instructive patterns:

Implementing write-through correctly requires attention to subtle details that are easy to overlook. A system that appears to be write-through but isn't can silently lose data.

Essential Verification Steps

1. Verify Device Cache Settings

   # Check HDD/SSD write cache status
   hdparm -W /dev/sda
   
   # Disable write cache for true write-through
   hdparm -W0 /dev/sda
   

2. Confirm File System Mount Options

   # Mount with sync option (all writes synchronous)
   mount -o sync /dev/sda1 /data
   
   # Or use dirsync for directory operations only
   mount -o dirsync /dev/sda1 /data
   

3. Validate at Application Level

   // Verify O_SYNC/O_DSYNC is honored
   int fd = open("/data/test", O_WRONLY | O_SYNC);
   
   struct timespec start, end;
   clock_gettime(CLOCK_MONOTONIC, &start);
   
   for (int i = 0; i < 1000; i++) {
       write(fd, buf, 4096);
   }
   
   clock_gettime(CLOCK_MONOTONIC, &end);
   
   double elapsed_ms = (end.tv_sec - start.tv_sec) * 1000 +
                       (end.tv_nsec - start.tv_nsec) / 1000000.0;
   
   // For NVMe: expect ~15-50ms (15-50µs × 1000)
   // If <<1ms total, writes are being cached!
   printf("1000 sync writes took %.2f ms
   

", elapsed_ms);

Error Handling Imperatives

Write-through is only as reliable as your error handling. Every synchronous write can fail, and the failure MUST be handled:

ssize_t safe_sync_write(int fd, const void *buf, size_t count) {
    ssize_t written = 0;
    
    while (written < count) {
        ssize_t result = write(fd, buf + written, count - written);
        
        if (result < 0) {
            if (errno == EINTR) {
                continue;  // Retry on interrupt
            }
            if (errno == EAGAIN || errno == EWOULDBLOCK) {
                // Should not happen with sync writes
                // Log and abort - something is wrong
                log_error("Unexpected EAGAIN on sync write");
                return -1;
            }
            
            // Hard errors: ENOSPC, EIO, EDQUOT, etc.
            log_error("Sync write failed: %s", strerror(errno));
            return -1;
        }
        
        written += result;
    }
    
    // For O_SYNC files, data is durable here
    // For regular files, still need fsync
    return written;
}

RAID Controller Considerations

Enterprise RAID controllers often have battery-backed write caches (BBU). This changes the durability equation:

• Write to BBU cache: ~microseconds, durable across power loss
• BBU destages to disk: happens in background
• From OS perspective: write-through to the controller

This is the best of both worlds—write-through semantics with write-back performance. However:

• BBU has limited capacity (256MB-4GB typically)
• BBU battery can fail (monitor health!)
• Some failure modes (double disk failure) still lose BBU data

We have comprehensively examined write-through as the most conservative write strategy—one that trades performance for immediate durability guarantees. Let's consolidate the essential insights:

What's Next

Write-through establishes our baseline—the safest, slowest approach. Next, we'll explore write-back caching, the strategy that most systems use by default. Write-back trades immediate durability for dramatically improved performance by allowing dirty data to accumulate in cache. Understanding write-back's mechanisms, risks, and proper usage is essential for any systems engineer working with storage.