Shared Memory via Virtual Memory

Consider a typical Linux server running 50 instances of a web application. Each process requires 100 MB of code and libraries. Without memory sharing, you'd need 5 GB of physical memory just for the redundant code. With page sharing, that same code occupies a mere 100 MB — a 50x reduction.

This isn't theoretical optimization; it's the difference between a $200/month cloud instance and a $4,000/month one. Page sharing is the fundamental mechanism that makes modern multi-process computing economically viable.

In this page, we'll dissect exactly how virtual memory enables this remarkable efficiency, exploring the hardware and software mechanisms that allow multiple processes to share physical memory pages while maintaining complete isolation and security.

Page sharing is made possible by the fundamental architecture of virtual memory systems. To understand it fully, we must first revisit the core abstraction that enables all modern memory management.

The Key Insight: Indirection

Virtual memory introduces an indirection layer between the addresses a process uses (virtual addresses) and the actual physical memory locations (physical addresses). This indirection is implemented through page tables, which the Memory Management Unit (MMU) consults on every memory access.

Here's the crucial observation: nothing in the virtual memory architecture requires that a physical frame be referenced by only one virtual page. The page table is simply a mapping structure, and multiple entries — whether in the same page table or in different processes' page tables — can point to the same physical frame.

This seemingly simple insight has profound implications. It means that two processes can have page table entries that contain the same physical frame number, allowing them to access the same physical memory through their own distinct virtual addresses.

Let's trace exactly how page sharing is established and maintained at the hardware and software levels.

Establishing a Shared Mapping

When the operating system decides to share a page between two processes, it performs the following operations:

1. Identify the physical frame to be shared (already allocated and containing the desired content)
2. Create or update a PTE in the second process's page table, pointing to the same physical frame
3. Set appropriate protection bits (typically read-only for safety)
4. Increment the reference count for the physical frame
5. Invalidate TLB entries if the page was previously mapped differently

Let's visualize this with a concrete example:

Virtual Address Independence

Notice that Process A maps the shared page at virtual address 0x1000, while Process B maps the same physical frame at virtual address 0x2000. The virtual addresses don't need to match — each process has complete autonomy over its virtual address space layout.

However, some types of sharing do require consistent virtual addresses across processes. Code that contains absolute addresses (rather than position-independent code) will only work correctly if mapped at the expected virtual address. This is why:

• Shared libraries are typically compiled as Position-Independent Code (PIC), allowing each process to map them at whatever virtual address is convenient
• The kernel is mapped at identical virtual addresses across all processes (in the kernel portion of the address space)
• Shared memory regions for IPC can be mapped at arbitrary addresses in each process

Page sharing in operating systems manifests in several distinct forms, each with specific use cases, implementation requirements, and performance characteristics. Understanding these categories is essential for systems programming and OS development.

For page sharing to work correctly and efficiently, the operating system must maintain sophisticated data structures that track which frames are shared, by whom, and with what properties. Let's examine this infrastructure in detail.

The Reference Counting Challenge

Reference counting for shared pages must handle several edge cases:

1. Transient references: The kernel often takes temporary references (e.g., during I/O). The _refcount must account for these beyond just page table mappings.

2. Split references: _mapcount counts page table entries, while _refcount counts all references. A page with _mapcount = 5 (five PTEs) might have _refcount = 7 (PTEs + 2 ongoing I/O operations).

3. Large pages: When using huge pages (2 MB), all component 4 KB pages must maintain consistent counts when the huge page is split.

4. NUMA proximity: Reference tracking must preserve NUMA node information to avoid migrating pages away from their optimal memory node.

The hardware plays a crucial role in enabling efficient page sharing. Modern CPUs provide several features specifically designed to support sharing semantics.

On Non-Uniform Memory Access (NUMA) systems, memory access time depends on which CPU socket is accessing which memory bank. Shared pages introduce interesting NUMA considerations.

The NUMA Sharing Dilemma

When a page is shared across processes running on different NUMA nodes:

For a shared library page accessed by processes on different nodes, someone will have remote access. The OS must decide where to place the page.

Linux AutoNUMA

Linux implements automatic NUMA balancing (AutoNUMA) that monitors access patterns and migrates pages to the accessing node. For shared pages, this creates interesting dynamics:

Page initially on Node 0
    |
    v
Process on Node 1 accesses frequently
    |
    v
AutoNUMA detects remote access pattern
    |
    v
Migrates page to Node 1
    |
    v
But now Process on Node 0 has remote access!

This back-and-forth ('NUMA ping-pong') can occur with shared pages accessed equally from multiple nodes. The solution is often to either:

• Accept remote access for shared pages as the cost of sharing
• Use huge pages to reduce migration decisions
• Replicate read-only pages (some research systems, not mainstream Linux)

Understanding sharing efficiency requires metrics to measure actual sharing in a running system. Operating systems provide various tools and interfaces to observe sharing behavior.

Limits on Sharing

Several factors limit how much memory can be shared:

1. Alignment requirements: Pages must be naturally aligned. Partially-overlapping data cannot share a page.

2. Write frequency: High-write pages break COW sharing immediately. Data structures with frequent writes are poor sharing candidates.

3. ASLR entropy: Address Space Layout Randomization places libraries at random addresses. Position-independent code handles this, but offset randomization within libraries can affect page alignment.

4. Execution state: Writable data (globals, static variables) in shared libraries is COW'd per-process.

5. Huge page granularity: 2 MB huge pages can only share if the entire 2 MB section is shareable.

6. Maximum mapcount: Linux limits how many PTEs can reference one page (128*1024 by default). Systems with extreme sharing (containers) can hit this limit.

We've established the foundational concepts of page sharing in virtual memory systems. Let's consolidate the key takeaways:

What's Next:

Now that we understand the fundamental mechanisms of page sharing, we'll explore one of its most important applications: shared libraries. We'll see how operating systems enable hundreds of processes to share a single copy of libc, how symbol resolution works with shared code, and the performance implications of library loading strategies.