Intel x86 Memory Model

When the Intel 80386 processor introduced protected mode in 1985, it fundamentally transformed how operating systems could manage memory and enforce security. Before protected mode, programs ran in a flat, unprotected address space where any process could read, write, or execute any memory location—including the operating system itself. A single buggy or malicious program could corrupt the entire system.

Protected mode changed everything. It introduced hardware-enforced memory protection, privilege levels, and segmentation that enabled operating systems to isolate processes, protect kernel memory, and build the secure, multitasking systems we rely on today. Understanding protected mode is essential for any systems engineer working with x86 architecture, operating system internals, or low-level security.

To truly understand protected mode, we must first understand what it replaced. The original Intel 8086 processor (1978) operated in what we now call real mode—a simple, direct memory addressing scheme that became the foundation of the IBM PC architecture.

Real Mode Characteristics:

In real mode, the processor uses a 20-bit address space, limiting memory to 1 MB (2²⁰ bytes = 1,048,576 bytes). Memory addresses are formed by combining a 16-bit segment register value with a 16-bit offset:

Physical Address = (Segment × 16) + Offset

This segment:offset scheme was originally designed to address more than 64 KB (the maximum addressable by a single 16-bit register) while maintaining backward compatibility with existing software.

The Problem This Created:

As personal computing evolved, the limitations of real mode became increasingly apparent:

1. No Memory Protection: Any program could overwrite any memory, including the operating system
2. Limited Address Space: Only 1 MB of addressable memory (minus reserved areas, leaving ~640 KB for applications—the infamous "640K barrier")
3. No Multitasking Support: No hardware support for process isolation or context switching
4. No Ring Protection: No distinction between privileged OS code and user applications
5. Single Address Space: All programs shared the same memory view with no isolation

These limitations made real mode unsuitable for serious multitasking operating systems. Intel's response was the introduction of protected mode with the 80286 processor (1982), significantly enhanced in the 80386 (1985).

Protected mode represents a fundamental reimagining of how the processor manages memory and enforces security. Unlike real mode's simple segment:offset calculation, protected mode treats segment registers as selectors that index into descriptor tables containing detailed information about each memory segment.

The Core Architectural Shift:

In protected mode, a segment register no longer contains a base address. Instead, it contains a selector—an index into a descriptor table. The actual segment information (base address, limit, access rights, privilege level) is stored in the descriptor table entry. This indirection is the cornerstone of protected mode's power.

Key Protected Mode Features:

The transition to protected mode brings several critical capabilities:

1. Extended Address Space: Up to 4 GB of linear address space (32-bit addressing)
2. Memory Protection: Hardware-enforced access control (read, write, execute permissions)
3. Privilege Levels: Four protection rings (0-3) for hierarchical privilege separation
4. Segment Limit Checking: Hardware verification that offsets don't exceed segment boundaries
5. Type Checking: Enforcement of segment types (code vs. data vs. system)
6. Virtual Memory Support: Foundation for paging and demand paging
7. Task Isolation: Hardware support for multitasking via Task State Segments (TSS)

One of the most important innovations of protected mode is the implementation of privilege levels, also known as protection rings. The x86 architecture provides four privilege levels, numbered 0 through 3, with ring 0 being the most privileged and ring 3 the least privileged.

This hierarchical privilege model enables the operating system to maintain strict separation between different types of code, ensuring that less privileged code cannot interfere with more privileged code or access resources beyond its authorization.

Why Only Ring 0 and Ring 3 Are Typically Used:

While the x86 architecture provides four privilege levels, most modern operating systems (Linux, Windows, macOS, BSD) use only Ring 0 (kernel mode) and Ring 3 (user mode). Rings 1 and 2 are rarely used for several reasons:

1. Complexity vs. Benefit: Managing four privilege levels adds significant complexity with minimal practical benefit for most OS designs
2. Performance Overhead: Additional ring transitions add overhead without corresponding security gains for typical workloads
3. Portability: Other processor architectures (ARM, MIPS, RISC-V) typically provide only two privilege levels, making code that depends on four levels non-portable
4. Virtualization Considerations: When hypervisors were introduced, they needed a privilege level above the guest OS kernel. With only two levels in use, the hypervisor could take Ring 0 while the guest OS used Ring 1 (now superseded by hardware virtualization extensions)

The two-ring model provides a clean separation: untrusted user code in Ring 3 and trusted kernel code in Ring 0.

Protected mode implements several hardware-enforced protection mechanisms that work together to ensure memory safety. These mechanisms operate at every memory access, with the processor performing multiple checks before allowing the access to proceed.

The Protection Checks:

When code attempts to access memory, the processor performs the following checks in sequence:

Privilege Level Comparisons:

The processor uses three privilege-related values for access control decisions:

• CPL (Current Privilege Level): The privilege level of the currently executing code (stored in CS selector bits 0-1)
• DPL (Descriptor Privilege Level): The privilege level required to access a segment (stored in the segment descriptor)
• RPL (Requested Privilege Level): An override privilege level specified in a segment selector (bits 0-1), used to prevent privilege escalation

The general rule for data segment access:

Access allowed if: MAX(CPL, RPL) ≤ DPL

For code segments, the rules differ depending on whether it's a conforming or non-conforming segment, and whether the access is via a call, jump, or return.

At the heart of protected mode is the segment descriptor—an 8-byte (64-bit) data structure that defines a memory segment's properties. Understanding the descriptor format is essential for comprehending how protected mode works at the hardware level.

Each descriptor contains:

• The segment's base address (32 bits, split across the structure for historical reasons)
• The segment's limit (20 bits, representing either bytes or 4KB pages)
• Access rights and type information
• Privilege level
• Various flag bits controlling behavior

Why the Fragmented Layout?

The segment descriptor's unusual layout—with base address and limit split across non-contiguous bits—is a result of Intel's commitment to backward compatibility. The 80286 introduced protected mode with 24-bit base addresses and 16-bit limits. When the 80386 extended this to 32-bit addressing, Intel added the additional bits in previously reserved fields rather than redesigning the entire structure. This preserved backward compatibility at the cost of a somewhat awkward format.

Limit Granularity:

The 20-bit limit field can represent:

• Byte granularity (G=0): Limits from 1 byte to 1 MB (2²⁰ bytes)
• Page granularity (G=1): Limits from 4 KB to 4 GB (2²⁰ × 4 KB pages)

The actual segment size equals (Limit + 1) × granularity. For a 4 GB segment, set Limit = 0xFFFFF (all 1s) with G=1, yielding (0xFFFFF + 1) × 4 KB = 4 GB.

The transition from real mode to protected mode is a critical bootstrap operation that every operating system must perform during startup. This transition must be done carefully—there's a brief moment where the processor is in an inconsistent state, and any mistakes will crash the system.

The Standard Transition Sequence:

Why Each Step is Necessary:

1. Disable Interrupts (CLI): During the mode switch, the interrupt handler infrastructure changes. An interrupt during transition would catastrophically fail.

2. Enable A20 Line: The 8086 compatibility feature that wraps addresses above 1 MB must be disabled. On original IBM PCs, this was done via the keyboard controller (!), creating the infamous "A20 gate" problem.

3. Load GDT: The processor needs valid descriptors before using selectors. The GDT must be loaded while still in real mode.

4. Set PE Bit: This single bit flip activates protected mode. From this moment, the processor interprets segment registers as selectors.

5. Far Jump: Flushes the pipeline and loads CS with a valid protected mode selector.

6. Reload Segment Registers: All other segment registers must be reloaded with valid protected mode selectors.

7. Set Up Stack: Establish a protected mode stack before calling any functions.

8. Re-enable Interrupts: Only after setting up the IDT can interrupts be safely re-enabled.

Although protected mode was designed over 40 years ago, its concepts remain fundamental to modern x86-64 systems. Understanding protected mode is essential because:

It's Still the Foundation:

Even in 64-bit long mode, x86-64 processors build upon protected mode concepts:

• Segment registers still exist (CS, DS, ES, FS, GS, SS)
• Privilege levels (rings) are unchanged (Ring 0-3, typically only 0 and 3 used)
• The GDT is still required (though with different descriptor formats for 64-bit code)
• Task State Segments (TSS) are still used for ring transitions and interrupt stack switching
• Protection faults (#GP, #SS) still occur when protection rules are violated

The Flat Memory Model:

Modern operating systems use protected mode (and long mode) with a flat memory model—all segments have base address 0 and extend to the full address space. This effectively disables segmentation's memory partitioning while retaining its protection features.

Why flatten segmentation but keep it?

1. Simplicity: A flat model simplifies memory management; only paging handles virtual memory
2. Performance: Segment limit checks add overhead; flat segments eliminate this
3. Portability: Other architectures don't have segmentation; flat models are more portable
4. Security is handled by paging: Page tables provide fine-grained protection (4KB granularity vs. segment-level)

However, segmentation features are still used for:

• Thread-Local Storage (TLS): FS and GS registers provide per-thread base addresses
• Kernel/User Separation: Different code segments for Ring 0 and Ring 3
• SYSENTER/SYSEXIT: Fast system call mechanisms that rely on segment selectors

We've explored the foundational architecture of Intel x86 protected mode—the mechanism that transformed the PC from a single-user, unprotected toy into a platform capable of running secure, multitasking operating systems.

What's Next:

With protected mode's foundational concepts established, we'll dive deeper into the Global Descriptor Table (GDT)—the system-wide table of segment descriptors that forms the backbone of x86 memory management. We'll examine its structure, entry types, and how operating systems configure it during boot.