Operating System Structure

Every time you boot your computer, launch an application, or simply move your mouse across the screen, you're interacting with one of the most complex pieces of software ever created: the operating system. But beneath the familiar desktop icons and command prompts lies a sophisticated architectural framework—a carefully designed structure that determines how millions of lines of code work together to manage hardware, execute programs, and keep your data safe.

The architecture of an operating system isn't an accident. It's the result of decades of engineering evolution, hard-won lessons from catastrophic failures, and countless tradeoffs between competing goals. Understanding OS structure isn't merely academic—it's the key to understanding why some systems crash while others run for years, why some are secure while others are vulnerable, and why the same hardware can feel blazingly fast or frustratingly slow depending on the OS running on it.

Before we explore specific architectural patterns, we must establish precisely what we mean by operating system architecture. In the broadest sense, OS architecture refers to the fundamental organizational structure of an operating system—how its components are organized, how they interact, and where the boundaries between them lie.

Think of it like building architecture. When an architect designs a skyscraper, they don't just arrange rooms randomly. They make deliberate decisions about:

• Which structural elements bear the load
• How utilities (water, electricity, HVAC) are distributed
• Where safety systems are placed
• How occupants move between areas

Similarly, OS architecture determines:

• What runs in kernel space vs. user space — Which code has unrestricted hardware access?
• How components communicate — Direct function calls? Message passing? Shared memory?
• How services are organized — Monolithic? Modular? Distributed across processes?
• Where trust boundaries exist — What can one component do to another if compromised?

The Three Fundamental Questions of OS Architecture:

1. Size Question: How much code runs in kernel mode?

   • Monolithic: Everything in the kernel
   • Microkernel: Minimal kernel, services in user space
   • Hybrid: Somewhere in between

2. Coupling Question: How tightly connected are components?

   • Tightly coupled: Components share data structures, call each other directly
   • Loosely coupled: Components communicate through well-defined interfaces

3. Trust Question: What happens if a component fails or is compromised?

   • Shared fate: One component's failure crashes the whole system
   • Isolated: Failures are contained to individual components

Every OS architecture is essentially a different answer to these three questions. There is no universally "correct" answer—each choice involves tradeoffs that we'll explore throughout this module.

You might wonder: does the average user or even the average developer really need to care about OS architecture? After all, most of us interact with operating systems through high-level APIs and never touch kernel code.

The answer is an emphatic yes—and here's why. OS architecture decisions made decades ago continue to shape your computing experience today. They determine:

The Maintenance Dimension:

Operating systems must evolve for decades. Windows NT's kernel architecture, designed in 1989, still forms the foundation of Windows 11. Linux's monolithic design from 1991 continues to power everything from smartphones to supercomputers.

Architectural choices made at the beginning become increasingly difficult—and expensive—to change:

• Early design flaws become technical debt that accumulates indefinitely
• Tightly-coupled components create dependency chains that break unexpectedly
• Poor abstraction boundaries cause compatibility constraints across versions

This is why kernel architects are among the most careful engineers in computing. They know their decisions will live for 30+ years.

Understanding OS architecture requires understanding its evolution. Today's architectural debates are shaped by lessons learned from decades of experimentation, failure, and refinement.

The First Generation: No Operating System (1940s-1950s)

The earliest computers had no operating system at all. Programmers had direct access to hardware, writing machine code that manipulated registers, memory addresses, and I/O ports directly. Each program was a complete system unto itself.

This approach was simple but incredibly inefficient:

• Expensive computers sat idle while programmers debugged
• Each program had to implement its own I/O routines
• Hardware differences meant programs were non-portable
• No protection—a bug in any program could corrupt anything

The Second Generation: Simple Batch Monitors (1950s-1960s)

The first "operating systems" were simple resident monitors—small programs that remained in memory and loaded/executed user programs in sequence. IBM's IBSYS and the FMS (Fortran Monitor System) were early examples.

These batch monitors introduced fundamental concepts:

• Privileged instructions reserved for the monitor
• Memory protection to prevent jobs from corrupting the monitor
• Job Control Language (JCL) to specify job sequences

But they were still essentially single programs with added supervisory code. There was no real "architecture" to speak of—just a growing collection of subroutines.

The Third Generation: Multiprogramming and the Monolith (1960s-1970s)

As computers became more capable and expensive, the need for multiprogramming—running multiple programs with overlapped I/O—drove OSes to become more sophisticated. IBM's OS/360, MULTICS, and eventually UNIX emerged during this period.

This is where the monolithic structure became dominant. Engineers needed to add features rapidly:

• Memory management
• File systems
• Device drivers
• Process scheduling
• Security mechanisms

The simplest approach? Put everything in a single program running in kernel mode, with direct access to all data structures and functions. This was fast, flexible, and easy to develop—but it created a growing ball of interconnected code.

The Fourth Generation: Questioning the Monolith (1980s-Present)

As monolithic systems grew to millions of lines of code, their problems became undeniable:

• Bugs in any component could crash the entire system
• Security vulnerabilities anywhere were exploitable everywhere
• Adding new features meant understanding vast interconnected codebases
• Testing became exponentially harder

This led to the exploration of alternative architectures: microkernels (Mach, MINIX, L4), hybrid systems (Windows NT, macOS), and modular approaches (Linux loadable modules).

The simple structure (sometimes called "primitive" or "unstructured" monolithic) represents the earliest approach to OS organization. In a simple structure:

• All OS code runs in a single address space with maximum privilege
• No clear separation between subsystems — the file system, memory manager, scheduler, and drivers are intermingled
• Interfaces are informal — any routine can call any other routine, access any data structure
• The "architecture" is essentially: one big program

MS-DOS is the canonical example of a simple-structure operating system. Let's examine why it was designed this way and what consequences followed.

MS-DOS: A Case Study in Simple Structure

MS-DOS (Microsoft Disk Operating System) was developed for the Intel 8088/8086 processors in 1981. These processors had significant limitations:

• Only 1 MB of addressable memory (and only 640 KB for programs)
• No hardware memory protection (no MMU in early versions)
• No distinction between kernel mode and user mode
• Limited processing power (4.77 MHz clock speed)

Given these constraints, a sophisticated layered or microkernel design was impractical. MS-DOS was designed as a minimal system that would:

1. Load and execute programs
2. Provide disk and screen I/O services
3. Manage a simple file system (FAT)
4. Fit in approximately 8 KB of memory

The MS-DOS Architecture (or Lack Thereof):

1. IBMBIO.COM / IO.SYS: Hardware abstraction layer providing basic I/O services
2. IBMDOS.COM / MSDOS.SYS: DOS services including file management, memory allocation, interrupt handling
3. COMMAND.COM: Command interpreter (shell)

The critical observation: there are no protection boundaries. An application program could:

• Directly access any hardware port
• Overwrite any memory location, including DOS itself
• Intercept and modify any interrupt handler
• Corrupt the file system structures directly

This wasn't a bug—it was a feature. The simplicity enabled DOS to run on minimal hardware, and it provided developers with maximum flexibility. Game developers loved it because they could bypass DOS entirely for direct hardware access, achieving better performance.

When Simple Structure Makes Sense:

Despite its limitations, simple structure remains appropriate in certain contexts:

1. Deeply Embedded Systems: A microcontroller running a coffee machine doesn't need protection—it runs a single program forever.

2. Real-Time Systems with Extreme Constraints: Some safety-critical systems (aircraft flight controls, anti-lock brakes) use minimal, thoroughly verified code where simplicity aids formal verification.

3. Bootloaders: The code that loads an OS before the OS exists must be simple—there's no OS to provide services.

4. Legacy Compatibility: Some industrial systems still run DOS-like OSes because they work perfectly well for their single-purpose applications.

As computing evolved, the limitations of simple structure became untenable. Three forces drove the transition to more sophisticated architectures:

1. Hardware Evolution:

• Protected mode processors (Intel 80286, 80386) provided memory protection, privilege levels, and virtual memory support
• More memory allowed larger, more complex OS code
• Faster processors made protection overhead negligible

2. Multiprogramming Requirements:

• Expensive computers needed to run multiple jobs simultaneously
• Interactive users demanded responsive systems
• Protection between programs became essential

3. Software Complexity:

• Feature demands grew exponentially
• Reliability expectations increased
• Security became critical as networks emerged

The Birth of True Kernel Architecture:

The response to these pressures was the development of true kernel architecture—a protected core that mediates all hardware access and enforces security policies. Key innovations included:

1. Dual-Mode Operation: CPU distinguishes between kernel mode (privileged) and user mode (restricted)

2. System Call Interface: Well-defined mechanism for user programs to request kernel services

3. Memory Protection: Hardware-enforced isolation between processes and between user/kernel space

4. Process Abstraction: Programs run in isolated virtual environments controlled by the kernel

These innovations didn't require abandoning the monolithic approach—they refined it. Modern Linux remains largely monolithic, but with protected mode separation between kernel and user space.

However, they also opened the door to more radical restructuring: layered systems, microkernels, and hybrid approaches.

It's crucial to distinguish between simple structure (no real architecture) and modern monolithic kernels (structured but integrated). Modern monolithic kernels like Linux are sophisticated, well-organized systems that simply choose to run most code in kernel space.

Characteristics of Modern Monolithic Kernels:

1. Internal Modularity: Code is organized into subsystems (VFS, memory manager, scheduler) with defined interfaces—even if those interfaces aren't enforced by hardware.

2. Loadable Modules: Device drivers and filesystems can be loaded/unloaded dynamically without rebooting.

3. Protection at the User/Kernel Boundary: While kernel code shares an address space, user processes are strictly isolated.

4. Preemptible Kernels: Modern Linux can be interrupted even during kernel operations, improving responsiveness.

5. Namespace Isolation: Even within the monolithic kernel, features like cgroups and namespaces provide process isolation.

We've established the foundational concepts of OS architecture and explored the simplest structural approach. Let's consolidate what we've learned:

What's Next: The Layered Approach

In the next page, we'll explore the layered architecture—a more disciplined approach to OS organization that influenced systems from MULTICS to THE operating system to modern security-focused designs. Layering introduced the concept of abstraction layers, where each layer only interacts with adjacent layers, creating cleaner interfaces and easier reasoning about system behavior.

We'll see how layering addressed many of simple structure's problems—and what new problems it introduced.