Buffer Overflow Concept - Buffer Overflow Attacks | OneNoughtOne

0/236

Operating SystemsBuffer Overflow Attacks

Buffer Overflow Attacks

LevelAdvanced

Duration90 mins

TopicBuffer Overflow Attacks

1 / 5

Buffer Overflow Concept

The Most Dangerous Bug in Computing History

On November 2, 1988, a 23-year-old Cornell graduate student named Robert Tappan Morris launched what would become the first major internet worm. The Morris Worm exploited a buffer overflow vulnerability in the fingerd daemon, spreading to approximately 6,000 machines—roughly 10% of the entire internet at the time. The estimated cost of cleanup exceeded $100 million.

This incident didn't introduce buffer overflows to the world; these vulnerabilities had existed since the earliest days of computing. But the Morris Worm demonstrated, with devastating clarity, that buffer overflows weren't merely academic curiosities—they were weapons capable of bringing down critical infrastructure.

Thirty-five years later, buffer overflow vulnerabilities continue to be discovered and exploited. Despite decades of research, countless tools, and multiple hardware and software mitigations, this fundamental class of vulnerability persists. Understanding why requires a deep dive into memory, pointers, and the assumptions that programming languages make about programmer behavior.

What You Will Learn

By the end of this page, you will understand what a buffer overflow is at the memory level, why they occur, how they violate program integrity, and why this class of vulnerability has proven so persistent. This foundation is essential before we explore specific exploitation techniques like stack smashing, code injection, and ROP.

What is a Buffer Overflow?

A buffer overflow occurs when a program writes data beyond the boundaries of an allocated memory region (the "buffer"), overwriting adjacent memory that the program did not intend to modify.

To understand this precisely, we must first understand what a buffer is:

Buffer: A contiguous region of memory allocated to hold a specific amount of data. In C, this might be:

A character array: char name[64];
A dynamically allocated block: char *data = malloc(1024);
A structure member: struct user { char username[32]; int age; };

Every buffer has a capacity—the maximum amount of data it can safely hold. A buffer overflow occurs when the amount of data written exceeds this capacity, causing writes to continue into adjacent memory regions.

buffer_overflow_example.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#include <stdio.h>
#include <string.h>
 
void vulnerable_function(char *user_input) {
    char buffer[64];  // Allocated 64 bytes
    
    // VULNERABILITY: No bounds checking!
    // If user_input exceeds 64 bytes, we overflow into adjacent memory
    strcpy(buffer, user_input);
    
    printf("You entered: %s
", buffer);
}
 
int main(int argc, char *argv[]) {
    if (argc > 1) {
        vulnerable_function(argv[1]);
    }
    return 0;
}

In this example, buffer is allocated with 64 bytes of capacity. The strcpy function copies bytes from user_input until it encounters a null terminator (\0). If user_input contains 100 bytes before its null terminator, strcpy will write all 100 bytes—64 into buffer and 36 into whatever memory lies beyond it.

The critical insight: The CPU has no concept of buffer boundaries. It doesn't know that buffer was "supposed" to be 64 bytes. Memory is just a continuous array of bytes, and write operations simply write to calculated addresses. The responsibility for staying within bounds falls entirely on the programmer or the programming language runtime.

Why strcpy is Dangerous

Functions like strcpy, gets, and sprintf are inherently unsafe because they have no way to know the destination buffer's size. They write until they encounter a terminator, regardless of available space. Modern code should use bounded variants like strncpy, fgets, and snprintf, though even these require careful attention to off-by-one errors and null termination.

Understanding Memory Layout

To fully grasp buffer overflows, we must understand how programs organize memory. A typical process's virtual address space is divided into distinct regions:

Text Segment (Code Segment): Contains the executable machine code instructions. This region is typically marked read-only and executable. Attempts to write here trigger a segmentation fault.

Data Segment: Divided into two sub-regions:

Initialized Data (.data): Global and static variables with explicit initial values
Uninitialized Data (.bss): Global and static variables without explicit initialization (implicitly zero-initialized)

Heap: Dynamically allocated memory. Grows upward (toward higher addresses) as the program calls malloc, new, etc. The heap is managed by the memory allocator (e.g., ptmalloc, jemalloc, tcmalloc).

Stack: Local variables, function parameters, and control flow information (return addresses, saved frame pointers). Grows downward (toward lower addresses) as functions are called. Each function invocation creates a new "stack frame".

Converting Mermaid diagram...

Why Layout Matters for Buffer Overflows

The key observation is that within each memory region, data is laid out contiguously. On the stack, local variables are allocated in adjacent memory. When you overflow a buffer on the stack, you overwrite whatever lies at higher addresses—which often includes:

Other local variables: You might corrupt application data
The saved frame pointer (EBP/RBP): Controls how the function returns to its caller's stack frame
The return address: Controls where execution jumps when the current function returns

This is what makes stack-based buffer overflows so dangerous: the return address sits at a predictable offset from local buffers, and overwriting it gives the attacker control over program execution flow.

Heap vs Stack Overflows

While stack overflows are the classic and most direct exploitation target, heap overflows are equally dangerous. They can corrupt heap metadata, function pointers stored in heap objects, or adjacent heap allocations. The exploitation techniques differ, but the fundamental issue—writing beyond buffer boundaries—is identical.

Anatomy of a Stack Frame

When a function is called, the CPU and compiler collaborate to create a stack frame (also called an activation record). This structure contains everything the function needs to execute and, critically, how to return to the caller.

Let's examine what happens when main calls vulnerable_function from our earlier example:

call_sequence.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// When main() executes: vulnerable_function(argv[1]);
// The following occurs at the assembly level:
 
// 1. Push argument(s) onto stack (or pass in registers for x64)
// 2. Execute CALL instruction:
//    - Push return address (address of next instruction after CALL)
//    - Jump to vulnerable_function
 
// 3. Function prologue (at start of vulnerable_function):
//    push rbp           ; Save caller's frame pointer
//    mov rbp, rsp       ; Set our frame pointer
//    sub rsp, 0x40      ; Allocate 64 bytes for 'buffer'
 
// 4. Function body executes...
 
// 5. Function epilogue (at end of vulnerable_function):
//    mov rsp, rbp       ; Deallocate local variables
//    pop rbp            ; Restore caller's frame pointer
//    ret                ; Pop return address into RIP, jump there

The resulting stack layout during vulnerable_function execution looks like this (for x86-64, with some simplifications):

Stack Frame Layout During Function Execution
Address (Example)	Content	Size	Role
0x7fff0080	buffer[0..7]	8 bytes	Start of local buffer
0x7fff0088	buffer[8..15]	8 bytes	...
0x7fff0090	buffer[16..23]	8 bytes	...
0x7fff0098	buffer[24..31]	8 bytes	...
0x7fff00a0	buffer[32..39]	8 bytes	...
0x7fff00a8	buffer[40..47]	8 bytes	...
0x7fff00b0	buffer[48..55]	8 bytes	...
0x7fff00b8	buffer[56..63]	8 bytes	End of local buffer
0x7fff00c0	Saved RBP	8 bytes	Caller's frame pointer
0x7fff00c8	Return Address	8 bytes	⚠️ CRITICAL TARGET
0x7fff00d0	Caller's stack frame...	...	main()'s local variables

The Exploitation Path

When buffer overflows, writes continue past buffer[63] into the saved RBP (8 bytes later) and then into the return address (16 bytes past buffer end). If an attacker provides exactly 72+ bytes of input:

Bytes 0-63: Fill the buffer
Bytes 64-71: Overwrite saved RBP (frame pointer)
Bytes 72+: Overwrite the return address

When vulnerable_function executes its epilogue, the ret instruction pops the corrupted return address into RIP (the instruction pointer). Execution then jumps to whatever address the attacker specified.

This is the fundamental buffer overflow exploit: the ability to redirect program execution by overwriting control flow data stored adjacent to a buffer.

Security Implications

Control over the return address means control over execution. An attacker can redirect execution to: (1) Injected shellcode in the buffer itself, (2) Existing code sequences in the program or libraries (return-to-libc, ROP), (3) System call wrappers to gain shell access, execute commands, or establish network connections. This transforms a memory corruption bug into complete system compromise.

Categories of Buffer Overflows

Buffer overflows are categorized based on where the vulnerable buffer is located and the nature of the overflow. Understanding these categories helps in both exploitation and defense.

Classification by Memory Region

•Stack-based Buffer Overflow — The buffer resides on the stack (local variable). These are the classic buffer overflows, with predictable layout and high-value targets (return addresses). Exploitation is often straightforward if mitigations are absent.
•Heap-based Buffer Overflow — The buffer was dynamically allocated. Exploitation typically involves corrupting heap metadata or adjacent heap objects. More complex to exploit but equally dangerous. Heap spraying and use-after-free often combine with heap overflows.
•BSS/Data Segment Overflow — The buffer is a global or static variable. These can corrupt other global state, function pointers, or GOT (Global Offset Table) entries.
•Format String Vulnerabilities — While not strictly buffer overflows, these memory corruption bugs share similar exploitation patterns. Functions like printf(user_input) can both read and write arbitrary memory.

Classification by Overflow Direction

•Linear Overflow — Writing sequentially past the buffer end. The classic case where strcpy just keeps writing until it finds a null byte.
•Integer Overflow Leading to Buffer Overflow — Arithmetic on size values wraps around (e.g., size + 1 becomes 0 for a 32-bit unsigned integer at max value). This leads to under-allocation, and subsequent writes overflow the undersized buffer.
•Off-by-One Overflow — Writing exactly one byte past the buffer end. Often occurs with fence-post errors in loop termination conditions. Surprisingly exploitable in many scenarios despite the small overflow size.
•Arbitrary Relative Write — When array indexing goes unchecked, allowing writes at attacker-controlled offsets relative to the buffer base. buffer[attacker_index] = attacker_value; becomes a powerful primitive.

overflow_categories.c
C
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
// Example: Integer overflow leading to buffer overflow
void process_data(size_t length, char *input) {
    // Integer overflow vulnerability
    // If length = SIZE_MAX, then length + 1 = 0 due to wraparound
    char *buffer = malloc(length + 1);  // Allocates 0 or small buffer!
    
    if (buffer == NULL) return;
    
    // Copies 'length' bytes into undersized buffer
    memcpy(buffer, input, length);  // HEAP OVERFLOW!
    buffer[length] = '\0';
    
    // ... process buffer ...
    free(buffer);
}
 
// Example: Off-by-one overflow
void get_username(char *dest, int size, const char *src) {
    int i;
    // Bug: Loop condition allows one extra iteration
    for (i = 0; i <= size; i++) {  // Should be i < size
        dest[i] = src[i];
        if (src[i] == '\0') break;
    }
    // May write null at dest[size], one byte past the buffer
}

Integer Overflows: The Hidden Danger

Integer overflows are a common source of buffer overflows in production code. Size calculations like malloc(user_count * sizeof(struct user)) can wrap around if user_count is large, resulting in a tiny allocation. Always validate sizes against reasonable maximums before arithmetic, and consider using compiler features like __builtin_mul_overflow for safe arithmetic.

Why Buffer Overflows Persist

Given that buffer overflows have been understood since the 1960s, one might expect they'd be eliminated by now. Their persistence stems from a combination of technical, economic, and historical factors that create a challenging security landscape.

Technical Factors

•Unsafe Languages Remain Prevalent — C and C++ remain foundational for operating systems, embedded systems, browsers, databases, and performance-critical code. These languages provide no built-in bounds checking, trusting programmers to manage memory correctly.
•Legacy Code Bases — Billions of lines of existing C/C++ code power critical infrastructure. Rewriting is often economically infeasible, and even careful auditing misses vulnerabilities. The Linux kernel has ~30 million lines; Windows has ~50 million.
•Performance Constraints — Runtime bounds checking has overhead. While often negligible (1-10%), certain domains (high-frequency trading, real-time systems, game engines) resist any performance penalty. Engineers make conscious tradeoffs.
•Complexity of Verification — Proving that all code paths respect buffer boundaries is undecidable in general. Even sophisticated static analysis tools (Coverity, CodeSonar, PVS-Studio) produce false negatives on complex code.
•ABI and FFI Boundaries — Safe languages interoperating with C libraries inherit C's unsafety at the boundary. Rust's unsafe blocks, Python's C extensions, and Java's JNI all expose programs to buffer overflow risk in native code.

Economic Factors

•Security is a cost center—revenue comes from features, not bug fixes
•Time-to-market pressure overrides thorough security review
•The cost of a breach is often externalized to users/customers
•Finding and exploiting vuln is concentrated; defense is distributed

Historical Factors

•Early systems had no concept of adversarial input or security
•C was designed for trusted programmers on single-user machines
•Standard library (strcpy, gets) prioritized convenience over safety
•Networked computing arrived long after language design solidified

The Defender's Dilemma

The asymmetry between attackers and defenders is stark:

Defenders must protect every buffer operation in millions of lines of code, including all third-party dependencies
Attackers need to find a single exploitable overflow in the entire attack surface

This asymmetry, combined with the technical debt of decades of unsafe code, ensures that buffer overflows will remain relevant for the foreseeable future. Our response must be layered defenses (defense in depth), architectural changes (memory-safe languages, sandboxing), and runtime mitigations (ASLR, stack canaries, DEP) that raise the exploitation bar while vendors work to eliminate the underlying bugs.

The Long Tail of Vulnerabilities

Even aggressive fuzzing and static analysis leave residual vulnerabilities. Google's syzkaller fuzzer finds new Linux kernel bugs weekly despite years of testing. The 2022 'Dirty Pipe' vulnerability existed in Linux since kernel 5.8 (2020), surviving two years of security scrutiny before discovery. Buffer overflows hide in obscure code paths, unusual input combinations, and rarely-tested configurations.

Historical Impact: Infamous Buffer Overflow Exploits

Buffer overflows have enabled some of the most impactful security incidents in computing history. These cases illustrate both the power of buffer overflow exploits and their evolution over time.

Landmark Buffer Overflow Incidents
Incident	Year	Impact	Technical Detail
Morris Worm	1988	First major internet worm; ~6,000 systems infected; $100M+ cleanup cost	Stack overflow in fingerd; also exploited sendmail and weak passwords
Code Red	2001	359,000 hosts infected in 14 hours; defaced websites; DDoS on whitehouse.gov	Buffer overflow in IIS web server's ISAPI extension parsing HTTP requests
SQL Slammer	2003	75,000 hosts in 10 minutes; $1B estimated damage; 5 of 13 DNS root servers affected	Single UDP packet exploit; entire worm fit in 376 bytes; stack overflow in SQL Server
Heartbleed	2014	17% of secure web servers vulnerable; private keys exposed; passwords leaked	Buffer over-read in OpenSSL's TLS heartbeat extension; read sensitive memory
EternalBlue	2017	Enabled WannaCry ransomware; $4B+ global damage; hospitals, banks affected	Integer and buffer overflow in Windows SMB; NSA-developed, leaked exploit

Pattern Recognition

Notice the pattern across these incidents:

Network-facing daemon/service — The vulnerable code processes attacker-controlled input from the network
Insufficient input validation — The service trusts input length or format claims
Stack or heap overflow — Memory corruption enables code execution
Wormable exploitation — The exploit can spread automatically without user interaction
Massive blast radius — Critical infrastructure is affected before patches deploy

Every major buffer overflow incident follows this template. Understanding it is the first step to preventing the next one.

The Irony of Heartbleed

Heartbleed is particularly instructive because it wasn't a write overflow but a read overflow. The bug allowed attackers to read up to 64KB of memory beyond the intended buffer, potentially exposing private keys, session tokens, and passwords. This demonstrates that buffer overflows aren't just about code execution—any boundary violation compromises system integrity.

Learning from History

Each major incident prompted improvements: Morris Worm led to CERT and increased security awareness; Code Red led to better IIS security; Heartbleed prompted OpenBSD to create LibreSSL and accelerated adoption of memory-safe alternatives. Progress is slow but real.

Detecting Buffer Overflows

Detection is crucial because eliminating all buffer overflows at the source is infeasible for large codebases. Multiple approaches complement each other, each with distinct tradeoffs.

Static Analysis

•Compiler Warnings — Modern compilers (GCC, Clang) warn about many unsafe patterns. Enable -Wall -Wextra -Werror and treat warnings as errors. Catches: obvious strcpy misuse, format string mismatches, suspicious sizeof usage.
•Static Analyzers — Tools like Coverity, PVS-Studio, Clang Static Analyzer, and Infer perform deeper analysis. They track data flow to find where unchecked input reaches buffer operations. False positive rates vary; tuning is essential.
•SAST in CI/CD — Integrate static analysis into continuous integration. Block merges that introduce new warnings. CodeQL, Semgrep, and Checkmarx offer pipeline integration.

Dynamic Analysis

•AddressSanitizer (ASan) — Compiler-based instrumentation (GCC, Clang) that inserts checks around memory operations. Detects stack, heap, and global buffer overflows with ~2x slowdown. Essential for testing.
•Valgrind / Memcheck — Heavyweight but thorough dynamic analysis. Runs code in a simulated CPU, tracking all memory accesses. 10-50x slowdown but catches subtle issues ASan may miss.
•Fuzzing — Automated testing with random or mutated input. AFL, libFuzzer, and Honggfuzz generate millions of test cases seeking crashes. Combines well with ASan for crash triage.
•Runtime Bounds Checking — Some systems (Softbound+CETS, MPX) add runtime checks to all pointer operations. High overhead but comprehensive. MPX is deprecated; software solutions remain niche.

build_with_asan.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#!/bin/bash
# Building with AddressSanitizer for buffer overflow detection
 
# Compile with ASan enabled
clang -fsanitize=address -fno-omit-frame-pointer -g -O1 \
    vulnerable_program.c -o vulnerable_program_asan
 
# Run - ASan will detect and report overflows
./vulnerable_program_asan "$(python3 -c 'print("A"*100)')"
 
# Example ASan output on overflow:
# ==12345==ERROR: AddressSanitizer: stack-buffer-overflow
# WRITE of size 100 at 0x7ffd12345678
#     #0 0x4567890 in vulnerable_function vulnerable_program.c:8
#     #1 0x456789a in main vulnerable_program.c:14
# [0x7ffd12345600,0x7ffd12345640) 'buffer' (64 bytes)
# [0x7ffd12345640,0x7ffd12345680) is the REDZONE

Defense in Depth for Detection

No single detection method is sufficient. Best practice combines: (1) Compiler warnings on every build, (2) Static analysis in code review, (3) ASan-enabled test suite for unit/integration tests, (4) Continuous fuzzing for critical parsers and network code, (5) Periodic Valgrind runs for deep memory checking.

Summary: The Buffer Overflow Foundation

We have established the conceptual foundation for understanding buffer overflow vulnerabilities. Let's consolidate the key insights before moving to specific exploitation techniques.

Key Takeaways

•A buffer overflow writes beyond allocated memory boundaries — The CPU has no concept of buffer size; it simply writes to calculated addresses. Bounds enforcement is the programmer's (or language's) responsibility.
•Memory layout determines exploitation targets — Stack overflows target return addresses; heap overflows corrupt metadata and adjacent objects; global overflows can corrupt GOT entries and function pointers.
•Stack frames contain control flow data — The return address on the stack is the attacker's primary target. Corrupting it redirects execution when the function returns.
•Buffer overflows persist due to systemic factors — Unsafe languages, legacy code, performance constraints, and economic incentives ensure this vulnerability class remains relevant.
•Historical exploits demonstrate devastating impact — From Morris Worm to EternalBlue, buffer overflows have enabled worms, ransomware, and infrastructure compromise.
•Detection requires layered approaches — Static analysis, AddressSanitizer, fuzzing, and runtime checks each catch different vulnerability subsets.

What's Next: Stack Smashing

With this conceptual foundation established, the next page dives into stack smashing—the classic technique of exploiting stack-based buffer overflows to overwrite return addresses and gain control of execution flow. We'll examine the exact byte-level mechanics, how attackers craft exploit payloads, and the practical challenges of reliable exploitation.

Page Complete

You now understand the fundamental concept of buffer overflows: what they are, why they occur, and why they represent an enduring security challenge. The next page will show you exactly how these vulnerabilities are exploited through stack smashing techniques.