Variable-length Encoding Problem - Huffman Coding — Optimal Prefix Codes | OneNoughtOne

Greedy AlgorithmsHuffman Coding — Optimal Prefix Codes

Huffman Coding — Optimal Prefix Codes

LevelIntermediate

Duration60 mins

TopicHuffman Coding — Optimal Prefix Codes

1 / 4

Variable-length Encoding Problem

The Challenge of Efficient Data Representation

Every file on your computer—every email, every image, every MP3—is ultimately a sequence of bits. But here's a profound question that sits at the heart of information theory: How do we represent data using the fewest possible bits?

This isn't merely an academic curiosity. When you compress a file, stream video, or transmit data over a network, every bit saved translates to faster transfers, reduced storage costs, and lower bandwidth consumption. The difference between a naive encoding and an optimal one can mean gigabytes saved across millions of users.

Huffman coding, discovered by David Huffman in 1952 while he was a graduate student at MIT, provides an elegant greedy algorithm that constructs provably optimal variable-length codes. To understand Huffman's genius, we must first understand the problem it solves.

What You Will Learn

By the end of this page, you will understand the fundamental distinction between fixed-length and variable-length encoding, why variable-length codes can be more efficient, the critical prefix-free property that makes decoding unambiguous, and the precise optimization problem that Huffman coding solves.

Fixed-Length Encoding: The Simple Approach

Let's begin with the most straightforward approach to encoding data. In fixed-length encoding, every symbol (character, byte, or whatever unit we're encoding) gets the same number of bits.

ASCII: The Classic Example

ASCII uses exactly 7 bits per character, allowing 2⁷ = 128 distinct characters. Extended ASCII uses 8 bits (1 byte), allowing 256 characters. This uniformity makes encoding and decoding trivially simple:

To encode: replace each character with its fixed bit pattern
To decode: read exactly n bits, look up the character, repeat

The simplicity is appealing. There's no ambiguity, no complex logic, and constant-time access to any position in the encoded string.

fixed_length_encoding.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# Fixed-length encoding example: 3 bits per character
# With 3 bits, we can represent 2³ = 8 distinct symbols
 
FIXED_CODE = {
    'A': '000',
    'B': '001', 
    'C': '010',
    'D': '011',
    'E': '100',
    'F': '101',
    'G': '110',
    'H': '111'
}
 
def encode_fixed(text: str) -> str:
    """Encode text using fixed-length 3-bit codes."""
    return ''.join(FIXED_CODE[char] for char in text)
 
def decode_fixed(bits: str) -> str:
    """Decode by reading exactly 3 bits at a time."""
    # Create reverse lookup
    reverse_code = {v: k for k, v in FIXED_CODE.items()}
    
    result = []
    for i in range(0, len(bits), 3):  # Step by exactly 3 bits
        chunk = bits[i:i+3]
        result.append(reverse_code[chunk])
    return ''.join(result)
 
# Example usage
message = "ABCABC"
encoded = encode_fixed(message)
print(f"Original: {message}")
print(f"Encoded:  {encoded}")
print(f"Bits used: {len(encoded)}")
print(f"Decoded:  {decode_fixed(encoded)}")
 
# Output:
# Original: ABCABC
# Encoded:  000001010000001010
# Bits used: 18
# Decoded:  ABCABC

The Calculation for Fixed-Length Codes

If we have an alphabet of n symbols, we need exactly ⌈log₂(n)⌉ bits per symbol to uniquely identify each one:

Alphabet Size	Bits Required	Distinct Patterns
2	1	0, 1
4	2	00, 01, 10, 11
8	3	000, 001, ..., 111
26	5	32 patterns (6 unused)
128 (ASCII)	7	128 patterns

For a message of m characters from an n-symbol alphabet, fixed-length encoding always uses exactly m × ⌈log₂(n)⌉ bits.

Fixed-Length Advantage: Random Access

Fixed-length encoding allows O(1) random access to any character. To find the 1000th character, just read bits 2997-2999 (for 3-bit codes). This property is lost with variable-length codes, which is why fixed-length encoding remains appropriate for many applications.

The Inherent Inefficiency of Fixed-Length Codes

Fixed-length encoding treats all symbols equally—each gets the same number of bits regardless of how often it appears. But in real data, symbol frequencies are rarely uniform.

Consider English text:

The letter 'E' appears about 12.7% of the time
The letter 'Z' appears only about 0.07% of the time
'E' is roughly 180 times more common than 'Z'

Yet in ASCII, both 'E' (01000101) and 'Z' (01011010) use exactly 8 bits. This uniformity seems wasteful—shouldn't the common letters use fewer bits?

A Concrete Example

Suppose we want to encode a message using only 4 characters: A, B, C, D. With fixed-length encoding, we need 2 bits per character.

Character Frequencies in a Sample Message
Character	Frequency	Percentage	Fixed Code (2 bits)
A	45	45%	00
B	13	13%	01
C	12	12%	10
D	30	30%	11

Fixed-Length Total: 100 characters × 2 bits = 200 bits

But notice that 'A' appears 45% of the time while 'C' appears only 12% of the time. If we could give 'A' a shorter code (fewer bits) and 'C' a longer code (more bits), the total might be smaller—because we'd be using fewer bits more often.

The Insight: In a variable-length scheme, the weighted average of code lengths matters more than the maximum code length. A 1-bit code used 45% of the time saves more bits than a 3-bit code used 12% of the time costs.

Shannon's Source Coding Theorem

Claude Shannon proved in 1948 that the theoretical minimum average bits per symbol equals the entropy of the source: H = -Σ p(x) log₂ p(x). For the example above, H ≈ 1.85 bits/symbol. No encoding can average below this, but Huffman coding achieves within 1 bit of optimal.

Computing the Theoretical Minimum

For our 4-character example with frequencies A:45%, B:13%, C:12%, D:30%:

H = -(0.45 × log₂(0.45) + 0.13 × log₂(0.13) + 0.12 × log₂(0.12) + 0.30 × log₂(0.30))
H = -(0.45 × (-1.152) + 0.13 × (-2.943) + 0.12 × (-3.059) + 0.30 × (-1.737))
H = -(-0.518 - 0.383 - 0.367 - 0.521)
H = 1.789 bits per symbol

The fixed-length encoding uses 2.0 bits per symbol. The optimal variable-length encoding should approach 1.789 bits per symbol—a potential savings of about 10%.

Variable-Length Encoding: The Core Idea

Variable-length encoding assigns different-length bit strings to different symbols. The guiding principle is intuitive:

Frequent symbols get short codes; rare symbols get long codes.

This is analogous to how natural languages evolve. The most common English words tend to be short ('the', 'a', 'is', 'it'), while rare words are longer ('surreptitious', 'quintessential'). Languages naturally optimize for efficient communication.

A First Attempt at Variable-Length Codes

Let's try assigning codes based on our frequency example:

Naive Variable-Length Encoding Attempt
Character	Frequency	Naive Code	Bits
A (most common)	45%	0	1 bit
D (second most)	30%	1	1 bit
B	13%	00	2 bits
C (least common)	12%	11	2 bits

The Decoding Disaster

Let's try to decode the bit string 0001:

Is it 0, 0, 0, 1 = A, A, A, D?
Is it 0, 0, 01 = A, A, ??? (01 isn't a code...)
Is it 0, 00, 1 = A, B, D?
Is it 00, 0, 1 = B, A, D?

The encoding is ambiguous! When we read '00', we can't tell whether it's:

The single code 'B' (00), or
Two consecutive 'A's (0, 0)

This is the fundamental challenge with variable-length codes: without additional structure, decoding becomes impossible or ambiguous.

The Ambiguity Problem

Variable-length codes introduce ambiguity because code boundaries are not marked. When you see a sequence of bits, you don't know where one code ends and the next begins. This problem must be solved for variable-length encoding to be practical.

Potential Solutions to Ambiguity

Separator symbols: Insert a special bit pattern between codes. Wasteful—we're adding bits we wanted to save.
Length prefixes: Encode the length of each code before the code itself. Also adds overhead.
Prefix-free codes: Design codes so that no code is a prefix of another code. This is the elegant solution used by Huffman coding.

The third approach is the key insight. If no code is a prefix of another, then as we read bits left-to-right, the moment we complete a valid code, we know it's the intended symbol—it can't be the beginning of a longer code.

Prefix-Free Codes: The Key to Unambiguous Decoding

A prefix-free code (also called a prefix code or instantaneous code) has a crucial property:

No codeword is a prefix of any other codeword.

This means that when reading a bit stream, the moment you recognize a valid code, you can immediately output the corresponding symbol without waiting to see what comes next. There's no ambiguity.

A Valid Prefix-Free Code

Let's fix our earlier example:

Prefix-Free Variable-Length Code
Character	Frequency	Prefix-Free Code	Bits
A	45%	0	1 bit
D	30%	10	2 bits
B	13%	110	3 bits
C	12%	111	3 bits

Verification that this is prefix-free:

'A' = 0: Is '0' a prefix of '10', '110', or '111'? No (they start with '1').
'D' = 10: Is '10' a prefix of '0', '110', or '111'? No.
'B' = 110: Is '110' a prefix of '0', '10', or '111'? No.
'C' = 111: Is '111' a prefix of '0', '10', or '110'? No.

✓ No code is a prefix of another. Decoding is unambiguous!

prefix_free_decode.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
# Prefix-free decoding: simple left-to-right scan
PREFIX_FREE_CODE = {
    '0': 'A',      # 1 bit
    '10': 'D',     # 2 bits  
    '110': 'B',    # 3 bits
    '111': 'C'     # 3 bits
}
 
def decode_prefix_free(bits: str) -> str:
    """
    Decode a prefix-free encoded bit string.
    
    Algorithm:
    1. Start with an empty current_code buffer
    2. Read bits one at a time, appending to current_code
    3. When current_code matches a valid code, output symbol, reset buffer
    4. If we reach end of bits with current_code empty, we're done
    5. If we reach end with current_code non-empty, encoding is invalid
    """
    result = []
    current_code = ""
    
    for bit in bits:
        current_code += bit
        
        # Check if current_code is a valid codeword
        if current_code in PREFIX_FREE_CODE:
            result.append(PREFIX_FREE_CODE[current_code])
            current_code = ""  # Reset for next symbol
    
    if current_code:
        raise ValueError(f"Invalid encoding: leftover bits '{current_code}'")
    
    return ''.join(result)
 
# Test decoding
test_cases = [
    "0",           # A
    "0010",        # A, A, D  
    "0101101110",  # A, D, B, C
    "1101110010",  # B, C, A, A, D
]
 
for bits in test_cases:
    decoded = decode_prefix_free(bits)
    print(f"{bits:15} -> {decoded}")
 
# Output:
# 0               -> A
# 0010            -> AAD
# 0101101110      -> ADBC
# 1101110010      -> BCAAD

Why 'Instantaneous'?

Prefix-free codes are also called instantaneous codes because decoding can happen immediately upon recognizing a complete code. There's no need to buffer additional bits to confirm the symbol. This property is essential for:

Real-time streaming: Decode and process data as it arrives
Sequential processing: No need to backtrack or look ahead
Simple implementation: Just maintain a current prefix and match against the code table

Huffman Codes Are Always Prefix-Free

The Huffman algorithm, which we'll study in depth, always produces prefix-free codes. This is guaranteed by the tree structure from which the codes are derived—no ancestor node can also be a leaf node.

Binary Trees and Prefix-Free Codes

There's a beautiful correspondence between prefix-free codes and binary trees. Every prefix-free code can be represented as a binary tree where:

Leaf nodes represent symbols
Left edges represent '0'
Right edges represent '1'
The path from root to leaf gives the code for that symbol

This representation makes prefix-free codes intuitive to visualize and manipulate.

code_tree_visualization.txt
# Binary tree for our prefix-free code:
#
#           (root)
#           /    \
#          0      1
#         /        \
#       [A]        ( )
#                 /   \
#                0     1
#               /       \
#             [D]       ( )
#                      /   \
#                     0     1
#                    /       \
#                  [B]      [C]
#
# Reading the codes:
#   A: 0           (left from root)
#   D: 10          (right, then left)
#   B: 110         (right, right, left)
#   C: 111         (right, right, right)
#
# The prefix-free property emerges naturally:
# - Symbols are ONLY at leaves
# - Every path continues until it hits a leaf
# - No leaf is an ancestor of another leaf
# - Therefore, no code is a prefix of another

Key Insight: Full Binary Trees

For optimal codes, the corresponding tree is a full binary tree—every internal node has exactly two children. If an internal node had only one child, we could remove that node and shorten the codes, which would be more efficient.

Tree Properties and Code Properties

Tree Property	Code Property
Leaf depth	Code length
Number of leaves	Alphabet size
Full binary tree	No wasted bit patterns
Weighted path length	Average code length

The weighted path length is the sum of (depth × frequency) for all leaves. Minimizing this is exactly the problem of finding the optimal prefix-free code!

Tree Perspective Enables Greedy Construction

Viewing codes as trees reveals the structure that Huffman's greedy algorithm exploits. The algorithm builds the tree bottom-up by repeatedly combining the two lowest-frequency nodes. This tree construction is the heart of Huffman coding.

Calculating Average Code Length

To compare different prefix-free codes, we need a metric. The natural choice is average code length—the expected number of bits per symbol.

Definition:

$$\text{Average Length} = \sum_{i=1}^{n} p_i \cdot l_i$$

Where:

$p_i$ is the probability (frequency) of symbol $i$
$l_i$ is the length (in bits) of the code for symbol $i$
$n$ is the number of symbols in the alphabet

For Our Example:

Calculating Average Code Length
Symbol	Frequency (pᵢ)	Code Length (lᵢ)	pᵢ × lᵢ
A	0.45	1	0.45
D	0.30	2	0.60
B	0.13	3	0.39
C	0.12	3	0.36
Total	1.00		1.80

Average code length = 1.80 bits per symbol

Compare this to:

Fixed-length encoding: 2.00 bits per symbol
Theoretical minimum (entropy): 1.789 bits per symbol

Our variable-length code achieves 10% compression compared to fixed-length, and it's remarkably close to the theoretical optimum!

average_length.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
import math
 
def calculate_average_length(frequencies: dict, codes: dict) -> float:
    """
    Calculate the average code length for a prefix-free code.
    
    Args:
        frequencies: dict mapping symbols to their probabilities
        codes: dict mapping symbols to their bit strings
        
    Returns:
        Average bits per symbol
    """
    total = sum(frequencies.values())
    return sum(
        (freq / total) * len(codes[symbol])
        for symbol, freq in frequencies.items()
    )
 
def calculate_entropy(frequencies: dict) -> float:
    """
    Calculate the entropy (theoretical minimum average bits).
    
    H = -Σ p(x) log₂ p(x)
    """
    total = sum(frequencies.values())
    entropy = 0
    for freq in frequencies.values():
        p = freq / total
        if p > 0:  # Avoid log(0)
            entropy -= p * math.log2(p)
    return entropy
 
# Example data
frequencies = {'A': 45, 'D': 30, 'B': 13, 'C': 12}
optimal_codes = {'A': '0', 'D': '10', 'B': '110', 'C': '111'}
fixed_codes = {'A': '00', 'D': '01', 'B': '10', 'C': '11'}
 
# Calculate metrics
avg_optimal = calculate_average_length(frequencies, optimal_codes)
avg_fixed = calculate_average_length(frequencies, fixed_codes)
entropy = calculate_entropy(frequencies)
 
print(f"Entropy (theoretical minimum): {entropy:.3f} bits/symbol")
print(f"Fixed-length average:          {avg_fixed:.3f} bits/symbol")
print(f"Variable-length average:       {avg_optimal:.3f} bits/symbol")
print(f"")
print(f"Compression vs fixed: {(1 - avg_optimal/avg_fixed)*100:.1f}%")
print(f"Efficiency vs entropy: {(entropy/avg_optimal)*100:.1f}%")
 
# Output:
# Entropy (theoretical minimum): 1.789 bits/symbol
# Fixed-length average:          2.000 bits/symbol
# Variable-length average:       1.800 bits/symbol
#
# Compression vs fixed: 10.0%
# Efficiency vs entropy: 99.4%

Huffman Codes Are Optimal

Huffman's algorithm always produces a prefix-free code with the minimum possible average length. No other prefix-free code can have a shorter average length for the same frequency distribution. This optimality is what makes Huffman coding a cornerstone of data compression.

The Optimization Problem: Formally Stated

We can now formally state the problem that Huffman coding solves:

Given:

An alphabet Σ = {c₁, c₂, ..., cₙ} of n symbols
A frequency distribution f = {f₁, f₂, ..., fₙ} where fᵢ is the frequency of symbol cᵢ

Find:

A prefix-free binary code that minimizes the average code length:

$$\text{minimize } \sum_{i=1}^{n} f_i \cdot l_i$$

Where lᵢ is the length of the codeword assigned to symbol cᵢ.

Equivalently (in tree terms):

Find a full binary tree with n leaves
Assign symbols to leaves
Minimize the weighted path length:

$$\text{minimize } \sum_{i=1}^{n} f_i \cdot d_i$$

Where dᵢ is the depth of the leaf containing symbol cᵢ.

Why Is This Hard?

At first glance, you might think we could just sort symbols by frequency and assign codes greedily: shortest codes to most frequent symbols. But the constraint that codes must form a valid prefix-free code complicates matters.

The Space of Possible Codes

For n symbols, the number of different prefix-free code structures grows exponentially. The number of different full binary trees with n leaves is the (n-1)th Catalan number:

$$C_{n-1} = \frac{1}{n} \binom{2n-2}{n-1}$$

For n = 10 symbols: C₉ = 4,862 different tree structures For n = 20 symbols: C₁₉ = 1,767,263,190 different structures

Exhaustive search is infeasible. We need a clever algorithm.

Greedy Algorithms to the Rescue

Despite the exponential search space, the optimal solution can be found in O(n log n) time using a greedy approach. Huffman's insight was that at each step, combining the two lowest-frequency symbols is always a safe choice that leads to an optimal solution.

Why This Matters: Real-World Applications

The variable-length encoding problem isn't just theoretical—it underpins much of modern data compression. Understanding it provides insight into technologies you use every day.

Huffman Coding in Practice

•JPEG Image Compression — After quantization, the frequency coefficients are Huffman-encoded. This final stage typically achieves 2-4x additional compression.
•DEFLATE (ZIP, gzip, PNG) — Combines LZ77 dictionary coding with Huffman coding. Huffman encodes both the literals and the back-references.
•MP3 Audio — Uses Huffman coding as the final entropy coding stage after the psychoacoustic model removes inaudible frequencies.
•Fax Machines (CCITT) — Early adoption of Huffman coding for encoding run lengths of black and white pixels.
•HTTP/2 Header Compression (HPACK) — Uses Huffman coding to compress HTTP headers, reducing overhead for web requests.
•Brotli Compression — Google's modern compression algorithm uses context-dependent Huffman codes for improved ratios.

Beyond Huffman: Arithmetic Coding

While Huffman coding is optimal among prefix-free codes with integer bit lengths, it can't achieve fractional bits per symbol. For example, if the optimal is 1.1 bits, Huffman must use either 1 or 2 bits.

Arithmetic coding overcomes this limitation by encoding entire messages as a single number, approaching the true entropy more closely. However, it's more computationally expensive and less intuitive.

For most practical applications, Huffman coding's simplicity and near-optimal performance make it the algorithm of choice.

Historical Note

David Huffman discovered this algorithm in 1952 as a term paper alternative to a final exam in Robert Fano's information theory class at MIT. Interestingly, his professor had been working on a similar problem (Fano coding) but Huffman's approach proved optimal where Fano's was merely good. Sometimes the student does surpass the teacher!

Summary: The Variable-Length Encoding Problem

We've established the foundational concepts that Huffman coding builds upon. Let's consolidate the key insights:

Key Takeaways

•Fixed-length codes are simple but wasteful — Every symbol uses the same bits regardless of frequency, leading to suboptimal compression.
•Variable-length codes can be more efficient — Assigning short codes to common symbols and long codes to rare symbols reduces average length.
•Arbitrary variable-length codes cause ambiguity — Without structure, we can't tell where one code ends and the next begins.
•Prefix-free codes solve ambiguity — If no code is a prefix of another, decoding is unambiguous and instantaneous.
•Binary trees represent prefix-free codes — Leaves are symbols, paths are codes, and weighted path length equals average code length.
•The optimization goal is minimum average length — This is equivalent to minimum weighted path length in the tree representation.
•Entropy sets the theoretical limit — No prefix-free code can achieve average length below the source entropy.

What's Next:

Now that we understand the problem, we're ready to learn the solution. In the next page, we'll discover how to construct the Huffman tree—a greedy algorithm that elegantly builds the optimal prefix-free code by repeatedly combining the lowest-frequency nodes.

Page Complete

You now understand the variable-length encoding problem: why fixed-length codes are inefficient, why naive variable-length codes are ambiguous, and how prefix-free codes enable unambiguous decoding. This foundation prepares you for the elegant Huffman tree construction algorithm.

1 / 4

Loading learning content...

Greedy AlgorithmsHuffman Coding — Optimal Prefix Codes

Huffman Coding — Optimal Prefix Codes

LevelIntermediate

Duration60 mins

TopicHuffman Coding — Optimal Prefix Codes

1 / 4

Variable-length Encoding Problem

The Challenge of Efficient Data Representation

What You Will Learn

Fixed-Length Encoding: The Simple Approach

Let's begin with the most straightforward approach to encoding data. In fixed-length encoding, every symbol (character, byte, or whatever unit we're encoding) gets the same number of bits.

ASCII: The Classic Example

To encode: replace each character with its fixed bit pattern
To decode: read exactly n bits, look up the character, repeat

The simplicity is appealing. There's no ambiguity, no complex logic, and constant-time access to any position in the encoded string.

fixed_length_encoding.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# Fixed-length encoding example: 3 bits per character
# With 3 bits, we can represent 2³ = 8 distinct symbols
 
FIXED_CODE = {
    'A': '000',
    'B': '001', 
    'C': '010',
    'D': '011',
    'E': '100',
    'F': '101',
    'G': '110',
    'H': '111'
}
 
def encode_fixed(text: str) -> str:
    """Encode text using fixed-length 3-bit codes."""
    return ''.join(FIXED_CODE[char] for char in text)
 
def decode_fixed(bits: str) -> str:
    """Decode by reading exactly 3 bits at a time."""
    # Create reverse lookup
    reverse_code = {v: k for k, v in FIXED_CODE.items()}
    
    result = []
    for i in range(0, len(bits), 3):  # Step by exactly 3 bits
        chunk = bits[i:i+3]
        result.append(reverse_code[chunk])
    return ''.join(result)
 
# Example usage
message = "ABCABC"
encoded = encode_fixed(message)
print(f"Original: {message}")
print(f"Encoded:  {encoded}")
print(f"Bits used: {len(encoded)}")
print(f"Decoded:  {decode_fixed(encoded)}")
 
# Output:
# Original: ABCABC
# Encoded:  000001010000001010
# Bits used: 18
# Decoded:  ABCABC

The Calculation for Fixed-Length Codes

If we have an alphabet of n symbols, we need exactly ⌈log₂(n)⌉ bits per symbol to uniquely identify each one:

Alphabet Size	Bits Required	Distinct Patterns
2	1	0, 1
4	2	00, 01, 10, 11
8	3	000, 001, ..., 111
26	5	32 patterns (6 unused)
128 (ASCII)	7	128 patterns

For a message of m characters from an n-symbol alphabet, fixed-length encoding always uses exactly m × ⌈log₂(n)⌉ bits.

Fixed-Length Advantage: Random Access

The Inherent Inefficiency of Fixed-Length Codes

Fixed-length encoding treats all symbols equally—each gets the same number of bits regardless of how often it appears. But in real data, symbol frequencies are rarely uniform.

Consider English text:

The letter 'E' appears about 12.7% of the time
The letter 'Z' appears only about 0.07% of the time
'E' is roughly 180 times more common than 'Z'

Yet in ASCII, both 'E' (01000101) and 'Z' (01011010) use exactly 8 bits. This uniformity seems wasteful—shouldn't the common letters use fewer bits?

A Concrete Example

Suppose we want to encode a message using only 4 characters: A, B, C, D. With fixed-length encoding, we need 2 bits per character.

Character Frequencies in a Sample Message
Character	Frequency	Percentage	Fixed Code (2 bits)
A	45	45%	00
B	13	13%	01
C	12	12%	10
D	30	30%	11

Fixed-Length Total: 100 characters × 2 bits = 200 bits

Shannon's Source Coding Theorem

Computing the Theoretical Minimum

For our 4-character example with frequencies A:45%, B:13%, C:12%, D:30%:

H = -(0.45 × log₂(0.45) + 0.13 × log₂(0.13) + 0.12 × log₂(0.12) + 0.30 × log₂(0.30))
H = -(0.45 × (-1.152) + 0.13 × (-2.943) + 0.12 × (-3.059) + 0.30 × (-1.737))
H = -(-0.518 - 0.383 - 0.367 - 0.521)
H = 1.789 bits per symbol

The fixed-length encoding uses 2.0 bits per symbol. The optimal variable-length encoding should approach 1.789 bits per symbol—a potential savings of about 10%.

Variable-Length Encoding: The Core Idea

Variable-length encoding assigns different-length bit strings to different symbols. The guiding principle is intuitive:

Frequent symbols get short codes; rare symbols get long codes.

A First Attempt at Variable-Length Codes

Let's try assigning codes based on our frequency example:

Naive Variable-Length Encoding Attempt
Character	Frequency	Naive Code	Bits
A (most common)	45%	0	1 bit
D (second most)	30%	1	1 bit
B	13%	00	2 bits
C (least common)	12%	11	2 bits

The Decoding Disaster

Let's try to decode the bit string 0001:

Is it 0, 0, 0, 1 = A, A, A, D?
Is it 0, 0, 01 = A, A, ??? (01 isn't a code...)
Is it 0, 00, 1 = A, B, D?
Is it 00, 0, 1 = B, A, D?

The encoding is ambiguous! When we read '00', we can't tell whether it's:

The single code 'B' (00), or
Two consecutive 'A's (0, 0)

This is the fundamental challenge with variable-length codes: without additional structure, decoding becomes impossible or ambiguous.

The Ambiguity Problem

Potential Solutions to Ambiguity

Separator symbols: Insert a special bit pattern between codes. Wasteful—we're adding bits we wanted to save.
Length prefixes: Encode the length of each code before the code itself. Also adds overhead.
Prefix-free codes: Design codes so that no code is a prefix of another code. This is the elegant solution used by Huffman coding.

Prefix-Free Codes: The Key to Unambiguous Decoding

A prefix-free code (also called a prefix code or instantaneous code) has a crucial property:

No codeword is a prefix of any other codeword.

This means that when reading a bit stream, the moment you recognize a valid code, you can immediately output the corresponding symbol without waiting to see what comes next. There's no ambiguity.

A Valid Prefix-Free Code

Let's fix our earlier example:

Prefix-Free Variable-Length Code
Character	Frequency	Prefix-Free Code	Bits
A	45%	0	1 bit
D	30%	10	2 bits
B	13%	110	3 bits
C	12%	111	3 bits

Verification that this is prefix-free:

'A' = 0: Is '0' a prefix of '10', '110', or '111'? No (they start with '1').
'D' = 10: Is '10' a prefix of '0', '110', or '111'? No.
'B' = 110: Is '110' a prefix of '0', '10', or '111'? No.
'C' = 111: Is '111' a prefix of '0', '10', or '110'? No.

✓ No code is a prefix of another. Decoding is unambiguous!

prefix_free_decode.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
# Prefix-free decoding: simple left-to-right scan
PREFIX_FREE_CODE = {
    '0': 'A',      # 1 bit
    '10': 'D',     # 2 bits  
    '110': 'B',    # 3 bits
    '111': 'C'     # 3 bits
}
 
def decode_prefix_free(bits: str) -> str:
    """
    Decode a prefix-free encoded bit string.
    
    Algorithm:
    1. Start with an empty current_code buffer
    2. Read bits one at a time, appending to current_code
    3. When current_code matches a valid code, output symbol, reset buffer
    4. If we reach end of bits with current_code empty, we're done
    5. If we reach end with current_code non-empty, encoding is invalid
    """
    result = []
    current_code = ""
    
    for bit in bits:
        current_code += bit
        
        # Check if current_code is a valid codeword
        if current_code in PREFIX_FREE_CODE:
            result.append(PREFIX_FREE_CODE[current_code])
            current_code = ""  # Reset for next symbol
    
    if current_code:
        raise ValueError(f"Invalid encoding: leftover bits '{current_code}'")
    
    return ''.join(result)
 
# Test decoding
test_cases = [
    "0",           # A
    "0010",        # A, A, D  
    "0101101110",  # A, D, B, C
    "1101110010",  # B, C, A, A, D
]
 
for bits in test_cases:
    decoded = decode_prefix_free(bits)
    print(f"{bits:15} -> {decoded}")
 
# Output:
# 0               -> A
# 0010            -> AAD
# 0101101110      -> ADBC
# 1101110010      -> BCAAD

Why 'Instantaneous'?

Real-time streaming: Decode and process data as it arrives
Sequential processing: No need to backtrack or look ahead
Simple implementation: Just maintain a current prefix and match against the code table

Huffman Codes Are Always Prefix-Free

Binary Trees and Prefix-Free Codes

There's a beautiful correspondence between prefix-free codes and binary trees. Every prefix-free code can be represented as a binary tree where:

Leaf nodes represent symbols
Left edges represent '0'
Right edges represent '1'
The path from root to leaf gives the code for that symbol

This representation makes prefix-free codes intuitive to visualize and manipulate.

code_tree_visualization.txt
# Binary tree for our prefix-free code:
#
#           (root)
#           /    \
#          0      1
#         /        \
#       [A]        ( )
#                 /   \
#                0     1
#               /       \
#             [D]       ( )
#                      /   \
#                     0     1
#                    /       \
#                  [B]      [C]
#
# Reading the codes:
#   A: 0           (left from root)
#   D: 10          (right, then left)
#   B: 110         (right, right, left)
#   C: 111         (right, right, right)
#
# The prefix-free property emerges naturally:
# - Symbols are ONLY at leaves
# - Every path continues until it hits a leaf
# - No leaf is an ancestor of another leaf
# - Therefore, no code is a prefix of another

Key Insight: Full Binary Trees

Tree Properties and Code Properties

Tree Property	Code Property
Leaf depth	Code length
Number of leaves	Alphabet size
Full binary tree	No wasted bit patterns
Weighted path length	Average code length

The weighted path length is the sum of (depth × frequency) for all leaves. Minimizing this is exactly the problem of finding the optimal prefix-free code!

Tree Perspective Enables Greedy Construction

Calculating Average Code Length

To compare different prefix-free codes, we need a metric. The natural choice is average code length—the expected number of bits per symbol.

Definition:

$$\text{Average Length} = \sum_{i=1}^{n} p_i \cdot l_i$$

Where:

$p_i$ is the probability (frequency) of symbol $i$
$l_i$ is the length (in bits) of the code for symbol $i$
$n$ is the number of symbols in the alphabet

For Our Example:

Calculating Average Code Length
Symbol	Frequency (pᵢ)	Code Length (lᵢ)	pᵢ × lᵢ
A	0.45	1	0.45
D	0.30	2	0.60
B	0.13	3	0.39
C	0.12	3	0.36
Total	1.00		1.80

Average code length = 1.80 bits per symbol

Compare this to:

Fixed-length encoding: 2.00 bits per symbol
Theoretical minimum (entropy): 1.789 bits per symbol

Our variable-length code achieves 10% compression compared to fixed-length, and it's remarkably close to the theoretical optimum!

average_length.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
import math
 
def calculate_average_length(frequencies: dict, codes: dict) -> float:
    """
    Calculate the average code length for a prefix-free code.
    
    Args:
        frequencies: dict mapping symbols to their probabilities
        codes: dict mapping symbols to their bit strings
        
    Returns:
        Average bits per symbol
    """
    total = sum(frequencies.values())
    return sum(
        (freq / total) * len(codes[symbol])
        for symbol, freq in frequencies.items()
    )
 
def calculate_entropy(frequencies: dict) -> float:
    """
    Calculate the entropy (theoretical minimum average bits).
    
    H = -Σ p(x) log₂ p(x)
    """
    total = sum(frequencies.values())
    entropy = 0
    for freq in frequencies.values():
        p = freq / total
        if p > 0:  # Avoid log(0)
            entropy -= p * math.log2(p)
    return entropy
 
# Example data
frequencies = {'A': 45, 'D': 30, 'B': 13, 'C': 12}
optimal_codes = {'A': '0', 'D': '10', 'B': '110', 'C': '111'}
fixed_codes = {'A': '00', 'D': '01', 'B': '10', 'C': '11'}
 
# Calculate metrics
avg_optimal = calculate_average_length(frequencies, optimal_codes)
avg_fixed = calculate_average_length(frequencies, fixed_codes)
entropy = calculate_entropy(frequencies)
 
print(f"Entropy (theoretical minimum): {entropy:.3f} bits/symbol")
print(f"Fixed-length average:          {avg_fixed:.3f} bits/symbol")
print(f"Variable-length average:       {avg_optimal:.3f} bits/symbol")
print(f"")
print(f"Compression vs fixed: {(1 - avg_optimal/avg_fixed)*100:.1f}%")
print(f"Efficiency vs entropy: {(entropy/avg_optimal)*100:.1f}%")
 
# Output:
# Entropy (theoretical minimum): 1.789 bits/symbol
# Fixed-length average:          2.000 bits/symbol
# Variable-length average:       1.800 bits/symbol
#
# Compression vs fixed: 10.0%
# Efficiency vs entropy: 99.4%

Huffman Codes Are Optimal

The Optimization Problem: Formally Stated

We can now formally state the problem that Huffman coding solves:

Given:

An alphabet Σ = {c₁, c₂, ..., cₙ} of n symbols
A frequency distribution f = {f₁, f₂, ..., fₙ} where fᵢ is the frequency of symbol cᵢ

Find:

A prefix-free binary code that minimizes the average code length:

$$\text{minimize } \sum_{i=1}^{n} f_i \cdot l_i$$

Where lᵢ is the length of the codeword assigned to symbol cᵢ.

Equivalently (in tree terms):

Find a full binary tree with n leaves
Assign symbols to leaves
Minimize the weighted path length:

$$\text{minimize } \sum_{i=1}^{n} f_i \cdot d_i$$

Where dᵢ is the depth of the leaf containing symbol cᵢ.

Why Is This Hard?

The Space of Possible Codes

For n symbols, the number of different prefix-free code structures grows exponentially. The number of different full binary trees with n leaves is the (n-1)th Catalan number:

$$C_{n-1} = \frac{1}{n} \binom{2n-2}{n-1}$$

For n = 10 symbols: C₉ = 4,862 different tree structures For n = 20 symbols: C₁₉ = 1,767,263,190 different structures

Exhaustive search is infeasible. We need a clever algorithm.

Greedy Algorithms to the Rescue

Why This Matters: Real-World Applications

The variable-length encoding problem isn't just theoretical—it underpins much of modern data compression. Understanding it provides insight into technologies you use every day.

Huffman Coding in Practice

•JPEG Image Compression — After quantization, the frequency coefficients are Huffman-encoded. This final stage typically achieves 2-4x additional compression.
•DEFLATE (ZIP, gzip, PNG) — Combines LZ77 dictionary coding with Huffman coding. Huffman encodes both the literals and the back-references.
•MP3 Audio — Uses Huffman coding as the final entropy coding stage after the psychoacoustic model removes inaudible frequencies.
•Fax Machines (CCITT) — Early adoption of Huffman coding for encoding run lengths of black and white pixels.
•HTTP/2 Header Compression (HPACK) — Uses Huffman coding to compress HTTP headers, reducing overhead for web requests.
•Brotli Compression — Google's modern compression algorithm uses context-dependent Huffman codes for improved ratios.

Beyond Huffman: Arithmetic Coding

For most practical applications, Huffman coding's simplicity and near-optimal performance make it the algorithm of choice.

Historical Note

Summary: The Variable-Length Encoding Problem

We've established the foundational concepts that Huffman coding builds upon. Let's consolidate the key insights:

Key Takeaways

•Fixed-length codes are simple but wasteful — Every symbol uses the same bits regardless of frequency, leading to suboptimal compression.
•Variable-length codes can be more efficient — Assigning short codes to common symbols and long codes to rare symbols reduces average length.
•Arbitrary variable-length codes cause ambiguity — Without structure, we can't tell where one code ends and the next begins.
•Prefix-free codes solve ambiguity — If no code is a prefix of another, decoding is unambiguous and instantaneous.
•Binary trees represent prefix-free codes — Leaves are symbols, paths are codes, and weighted path length equals average code length.
•The optimization goal is minimum average length — This is equivalent to minimum weighted path length in the tree representation.
•Entropy sets the theoretical limit — No prefix-free code can achieve average length below the source entropy.

What's Next:

Page Complete

1 / 4