Positional Encoding

Transformers have revolutionized machine learning, achieving state-of-the-art results across language, vision, and multimodal domains. Yet beneath their remarkable success lies a fundamental architectural blind spot: transformers are inherently position-agnostic.

Unlike recurrent neural networks that process tokens sequentially—one after another in strict temporal order—or convolutional networks that operate on local spatial neighborhoods, the transformer's self-attention mechanism treats all input positions as an unordered set. Every token attends to every other token simultaneously, with no built-in mechanism to distinguish whether a word appears at the beginning, middle, or end of a sentence.

This presents a profound challenge for any task where order matters—which is to say, virtually every task involving sequences.

To understand why positional encoding is necessary, we must first understand what the self-attention mechanism actually computes and why it lacks positional awareness by design.

Self-Attention: A Set Operation

Recall the core self-attention computation:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Where:

• $Q = XW_Q$ (queries)
• $K = XW_K$ (keys)
• $V = XW_V$ (values)
• $X \in \mathbb{R}^{n \times d}$ is the input sequence

The critical observation is that this operation is permutation equivariant. If we permute the input sequence $X$ to get $X' = PX$ (where $P$ is a permutation matrix), the output is similarly permuted:

$$\text{Attention}(PX) = P \cdot \text{Attention}(X)$$

This means shuffling the input tokens shuffles the output in exactly the same way—the relative relationships between positions are preserved, but the absolute positions carry no special meaning.

Mathematical Demonstration

Consider a simple three-token sequence: ["A", "B", "C"]. The attention weight from token $i$ to token $j$ is:

$$\alpha_{ij} = \frac{\exp(q_i \cdot k_j / \sqrt{d_k})}{\sum_{m} \exp(q_i \cdot k_m / \sqrt{d_k})}$$

Note that this formula depends only on the content of the queries and keys—the embedding vectors $q_i$ and $k_j$—not on the indices $i$ and $j$ themselves. If we swap the positions of "B" and "C" to get ["A", "C", "B"], the attention computation proceeds identically; only the final output ordering changes.

This is in stark contrast to an RNN, where the hidden state $h_t$ is computed as:

$$h_t = f(h_{t-1}, x_t)$$

Here, the output at position $t$ explicitly depends on the output at position $t-1$, creating an unavoidable sequential dependency that encodes position implicitly through the order of computation itself.

Before diving into technical solutions, let's examine why position matters from a linguistic perspective. Understanding these phenomena motivates the design of positional encoding schemes that must capture subtle positional dependencies.

Syntactic Structure Depends on Position

Natural language encodes meaning through word order. Consider these examples:

Long-Range Dependencies

Position matters not just for adjacent words but across arbitrary distances. Consider:

"The keys to the cabinet that was bought yesterday were lost."

The verb "were" must agree with "keys" (plural), not "cabinet" (singular), despite "cabinet" being much closer. A model must track that "keys" is the syntactic subject across an intervening relative clause. This requires understanding both the absolute position of "keys" and its relationship to "were" across multiple intervening positions.

Relative vs. Absolute Position

Linguistic phenomena often care about relative rather than absolute position:

• Adjacency constraints (determiners precede nouns in English)
• Bounded movement (pronoun reference typically within a clause)
• Distance-sensitive processing (garden-path effects)

This insight has driven the development of relative positional encodings, which encode the distance between token pairs rather than their absolute indices.

To appreciate positional encoding's importance, let's examine what a transformer actually computes when no positional information is provided. This pathological case illuminates exactly what is missing.

The Bag-of-Words Collapse

Without positional encoding, a transformer's output for each position depends only on:

1. The content of that position's token
2. The content of all other tokens (via attention)

This creates effective bag-of-words representations—useful for some tasks, but catastrophically insufficient for most sequence understanding.

Consider the self-attention output for position $i$ without positional encoding:

$$z_i = \sum_{j=1}^{n} \alpha_{ij} v_j$$

where the attention weights $\alpha_{ij}$ depend only on: $$\alpha_{ij} \propto \exp\left(\frac{(x_i W_Q)(x_j W_K)^T}{\sqrt{d_k}}\right)$$

If tokens "dog" and "cat" appear in a sentence, the attention from any other token to "dog" vs. "cat" depends solely on their respective embeddings—not on whether "dog" precedes or follows "cat".

Empirical Consequences

Experiments removing positional encoding from trained transformers reveal severe performance degradation on order-sensitive tasks:

Given that position information is essential, what properties should a positional encoding scheme have? The original transformer paper and subsequent research have identified several desiderata that guide the design of effective positional representations.

Core Requirements

An ideal positional encoding should satisfy:

The Encoding Interface

Mathematically, positional encoding is typically formalized as a function that maps position indices to vectors:

$$PE: \mathbb{N} \rightarrow \mathbb{R}^d$$

where $d$ is the model dimension. The encoded positions are then combined with token embeddings:

$$\tilde{x}_i = x_i + PE(i)$$

This additive combination is the standard approach in the original transformer. Alternative combination strategies include:

• Concatenation: $\tilde{x}_i = [x_i; PE(i)]$ (doubles dimension)
• Multiplicative: $\tilde{x}_i = x_i \odot PE(i)$ (element-wise product)
• Attention integration: Inject into attention scores rather than embeddings

Each approach has tradeoffs we will explore in subsequent sections.

Before the transformer, various neural architectures addressed positional information in different ways. Understanding these precedents helps contextualize the transformer's positional encoding design.

Recurrent Neural Networks (Pre-Transformer)

RNNs encode position implicitly through their sequential computation structure. The hidden state at position $t$ is computed as:

$$h_t = f(W_h h_{t-1} + W_x x_t + b)$$

Position information is captured in:

1. The recurrent connection $h_{t-1}$, which carries information about all preceding positions
2. The sequential processing order, which is deterministic

Advantages: Position is implicit and unlimited—RNNs can, in principle, process arbitrarily long sequences.

Disadvantages: Sequential processing prevents parallelization; gradients vanish or explode over long distances; the hidden state becomes a bottleneck for long-range information.

Convolutional Approaches

CNNs for sequence modeling (e.g., WaveNet, ByteNet) encode position through:

1. Local receptive fields—each output depends on a fixed window of nearby inputs
2. Stacking—deeper layers have larger effective receptive fields

Some CNN approaches add explicit position encodings similar to transformers, but the convolution structure provides some implicit positional bias toward local context.

Simple Index Encoding (Naive Baseline)

The simplest possible positional encoding is just the position index itself:

$$PE_{\text{simple}}(i) = [i, 0, 0, \ldots, 0] \in \mathbb{R}^d$$

or normalized:

$$PE_{\text{normalized}}(i) = [i / n_{\max}, 0, 0, \ldots, 0]$$

Problems with simple indexing:

1. Unbounded magnitude: Position 1000 has encoding 1000, which may overwhelm token embeddings
2. Poor extrapolation: Trained on positions 1-512, the model sees very different values for position 1000
3. Sparse utilization: Only one dimension encodes position, wasting model capacity
4. No smoothness control: Linear spacing may not match linguistic distance perception

These shortcomings motivated the development of more sophisticated encoding schemes, which we will explore in the following pages.

The research community has developed diverse approaches to positional encoding, which can be organized into several broad categories. This taxonomy provides a roadmap for the detailed treatments in subsequent pages.

Taxonomy of Positional Encoding Approaches

Absolute vs. Relative: The Central Divide

The most fundamental distinction in positional encoding is between absolute and relative approaches:

Absolute Positional Encoding:

• Assigns a unique vector to each position index
• Position 5 always has the same encoding, regardless of sequence length or context
• Examples: Sinusoidal (original transformer), learned embeddings (BERT)

Relative Positional Encoding:

• Encodes the distance or relationship between positions
• The encoding of position 5 relative to position 3 is the same as position 105 relative to position 103
• Examples: T5 relative bias, Rotary Position Embedding (RoPE)

Each approach has implications for generalization, extrapolation, and computational efficiency that we will explore in detail.

While this module focuses primarily on sequence (text) transformers, positional encoding generalizes to other domains where transformers have been applied. Understanding these extensions reveals the core principles underlying all positional encoding schemes.

Vision Transformers (ViT)

In Vision Transformers, images are divided into patches that are flattened and treated as tokens. Position now involves 2D spatial coordinates:

$$PE_{\text{2D}}(i, j) = [PE_{\text{row}}(i); PE_{\text{col}}(j)]$$

or more sophisticated 2D sinusoidal encodings:

$$PE_{\text{2D}}(x, y) = [\sin(\omega_1 x), \cos(\omega_1 x), \sin(\omega_2 y), \cos(\omega_2 y), \ldots]$$

Audio and Speech

Audio transformers (e.g., Wav2Vec, Whisper) process spectrograms or raw waveforms. Position corresponds to time, often at very high resolution (e.g., 16kHz sampling). Efficient positional encoding is critical given the long sequences involved.

Video

Video transformers must encode position in three dimensions: two spatial and one temporal. Strategies include:

• Factorized encodings: Separate spatial and temporal position
• Joint 3D positional encoding
• Frame-level vs. token-level position

Graphs and Sets

Some transformer variants operate on graphs (e.g., message passing transformers) or sets (e.g., set transformers). Here, absolute position is undefined—there is no canonical ordering. Instead:

• Graph structure provides implicit relational position
• Learnable positional encodings based on node features
• Position derived from graph algorithms (e.g., random walk positions)

We have established why positional encoding is fundamental to transformer architectures. Let's consolidate the key insights:

What's Next: Sinusoidal Encoding

In the next page, we dive deep into the original transformer's positional encoding: the sinusoidal scheme. We will:

• Derive the sinusoidal encoding formulas and their mathematical properties
• Understand why sine and cosine functions were chosen
• Explore the frequency selection and its implications for learning
• Analyze the scheme's strengths, limitations, and the insight that led to subsequent innovations

The sinusoidal encoding remains influential despite newer alternatives. Understanding it provides essential intuition for all subsequent positional encoding research.