Gated Recurrent Units

The Long Short-Term Memory (LSTM) architecture revolutionized sequence modeling by introducing gating mechanisms that allow gradients to flow unimpeded across long temporal spans. However, LSTMs come with substantial complexity: three distinct gates (input, forget, output), a separate cell state, and intricate interactions between components that make both implementation and intuition challenging.

In 2014, Cho et al. posed a provocative question: Can we achieve comparable performance with a simpler architecture? The answer was the Gated Recurrent Unit (GRU)—a design that distills the essence of gating into a more elegant form while maintaining the capacity to capture long-range dependencies.

This page examines the principles behind GRU's design, exploring how strategic simplification can yield architectures that are not merely "good enough" but often preferred in practice.

Before appreciating GRU's elegance, we must understand what it simplifies. The LSTM architecture, while powerful, introduces considerable complexity that manifests in multiple dimensions.

Architectural Complexity

The standard LSTM cell maintains two distinct state vectors:

• Cell state (c_t): The long-term memory highway, modified through element-wise operations
• Hidden state (h_t): The short-term output, computed from the cell state through the output gate

These states interact through three gating mechanisms:

$$\begin{aligned} \mathbf{f}_t &= \sigma(\mathbf{W}f \cdot [\mathbf{h}{t-1}, \mathbf{x}_t] + \mathbf{b}_f) \quad \text{(forget gate)} \ \mathbf{i}_t &= \sigma(\mathbf{W}i \cdot [\mathbf{h}{t-1}, \mathbf{x}_t] + \mathbf{b}_i) \quad \text{(input gate)} \ \mathbf{o}_t &= \sigma(\mathbf{W}o \cdot [\mathbf{h}{t-1}, \mathbf{x}_t] + \mathbf{b}_o) \quad \text{(output gate)} \end{aligned}$$

Plus a candidate cell state and the final computations:

$$\begin{aligned} \tilde{\mathbf{c}}_t &= \tanh(\mathbf{W}c \cdot [\mathbf{h}{t-1}, \mathbf{x}_t] + \mathbf{b}_c) \ \mathbf{c}_t &= \mathbf{f}t \odot \mathbf{c}{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t \ \mathbf{h}_t &= \mathbf{o}_t \odot \tanh(\mathbf{c}_t) \end{aligned}$$

This gives us four weight matrices (W_f, W_i, W_o, W_c), four bias vectors, and multiple nonlinear transformations per timestep.

Practical Implications

This complexity manifests in several ways:

1. Training Time: More parameters mean more gradients to compute. Each backward pass through an LSTM requires computing gradients through all gate operations, multiplying training time.

2. Hyperparameter Sensitivity: More components mean more interactions. The forget gate bias initialization, cell state clipping, gradient clipping thresholds—all require careful tuning.

3. Overfitting Risk: With ~525K parameters for a single 256-unit layer, LSTMs can easily overfit on smaller datasets.

4. Hardware Utilization: The sequential nature of recurrence already limits parallelization; complex cell computations compound this limitation.

5. Interpretability: Understanding what an LSTM has learned requires analyzing three gates and two state vectors—a daunting task for practitioners.

The Birth of GRU

The Gated Recurrent Unit was introduced by Cho et al. in their 2014 paper "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation". The context is crucial: this was not a paper about recurrent architectures per se, but about building effective encoder-decoder systems for machine translation.

The authors needed an efficient recurrent unit that could:

1. Handle variable-length input sequences
2. Capture long-range dependencies in source sentences
3. Train efficiently on large parallel corpora
4. Generalize well to unseen sentence structures

Rather than defaulting to LSTM, they designed a new unit from first principles, asking: What is the minimal gating structure that preserves the essential properties of gated recurrence?

The Design Principles

Cho et al. identified several key insights:

1. The forget-input gate redundancy: In LSTM, the forget gate decides what to discard, while the input gate decides what to add. But these decisions are inherently coupled—discarding old information often correlates with adding new information. Why not model this directly?

2. The cell-hidden state distinction: LSTM maintains separate cell and hidden states, but the output gate's job is largely to mediate between them. What if we could eliminate this indirection?

3. The output gate's role: The output gate in LSTM controls how much of the cell state to expose. But if we unify cell and hidden states, this gate becomes redundant.

Concurrent Developments

GRU was not developed in isolation. Around the same time:

• Bahdanau et al. (2014) introduced attention mechanisms for translation, which would eventually supersede both LSTM and GRU for many NLP tasks
• Sutskever et al. (2014) demonstrated that deep LSTMs could achieve state-of-the-art translation
• Zaremba et al. (2014) showed that dropout could be applied effectively to LSTMs

This context is important: GRU emerged during a period of rapid innovation in sequence modeling, and its success was not guaranteed. That it has remained relevant despite the attention revolution speaks to the soundness of its design principles.

GRU achieves its simplification through two fundamental design decisions:

Simplification 1: Merge the Cell State and Hidden State

In LSTM: $$\mathbf{c}_t eq \mathbf{h}_t$$

The cell state is a protected channel for gradient flow, while the hidden state is the externally-visible output. This separation enables the LSTM to maintain information without exposing it, but it doubles the state dimensionality.

GRU unifies these: $$\mathbf{h}_t \text{ serves both roles}$$

The hidden state directly carries long-term information. There is no separate "memory highway"—the highway and the output are one.

Why This Works

The key insight is that the cell state's protection in LSTM comes primarily from the gating mechanism, not from its separation from the hidden state. If we design our gates correctly, we can protect information while keeping a single state vector.

Simplification 2: Couple the Forget and Input Gates

In LSTM, the forget gate (f_t) and input gate (i_t) operate independently:

• f_t decides how much old information to retain
• i_t decides how much new information to add

These can sum to more than 1 (adding information without removing) or less than 1 (removing without adding). This flexibility may seem beneficial, but it also means the network must learn to coordinate these gates properly.

GRU introduces a single update gate (z_t) that inherently couples these decisions:

$$\mathbf{h}_t = (1 - \mathbf{z}t) \odot \mathbf{h}{t-1} + \mathbf{z}_t \odot \tilde{\mathbf{h}}_t$$

When z_t is close to 0:

• $(1 - z_t) \approx 1$: Almost all of the previous state is retained
• $z_t \approx 0$: Almost none of the candidate is added

When z_t is close to 1:

• $(1 - z_t) \approx 0$: Almost all of the previous state is discarded
• $z_t \approx 1$: Almost all of the candidate is used

This creates a conservation law: the total "weight" on old and new information always sums to 1. The network cannot simultaneously retain everything and add everything—it must make tradeoffs.

With the design principles established, let us formalize the complete GRU architecture. The following equations define the forward computation of a single GRU cell:

Update Gate

$$\mathbf{z}_t = \sigma(\mathbf{W}_z \mathbf{x}_t + \mathbf{U}z \mathbf{h}{t-1} + \mathbf{b}_z)$$

The update gate controls the interpolation between old and new states. The sigmoid activation ensures $z_t \in (0, 1)^d$, enabling smooth interpolation.

Reset Gate

$$\mathbf{r}_t = \sigma(\mathbf{W}_r \mathbf{x}_t + \mathbf{U}r \mathbf{h}{t-1} + \mathbf{b}_r)$$

The reset gate determines how much of the previous hidden state to expose when computing the candidate. This is GRU's mechanism for selectively "forgetting" parts of the past when relevant.

Candidate Hidden State

$$\tilde{\mathbf{h}}_t = \tanh(\mathbf{W}_h \mathbf{x}_t + \mathbf{U}_h (\mathbf{r}t \odot \mathbf{h}{t-1}) + \mathbf{b}_h)$$

The candidate is computed from the current input and a reset-modulated version of the previous state. When r_t ≈ 0, the candidate ignores history—useful when the past is irrelevant to the current context.

Final Hidden State

$$\mathbf{h}_t = (1 - \mathbf{z}t) \odot \mathbf{h}{t-1} + \mathbf{z}_t \odot \tilde{\mathbf{h}}_t$$

The final state interpolates between the previous state and the candidate, controlled element-wise by the update gate.

Parameter Count Comparison

For a hidden dimension $d_h$ and input dimension $d_x$:

GRU parameters (single layer): $$3 \times (d_h \times d_x + d_h \times d_h + d_h) = 3(d_h \cdot d_x + d_h^2 + d_h)$$

LSTM parameters (single layer): $$4 \times (d_h \times d_x + d_h \times d_h + d_h) = 4(d_h \cdot d_x + d_h^2 + d_h)$$

Example: For $d_h = 256$ and $d_x = 128$:

• GRU: $3(256 \cdot 128 + 256^2 + 256) = 3(32,768 + 65,536 + 256) = 295,680$ parameters
• LSTM: $4(32,768 + 65,536 + 256) = 394,240$ parameters

GRU uses 25% fewer parameters than LSTM. This reduction is not marginal—it translates directly to faster training, lower memory consumption, and reduced overfitting risk.

The reset gate is perhaps the most distinctive feature of GRU's design. Unlike LSTM's three gates which all modulate state flow, the reset gate specifically controls how much history to consider when forming the candidate update.

Intuition: Selective Amnesia

Consider processing a sentence:

"The cat, which was very old and had lived through many adventures, finally sat down."

When computing the representation for "sat," how much of "The cat" matters versus "many adventures"? The reset gate enables the model to selectively "forget" irrelevant parts of history when computing updates.

$$\tilde{\mathbf{h}}_t = \tanh(\mathbf{W}_h \mathbf{x}_t + \mathbf{U}_h (\mathbf{r}t \odot \mathbf{h}{t-1}) + \mathbf{b}_h)$$

When $\mathbf{r}_t \approx \mathbf{0}$:

• The previous hidden state is zeroed out
• The candidate depends only on the current input
• The unit can "start fresh" if history is irrelevant

When $\mathbf{r}_t \approx \mathbf{1}$:

• The full previous state is used
• The candidate incorporates all accumulated context
• Continuity with history is maintained

Comparison with LSTM's Forget Gate

The LSTM forget gate operates differently:

$$\mathbf{c}_t = \mathbf{f}t \odot \mathbf{c}{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t$$

The forget gate directly discards information from the cell state. In contrast, GRU's reset gate affects only how the candidate is computed—it doesn't directly discard information from the state. The actual forgetting in GRU happens through the update gate.

This is a subtle but important distinction:

• LSTM forget gate: "Delete this information from memory"
• GRU reset gate: "Ignore history when deciding what to add"

The GRU approach is arguably more nuanced. Even when the reset gate is closed, the update gate can still choose to retain the previous state. The reset gate affects the proposal, not the decision.

Understanding how information flows through a GRU is essential for developing intuition about its behavior. Let us trace both the forward and backward passes to understand the architecture's computational structure.

Forward Pass: Information Aggregation

At each timestep, information from the input $x_t$ and history $h_{t-1}$ is combined through a carefully orchestrated sequence:

1. Gate Computation (parallel):

   • Update gate $z_t$ assesses: "Should I update the state?"
   • Reset gate $r_t$ assesses: "Should I consider history for the update?"

2. Candidate Formation:

   • History is optionally masked via $r_t$
   • Input and (masked) history combine to form proposal $\tilde{h}_t$

3. State Update:

   • Update gate interpolates between $h_{t-1}$ and $\tilde{h}_t$
   • Final state $h_t$ results

The Gradient Highway

The critical question for any recurrent architecture: how do gradients flow backward through time? In GRU, the update equation provides a direct path:

$$\mathbf{h}_t = (1 - \mathbf{z}t) \odot \mathbf{h}{t-1} + \mathbf{z}_t \odot \tilde{\mathbf{h}}_t$$

Taking the gradient of a loss $L$ with respect to $h_{t-1}$:

$$\frac{\partial L}{\partial \mathbf{h}_{t-1}} = \frac{\partial L}{\partial \mathbf{h}_t} \cdot \frac{\partial \mathbf{h}t}{\partial \mathbf{h}{t-1}}$$

The key term is:

$$\frac{\partial \mathbf{h}t}{\partial \mathbf{h}{t-1}} = \text{diag}(1 - \mathbf{z}_t) + \text{(terms involving } \tilde{\mathbf{h}}_t, \mathbf{z}_t, \mathbf{r}_t \text{)}$$

The crucial observation: when $z_t \approx 0$, the gradient through the direct path $(1 - z_t) \odot h_{t-1}$ approaches identity. This creates the "gradient highway" that allows information to flow backward through time without vanishing.

The Interpolation Advantage

GRU's update rule can be viewed as a leaky integration:

$$\mathbf{h}t = \mathbf{h}{t-1} + \mathbf{z}_t \odot (\tilde{\mathbf{h}}t - \mathbf{h}{t-1})$$

This alternative form reveals that GRU computes updates to the state, gated by $z_t$. When updates are small (small $z_t$), the state changes slowly, enabling long-term memory. When updates are large (large $z_t$), the state can rapidly adapt.

This interpolation structure ensures:

1. States remain bounded (convex combinations of bounded values)
2. Gradients have a direct path that doesn't require matrix multiplication
3. The network can smoothly transition between "remember" and "update" modes

Computational Graph Structure

The GRU computational graph is simpler than LSTM's:

• Only one state vector to maintain and update
• Two gates instead of three
• Fewer intermediate values to compute and store
• More regular structure aids hardware optimization

The original GRU formulation has spawned numerous variants, each exploring different points in the simplicity-expressivity tradeoff space.

Minimal Gated Unit (MGU)

The Minimal Gated Unit takes simplification further by using a single gate for both reset and update:

$$\begin{aligned} \mathbf{f}_t &= \sigma(\mathbf{W}_f \mathbf{x}_t + \mathbf{U}f \mathbf{h}{t-1} + \mathbf{b}_f) \ \tilde{\mathbf{h}}_t &= \tanh(\mathbf{W}_h \mathbf{x}_t + \mathbf{U}_h (\mathbf{f}t \odot \mathbf{h}{t-1}) + \mathbf{b}_h) \ \mathbf{h}_t &= (1 - \mathbf{f}t) \odot \mathbf{h}{t-1} + \mathbf{f}_t \odot \tilde{\mathbf{h}}_t \end{aligned}$$

MGU uses the same gate for both modulating history in the candidate and interpolating the final state. This further reduces parameters but may limit flexibility.

Type-1 and Type-2 GRU (Dey & Salem, 2017)

These variants explore different reset gate formulations:

Type-1: Reset gate affects only the recurrent connection (standard GRU) Type-2: Reset gate affects both input and recurrent connections Type-3: No reset gate (equivalent to a simpler interpolation)

Empirical studies show that the standard GRU (Type-1) performs best on most tasks.

Initialization and Regularization Considerations

Unlike LSTM, which benefits from specific forget gate bias initialization (typically 1.0 to encourage remembering), GRU is more robust to initialization:

• Weight initialization: Standard Xavier or He initialization works well
• Bias initialization: Zeros generally work; no special initialization required
• Regularization: Dropout, weight decay, and layer normalization all apply straightforwardly

This robustness is another practical advantage of GRU's simplified design—fewer hyperparameters to tune means faster development cycles.

We have explored the design philosophy and mathematical structure underlying the Gated Recurrent Unit. The key lessons extend beyond GRU itself to principles of neural architecture design:

Simplification Strategies That Work

1. Identify redundancy: LSTM's forget and input gates often correlate. GRU couples them, eliminating redundant parameters.

2. Question separations: LSTM's cell/hidden state distinction adds complexity. GRU proves it's often unnecessary.

3. Preserve essential structure: GRU maintains the gradient highway property through its interpolation mechanism, preserving LSTM's key advantage.

4. Embrace constraints: The convex combination constraint in GRU's update rule regularizes learning, preventing degenerate configurations.

What's Next

Having established the design philosophy behind GRU, the next page provides a deep dive into the update and reset gates themselves. We will examine:

• Detailed mathematical analysis of each gate's role
• Visualizations of gate activations on real sequences
• Intuition for what gates learn to detect
• How gates coordinate to model complex temporal patterns

Understanding these gates at a deep level is essential for debugging GRU models, interpreting their decisions, and knowing when simpler or more complex alternatives are appropriate.