RNNs and sequence models

In plain English

Many problems unfold over time: sentences, stock prices, sensor streams. Sequence models read one step at a time while remembering something about what came before.

Recurrent Neural Networks (RNNs) feed each step's output back as input to the next step, building a hidden state summary of the past.

LSTMs and GRUs added gates to remember or forget information, helping longer dependencies. Transformers later replaced most RNN stacks for language with parallel self-attention.

How recurrence works

At each timestep t, an RNN takes input x_t and previous hidden state h_{t-1}, producing h_t and optionally an output y_t. Same weights reused across time—parameter sharing for sequences of varying length.

Backpropagation through time (BPTT) unrolls the sequence to compute gradients. Long sequences cause vanishing/exploding gradient issues without gating.

Bidirectional RNNs read forward and backward for encoding tasks like tagging; decoders generate outputs step by step (early machine translation).

import math

def tanh(x):
    return math.tanh(x)

h = 0.0  # initial hidden state
Wx, Wh, b = 0.5, 0.8, 0.0

for x_t in [1.0, 0.0, -1.0]:  # sequence of inputs
    h = tanh(Wx * x_t + Wh * h + b)
    print("x_t:", x_t, "-> h:", round(h, 4))

Going deeper

RNNs remain useful for low-latency streaming, small on-device models, and some time-series baselines where attention is overkill.

CNNs over text (temporal convolutions) and structured state-space models offer additional sequence tooling beyond classic RNNs.

Reading RNN-era papers clarifies why attention fixes bottlenecks: hidden states struggled to carry very long context.

Common misconceptions

RNNs are obsolete and never used.

Transformers dominate large language modeling, but RNN variants still appear in edge, audio, and specialized sequence tasks.

LSTMs can remember arbitrary-length context perfectly.

Gating helps but does not remove all long-range difficulty; extremely long dependencies remain hard.

Key facts

• RNNs maintain a hidden state updated at each timestep.
• LSTMs and GRUs use gates to mitigate vanishing gradient problems.
• Backpropagation through time trains recurrent models on sequences.
• Bidirectional encoders combine past and future context for labeling tasks.
• Transformers largely supplanted RNNs for large-scale language modeling.