Vanishing Gradients Strike Back - The Pain of Training RNNs

Last month, I finally understood what an RNN is:

• a shared-weight cell
• unrolled across time
• producing a hidden state that acts like learned memory

This month, I learned what an RNN does to you:

It turns training into a fight with gradients.

And not in a theoretical way. In a “my loss is NaN” way.

The Problem: Depth in Time Is Still Depth

In March I wrote about why deeper networks were harder to train than I expected.

October felt like that lesson returning… with a twist.

Because with RNNs, you can have a network that is shallow in layers, but extremely deep in time:

• 1 recurrent layer
• unrolled for 100 steps
• behaves like a 100-layer deep computation graph (with shared weights)

So all the trouble we saw in deep feedforward nets shows up again:

• gradients shrink until nothing learns
• gradients explode until everything breaks

Just… now it’s happening through time.

What I Observed First (Before I Understood Why)

I ran a simple character-level RNN again, but pushed it slightly harder:

• longer sequences
• slightly larger hidden state
• more training steps

And I saw three recurring behaviors:

1. The model learns short-term patterns but ignores long-term ones
2. Training becomes unstable: loss spikes suddenly
3. Changing tiny hyperparameters feels like switching realities

At first this looked like randomness.
Then I realized it was the same underlying issue:

Gradient flow across long chains is fragile.

Vanishing Gradients: When Learning Can’t Reach the Past

Here’s how it felt in practice:

• early in training, the RNN becomes decent at local structure
• it predicts plausible next characters for short contexts
• but it fails at anything requiring a long memory

The key symptom:

Improvements in prediction happen mostly for nearby dependencies.

It’s like the network can only “hear” the last few time steps.

Everything earlier becomes muffled.

My mental model (no math, just intuition)

Every time step applies another transformation to the signal (forward) and to the gradient (backward).

If that transformation tends to shrink values, then after enough steps:

• hidden state forgets
• gradients fade
• early time steps get almost no learning signal

So the network may technically “have memory” — but training never teaches it to use it.

Exploding Gradients: When One Bad Update Destroys the Run

The other failure was more dramatic.

Everything looks fine… then:

• loss jumps by an order of magnitude
• weights become huge
• hidden state norms blow up
• the next update produces NaNs

This felt like training “falls off a cliff”.

And the mechanism is the same story in reverse:

If a transformation tends to amplify values, repeated over many steps it can turn small numbers into huge ones.

What it looked like in my logs

• hidden state norm grows steadily during a batch
• gradient norm spikes
• loss spikes right after
• and training becomes unrecoverable

In classical ML I almost never saw “catastrophic training collapse.”
Here it was… normal.

The Engineering Response: Instrument Everything

My 2016 “treat it like a system” mindset saved me here.

Instead of guessing, I added instrumentation.

What I tracked

• loss per iteration
• hidden state norm (average and max across time)
• gradient norm (approximate is fine)
• how these values change as I increase sequence length

The Practical Fixes I Used (And Why They Help)

I didn’t want to jump straight to LSTMs yet.
I wanted to understand what you can do with a “vanilla” RNN first.

Here’s the short list that actually made my runs survivable.

Why Long-Term Dependencies Are So Hard (In Plain Words)

I kept trying to get the vanilla RNN to remember a detail from far back in a sequence.

And it clicked that the network isn’t failing because it’s dumb.

It’s failing because the training signal has to travel through too many transformations.

Even if the architecture could represent the dependency, the optimizer can’t reliably deliver learning to the earlier steps.

This was the crucial realization:

Architecture and optimization are inseparable.

You can’t talk about “sequence modeling” without talking about gradient flow.

How This Connected Back to My 2016 Foundation

This month didn’t feel like “new ML”.

It felt like the same fundamentals—just under more stress:

• Optimization: gradient descent is still the engine, but now it’s unstable
• Diagnostics: you must instrument and interpret failure modes
• Regularization mindset: constraints can stabilize learning
• System thinking: treat training like a process you can observe and debug

It also echoed CNNs:

CNNs succeed partly because their structure improves learnability.

RNNs struggle partly because their structure makes learnability harder.

Same principle, opposite outcome.

What Changed in My Thinking (October Takeaway)

I used to think:
“If the model is expressive enough, training will figure it out.”

After this month I think:

Expressiveness doesn’t matter if gradients can’t deliver learning to where it’s needed.

Depth in time is not “just another dimension.”
It’s a multiplier on optimization pain.

What’s Next

Vanilla RNNs taught me the problem.

Now I want the engineering solution that made sequence modeling practical:

LSTMs: Engineering Memory into the Network

Gates, memory cells, and a design that’s basically built to keep gradients alive.

FAQ