Normal Equation vs Gradient Descent (Choosing Tools Like an Engineer)

After implementing linear regression with gradient descent (single variable, then multiple variables), I hit the most “engineer” question in the course:

If two methods solve the same problem, how do I pick the right one?

In this part of Andrew Ng’s ML course, linear regression can be trained using either:

• Gradient Descent (iterative optimization)
• Normal Equation (solve for parameters in one shot)

Both produce theta (the parameter vector). Both can predict y from X. But the developer experience and operational tradeoffs are totally different.

Same model, two training strategies

No matter which training method you choose, the prediction step is the same:

predictions = X * theta;

Option A — Gradient Descent

Gradient descent starts from an initial theta (often zeros) and updates it step-by-step.

What you implement

In Octave, a typical vectorized update loop looks like this:

for iter = 1:num_iters
  errors = (X * theta) - y;
  theta = theta - (alpha/m) * (X' * errors);
  J_history(iter) = computeCostMulti(X, y, theta);
end

Why it’s used

• Works for many models, not just linear regression
• Scales to large feature sets (where one-shot matrix operations get expensive)
• Teaches you how optimization behaves

What can go wrong

• you need to choose a good alpha
• you often need feature normalization
• convergence can be slow or unstable without telemetry

Treat J_history like logs. If you can’t see whether cost is decreasing, you’re guessing.

Option B — Normal Equation

The normal equation computes theta directly (no iterations).

In Octave, the common implementation uses the pseudo-inverse:

theta = pinv(X' * X) * X' * y;

Why it feels great

• no learning rate to tune
• no iterations
• no “is it converging?” anxiety

What can go wrong

• computing X' * X and inverting/solving it becomes expensive as feature count grows
• you can run into numerical issues if features are highly correlated (the pseudo-inverse helps)

The course typically recommends pinv instead of directly computing an inverse. pinv is more numerically stable and works better when matrices are not nicely invertible.

Feature scaling: the biggest practical difference

This is one of the most useful rules of thumb I took from the course:

• Gradient descent usually benefits massively from feature normalization.
• Normal equation does not require feature normalization to converge (because it doesn’t iterate).

That said, scaling can still be helpful for interpretability and numerical conditioning, but it’s not a hard requirement the way it is with gradient descent.

If gradient descent “doesn’t learn,” check feature scales first. The most common failure mode is mixing features with very different ranges.

Choosing like an engineer

Here’s how I frame the choice in practice.

Decision table

Rule of thumb

• If the problem is small and strictly linear regression, normal equation is a clean baseline.
• If I want a workflow that generalizes to other models (logistic regression, neural nets), gradient descent is the habit-builder.

How I compare them (a practical workflow)

Operational thinking (even in coursework)

This is where I started thinking beyond the math.

Monitoring

• Gradient descent gives you a built-in health signal: cost over time.
• Normal equation is “silent” — you get a theta, but you don’t observe a learning process.

Reproducibility

• Gradient descent depends on alpha, iterations, and initialization.
• Normal equation is deterministic given X and y.

Performance

• Gradient descent cost per iteration is predictable.
• Normal equation can be fast for small n, but becomes heavy as n grows.

The method that’s simplest for a homework dataset might not be simplest in production. Production constraints usually push you toward iterative methods and good monitoring.

What I’m keeping from this lesson

• “One-shot” vs “iterative” is a recurring theme in engineering.
• Normal equation is a great baseline for small linear regression.
• Gradient descent is the reusable tool that unlocks most of the course.
• Telemetry matters: if you can’t observe training behavior, debugging gets expensive.

What’s Next

Next up is my first real classifier: Logistic Regression.

I’ll predict admissions from exam scores, then push into a non-linear dataset and learn why regularization is the difference between a model that generalizes and one that memorizes.