Gradient descent
you can feel
Training a model is a ball rolling downhill on a surface it cannot see. Here is the surface. Drag to look around it, then go and push the ball yourself.
Drag to orbit · 6 min
The model is blind. It only feels the slope.
Flattened out, the surface becomes a contour map — dark valleys, bright ridges. At every step the model learns one thing: which way is downhill from exactly where it stands. Not where the bottom is. Just the slope under its feet. Tap anywhere to drop the ball and watch it work.
Loss over steps
Optimiser
Start somewhere hard
The easy case. Almost anything works from here.
Two parameters, so the landscape can be drawn. A real model has millions, and the surface cannot be pictured — but the mechanism on screen is exactly the one running inside it.
One number decides whether it works at all.
Same surface, same start, same optimiser. The only difference is step size. This is why the learning rate is the first thing anyone tunes, and why a model that will not train is usually not a broken model.
Now the training log reads differently.
Why your loss curve plateaus
A flat stretch is not always progress stalling — often the ball is crossing a wide, shallow region where gradients are tiny. Lowering the learning rate there makes it worse, not better.
Why the learning rate is the first thing to tune
Nothing else you change matters if this is wrong. Too small and training never finishes; too large and it never starts. Every other hyperparameter is a refinement of this one.
Why batch size shows up in the loss
Small batches make the gradient an estimate rather than the truth, which is the noise slider. That noise is a cost — and occasionally the thing that shakes you out of a bad minimum.
Why schedules exist
Large steps early to cover ground, small steps late to settle. Once you have watched a fixed rate fail at both ends, decay stops being a trick you copy and becomes the obvious fix.
Next
You just tuned a learning rate by feel.
In Module III you write this loop yourself — the forward pass, the backward pass, and the optimiser — before you are allowed to import one.