Optimization algorithm · BEGINNER LESSON
Stochastic gradient descent
Stochastic gradient descent (SGD) is a first-order optimization method that updates parameters using a cheap random estimate of the true gradient instead of the full gradient. It is used when the exact gradient is expensive to compute, especially when the objective is an average or sum over many training examples.
What it is and what it measures
Stochastic gradient descent is a way to minimize a differentiable function, meaning a function with a gradient. The gradient is a vector that points in the direction of steepest increase, so moving in the opposite direction can reduce the function value. In machine learning, the function to minimize often has many terms, one per training example, so computing the full gradient can be too slow. SGD measures only a cheap estimate of that gradient at each step, rather than the exact full gradient.
[1] www.mit.edu › ~gfarina › 2025Lecture 15 Stochastic gradient descent - MIT [2] Stochastic gradient descent Lecture 10 - MIT [3] theory.stanford.edu · ~tim · s16CS168: The Modern Algorithmic Toolbox Lecture #6: Stochastic...
How it works
The basic update is: new parameters = old parameters minus a step size times an estimated gradient. The step size is also called the learning rate. The estimate is usually unbiased, which means that if you averaged many such random estimates, you would recover the true gradient on average. A common version is mini-batch SGD, where the estimate comes from a small randomly chosen subset of training data instead of a single example or the full data set.
[1] www.mit.edu › ~gfarina › 2025Lecture 15 Stochastic gradient descent - MIT [2] Stochastic gradient descent Lecture 10 - MIT [3] theory.stanford.edu · ~tim · s16CS168: The Modern Algorithmic Toolbox Lecture #6: Stochastic...
Why randomness helps
Using a small sample makes each step much cheaper, so the algorithm can take many more steps in the same time than full gradient descent. The randomness also adds noise to the updates. In machine learning, that noise has been linked to reduced overfitting and to escaping local minima and saddle points, which are points where the loss is not globally best but the landscape can trap simpler methods.
[1] www.mit.edu › ~gfarina › 2025Lecture 15 Stochastic gradient descent - MIT [2] Stochastic gradient descent Lecture 10 - MIT [3] theory.stanford.edu · ~tim · s16CS168: The Modern Algorithmic Toolbox Lecture #6: Stochastic...
Limitations and trade-offs
SGD trades exactness for speed. Because the gradient estimate has variance, updates can look erratic instead of smooth. If the step size is too large, SGD may fail to converge. For very noisy estimates, the amount of safe progress per step can become smaller, so the choice of learning rate and batch size matters.
[1] www.mit.edu › ~gfarina › 2025Lecture 15 Stochastic gradient descent - MIT [2] Stochastic gradient descent Lecture 10 - MIT [3] theory.stanford.edu · ~tim · s16CS168: The Modern Algorithmic Toolbox Lecture #6: Stochastic...
SEE THE IDEA
How stochastic gradient descent updates a model
Start with the model’s current values.
Pick one example or a mini-batch from the training set.
Compute a cheap unbiased approximation instead of the full gradient.
Move opposite the estimate using the learning rate.
Each step uses a cheaper estimate to replace an expensive exact gradient, trading smoothness for speed.
SGD repeats this cycle many times to reduce the objective function on large data sets.
TRY IT YOURSELF
Test your understanding
Choose an answer, then check the explanation.
PUT IT TOGETHER
Match each concept to its meaning
Drag a label onto its description, or select a label and then select a description. This also works with a keyboard.
4 pairs to match
Sources and further reading
This general lesson was generated from public source material. It explains a concept; it does not interpret a private paper or provide medical advice.