Machine learning regularization · BEGINNER LESSON
Weight decay
Weight decay is a regularization method that reduces the size of a model’s weights during training. In these sources, it is described as removing a proportion of the current weight matrix at each update, which helps keep weights small and can make training less dependent on initialization.
What it is and what is measured
Weight decay is a rule added to training updates in which the weights are pulled downward in size over time. The measurable quantity is the magnitude, or norm, of the weights: a norm is a number that summarizes how large a vector or matrix is. In these sources, weight decay is linked to smaller weight norms, reduced overfitting, and more stable training behavior. It is discussed together with stochastic gradient descent (SGD), which is an optimization method that updates parameters using noisy minibatches of data, and with weight normalization (WN) or batch normalization (BN), which rescale weights or activations during training. [0,1,2]
[1] [PDF] Dynamics and Neural Collapse in Deep Classifiers trained with the ... [2] New insights into training dynamics of deep classifiers [3] Hebbian Descent: A Unified View on Log-Likelihood Learning
How it works
The sources describe weight decay as subtracting a fraction of the current weights at each update. That simple shrinkage can change the path of learning: it can promote minimum-norm solutions, help make final solutions less sensitive to the starting weights, and in some settings support convergence toward normalized critical points. In deep classifiers trained with square loss, the paper reports that SGD, weight decay, and weight normalization together can drive neural collapse, a training outcome where classes become highly structured in the learned representation. [0,1]
[1] [PDF] Dynamics and Neural Collapse in Deep Classifiers trained with the ... [2] New insights into training dynamics of deep classifiers
Why it can help
One benefit is regularization, which means discouraging overly complex fitted models. The MIT news summary states that weight decay reduces the magnitude of weights and helps prevent overfitting, which is fitting the training examples too closely instead of learning patterns that generalize. The research excerpt also says that weight decay can make large-margin or minimum-norm solutions more robust to initialization, and can stabilize normalized weights. In another setting, it is used as an explicit forgetting mechanism, removing a proportion of the current weights each step. [0,1,2]
[1] [PDF] Dynamics and Neural Collapse in Deep Classifiers trained with the ... [2] New insights into training dynamics of deep classifiers [3] Hebbian Descent: A Unified View on Log-Likelihood Learning
Limits and assumptions
Weight decay is not a guarantee of perfect learning or generalization. The theoretical analysis in the research excerpt depends on assumptions, and some results are shown for specific toy models, square loss, or particular combinations such as BN plus weight decay or weight normalization plus weight decay. The forgetting paper also notes that different learning rules can behave differently when activation functions saturate or when a zero target produces no update. So weight decay is a useful training control, but its effect depends on the model, loss, data, and other training choices. [0,2]
[1] [PDF] Dynamics and Neural Collapse in Deep Classifiers trained with the ... [3] Hebbian Descent: A Unified View on Log-Likelihood Learning
SEE THE IDEA
How weight decay changes training
The parameters being learned in the model.
A fraction of the current weights is removed each step.
Weights stay smaller in magnitude over training.
This can reduce overfitting or stabilize certain solutions.
Current weights are repeatedly shrunk by the update rule, which pushes the model toward smaller norms and can change which solution training reaches.
A simple view of weight decay as repeated shrinking of weights during optimization.
TRY IT YOURSELF
Test your understanding
Choose an answer, then check the explanation.
PUT IT TOGETHER
Match each concept to its meaning
Drag a label onto its description, or select a label and then select a description. This also works with a keyboard.
4 pairs to match
Sources and further reading
This general lesson was generated from public source material. It explains a concept; it does not interpret a private paper or provide medical advice.