Machine Learning / Hyperparameter · BEGINNER LESSON
Batch size
Batch size is a machine learning hyperparameter that tells a model how many training examples to process before it updates its weights and bias. It controls how often training updates happen and helps shape training speed and noise.
What batch size is
Batch size is a hyperparameter, meaning it is an external setting that controls part of training. In linear regression training, it is the number of examples the model processes before it updates its weights and bias. A weight and bias are the model’s learned numbers; training changes them to reduce loss, which is a measure of how wrong the model’s predictions are. [source 0]
How it works during training
If batch size is 1, the model uses stochastic gradient descent, or SGD, where each update is based on a single randomly chosen example. If batch size is larger than 1 but smaller than the full dataset, the model uses mini-batch stochastic gradient descent, or mini-batch SGD, and averages the gradients from the examples in that batch before updating once. A gradient is the direction and size of change used to improve the model. [source 0]
Why it matters
Batch size affects training speed and noise. Smaller batches usually create noisier training because each update is based on fewer examples, so the loss can rise during an iteration more often. Larger batches average more gradients, which can reduce the effect of outliers, but the best choice depends on the dataset and available compute resources, meaning the hardware and processing power available for training. [source 0]
Limits and trade-offs
There is no single ideal batch size for every problem. Very large full-batch training may be impractical when datasets contain hundreds of thousands or millions of examples. Mini-batch sizes are chosen between 1 and the dataset size, and the practical choice depends on the dataset and compute resources rather than a fixed rule. [source 0]
SEE THE IDEA
How batch size changes training
The dataset provides examples for the model to learn from.
The model groups 1, a few, or many examples before an update.
The model combines the gradients from the batch.
The model updates weights and bias once per batch.
Smaller batches mean more frequent updates with more noise; larger batches mean fewer updates with more averaging.
Batch size determines how many examples are used to make one training update.
TRY IT YOURSELF
Test your understanding
Choose an answer, then check the explanation.
PUT IT TOGETHER
Match each concept to its meaning
Drag a label onto its description, or select a label and then select a description. This also works with a keyboard.
4 pairs to match
Sources and further reading
This general lesson was generated from public source material. It explains a concept; it does not interpret a private paper or provide medical advice.