Machine learning · BEGINNER LESSON
Hyperparameter
A hyperparameter is a model setting chosen before training starts, not learned from the data during training. It shapes how a machine learning model learns, how complex it can become, and sometimes how fast training runs.
What it is
In machine learning, a hyperparameter is a configuration value set before training begins. Examples from the sources include maximum tree depth, number of samples used for split rules, temperature, top_p, max_length, min_length, repetition_penalty, and num_beams. By contrast, model parameters are learned from the training data during training. Hyperparameters guide the learning process rather than being directly estimated by it. [0,1]
[1] Loss function influence on hyperparameter optimization ... - PMC [2] Table 2. LLM Hyperparameter configuration.
What is measured or tuned
Hyperparameter optimization, often shortened to HPO, means searching for hyperparameter values that improve a chosen performance measure. The measure may be a loss function, which is a score used to judge how well a model performs. Different loss functions emphasize different goals: probability-based metrics such as Brier score and log loss focus on calibration, while threshold-based metrics such as precision, recall, and F1 score focus on classification at a decision boundary. [0,2]
[1] Loss function influence on hyperparameter optimization ... - PMC [3] A new good point set stepwise shrinkage optimization in machine ...
How hyperparameters affect models
The sources say hyperparameters influence model complexity, behavior, and training speed, so changing them can change the learned parameter values and the final model output. Tuning can improve performance compared with default settings, but the best setting depends on the algorithm and the metric used for optimization. In one study, the hyperparameters chosen under one metric did not always produce the best result on that same metric after evaluation. [0,2]
[1] Loss function influence on hyperparameter optimization ... - PMC [3] A new good point set stepwise shrinkage optimization in machine ...
How tuning is done
A common approach is to select hyperparameters, train the model, and then check performance on validation data. Cross-validation divides data into subsets and tests performance across them, which can help find a stable setting instead of one that works only for a single split. The sources also describe hyperparameter optimization as a bi-level process: an inner loop minimizes training loss and an outer loop minimizes validation loss. [1,2]
[2] Table 2. LLM Hyperparameter configuration. [3] A new good point set stepwise shrinkage optimization in machine ...
Limitations and cautions
Tuning is not neutral. The chosen loss function can change which model is selected and how it behaves in practice, including the predicted risk for an individual case. Cross-validation can add randomness, raise computational cost, and still leave uncertainty about the globally best setting. One source also notes that a threshold used in threshold-based metrics can itself be treated as a hyperparameter. [0,1,2]
[1] Loss function influence on hyperparameter optimization ... - PMC [2] Table 2. LLM Hyperparameter configuration. [3] A new good point set stepwise shrinkage optimization in machine ...
SEE THE IDEA
How a hyperparameter influences model development
Choose values such as depth, temperature, or beam number before fitting the model.
The model learns its parameters while the chosen hyperparameters guide the process.
A loss function or score checks performance on validation data.
The next trial may use different hyperparameters based on the evaluation result.
Sequential process: earlier settings guide later learning and evaluation.
Hyperparameters are chosen first, then they shape training and the final evaluation outcome.
TRY IT YOURSELF
Test your understanding
Choose an answer, then check the explanation.
PUT IT TOGETHER
Match each concept to its meaning
Drag a label onto its description, or select a label and then select a description. This also works with a keyboard.
4 pairs to match
Sources and further reading
This general lesson was generated from public source material. It explains a concept; it does not interpret a private paper or provide medical advice.