Machine learning loss function · BEGINNER LESSON
Binary cross-entropy
Binary cross-entropy is a loss and performance measure for a two-class classifier that compares the true class with the predicted probability for that class. It is computed from the negative log of the probability assigned to the correct label, with optional class weighting in imbalanced data.
What it is and what is measured
Binary cross-entropy is used in binary classification, where each example belongs to one of two classes. It measures how much probability the model assigns to the correct class: predictions that give high probability to the true label get low loss, while confident wrong predictions get high loss. In one common form, if y is the true label (0 or 1) and p is the predicted probability for class 1, the loss is −(y log(p) + (1−y) log(1−p)). In the cited classifier study, cross-entropy is also treated as a performance measure and an optimization objective. [0,1,2]
[1] Deconstructing Cross-Entropy for Probabilistic Binary ... [2] Drug Resistance Prediction Using Deep Learning ... [3] Automatic recognition of complementary strands: lessons regarding machine learning abilities in RNA folding
How it works
The logarithm makes the penalty grow quickly when the model is very confident but wrong. If the true label is 1, the loss depends on log(p); if the true label is 0, it depends on log(1−p). This makes the training process push predicted probabilities toward the correct class. One source also describes a weighted version for imbalanced data, where positive and negative examples do not appear in the same proportion, so the loss can be adjusted to reduce bias toward the larger class. [1,2]
[2] Drug Resistance Prediction Using Deep Learning ... [3] Automatic recognition of complementary strands: lessons regarding machine learning abilities in RNA folding
Interpretation in probabilistic classification
A more detailed source explains cross-entropy in Bayesian decision theory, meaning a framework for decision-making with probabilities. In that view, the loss reflects two broad ingredients: prior knowledge about class frequencies and the information from the observation, expressed through a likelihood ratio, which compares how strongly the features support one class over the other. The same source also separates the loss into discrimination and calibration components. Discrimination is about separating the classes well; calibration is about whether the predicted probabilities are reliable. [0]
[1] Deconstructing Cross-Entropy for Probabilistic Binary ...
Limitations and caution
Binary cross-entropy is useful during training, but low training loss does not guarantee good testing performance. One cited study notes that performance can be limited by small datasets, class imbalance, and overfitting, where a model fits training examples too closely and generalizes poorly to new data. Another limitation is that a model can separate classes well without producing well-calibrated probabilities, so a single loss value may not reveal every weakness. [1,2]
[2] Drug Resistance Prediction Using Deep Learning ... [3] Automatic recognition of complementary strands: lessons regarding machine learning abilities in RNA folding
SEE THE IDEA
Binary cross-entropy in binary classification
The correct class for one example: 0 or 1.
The model’s probability for class 1.
Penalty from the negative log of the probability given to the correct label.
The loss guides parameter updates so probabilities move toward the correct class.
True label and predicted probability are combined by log loss, which becomes the training signal.
Binary cross-entropy compares the correct label with predicted probability and turns mismatch into a penalty.
TRY IT YOURSELF
Test your understanding
Choose an answer, then check the explanation.
PUT IT TOGETHER
Match each concept to its meaning
Drag a label onto its description, or select a label and then select a description. This also works with a keyboard.
4 pairs to match
Sources and further reading
This general lesson was generated from public source material. It explains a concept; it does not interpret a private paper or provide medical advice.