Statistics · BEGINNER LESSON
Regression
Regression is a statistical way to describe how a dependent variable changes with one or more independent variables. It estimates the average relationship, not the value for any one person or sample.
What regression is and what it measures
Regression is a structure for a relationship between variables. The variable on the left side of the model is the dependent variable, also called the response or outcome; the variables on the right side are independent variables, also called predictors, regressors, risk factors, or antecedents. In the common ordinary least squares form, the dependent variable is quantitative, meaning it is measured on a numerical scale. In logistic regression, the dependent variable is qualitative, meaning it is categorical and often described as a proportion or probability. Regression mainly measures the average effect of the predictors on the outcome, not individual outcomes. It also uses coefficients to show how the outcome changes when a predictor changes, and fit measures such as R-squared, written R^2, or eta-squared, written η^2, to show how much variation in the outcome is explained by the predictors together.
[1] 6. The conundrum of regression, correlation, association, and ... - PMC [2] Table 2.
How it works
In a linear regression, the model can be written as y = a + bx for one predictor, or extended to several predictors for multiple regression. The result is a line in the one-predictor case or a flat surface, called a plane, when there are two predictors. Each regression coefficient describes the average contribution of one predictor while the others are included in the model. If the predictors are transformed with terms such as square, cube, logarithm, or reciprocal, the regression is nonlinear. The best-fitting linear model is often found by the least squares method, which chooses the line or surface that minimizes error. Some other regression methods in machine learning, such as support vector regression, Gaussian process regression, and decision-tree-based regression, are built to handle more complex patterns.
[1] 6. The conundrum of regression, correlation, association, and ... - PMC [2] Table 2. [3] Table 3.
Limits and interpretation
Regression coefficients depend on the measurement units, so raw coefficients are not always easy to compare across predictors. Standardized coefficients remove unit differences by rescaling variables to mean 0 and standard deviation 1, where standard deviation is a measure of spread. Statistical significance for a regression coefficient is often tested with a Student t-test under an assumption of Gaussian, meaning bell-shaped, data for the dependent variable. Even when a coefficient is significant, that does not prove cause and effect. Correlation and association describe strength of relationship, but they do not show causation unless confounding factors are ruled out. Regression models can also overfit noisy or high-dimensional data, meaning they fit the sample too closely and may work poorly on new data.
[1] 6. The conundrum of regression, correlation, association, and ... - PMC [2] Table 2. [3] Table 3.
SEE THE IDEA
Regression links predictors to an average outcome
The outcome on the left side of the model, such as a measured response.
Predictors on the right side, such as age or body mass index.
Numbers that estimate each predictor’s average contribution.
R^2, standardized coefficients, and checks for overfitting or weak interpretation.
Predictors are combined in a model to estimate the outcome’s average pattern and judge how well the model explains variation.
Regression turns several measured inputs into an estimated average outcome, but the meaning of the result depends on units, assumptions, and data quality.
TRY IT YOURSELF
Test your understanding
Choose an answer, then check the explanation.
PUT IT TOGETHER
Match each concept to its meaning
Drag a label onto its description, or select a label and then select a description. This also works with a keyboard.
4 pairs to match
Sources and further reading
This general lesson was generated from public source material. It explains a concept; it does not interpret a private paper or provide medical advice.