Statistical inference · BEGINNER LESSON
Multiple testing
Multiple testing means performing more than one statistical test in the same study or report. It matters because picking out a significant result from several tests can increase the chance of a false positive finding.
What it is
Multiple testing is the situation where a study runs several statistical tests rather than just one. The key issue is not only how many tests are done, but whether the results from some tests are given extra importance when the study is reported or interpreted. In that case, the chance of drawing a wrong conclusion from a chance result can increase. Here, a p-value is a number that helps judge how unusual the data are if there were actually no true effect. A false positive means concluding there is an effect when there is not one.
Why it happens in trials
In complex clinical trials, several treatment arms, several endpoints, or several related comparisons may be studied at once. Examples include umbrella trials, where multiple treatments are tested for one disease, and basket trials, where one treatment is tested across several disease groups. When a common control group is shared, the comparisons can become statistically dependent rather than fully separate, which is one reason multiplicity control is discussed. Multiplicity control means using a planned method to account for the fact that several tests are being considered together.
[1] The use of complex clinical trials: a regulatory review - PMC [2] pmc.ncbi.nlm.nih.gov · articles · PMC13324398When to Adjust for Multiple Testing: A Unifying Guiding ...
How the problem is handled
The sources describe a guiding principle: adjust for multiple testing if authors put more emphasis on one or more results because those tests have small p-values. In practice, methods mentioned include Bonferroni adjustment, Dunnett’s test, closed testing, and hierarchical testing. Regulatory reviewers also stress pre-planning, especially when new treatment arms are added or a shared control group is used. The FDA is described as often not requiring adjustment when comparisons are like separate independent trials with distinct objectives, but exceptions apply when treatments are closely related, such as different doses of the same drug.
[1] The use of complex clinical trials: a regulatory review - PMC [2] pmc.ncbi.nlm.nih.gov · articles · PMC13324398When to Adjust for Multiple Testing: A Unifying Guiding ...
Limits and cautions
There is no simple rule that fits every study. The sources note that whether tests are “related” can be subjective, and the amount of emphasis in a paper can depend on titles, abstracts, tables, and figures. Also, the guiding principle is not useful when testing is done in a biased or spurious way from the start. In complex trials, extra operational issues can also affect trial integrity and data quality, so the statistical plan and the study design must be considered together.
[1] The use of complex clinical trials: a regulatory review - PMC [2] pmc.ncbi.nlm.nih.gov · articles · PMC13324398When to Adjust for Multiple Testing: A Unifying Guiding ...
SEE THE IDEA
From several tests to a decision about adjustment
A study runs more than one statistical comparison.
Some results are highlighted more because they are significant.
A planned adjustment method is used to account for the set of tests.
The final interpretation is made with the testing structure in mind.
Several tests can create a need for adjustment when one or more significant results receive special emphasis.
The diagram shows that the need for adjustment depends on both the number of tests and how the results are reported and interpreted.
TRY IT YOURSELF
Test your understanding
Choose an answer, then check the explanation.
PUT IT TOGETHER
Match each concept to its meaning
Drag a label onto its description, or select a label and then select a description. This also works with a keyboard.
4 pairs to match
Sources and further reading
This general lesson was generated from public source material. It explains a concept; it does not interpret a private paper or provide medical advice.