1
What a P-Value Can and Cannot Tell You
Student Name
Department, University Name
STAT 200: Introductory Statistics
Professor Name
12 November 2026
What a P-Value Can and Cannot Tell You
The p-value is one of the most widely reported numbers in research and one of the most widely misunderstood. Many students learn to compare it with 0.05 and declare a result significant or not, without asking what the number means. This essay explains the definition of a p-value, sets out common misinterpretations, considers the guidance issued by the American Statistical Association, and argues that p-values are useful only when reported alongside effect sizes, confidence intervals and an honest account of the analysis.
What a P-Value Means
A p-value is the probability of obtaining a result at least as extreme as the one observed, assuming that the null hypothesis and the other assumptions of the statistical model are true. A small p-value indicates that the data would be unusual if the null hypothesis held. It is a statement about the data given a hypothesis, not about the hypothesis given the data.
Common Misinterpretations
The most common error is to treat the p-value as the probability that the null hypothesis is true. A p-value of 0.03 does not mean there is a 3% chance the effect is absent. Cohen (1994) argued forcefully that this confusion reverses the logic of the test and leads researchers to overstate their findings. A second error is to treat statistical significance as practical importance. With a very large sample, a trivial difference can produce a tiny p-value, while an important effect may fail to reach significance in a small study.
A third problem is the practice of treating 0.05 as a bright line. Results just below and just above the threshold provide almost identical evidence, yet they are often reported as opposite conclusions. When researchers run many tests or try several analyses and report only those that cross the threshold, the chance of false positives rises sharply.
This problem contributes to what is often called the replication crisis. In several fields, attempts to repeat published studies have produced weaker or absent effects. Part of the explanation lies in the combination of small samples, flexible analysis and a publication system that favours significant results. When only studies that cross the threshold are published, the published literature overstates the size and reliability of effects. Pre-registration of hypotheses and analysis plans, sharing of data and greater willingness to publish null results are among the responses researchers have adopted.
Professional Guidance
Concern about these problems led the American Statistical Association to issue a formal statement on p-values (Wasserstein & Lazar, 2016). Among its principles are that p-values do not measure the probability that a hypothesis is true, that scientific conclusions should not rest only on whether a p-value passes a threshold, and that a p-value does not measure the size of an effect or the importance of a result. The statement also stresses full reporting and transparency about every analysis conducted.
Conclusion
A p-value is a limited but legitimate tool. It indicates how compatible the data are with a specific model, nothing more. Good statistical practice reports it together with the estimated effect size and a confidence interval, explains how the analysis was chosen, and interprets results in the context of the research question. Students who learn to read p-values this way will avoid the most common errors in published research. They will also be better prepared to read research critically, asking not only whether a result is significant but how large it is, how precisely it was estimated, and whether the study would be likely to give the same answer if it were repeated.
References
Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49(12), 997–1003.
Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133.
Sample paper by Professor Eric · For study and reference only. Do not submit it as your own work.