Categorical Perception of p-Values

Abstract

Traditional statistics instruction emphasizes a .05 significance level for hypothesis tests. Here, we investigate the consequences of this training for researchers’ mental representations of probabilities — whether .05 becomes a boundary, that is, a discontinuity of the mental number line, and alters their reasoning about p-values. Graduate students with statistical training (n = 25) viewed pairs of p-values and judged whether they were “similar” or “different.” After controlling for several covariates, participants were more likely and faster to judge p-values as “different” when they crossed the .05 boundary (e.g., .046 vs. .052) compared to when they did not (e.g., .026 vs. .032). This result suggests a categorical perception-like effect for the processing of p-values. It may be a consequence of traditional statistical instruction creating a psychologically real divide between so-called statistical “significance” and “nonsignificance.” Such a distortion is undesirable given modern approaches to statistical reasoning that de-emphasize dichotomizing the p-value continuum.