Problems with p-values

Nick Griffiths · Jun 16, 2020

Invalidation

  • Cherry-picking good data that supports the hypothesis and ignoring the rest. This includes obvious hunting for p-values as well as subtler issues. One example is doing a meta-analysis on a topic where many non-significant results were obtained by researchers but left unpublished.

  • Changing the hypothesis based on the data. Like with cherry-picking, this can overstate the significance of results. It includes things like removing terms from a model after deciding they aren’t significant. (However, adding new terms won’t cause problems).

Misinterpretation

There are lots of ways to misinterpret p-values. In a 2019 article, Sander Greenland gives a good overview with more detail.

  • Probability of data due to chance. The p-value is not the probability that chance alone produced the data. That is the probability that the null hypothesis is true, and it is always unknown. P-values assume the null and measure how compatible the data are with this pre-specified hypothesis.

  • Scale issues. The p-value is not additive, it is multiplicative. By taking the logarithm it becomes additive. (Greenland 2019 recommends s-values, \(s = -log_2(p)\)).