If you are not a statistician, this cartoon is mocking statistical significance used in most frequentist hypothesis tests. If we use a threshold of \(\)P<0.05[/latex] then we accept that on average we will falsely declare there to be a significant departure from the null approximately one time in twenty ([latex]1/20 = 0.05[/latex]). In lay terms we will decide there is a difference when there really is not one time in twenty.
Thanks Randall.

The same thing can happen to a Bayesian too. If they applied the same analysis, say with a 50% prior for the effect being nonzero, then they could cherry pick the data set that happens to make the posterior probability high.

Viewed from the point of view of the question “are there ANY nonzero effects”, the 50% prior looks very stupid, because it already confidently asserts that some effect exists. The key to “multiple testing” situations is not being an idiot about priors.

Agreed, although I find those slab and spike priors are used a little too often to highlight situations where decisions based Frequentist and Bayesian tests give conflicting information.

I guess the cartoon is playing up publication bias too.

The same thing can happen to a Bayesian too. If they applied the same analysis, say with a 50% prior for the effect being nonzero, then they could cherry pick the data set that happens to make the posterior probability high.

Viewed from the point of view of the question “are there ANY nonzero effects”, the 50% prior looks very stupid, because it already confidently asserts that some effect exists. The key to “multiple testing” situations is not being an idiot about priors.

Agreed, although I find those slab and spike priors are used a little too often to highlight situations where decisions based Frequentist and Bayesian tests give conflicting information.

I guess the cartoon is playing up publication bias too.