12  Why Statistical Significance is Useless

Rather than writing a new chapter here, I will share my thoughts in class.

Two important references for what I’ll be saying are:

• Ronald L. Wasserstein, Allen L. Schirm & Nicole A. Lazar (2019) Moving to a World Beyond “p < 0.05”, The American Statistician, 73:sup1, 1-19, DOI: 10.1080/00031305.2019.1583913.
  • Ron gave a one-hour talk you can watch here on “Helping to move to a world beyond p < 0.05” which I cannot recommend enough.
• Ronald L. Wasserstein & Nicole A. Lazar (2016) The ASA’s Statement on p-Values: Context, Process, and Purpose, The American Statistician, 70:2, 129-133, DOI: 10.1080/00031305.2016.1154108.

12.0.1 Learning More about Statistical Significance (and related issues)

All of this material is also found on our Resources page.

• The Growing Importance of Reproducibility and Responsible Workflow in the Data Science and Statistics Curriculum by Nicholas J. Horton, Rohan Alexander, Micaela Parker, Aneta Piekut & Colin Rundel (2022) Journal of Statistics and Data Science Education, 30:3, 207-208, DOI: 10.1080/26939169.2022.2141001
• Frank E. Harrell’s A Litany of Problems with p-values blog post originally written in 2017, with more recent updates.
• Jeffrey Leek and Roger Peng P-values are just the tip of the iceberg
• Jeffrey D Blume, Lucy D’Agostino McGowan, William D. Dupont, Robert A Greevy Second-generation p values: Improved rigor, reproducibility and transparency in statistical analyses
• Andrew Gelman and John Carlin Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors
• Scientists rise up against statistical significance in Nature 2019-03-20
• It’s time to talk about ditching statistical significance also in Nature 2019-03-19.
• Briggs, William M., 2019. Everything Wrong with P-Values Under One Roof. In Beyond Traditional Probabilistic Methods in Economics, V Kreinovich, NN Thach, ND Trung, DV Thanh (eds.), pp 22–44. DOI 978-3-030-04200-4_2
• the “PROBABLE CAUSE” graphic reprinted in this Nature piece by Regina Nuzzo, originally from T. Sellke et al. in The American Statistician, 2001.
• and several great pieces by Christie Aschwanden archived from the late, lamented 538 website:
  • “Not Even Scientists Can Easily Explain P-Values”, and
  • “Statisticians Found One Thing They Can Agree On: It’s Time To Stop Misusing P-values”, and
  • “Science Isn’t Broken” with graphics by Ritchie King.
• You may also be interested in this piece at pbs.org about a NOVA program entitled “Rethinking Science’s Magic Number”.
• Here is a nice Editorial by Jeff Witmer (from Oberlin College) at the Journal of Statistics Education which is a position paper on the use of p-values, statistical inference, terminology, and related ideas. I got a lot out of it, and there’s a lot here related to our discussion.
• EJ Daza, Ditch “Statistical Significance” — But Keep Statistical Evidence. How? A statistician shares a writing sample, 2021.
• Why p values are like puppies is a 3:29 YouTube Video by Cassie Kozyrkov, MS, Chief Decision Scientist, Google Inc. It explains how to understand and interpret p values in an intuitive way using an example based on puppies.

I have given several talks on “Rethinking Statistical Significance” over the years. The Github repository (90 minutes at MetroHealth Medical Center and the Center for Health Care Research and Policy, with an audio recording) is at https://github.com/THOMASELOVE/rethink, if you’re a glutton for punishment.