Sample size can seem deceptively simple. It's just a number! Yet, that number has a major impact on what our statistical analysis can and can't tell us about the results of our study. In this unit, we explore the impact sample size can have, learn about various calculation methods and their pitfalls, and explore how reading and reporting about sample size can help us to improve the rigor of our work more broadly. This unit includes advice that applies across research contexts, from small to large n and exploratory to confirmatory work.
The Biased Brain
Explore how cognitive biases shape our perception and decision-making, and why even trained researchers are not immune.
Framing Effects & Confirmation Bias
Discover how the way a question is framed can steer conclusions before any data is collected.
Spotting Bias in the Wild
Apply what you've learned by identifying real examples of confirmation bias in published studies and everyday research scenarios.
Masking & Blinding Techniques
Learn practical strategies like masking and blinding that researchers use to reduce the influence of confirmation bias.
Building a Rigorous Research Practice
Bring everything together and develop a personal checklist for designing bias-aware, reproducible research.
Lesson title
Lesson description
Lesson title
Lesson description
Lesson title
Lesson description
Lesson title
Lesson description
References:
Sample Size
Adolfi, F., van de Braak, L., & Woensdregt, M. (2024). From Empirical Problem-Solving to Theoretical Problem-Finding Perspectives on the Cognitive Sciences. Computational Brain & Behavior, 7(4), 572–587. https://doi.org/10.1007/s42113-024-00216-6
Albers, C. (2019). The problem with unadjusted multiple and sequential statistical testing. Nature Communications, 10(1), 1921. https://doi.org/10.1038/s41467-019-09941-0
Bazerman, M. H., & Samuelson, W. F. (1983). I won the auction but don't want the prize. Journal of Conflict Resolution, 27(4), 618–634.
Brownell, S. E., Wenderoth, M. P., Theobald, R., Okoroafor, N., Koval, M., Freeman, S., Walcher-Chevillet, C. L., & Crowe, A. J. (2014). How Students Think about Experimental Design: Novel Conceptions Revealed by in-Class Activities. BioScience, 64(2), 125–137. https://doi.org/10.1093/biosci/bit016
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376. https://doi.org/10.1038/nrn3475
Cano-Corres, R., Sánchez-Álvarez, J., & Fuentes-Arderiu, X. (2012). The Effect Size: Beyond Statistical Significance. EJIFCC, 23(1), 19–23.
Cohen, J. (1977). Statistical Power Analysis for the Behavioral Sciences. Elsevier Science & Technology. https://ebookcentral-proquest-com.proxy.library.upenn.edu/lib/upenn-ebooks/detail.action?docID=1882849
Confirmatory versus Exploratory Research. (2024, April 5). Irving Institute for Clinical and Translational Research. https://www.irvinginstitute.columbia.edu/confirmatory-versus-exploratory-research
Danziger, M., Collazo, A., Dirnagl, U., & Toelch, U. (2022). Balancing sensitivity and specificity in preclinical research (p. 2022.01.17.476585). bioRxiv. https://doi.org/10.1101/2022.01.17.476585
Devezer, B., & Buzbas, E. O. (2023). Rigorous exploration in a model-centric science via epistemic iteration. Journal of Applied Research in Memory and Cognition, 12(2), 189–194. https://doi.org/10.1037/mac0000121
Devezer, B., Navarro, D. J., Vandekerckhove, J., & Ozge Buzbas, E. (n.d.). The case for formal methodology in scientific reform. Royal Society Open Science, 8(3), 200805. https://doi.org/10.1098/rsos.200805
Experimental design Landing page | NC3Rs EDA. (n.d.). Retrieved March 20, 2025, from https://eda.nc3rs.org.uk/index.php/experimental-design
Gigerenzer, G. (2018). Statistical Rituals: The Replication Delusion and How We Got There. Advances in Methods and Practices in Psychological Science, 1(2), 198–218. https://doi.org/10.1177/2515245918771329
Heinsberg, L. W., & Weeks, D. E. (2022). Post hoc Power is Not Informative. Genetic Epidemiology, 46(7), 390–394. https://doi.org/10.1002/gepi.22464
Hoenig, J. M., & Heisey, D. M. (2001). The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis. The American Statistician, 55(1), 19–24. https://doi.org/10.1198/000313001300339897
Ioannidis, J. P. A. (2005). Why Most Published Research Findings Are False. PLOS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124
Jaeger, R. G., & Halliday, T. R. (1998). On Confirmatory versus Exploratory Research. Herpetologica, 54, S64–S66.
Krzywinski, M., & Altman, N. (2013). Power and sample size. Nature Methods, 10(12), 1139–1140. https://doi.org/10.1038/nmeth.2738
May, J., Ketchum, J., & Looney, S. (2016). Simplified Tools for Sample Size Determination for Correlation Coefficient Inference. Proceedings of the Joint Statistical Meetings. https://ww2.amstat.org/meetings/proceedings/2016/data/presinfo231324.cfm
Negida, A. (2020). Sample Size Calculation Guide - Part 7: How to Calculate the Sample Size Based on a Correlation. Advanced Journal of Emergency Medicine, 4(2), e34. https://doi.org/10.22114/ajem.v0i0.344
Reinagel, P. (2023). Is N-Hacking Ever OK? The consequences of collecting more data in pursuit of statistical significance. PLOS Biology, 21(11), e3002345. https://doi.org/10.1371/journal.pbio.3002345
Savitz, D. A., Wise, L. A., Bond, J. C., Hatch, E. E., Ncube, C. N., Wesselink, A. K., Willis, M. D., Yland, J. J., & Rothman, K. J. (2024). Responding to Reviewers and Editors About Statistical Significance Testing. Annals of Internal Medicine, 177(3), 385–386. https://doi.org/10.7326/M23-2430
Schwab, S., & Held, L. (2020). Different worlds Confirmatory versus exploratory research. Significance, 17(2), 8–9. https://doi.org/10.1111/1740-9713.01369
Scott, S., Kranz, J. E., Cole, J., Lincecum, J. M., Thompson, K., Kelly, N., Bostrom, A., Theodoss, J., Al‐Nakhala, B. M., Vieira, F. G., Ramasubbu, J., & Heywood, J. A. (2008). Design, power, and interpretation of studies in the standard murine model of ALS. Amyotrophic Lateral Sclerosis, 9(1), 4–15. https://doi.org/10.1080/17482960701856300
Serdar, C. C., Cihan, M., Yücel, D., & Serdar, M. A. (2021). Sample size, power and effect size revisited: Simplified and practical approaches in pre-clinical, clinical and laboratory studies. Biochemia Medica, 31(1), 010502. https://doi.org/10.11613/BM.2021.010502
Sert, N. P. du, Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., Garner, P., Holgate, S. T., Howells, D. W., Hurst, V., Karp, N. A., Lazic, S. E., Lidster, K., MacCallum, C. J., Macleod, M., … Würbel, H. (2020). Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0. PLOS Biology, 18(7), e3000411. https://doi.org/10.1371/journal.pbio.3000411
Sullivan, G. M., & Feinn, R. (2012). Using Effect Size—Or Why the P Value Is Not Enough. Journal of Graduate Medical Education, 4(3), 279–282. https://doi.org/10.4300/JGME-D-12-00156.1
Szucs, D. (2016). A Tutorial on Hunting Statistical Significance by Chasing N. Frontiers in Psychology, 7. https://doi.org/10.3389/fpsyg.2016.01444
Thibault, R. T., Zavalis, E. A., Malicki, M., & Pedder, H. (2024). An evaluation of reproducibility and errors in published sample size calculations performed using G*Power (p. 2024.07.15.24310458). medRxiv. https://doi.org/10.1101/2024.07.15.24310458
Waldron, S., & Allen, C. (2022). Not all pre-registrations are equal. Neuropsychopharmacology, 47(13), 2181–2183. https://doi.org/10.1038/s41386-022-01418-x
Yarkoni, T. (2022). The generalizability crisis. Behavioral and Brain Sciences, 45, e1. https://doi.org/10.1017/S0140525X20001685
Instructor guide:
Sample Size
Overview and Introduction
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Block quote
Ordered list
- Item 1
- Item 2
- Item 3
Unordered list
- Item A
- Item B
- Item C
Bold text
Emphasis
Superscript
Subscript
This is some text inside of a div block.
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Block quote
Ordered list
- Item 1
- Item 2
- Item 3
Unordered list
- Item A
- Item B
- Item C
Bold text
Emphasis
Superscript
Subscript
This is some text inside of a div block.
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Block quote
Ordered list
- Item 1
- Item 2
- Item 3
Unordered list
- Item A
- Item B
- Item C
Bold text
Emphasis
Superscript
Subscript
This is some text inside of a div block.
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Block quote
Ordered list
- Item 1
- Item 2
- Item 3
Unordered list
- Item A
- Item B
- Item C
Bold text
Emphasis
Superscript
Subscript
This is some text inside of a div block.
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Block quote
Ordered list
- Item 1
- Item 2
- Item 3
Unordered list
- Item A
- Item B
- Item C
Bold text
Emphasis
Superscript
Subscript
This is some text inside of a div block.
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Block quote
Ordered list
- Item 1
- Item 2
- Item 3
Unordered list
- Item A
- Item B
- Item C
Bold text
Emphasis
Superscript
Subscript
This is some text inside of a div block.
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Block quote
Ordered list
- Item 1
- Item 2
- Item 3
Unordered list
- Item A
- Item B
- Item C
Bold text
Emphasis
Superscript
Subscript
This is some text inside of a div block.
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Block quote
Ordered list
- Item 1
- Item 2
- Item 3
Unordered list
- Item A
- Item B
- Item C
Bold text
Emphasis
Superscript
Subscript
This is some text inside of a div block.
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Block quote
Ordered list
- Item 1
- Item 2
- Item 3
Unordered list
- Item A
- Item B
- Item C
Bold text
Emphasis
Superscript
Subscript
This is some text inside of a div block.
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Block quote
Ordered list
- Item 1
- Item 2
- Item 3
Unordered list
- Item A
- Item B
- Item C
Bold text
Emphasis
Superscript
Subscript