Material for Statistics Workshops

James Hanley
Dept of Epidemiology, Biostatistics and Occupational Health
Faculty of Medicine

http://www.medicine.mcgill.ca/epidemiology/hanley

Computational and Data Systems Initiative: Workshop
Faculty of Science
2023.01.18

MInimizing Statistical Non-Reproducibilty

Content: to come

Teaching objectives: to come

Pre-requisites: to come

Resources: to come

Computational and Data Systems Initiative: Workshop
Faculty of Science
2022.01.10

Pitfalls in Statistics

Content

In this workshop you will see examples of statistical/scientific errors of various types. The session will begin with ones produced by chance variation, with probabilities we think we can control. But more important are ones produced by faulty reasoning, overuse of our computing power and over-confidence in what we can learn from big data.

Teaching objectives

At the end of this workshop, you will be able to better appreciate

what factors produce 'standard' (statistical) errors

how modern 'data-analysis' practices can produce even more serious errors

that the probability of such errors can be larger with larger amounts of data

Pre-requisites

None

Resources

P-values: what they are and what they aren't [see under 'Introduction to Statistics' below]

Common fallacies and difficulties:
3 articles 1 2 3 from the 1937 Lancet series that served as the basis for
Austin Bradford Hill's classic textbook of medical statistics
The 6th edition of the book can be found here here

Coincidences: 1 2

Statistical Behaviors: Personal and Computer-Aided Observations

Immortal Time' Blunders: history, identification, severity

Giving science the finger - is the second-to-fourth digit ratio (2D:4D) a biomarker of good luck?

SLIDES 2022

Biostatistics Workshop
Department of Anatomy and Cell Biology.
2017.10.06 2018.09.28 2018.10.05

Introduction to Statistics

Introduction to Statistics: Example
Definition of a (statistical) parameter
Factors that influence how precisely we can measure a parameter
Confidence intervals and margins of error

Hypothesis tests:

Definition, specifications, performance
Link with medical diagnostic tests
p-values: what they are and what they aren't

What a nerdy debate about p-values shows about science - and how to fix it

(The Economist 2013) Problems with scientific research: How science goes wrong
Not Even Scientists Can Easily Explain P-values

The Texas sharpshooter

ONLINE RESOURCES

Link to JH's Introductory Statistics course, 2001
Link to Sahir Bhatnagar's Introductory Statistics course, 2018
Link to Against All Odds: Inside Statistics (online video instructional series on statistics)

JH's 2018 Notes re 1-sample inference for a mean:

Material for Bionano Workshop:
Biostatistics
November 27, 2013

Measuring Statistical Parameters; Precision/Resolution

Bottom Line

Measurement error Therapeutic Trials in Advanced Cancer
Biological variability
Role of sqrt(n)

'Testing' vs 'Estimation'

phychophysics ; linguistics adjuvant tx for primary breast cancer [p8/9]
breast-feeding inhibitors; starch blockers
If nothing goes wrong, is everything all right? Interpreting zero numerators.
(Quand rien ne va mal, peut-on dire que tout va bien?)
If we're so different, why do we keep overlapping? When 1 plus 1 doesn't make 2.

Sharper and Fairer

MMR and autism Diagram 1 Diagram 2
Longevity cost of sexual activity

Breast Milk and Subsequent IQ in Children Born Preterm
Exposure to Scientific Theories: Women's Math Performance
Trials of vaccines & other prophylactics against: polio hiv hpv hsv hpv

Appropriate uses of multivariate analysis

Capitalization on chance (?)

Jumping to Coincidences e.g.'s

Assisted Reproduction & Cancer

2012: for Residents in Internal Medicine
Material for Academic 1/2 day:
Statistics in the Medical Literature and in Medical Practice
October 18, 2012

'Averages'

Why do statisticians commonly limit their inquiries to averages?
Salaries of professional baseball players 1994 Raw Data(.xls)
"Average" height of basketball team
Number of authors on medical articles (Fletcher NEJM 1979)

Year No. articles No. au's No. Subjects
      examined     mean SD    Median

1946     151        2.0 1.4      25
1956     149        2.3 1.6      36
1966     157        2.8 1.2      16
1976     155        4.9 7.3      30

A new sex survey of American men (1993)
The Myth, the Math, the Sex - NY Times
The Median, the Math and the Sex - NY Times
Statistical methods Radiology readers need to understand
Visualizing the Median as the Minimum-Deviation Location
Fig 1 [colour]; java applet

There are Probabilities and Probabilities

Different Probabilities

The Place of Statistical Methods in Radiology: & the Bigger Picture (Int. Med!)

article

'P-values'

Notes from JH's Intro Biostatistics course, 2001

Notes: Introduction to Statistical Inference. 2nd year Med Students, 2002.

What the P-Value IS and IS NOT

L-Phenylalanine Mustard (L-PAM) in Management of Primary Breast Cancer
(Fisher's Exact Test)

The truth about doctors' handwriting: a prospective study

Are all P values created equal ?

Sifting the evidence -- what's wrong with significance tests?

BMJ Editor's choice: Some gentle statistics

Confidence Intervals / Statistical Power / Clinical Important Differences

Notes from JH's Intro Biostatistics course, 2001

Do free infant formula samples reduce duration of breastfeeding?

Do starch blockers really block calorie absorption?

Standard deviation, standard error. Which 'standard' should we use?

Rather than compare 2 CIs, look at the single CI for the difference:
If we're so different, why do we keep overlapping? When 1 plus 1 doesn't make 2.

NNT

Julien M and Hanley JA. Profile-specific survival estimates: Making reports of clinical trials more patient-relevant Clinical Trials 2008; 5: 107-115.

Hanley J Presentation on profile-specific survival estimates

'Hazard' rates and 'Hazard' Ratios

Weekend versus Weekday Admission and Mortality from Myocardial Infarction

Ticagrelor vs. clopidogrel in patients with a planned invasive strategy for acute coronary syndromes (PLATO): a randomised double-blind study

Rosuvastatin to Prevent Vascular Events in Men and Women with Elevated C-Reactive Protein

Loneliness of the Long-distance Trialists [Screening for Pr Ca, AAA]

Other examples of 'Instant' vs. delayed reductions

Summary

1 2 3

2013
Material for Academic 1/2 day:
Regression and Multivariable Analysis
September 26, 2013

DIA-gnosis

Pneumonia Pubmed Article
.
Acute bacterial rhinosinusitis Pubmed Article
.
Acute coronary heart disease Pubmed Article
.
Deep Venous Thrombosis Risk Stratification Website
.
Prostate Cancer: presence; extension; etc Partin Tables
.
Catheter Lengths
. Pretracheal tissue thickness Pubmed Article
.

PRO-gnosis

Heart Disease Risk Calculators UpToDate
.
Breast Cancer Risk Assessment Tool Website
.
Lung cancer Various Risk Models
.
Profile-specific Info from Trials :

Profile-specific survival estimates: Making reports of clinical trials more patient-relevant Julien M and Hanley JA. Clinical Trials 2008; 5: 107-115.

Presentation on profile-specific survival estimates -- and concept of 'borrowing strength' from data in 'neighbouring' cells Hanley J
.

ETIO-gnosis

Breast Milk and Subsequent IQ in Children Born Preterm
Making comparison FAIRER (and sharper)
.
Exposure to Scientific Theories Affects Women's Math Performance
Article, Data, Remarks on Confounding
.
Rates of Fatal Crashes in 5 y. with 55 mph and 1 y. with 65 mph limit:
a naive -- and a FAIRER -- Comparison
.
Sexual Activity Reduces Lifespan of Male Fruitflies Article
-- SHARPER (& fairer) Comparisons
.

(Technical)

Fitting Regression Models `by Eye':
Canadian BirthWeights Danish Mortality Rates
.
Short Course on Analysis of Multivariable (one Y, multiple Xs) Data
. Notes
.
Appropriate uses of multivariate analysis.
Article: Hanley JA. Annual Review Public Health. 1983;4:155-80.
.
"Transmuting" women into men: Galton's family data on human stature.
Article Hanley JA. The American Statistician, 58(3) 237-243. August 2004.
poster [ 57MB !!! ] (including first ever 'regression' line involving human data.
.
Reducing Confounding by Standardization / Regression:
Textbook Chapter (Clayton&Hills), and Notes and Diagram by JH
.

Earlier material -- for year 2 medical students

Statistics CoursePack / Lecture / Small Group Assignment in Unit 8b

2010 Coursepack Assignment(+Ans)

2009 Lecture

2008 Lecture

2007 Lecture

Material for Unit 7 and for Unit 8b
INTRODUCTION TO EPIDEMIOLOGY
Fall 2002

Frequently Asked Questions

Other textbooks (besides the recommended Fletcher et al.)

Hennekens, Charles H. Epidemiology in medicine /
WA 950 H515e 1987 [2 Hour Same Day] Health Sciences Reserves

Friedman, Gary D., 1934- Primer of epidemiology /
WA 105 F911p 1994 [Regular Loan] Health Sciences
Health-Sciences( 1/ 0) 1994 and earlier editions

*
Sept 03 lecture [one type of study]    Handout

*
Types of Epidemiologic Studies     material from Rothman and Greenland

*
THE BIGGEST PUBLIC HEALTH EXPERIMENT EVER: The 1954 Field Trial of the Salk Poliomyelitis Vaccine     Book Chapter      polio_cases_US_1930-1956

*
1-Phenylalanine mustard (L-PAM) in the management of primary breast cancer. NEJM 292:117-122 January 16, 1975       Link

*
Twenty-five-year follow-up of a randomized trial comparing radical mastectomy, total mastectomy, and total mastectomy followed by irradiation. Fisher B, et al. N Engl J Med. 2002 Aug 22; 347(8):567-75.       Link      Full article

*
Final report on the aspirin component of the ongoing Physicians' Health Study. Steering Committee of the Physicians' Health Study Research Group.N Engl J Med 1989 Jul 20; 321(3): 129 - 35.   Link

*
Randomised trial of prophylactic daily aspirin in British male doctors. Peto R, et al Br Med J 1988 Jan 30;296(6618):313-6      Link

See also: An overview of the British and American aspirin studies. Hennekens CH, Peto R, Hutchison GB, Doll R. N Engl J Med 1988 Apr 7;318(14):923-4

*
Effects of stretching before and after exercising on muscle soreness and risk of injury
     Article by R.D. Herbert and M Gabriel, BMJ, Aug 31, 2002
     Cumulative Incidence Graphs (modified from figure in article)     1     2

*
Sept 04 lecture [another type of study]    Handout

*
John Snow: On the mode of communication of cholera       Excerpt from Hennekens and Buring

*
Dr. John Snow (1813-1858), a legendary figure in the history of public health, epidemiology and anesthesiology     website

*
Framingham Heart Study

*
Quebec_Asbestos_cohort

*
The Stress and Health Study (also known as the Whitehall II Study)     Link

Nurses Health Study

Outcomes of pregnancy in a national sample of resident physicians

Vasectomy and health. Results from a large cohort study

*
Medical students' compliance with simple administrative tasks and success in final examinations: retrospective cohort study     Article

*
Sept 05 lecture [case-control studies]    Handout

*
Pour battre Patrick Roy, mieux vaut lancer bas       La Presse

*
Women are better pilots     newspaper item

*
PubMed search on case control study quebec

*
The role of woodstoves in the etiology of polyposis     article

*
Pedestrians Fatally Injured by motor vehicles   From Friedman

*
Sept 06 lecture   More on case-control studies

*
Nov 11 lecture [Statistical Inference]     Handout    (updated Nov 10)

*
If nothing goes wrong is everything alright? interpreting zero numerators       article

*
A POPULATION-BASED STUDY OF MEASLES,MUMPS,AND RUBELLA VACCINATION AND AUTISM       article       edtorial

How could one get a crude Rate Ratio (RR) of 1.45?

Why can the crude Rate Ratio (RR) be 1.45 if RR=1 at all ages and in all years?

*
Explaining risks: turning numerical data into meaningful pictures       article

*
Nov 12 Small Group Session

*
Material for question 3 ( on sampling / confidence intervals )

      BirthWeights_AdultHeights.pdf       RandomDigits.pdf

*
Nov 21 lecture       Handout

*
A CONTROLLED TRIAL OF A HUMAN PAPILLOMAVIRUS TYPE 16 VACCINE       article

Updated: Sept 28, 2018

Computational and Data Systems Initiative: Workshop Faculty of Science 2023.01.18
MInimizing Statistical Non-Reproducibilty
	Content: to come Teaching objectives: to come Pre-requisites: to come Resources: to come

Computational and Data Systems Initiative: Workshop Faculty of Science 2022.01.10
Pitfalls in Statistics
	Content In this workshop you will see examples of statistical/scientific errors of various types. The session will begin with ones produced by chance variation, with probabilities we think we can control. But more important are ones produced by faulty reasoning, overuse of our computing power and over-confidence in what we can learn from big data. Teaching objectives At the end of this workshop, you will be able to better appreciate what factors produce 'standard' (statistical) errors how modern 'data-analysis' practices can produce even more serious errors that the probability of such errors can be larger with larger amounts of data Pre-requisites None Resources P-values: what they are and what they aren't [see under 'Introduction to Statistics' below] Common fallacies and difficulties: 3 articles 1 2 3 from the 1937 Lancet series that served as the basis for Austin Bradford Hill's classic textbook of medical statistics The 6th edition of the book can be found here here Coincidences: 1 2 Statistical Behaviors: Personal and Computer-Aided Observations Immortal Time' Blunders: history, identification, severity Giving science the finger - is the second-to-fourth digit ratio (2D:4D) a biomarker of good luck? SLIDES 2022

Biostatistics Workshop Department of Anatomy and Cell Biology. 2017.10.06 2018.09.28 2018.10.05
Introduction to Statistics
	Introduction to Statistics: Example Definition of a (statistical) parameter Factors that influence how precisely we can measure a parameter Confidence intervals and margins of error Hypothesis tests: Definition, specifications, performance Link with medical diagnostic tests p-values: what they are and what they aren't What a nerdy debate about p-values shows about science - and how to fix it (The Economist 2013) Problems with scientific research: How science goes wrong Not Even Scientists Can Easily Explain P-values The Texas sharpshooter ONLINE RESOURCES Link to JH's Introductory Statistics course, 2001 Link to Sahir Bhatnagar's Introductory Statistics course, 2018 Link to Against All Odds: Inside Statistics (online video instructional series on statistics) JH's 2018 Notes re 1-sample inference for a mean:

Material for Bionano Workshop: Biostatistics November 27, 2013
Measuring Statistical Parameters; Precision/Resolution
	Bottom Line Measurement error Therapeutic Trials in Advanced Cancer Biological variability Role of sqrt(n) 'Testing' vs 'Estimation' phychophysics ; linguistics adjuvant tx for primary breast cancer [p8/9] breast-feeding inhibitors; starch blockers If nothing goes wrong, is everything all right? Interpreting zero numerators. (Quand rien ne va mal, peut-on dire que tout va bien?) If we're so different, why do we keep overlapping? When 1 plus 1 doesn't make 2. Sharper and Fairer MMR and autism Diagram 1 Diagram 2 Longevity cost of sexual activity Breast Milk and Subsequent IQ in Children Born Preterm Exposure to Scientific Theories: Women's Math Performance Trials of vaccines & other prophylactics against: polio hiv hpv hsv hpv Appropriate uses of multivariate analysis Capitalization on chance (?) Jumping to Coincidences e.g.'s Assisted Reproduction & Cancer

2012: for Residents in Internal Medicine Material for Academic 1/2 day: Statistics in the Medical Literature and in Medical Practice October 18, 2012
'Averages'
	Why do statisticians commonly limit their inquiries to averages? Salaries of professional baseball players 1994 Raw Data(.xls) "Average" height of basketball team Number of authors on medical articles (Fletcher NEJM 1979) Year No. articles No. au's No. Subjects examined mean SD Median 1946 151 2.0 1.4 25 1956 149 2.3 1.6 36 1966 157 2.8 1.2 16 1976 155 4.9 7.3 30 A new sex survey of American men (1993) The Myth, the Math, the Sex - NY Times The Median, the Math and the Sex - NY Times Statistical methods Radiology readers need to understand Visualizing the Median as the Minimum-Deviation Location Fig 1 [colour]; java applet
There are Probabilities and Probabilities
	Different Probabilities
The Place of Statistical Methods in Radiology: & the Bigger Picture (Int. Med!)
	article
'P-values'
	Notes from JH's Intro Biostatistics course, 2001 Notes: Introduction to Statistical Inference. 2nd year Med Students, 2002. What the P-Value IS and IS NOT L-Phenylalanine Mustard (L-PAM) in Management of Primary Breast Cancer (Fisher's Exact Test) The truth about doctors' handwriting: a prospective study Are all P values created equal ? Sifting the evidence -- what's wrong with significance tests? BMJ Editor's choice: Some gentle statistics
Confidence Intervals / Statistical Power / Clinical Important Differences
	Notes from JH's Intro Biostatistics course, 2001 Do free infant formula samples reduce duration of breastfeeding? Do starch blockers really block calorie absorption? Standard deviation, standard error. Which 'standard' should we use? Rather than compare 2 CIs, look at the single CI for the difference: If we're so different, why do we keep overlapping? When 1 plus 1 doesn't make 2.
NNT
	Julien M and Hanley JA. Profile-specific survival estimates: Making reports of clinical trials more patient-relevant Clinical Trials 2008; 5: 107-115. Hanley J Presentation on profile-specific survival estimates
'Hazard' rates and 'Hazard' Ratios
	Weekend versus Weekday Admission and Mortality from Myocardial Infarction Ticagrelor vs. clopidogrel in patients with a planned invasive strategy for acute coronary syndromes (PLATO): a randomised double-blind study Rosuvastatin to Prevent Vascular Events in Men and Women with Elevated C-Reactive Protein Loneliness of the Long-distance Trialists [Screening for Pr Ca, AAA] Other examples of 'Instant' vs. delayed reductions
Summary
	1 2 3

2013 Material for Academic 1/2 day: Regression and Multivariable Analysis September 26, 2013

DIA-gnosis
	Pneumonia Pubmed Article . Acute bacterial rhinosinusitis Pubmed Article . Acute coronary heart disease Pubmed Article . Deep Venous Thrombosis Risk Stratification Website . Prostate Cancer: presence; extension; etc Partin Tables . Catheter Lengths . Pretracheal tissue thickness Pubmed Article .
PRO-gnosis
	Heart Disease Risk Calculators UpToDate . Breast Cancer Risk Assessment Tool Website . Lung cancer Various Risk Models . Profile-specific Info from Trials : Profile-specific survival estimates: Making reports of clinical trials more patient-relevant Julien M and Hanley JA. Clinical Trials 2008; 5: 107-115. Presentation on profile-specific survival estimates -- and concept of 'borrowing strength' from data in 'neighbouring' cells Hanley J .
ETIO-gnosis
	Breast Milk and Subsequent IQ in Children Born Preterm Making comparison FAIRER (and sharper) . Exposure to Scientific Theories Affects Women's Math Performance Article, Data, Remarks on Confounding . Rates of Fatal Crashes in 5 y. with 55 mph and 1 y. with 65 mph limit: a naive -- and a FAIRER -- Comparison . Sexual Activity Reduces Lifespan of Male Fruitflies Article -- SHARPER (& fairer) Comparisons .
(Technical)
	Fitting Regression Models `by Eye': Canadian BirthWeights Danish Mortality Rates . Short Course on Analysis of Multivariable (one Y, multiple Xs) Data . Notes . Appropriate uses of multivariate analysis. Article: Hanley JA. Annual Review Public Health. 1983;4:155-80. . "Transmuting" women into men: Galton's family data on human stature. Article Hanley JA. The American Statistician, 58(3) 237-243. August 2004. poster [ 57MB !!! ] (including first ever 'regression' line involving human data. . Reducing Confounding by Standardization / Regression: Textbook Chapter (Clayton&Hills), and Notes and Diagram by JH .

Earlier material -- for year 2 medical students
Statistics CoursePack / Lecture / Small Group Assignment in Unit 8b
	2010 Coursepack Assignment(+Ans)
	2009 Lecture
	2008 Lecture
	2007 Lecture
Material for Unit 7 and for Unit 8b INTRODUCTION TO EPIDEMIOLOGY Fall 2002
	Frequently Asked Questions Other textbooks (besides the recommended Fletcher et al.) Hennekens, Charles H. Epidemiology in medicine / WA 950 H515e 1987 [2 Hour Same Day] Health Sciences Reserves Friedman, Gary D., 1934- Primer of epidemiology / WA 105 F911p 1994 [Regular Loan] Health Sciences Health-Sciences( 1/ 0) 1994 and earlier editions

*	Sept 03 lecture [one type of study] Handout
*	Types of Epidemiologic Studies material from Rothman and Greenland
*	THE BIGGEST PUBLIC HEALTH EXPERIMENT EVER: The 1954 Field Trial of the Salk Poliomyelitis Vaccine Book Chapter polio_cases_US_1930-1956
*	1-Phenylalanine mustard (L-PAM) in the management of primary breast cancer. NEJM 292:117-122 January 16, 1975 Link
*	Twenty-five-year follow-up of a randomized trial comparing radical mastectomy, total mastectomy, and total mastectomy followed by irradiation. Fisher B, et al. N Engl J Med. 2002 Aug 22; 347(8):567-75. Link Full article
*	Final report on the aspirin component of the ongoing Physicians' Health Study. Steering Committee of the Physicians' Health Study Research Group.N Engl J Med 1989 Jul 20; 321(3): 129 - 35. Link
*	Randomised trial of prophylactic daily aspirin in British male doctors. Peto R, et al Br Med J 1988 Jan 30;296(6618):313-6 Link See also: An overview of the British and American aspirin studies. Hennekens CH, Peto R, Hutchison GB, Doll R. N Engl J Med 1988 Apr 7;318(14):923-4
*	Effects of stretching before and after exercising on muscle soreness and risk of injury Article by R.D. Herbert and M Gabriel, BMJ, Aug 31, 2002 Cumulative Incidence Graphs (modified from figure in article) 1 2

*	Sept 04 lecture [another type of study] Handout
*	John Snow: On the mode of communication of cholera Excerpt from Hennekens and Buring
*	Dr. John Snow (1813-1858), a legendary figure in the history of public health, epidemiology and anesthesiology website
*	Framingham Heart Study
*	Quebec_Asbestos_cohort
*	The Stress and Health Study (also known as the Whitehall II Study) Link
	Nurses Health Study
	Outcomes of pregnancy in a national sample of resident physicians
	Vasectomy and health. Results from a large cohort study
*	Medical students' compliance with simple administrative tasks and success in final examinations: retrospective cohort study Article

*	Sept 05 lecture [case-control studies] Handout
*	Pour battre Patrick Roy, mieux vaut lancer bas La Presse
*	Women are better pilots newspaper item
*	PubMed search on case control study quebec
*	The role of woodstoves in the etiology of polyposis article
*	Pedestrians Fatally Injured by motor vehicles From Friedman

*	Sept 06 lecture More on case-control studies

*	Nov 11 lecture [Statistical Inference] Handout (updated Nov 10)
*	If nothing goes wrong is everything alright? interpreting zero numerators article
*	A POPULATION-BASED STUDY OF MEASLES,MUMPS,AND RUBELLA VACCINATION AND AUTISM article edtorial
	How could one get a crude Rate Ratio (RR) of 1.45?
	Why can the crude Rate Ratio (RR) be 1.45 if RR=1 at all ages and in all years?
*	Explaining risks: turning numerical data into meaningful pictures article

*	Nov 12 Small Group Session
*	Material for question 3 ( on sampling / confidence intervals ) BirthWeights_AdultHeights.pdf RandomDigits.pdf

*	Nov 21 lecture Handout
*	A CONTROLLED TRIAL OF A HUMAN PAPILLOMAVIRUS TYPE 16 VACCINE article