Critical appraisal
Critical appraisal
BRINING IT ALL TOGETHER
To be able to:
1. Interpret the results of epidemiological papers considering the
potential for:
• Selection bias
• Measurement bias / misclassification
• Confounding
• Chance
2. Understand the concepts of internal and external validity
(generalisability)
Critical appraisal
RESEARCH QUESTION/STUDY DESIGN
• What is the research question?
• Study factor/s
• Outcome factor/s
• Is the study design the ‘best’ way to answer the research question?
• Explorative/Causal
• Feasibility – duration, costs
• Ethics
Critical appraisal
4
VALIDITY
• Has everything possible been done to ascertain a ‘true’ outcome?
• Appropriate selection of participants
• Correct measurement
• Minimised impact of confounding
• Minimise risk of ‘chance’ finding
FACULTY OF MEDICINE AND HEALTH SCIENCES
Critical appraisal
CHECK LIST
For all studies
• What is the study question?
• What are the study factors and outcome factors?
• Who are the study participants?
• Are they likely to be representative of the population of interest?
For each item
• Was the item addressed (yes/partially/no/unclear)
• How may it have affected the study’s validity
• Strengths and weaknesses
CAMBRIDGE Medicine ESSENTIAL Epidemiology
Issues to consider when reading epidemiological papers
Insert Figure 9.1 here
Critical appraisal
RCT
Selection bias
• Were participants appropriately randomised?
• Method of randomisation
• Demonstration of balance at baseline
• Was every one accounted for (loss to follow up)?
• Was loss balanced between arms?
• Were those lost similar to the rest of the cohort?
8
FACULTY OF MEDICINE AND HEALTH SCIENCES 9
Effect of Fresh
Red Blood Cell
Transfusions on
Clinical Outcomes
in Premature,
Very Low-BirthWeight InfantsThe
ARIPI Randomized
Trial
JAMA. 2012;308(14):
1443-1451
Critical appraisal
RCT
Measurement error (outcome factor)
• Who was blind to treatment allocation?
• Participants, health workers, study personnel
• Possibility of becoming unblinded
• Were there systematic differences in care?
• Was there any error in assessment of outcome?
• Was it systematic?
• Was it differential?
Critical appraisal
RCT
Confounding
• Is there any evidence of potential confounding?
• Balance in measured baseline characteristics
Critical appraisal
COHORT STUDIES
Selection bias
• Were the exposed and unexposed similar at baseline?
• Was exposure prevalent or incident?
• Was everyone free of the outcome at entry?
• Was every one accounted for (loss to follow up)?
• Was loss balanced between exposed and unexposed?
• Were those lost similar to the rest of the cohort?
Critical appraisal
COHORT
Measurement error (study factor and outcome factor)
• Were outcome assessors blind to exposure?
• Were there systematic differences in care?
• Was there any error in assessment of study factor or outcome factor?
• Was it systematic?
• Was it differential?
Critical appraisal
COHORT
Confounding
• Was there any evidence of potential confounding?
• Were all potential confounders measured?
• Were they measured accurately and without bias?
• Were important risk factors equally distributed among exposed and
unexposed?
• Was confounding managed by design or in analysis?
Statins and atrial fibrillation
• Prognostic cohort study of 449 patients with coronary artery disease
• Rate of arterial fibrillation (AF) lower among people using statins
• Unadjusted RR = 0.5 (95%CI 0.3-0.8)
• Adjusted OR = 0.4 (95%CI 0.2-0.8)
• Do statins (cholesterol-lowering drugs) lower risk of AF?
• Adjusting for confounding strengthened the association
• BUT doctors prescribe statins to people with CAD based on their judgement of
patient risks / prognosis
• May be confounding by indication
Young-Xu et al. (2003). Am J Cardiol, 92(12): 1379-83
EXAMPLE: CONFOUNDING (BY INDICATION)
Critical appraisal
CASE CONTROL
Selection bias
• Did cases and controls arise from the same population?
• Is it a population based study?
• Were only incident cases selected?
• Was the response rate in cases similar to controls?
FACULTY OF MEDICINE AND HEALTH SCIENCES 17
Risk Factors for Menstrual Toxic Shock Syndrome: Results of a Multistate CaseControl Study
REVIEWS OF INFECTIOUS DISEASES * VOL. 11, SUPPLEMENT 1 * JANUARY-FEBRUARY 1989
Smoking & oesophageal cancer
• Case-control study
• 70% of eligible cases took part
• 49% of contacted controls took part
• Smoking rates were higher among cases than controls
• OR = 2.7 (95%CI 1.9-3.9) for current vs. never smoking
• Does smoking cause oesophageal cancer?
• Might low response rates have affected the results?
• Sensitivity analysis using smoking rates from a national
survey with a much higher response rate
• OR = 2.4 (95%CI 1.7-3.4)
• Were the low response rates a problem?
Pandeya et al. (2009). ANZJPH, 33(4): 312-19

Critical appraisal
CASE CONTROL
Measurement error (study factor and outcome factor)
• Was there any error in assessment of study factor or outcome factor?
• Was it systematic?
• Was it differential?
Differential error in the outcome factor is rare in case control studies
Assessment of study factor can be particularly difficult
Obesity and asthma
• Case-control study in Mexico
• Body-mass index from self-reported height & weight
• BMI was slightly higher among cases than controls
• Women OR
adj = 1.7 (95%CI 1.1-2.7)
• Men OR
adj = 1.3 (95%CI 0.6-2.9)
• Does obesity really cause asthma?
• Using measured height & weight: people said they were taller & lighter than they
were (esp. men and cases > controls)
• How might this affect the results?
• Error is differential – it would weaken associations

EXAMPLE: MEASUREMENT ERROR
Critical appraisal
CASE CONTROL
Confounding
• Was there any evidence of potential confounding?
• Were all potential confounders measured?
• Were they measured accurately and without bias?
• Were important risk factors equally distributed among exposed and
unexposed?
• Was confounding managed by design or in analysis?
o Matching is a common design strategy to manage confounding in case
control studies
Critical appraisal
CROSS SECTIONAL
Is the study descriptive or analytic?
Selection bias
• Were the study participants a random selection of the population of interest?
• What was the response rate for participation?
Critical appraisal
CROSS SECTIONAL
Measurement error (study factor and outcome factor)
• Was there any error in assessment of study factor or outcome factor?
• Was it systematic?
• Was it differential?
Critical appraisal
CROSS SECTIONAL
Confounding
• Was there any evidence of potential confounding?
• Were all potential confounders measured?
• Were they measured accurately and without bias?
• Were important risk factors equally distributed among exposed and
unexposed?
• Was confounding managed by design or in analysis?
FACULTY OF MEDICINE AND HEALTH SCIENCES 25
Critical appraisal
CHECK LIST
For all studies
• How strong is the strength of association?
• Consider the range of the confidence interval
• (Could the findings have occurred by chance?)
• Is residual confounding likely to be an issue?
• What is the likely impact of error?
• Are the conclusions valid?
• Internal
• External- generalisability
Do the strengths over come the weaknesses?
Statistical vs. biological significance
Statistically and clinically
significant
May be clinically
significant but NOT
statistically significant