Interpretation of Medical Literature — Master Study Guide

1 · Introduction, Bias & Validity

Megan B. Finck, MMS, PA-C · Session 1

What epidemiology is

The study of the distribution and determinants of health-related states among specified populations, and the application of that study to the control of health problems. The second half of the definition is the part people forget: it is an applied discipline, not a descriptive one.

Purposes in public health practice: discover agent, host and environmental factors affecting health; determine the relative importance of causes of illness, disability and death; identify the population segments at greatest risk; evaluate whether health programs and services actually improve population health.

Contributions to medicine: investigating new disease transmission · determining preventable causes · determining natural history · studying the biological spectrum of disease · evaluating community interventions · setting disease control priorities · improving health services research · providing expert testimony.

IO 1 — Epidemiologic data sources

Clinical epidemiology and evidence-based medicine

Clinical epidemiology makes predictions about individual patients from events in groups of similar patients. Strong scientific methods are what make those predictions accurate.

Evidence-based medicine applies clinical epidemiology to patient care: using clinical judgment to assess available research for validity and applicability, then applying it to decisions.

The alternatives to evidence (know these by name)

The five As

Ask (define the clinical question) → Acquire (systematic retrieval) → Appraise (validity and applicability) → Apply (change practice) → Assess (evaluate outcomes on patient care).

IO 4 — Purposes of research

• Exploration — new or understudied topic; satisfy curiosity; test feasibility of a more careful study; develop its methods.
• Description — careful deliberate observation and a description of what is found (disease rates, demographics of those affected).
• Explanation — attempt to explain why.

Primary research uses data the researcher collects personally (questionnaires, surveys, clinical measurements, laboratory results) — e.g. original research articles. Secondary research analyses existing data — meta-analyses, systematic reviews.

Basic vocabulary

• Independent variable — the possible cause.
• Dependent variable — the possible effect.
• Extraneous variable (covariate) — may impact the relationship between the two.
• Population — all people in a defined setting or with defined characteristics. Sample — a subset of them.

IO 5 — The bias taxonomy

Bias is systematic error producing conclusions that differ from the true results. It can enter at any stage: conception and design, data collection, data analysis, data interpretation. Potential for bias does not always mean bias is present — but it should be considered, and it is often detectable when reading the article.

IO 6 — Chance

Random variation producing conclusions that differ from the true results. Statistical analysis can estimate the degree; good design reduces it but never eliminates it; it may coexist with bias.

IO 7 — Internal and external validity

Two questions to ask of any study:

1. Are the conclusions valid for the people in the sample? → internal validity, the degree results are correct for the sample studied.
2. Does the sample represent my patients? → external validity, the degree results are also true in other clinical settings. Also called generalisability, and rarely determined from a single study.

2 · Evidence

Megan B. Finck, MMS, PA-C · Session 2

Origins and definition

Evidence-based medicine originated at McMaster University in the 1970s. Dr David Sackett's seminal work provided the core concepts and definition; Dr Archie Cochrane advocated randomized controlled trials and systematic reviews as tools for verifying treatment effectiveness.

The framing question: how do you know that what you do and how you do it really works?

Challenges of implementing it

• Searching and finding the best evidence is time consuming
• Scientific evidence may not be available
• Conflicting evidence
• Outdated evidence
• Findings presented in language clinicians may have difficulty understanding

Three types of evidence

The evidence hierarchy

Studies at higher levels are least vulnerable to bias, more generalizable, and their outcomes are more likely attributable to the construct being studied.

Clinical studies pyramid — bottom to top

USPSTF levels of certainty

Certainty = the likelihood that the assessment of net benefit is correct. Net benefit = benefit minus harm of the service as implemented in a general, primary care population.

USPSTF grades of recommendation

Synthesis

• Systematic review — systematically identifying and evaluating multiple studies to answer a specific, focused question.
• Meta-analysis — qualitative or quantitative synthesis; “pools” data from multiple studies to produce a single large study. Greater n usually means greater power.

IO 3 — Judging sources

Quality markers: peer-reviewed · government databases · be cautious of websites.

The ABCD method for web resources

The library also provides the CRAAP test.

IO 4 — Databases and guidelines

Joint decision making and the five steps

Three primary elements of joint clinical decision making: disclosure of risks and benefits of therapeutic alternatives → exploration of patient values about the therapy and potential outcomes → the actual decision.

Steps to evidence-based practice: create a focused clinical question → systematic review of the literature or an evidence-based synopsis from a credible resource → evaluate the validity (truth) of the research → apply to individual patient encounters → evaluate performance or outcome.

3 · Study Design & Research Methods

Megan B. Finck, MMS, PA-C · Session 2

IO 2 — Research paradigms

What study design is, and why it matters

The organization and plan for data collection and analysis. It is directly related to the validity and reliability of data:

The two axes

Cross-sectional appears under both descriptive and analytic, and is always observational.

IO 3 — Other research methods

The goal of epidemiologic and clinical research

To permit an unbiased comparison between a group with the factor and a group without it. Good design allows: comparisons between two or more groups · quantification of the risk difference · determination of temporal sequence · minimization of biases.

Typical study sequence: Clinical observations → Available data → Case-control studies → Cohort studies → Randomized trials.
Trajectory toward EBM: Ideas (from biology, epidemiology, clinical observation, imagination, reasoning) → Hypotheses → Observational studies → Experimental studies → Evidence-based information.

IO 1 — The designs in detail

Cross-sectional — observational, for generating hypotheses

• Cross-sectional survey — data at a single point in time; risk factor and disease measured at the same time; individual level data. Examples: obesity in a population at a point in time; colorectal cancer screening by fecal occult blood; varicella titres for population immunity.
• Cross-sectional ecologic — relates the frequency of a characteristic to an outcome; population level data. Example: the 1996 Atlanta Olympics — streets blocked off, less car emission, fewer emergency visits for childhood asthma.
• Longitudinal ecologic survey — ongoing surveillance or frequent cross-sectional surveys; measures trends. Example: as smoking rises among women, does lung cancer incidence rise?

Case-control — observational, for generating or testing hypotheses

Groups defined by outcome (disease or no disease), then assessed for a difference in exposure or risk factors. It looks backward: from outcome to cause.

Cohort — observational, for testing hypotheses

A cohort is a group of people with a common characteristic assembled at a similar point and followed over time. The characteristic may be age, date of birth, exposure to a risk factor, occupation, religious affiliation, having a specific disease, or participation in an intervention.

• Subjects grouped on the basis of exposure rather than outcome.
• The comparison group should be as similar as possible except for the exposure.
• Must be followed long enough for the expected outcome to appear, and completely, so attrition does not falsely skew the outcome data.
• Prospective (concurrent) — assemble groups in the present by exposure, collect baseline data, follow forward.
• Retrospective (historical) — go back in history to define a risk group, then follow its members forward to the present.

Randomized controlled trial — experimental, for testing hypotheses

• Clearly formed hypothesis · development of a study protocol
• Results must be applicable to a larger population
• Subjects randomly assigned to treatment and control
• Can be double or single blinded · needs adequate sample size
• Should account for withdrawals and drop-outs

Systematic review and meta-analysis

As in section 2. The review identifies and evaluates; the meta-analysis pools the data into a single larger study.

4 · Rates & Disease Measurement

Megan B. Finck, MMS, PA-C · Session 3 · Fletcher ch. 2–3

Rates

A rate is the number of times an event occurs during a fixed time period. Rates allow comparisons — that is why they exist rather than raw counts.

Age-specific mortality is limited to a particular age group. Infant mortality is commonly used to compare health status between nations. Age-adjusted rates remove the effect of a population's age structure — the US crude death rate of about 794 per 100,000 in 2009 corresponded to an age-adjusted 741.

Case fatality = a measure of disease severity: the proportion diagnosed who die from it. Years of potential life lost = death at a younger age loses more productive years; used to target interventions.

IO 1 — Incidence and prevalence

Incidence density is used in dynamic populations and often in clinical trials, where people move in and out. Its denominator is person-time: one person followed 10 years contributes 10 person-years. Error can arise from variation in duration of follow-up.

Why prevalence matters clinically: it prioritizes the differential (horses, not zebras) and it sets the pretest probability, which changes how a test result should be read.

IO 5 — What influences measurement

• Cases on a continuum — most clinical information is not binary. The case definition has a marked impact on measured frequency.
• Population at risk — should include only those susceptible (men are not in a study of ovarian cancer). Limiting the population may curtail generalisability.

Sampling

Distribution of disease — person, place, time

IO 2 — Surveillance

May be active or passive. Helps establish a baseline. Operates at national, state and county level.

Epidemiologic functions

• Determine magnitude of a health problem
• Understand the natural history of a disease
• Detect outbreaks/epidemics — foodborne gastrointestinal disease; Vibrio after Hurricane Katrina floods
• Document geographic distribution and spread — tuberculosis on an international flight
• Allow hypothesis testing for etiology — via case control or cohort studies
• Evaluate control strategies
• Monitor changes in an infectious agent — emerging antibiotic resistance

Strategies

• Screening high-risk persons or groups — influenza in a nursing home resident or worker; restaurant worker with hepatitis A
• Clinical symptoms/syndrome — ask about persons with similar problems
• Mortality — some cases are detected only at death
• Provider reporting of morbidity — allows tracking of contacts and exposures
• Laboratory reporting of positive results

IO 3 — Investigating an outbreak

1. Determine epidemic vs endemic
2. Determine the attack rate
3. Establish the diagnosis
4. Establish the case definition
5. Determine if an epidemic truly exists
6. Characterize by time, place and person
7. Characteristics: type of exposure · route of spread · timing of exposure
8. Develop an epi curve — number of cases plotted against time
9. Develop hypotheses about source, type and route of spread
10. Test hypotheses
11. Initiate control measures

The investigation arc: detect possible outbreak → find cases → generate hypotheses → test hypotheses → solve original source → control outbreak → decide the outbreak is over.

Classic epi curve: Legionnaires' disease by date of onset, Philadelphia, 1 July – 18 August 1976, following an American Legion convention.

5 · Data, Validity & Variation

Megan B. Finck, MMS, PA-C · Session 3 · Fletcher ch. 2–3

IO 2 — Levels of measurement

IO 3 — Validity and reliability

Validity without physical verification

IO 5 — Types of variation

Fluctuation reflects two phenomena combined: biologic differences and variation in measurement.

Inter-patient variation may be due to inherent biological differences, systematic differences in the conditions of measurement, and/or measurement error. Answers to clinical questions about individuals are most often derived from groups of patients with similar conditions.

IO 4 & 6 — Distribution and dispersion

Frequency distributions portray the quantity of each category, class or interval — the number or proportion of people with different values. Usable on nominal and on interval data (more commonly the latter in clinical studies). Described by central tendency (the middle) and dispersion (the spread).

IO 7 — Normal versus abnormal

There is no clear-cut point at which pathology or disease exists. Disease is often acquired by degrees:

Susceptibility → Pre-symptomatic → Clinical Disease → Dysfunction → Death

• Statistical abnormality does not necessarily mean clinical abnormality. A serum cholesterol of 201 is above “desired” but not in itself clinically significant — though being high-normal, the risk is higher than at low-normal.
• You can have an abnormal finding and not be sick — a sickle cell trait carrier.
• Value extremes may be desirable — the lower end of the systolic blood pressure range; the higher end of the high-density lipoprotein range.
• A normal test does not necessarily rule out disease.

Defining abnormality

• Determine it in the context of values known to be associated with disease, disability or death
• A clinically recognizable difference from healthy
• Abnormal tests should be confirmed by repeating the test — regression to the mean
• Cutoff points for normal/abnormal are often arbitrary
• Be careful of labeling.

6 · Diagnostic Tests

Megan B. Finck, MMS, PA-C · Session 4 · Fletcher ch. 4 + the 2×2 / likelihood ratio reading

Before any test is ordered

• Common diseases occur commonly — horses, not zebras
• Uncommon presentations of common disease occur more frequently than common presentations of uncommon disease
• No disease is rare to the patient who has it

Considerations for ordering: avoid ordering tests indiscriminately · risk : benefit ratio · know the limitations of the test · avoid repeating a test without an indication · consider cost.

Diagnostic tests are for symptomatic individuals; screening tests for asymptomatic ones.

The gold standard

Any diagnostic procedure believed to identify the disease with certainty: culture for streptococcal throat infection · biopsy for cancer · cardiac catheterization for coronary artery disease · endoscopy for peptic ulcer disease.

Limitations: often expensive · often invasive or risky · may not be acceptable to patients · may not be easily completed. So alternative tests are often the initial choice — and are judged by how they compare to the gold standard.

The 2×2 table

Worked example — HUGIT vs endoscopy for peptic ulcer disease

10,000 patients had both. 25 ulcers by the gold standard, 20 correctly identified by the new test. The new test also suggested ulcer in 80 with a negative endoscopy.

Sensitivity = 20/(20+5) = 0.8 = 80%.   Specificity = 9,895/9,975 = 0.99 = 99%. Prevalence = 25/10,000 = 0.25% (low) — which is why the 80 false positives outnumber the 20 true positives.

SnOUT and SpIN

Sensitivity and specificity are inversely related. Ideally maximize both; the receiver operating characteristic (ROC) curve plots true positive rate against false positive rate (1 − specificity) and lets clinicians compare alternative tests for the same diagnosis. The better test's curve lies closest to the upper left and has the greatest area beneath it. ROC curves also help determine where to set the cutoff point. To avoid the trade-off altogether, use serial testing.

Screening strategy: screen with a highly sensitive test and confirm with a highly specific one.

Predictive values and prevalence

PPV and NPV are the numbers that matter in the exam room: “if PPV = 85%, my patient with a positive result has an 85% chance of really having the disease.”

Pretest and posttest probability

• Pretest probability — the chance the patient has the disease based on what is known before the test; often the prevalence.
• Posttest probability — the chance after the test. This is what informs treatment decisions.

Likelihood ratios

How much the probability that a patient has a condition changes once the test result is available. Not affected by prevalence.

Getting to posttest probability. Mathematically: convert pretest probability → pretest odds; multiply by the LR → posttest odds; convert back to posttest probability. Or use the nomogram: find the pretest probability in the left column, the LR in the center, draw a straight line through both — the posttest probability is where the line crosses the right-hand column.

• LR > 1 → posttest probability higher than pretest. LR < 1 → lower.
• A very low LR (below ~0.1) virtually rules out the disease.
• When pretest probability is 30–70%, a very high LR (above ~10) rules disease in.

Worked example — the 8-year-old with a sore throat

Oral temperature 101°F, enlarged tender anterior cervical nodes, exudative tonsils. Rapid strep antigen test: sensitivity 65%, specificity 96%. Throat culture is the gold standard but takes 24–48 hours.

1. Pretest probability — modified Centor score:

2. Calculate the likelihood ratios:
LR+ = 0.65 / (1 − 0.96) = 0.65 / 0.04 = 16.25 — a positive result makes strep 16× more likely.
LR− = (1 − 0.65) / 0.96 = 0.35 / 0.96 = 0.36 — a negative result makes it about ⅓ as likely.

3. Read the posttest probability off the nomogram: ~93% with a positive result, ~22% with a negative one.

Using multiple tests

If multiple tests all agree, interpretation is easy. If they disagree, consider how they were conducted.

IO 1 — Questions to ask about a new test

1. What is the disease of interest?
2. Did the study include a spectrum of patients?
3. What is the new diagnostic test?
4. Did every subject undergo both the new test and the gold standard?
5. Were those conducting the tests blinded to the results of the other test?
6. Benefits and risks of the new test — is it reasonable?
7. Sensitivity, specificity, PPV, NPV?
8. Is the prevalence of disease similar in your patient population?

7 · Risk & Ratios

Megan B. Finck, MMS, PA-C · Session 7 · Fletcher ch. 6-7

IO 1 — What a risk factor is

A risk factor is a factor associated with an increase in the probability of developing the disease later. It raises the probability; it does not make the disease certain, so risk factors describe groups of people rather than guaranteeing an outcome for one person. A factor that is present only after the disease has begun is not a risk factor, and a treatment that lowers the chance of disease is a protective intervention rather than a risk factor.

Example — congestive heart failure: smoking · hypertension · hypercholesterolemia · family history · diabetes · valvular disease. For other diseases the risk factors may be less well known.

IO 1 — Recognizing risk

IO 1 — What risk is used for

IO 1 — How risk is studied

Experimental studies of risk in humans are limited by ethical constraints: investigators cannot ethically assign people to a harmful exposure. So observational studies are used: a concurrent (prospective) cohort, a historical (retrospective) cohort, or a case-control study.

IO 2 — Relative risk and what its values mean

Relative risk asks how many times more likely exposed persons are than non-exposed persons. It is expressed as a ratio of the incidence in exposed persons to the incidence in non-exposed persons, and it examines the strength of association between the independent variable (exposure) and the dependent variable (outcome).

• For prevention of disease we expect the relative risk to be less than one.
• Relative risk is a ratio of two incidences, and incidences cannot be negative, so relative risk can take any value above zero but cannot be negative (and is not capped at 2 or clustered near 1).
• Reading the number: a relative risk of 2.5 means exposed persons are 2.5 times as likely to develop the disease. Above 1, the excess is read as a percentage (1.25 is 25 percent higher, 1.4 is 40 percent higher); below 1, the shortfall is read the same way, so a relative risk of 0.6 means the risk in the exposed is 0.6 times that in the unexposed, that is 40 percent lower.
• Relative risk is a ratio, not a percentage: a 2.5 percent increase in risk would be a relative risk of 1.025.
• Raw case counts cannot be compared directly, because the groups differ in size; each count is first turned into an incidence (cases divided by the people in the group) and then the two incidences are divided.

Worked example — birth control pills and thrombophlebitis

For 10 years, 2,000 women were followed: 1,000 randomly selected young women on birth control pills and 1,000 randomly selected young women who were nonusers. 30 of the women on the pills developed thrombophlebitis over the 10-year period; 3 of the nonusers developed it over the same period.

Risk in users = 30/1,000 = 0.03. Risk in non-users = 3/1,000 = 0.003. Relative risk = risk in users ÷ risk in non-users = 0.03 ÷ 0.003 = 10. Expressed: women on the pill are 10 times more likely to develop thrombophlebitis than those not on the pill. Had the relative risk equaled 1, the pill would not increase the risk of thrombophlebitis: there would be no association between pill use and the development of thrombophlebitis.

IO 3 — Predicting risk from groups

Risk is used for prediction of a future event, and the probabilities are based on groups of people, not on one patient followed in detail. Relative risk is calculated to determine the probability. For an individual patient, the best predictor is the presence of early clinical disease, not the presence of a single risk factor and not the relative risk or the incidence of the wider population. A probability quoted from cohort data, such as the chance a smoker develops lung disease within ten years, is prediction.

IO 3 — Absolute risk, and the cohort that measures it

Absolute risk is the possibility of an event in the population under study: the incidence of disease in that population. It does not indicate whether exposure is associated with an increased risk of disease.

Among the observational designs, the cohort study establishes incidence directly: following exposed and unexposed persons forward shows who develops the disease (the printed table of cohort study advantages calls it the only way of establishing incidence, that is absolute risk, directly). A case-control study starts from persons who already have the condition, a case report describes a single patient, and expert opinion is not a followed group, so none of them supplies incidence.

Predicting cases from a relative risk. Because relative risk = exposed incidence ÷ unexposed incidence, the exposed incidence = relative risk × unexposed incidence. Multiply that incidence by the number of people in the group to get the expected number of cases. Example: unexposed incidence 0.05 and relative risk 3 gives an exposed incidence of 0.15, so 200 exposed persons are expected to produce 0.15 × 200 = 30 cases. The relative risk multiplies the incidence, not the head count.

IO 4 — What a cohort is, and how it is followed

A cohort is a group of people with a common characteristic assembled at a similar point and followed over time. The characteristic may be age, date of birth, exposure to a risk factor, having a specific disease, or participation in a preventive or therapeutic intervention. It can be any characteristic that is followed for a significant period of time.

Design of a cohort study of risk: persons without disease are divided into two groups — those exposed to a risk factor and those not exposed. Both groups are followed over time to determine what proportion of each group develops disease. Relative risk can be calculated.

• The cohort must be followed over a sufficient period of time to observe the expected outcome.
• It must have complete follow-up so that attrition (people dropping out) does not falsely skew the outcome data. A larger control group does not replace lost participants; the safeguard is following everyone through to the outcome.
• Check what was actually measured in the follow-up: were data points collected that relate to the study?

IO 4 — Historical and concurrent cohorts

Be careful: a case-control study grouped by disease is sometimes called “retrospective”. A historical cohort is grouped by exposure and followed forward, so it is not a case-control study. It is also not a randomized trial, which assigns the exposure by chance.

IO 4 — Advantages and limitations of cohort studies

IO 5 — Three types of risk

• Attributable risk is a difference (subtraction, exposed minus unexposed, in that order); relative risk is a ratio of two incidences. Adding the incidences gives nothing meaningful.
• Example: incidence 15 per 500 (0.03) in exposed persons and 5 per 500 (0.01) in unexposed persons gives an attributable risk of 0.03 − 0.01 = 0.02, while the relative risk is 0.03 ÷ 0.01 = 3.0.
• Example: 36 of 300 exposed (0.12) and 20 of 400 unexposed (0.05) persons developed disease. The relative risk is 0.12 ÷ 0.05 = 2.4 and the attributable risk is 0.07. Dividing the raw counts (36 by 20) ignores the group sizes.
• Example: an unexposed incidence of 0.02 and a relative risk of 4 give an exposed incidence of 4 × 0.02 = 0.08; the attributable risk is 0.08 − 0.02 = 0.06.
• Example: an incidence of 8 percent in exposed and 2 percent in unexposed persons gives a relative risk of 8 ÷ 2 = 4.0; the difference of 6 percentage points is the attributable risk, and the sum of the two incidences (10) has no meaning. In a cohort where 10 of 500 vaccinated and 40 of 500 unvaccinated persons developed influenza, the relative risk with vaccination is 0.02 ÷ 0.08 = 0.25.

IO 5 — Same relative risk, different attributable risk

The same relative risk can hide very different numbers of excess cases, because relative risk cancels out incidence while attributable risk depends on it. The higher the baseline incidence, the greater the attributable risk for the same relative risk.

For example, with a relative risk of 2.0 the attributable risk is 0.001 when the baseline incidence is 1 in 1,000 (0.002 minus 0.001) but 0.1 when the baseline is 1 in 10 (0.2 minus 0.1). The exposure with the higher baseline incidence has the far greater attributable risk; the relative risks are equal, so equal relative risks do not mean equal attributable risks.

IO 6 — Confounding variables

Confounding causes systematic error (bias). A confounder is an additional independent factor related to both the exposure and the outcome, so it can distort the apparent association and produce a false one. It is not random error: chance produces variation equally likely above or below the truth, small samples add random error, and dropout is the attrition effect.

Example: if smoking is more common among coffee drinkers and also raises the risk of pancreatic cancer, smoking is related to both the exposure (coffee) and the outcome (cancer) and is a confounder of the coffee-cancer association.

IO 6 — Methods for controlling confounding

• Only randomization and restriction are design-phase-only methods.
• Matching works at either phase, design or analysis.
• Stratification, simple adjustment, multivariable adjustment and best-case/worst-case analysis are applied only in the analysis phase.

IO 7 — Effect modification

Effect modification means the presence or absence of a variable changes the effect of the exposure. The change may be synergistic (the variable strengthens the effect) or antagonistic (it weakens or opposes the effect). It is NOT a bias: it is a real change in the effect, whereas confounding is a systematic error to be controlled.

Example: aspirin, age and prior history and their effect on the development of upper gastrointestinal bleeding — age and prior history modify how strongly aspirin affects the chance of bleeding. Likewise, if a drug lowers blood pressure far more in younger than in older adults, age is an effect modifier of the drug's effect, not a confounder.

IO 8 — Case-control design

In a case-control study the case group has the condition and the control group does not. The groups are chosen on the basis of whether they have the condition (outcome), not on the basis of exposure. Controls may be drawn from the same cohort, from the community or a hospital, or matched to the cases.

Direction: cases and controls are identified first and each is then compared for earlier exposure to the risk factor (exposed or not). Time runs forward for events but the research runs backward, from disease to past exposure — the reverse of a cohort study, which looks forward from exposure to disease. The design ends in an estimate of relative risk. The exposure is the factor compared between the groups (for example, past sunburns in a study comparing patients with and without melanoma); the presence or absence of the condition defines the groups.

Exposure information. Exposure is ideally measured, but oftentimes subjects are asked about their exposure, which brings recall bias: the accuracy of memory and the ability to recall specific information may not be perfect, especially once subjects know the disease studied.

Example of the design: a case-control study of helmet use and head injury among skiers and snowboarders. Cases had a head injury and controls did not; each was compared for whether a helmet had been used, with age, sex, nationality, skill level, equipment used, ski school attendance and rented-or-owned equipment controlled.

IO 8 — Advantages and limitations of case-control studies

Cohort studies are not ideal for capturing causation: there is a latency period between exposure and pathological effect, and a large sample is required although only a limited number of participants get the condition. Case-control studies are practical for this situation, and better suited to a rare disease with a long latency:

A case-control study still needs a defined case group and a control group, and still asks about earlier exposure; what it avoids is waiting for the condition to develop. Once the exposures that increase the risk of disease are understood, the exposure is followed to determine causation.

IO 9 — Which measure for which design

The odds ratio (relative odds) is the measure of association for case-control studies. For cohort studies, the incidence of disease in the exposed versus non-exposed can be calculated directly, so relative risk is usually used rather than the odds ratio. Relative risk and odds ratio allow numerical expression of the likelihood of a risk outcome: relative risk for cohort studies, odds ratio for case-control studies. The odds ratio is an estimated relative risk.

Example: researchers enroll 150 patients with a rare cancer and 300 without it and ask about past exposure. This is a case-control study, so the measure of association is the odds ratio, not the relative risk, attributable risk or absolute risk.

IO 9 — The lettered 2×2 table and the formulas

• Odds compare those with the exposure to those without it: exposed ÷ not exposed (48 exposed and 12 unexposed cases give odds of 48/12 = 4, not the proportion 0.8).
• Odds ratio = odds of exposure in cases ÷ odds of exposure in controls. The case odds go on top; inverting the ratio is a slip.
• Example: 60 of 100 cases and 30 of 100 controls exposed. Odds in cases 60/40 = 1.5; odds in controls 30/70 = about 0.43; odds ratio = 1.5 ÷ 0.43 = 3.5, matching ad/bc = (60 × 70)/(40 × 30) = 4,200/1,200.
• Example: 20 of 100 cases and 40 of 100 controls exposed. Odds in cases 20/80 = 0.25; odds in controls 40/60 = about 0.67; odds ratio = about 0.38, which is below 1, so exposure is associated with lower odds of disease.
• Dividing the exposed counts alone (60 by 30) ignores the unexposed counts and is not the odds ratio.

IO 9 — A worked odds ratio, and how to state it

200 women were selected: 100 young women with thrombophlebitis and 100 without. History of prior birth control pill use in the past 5 years was carefully obtained. 90 of the 100 women with thrombophlebitis were using pills; 45 of the 100 women without thrombophlebitis had used them.

Odds of pill use in cases = 90/10 = 9. Odds of pill use in controls = 45/55 = 0.82. Odds ratio = 9 ÷ 0.82 = 11.0.

Expressing the result. Women with thrombophlebitis have 11 times greater odds of having taken birth control pills than women without thrombophlebitis. The version to use, and the one we actually want to know, is: women who take birth control pills have 11 times greater odds of developing thrombophlebitis. The odds ratio is the same whichever way it is read, so a result such as “patients with lung disease had six times the odds of prior asbestos exposure” is stated as: exposed persons have six times the odds of the disease.

• An odds ratio is a ratio of odds. It is not a percentage of exposed persons who are affected, and an odds ratio of six is not a one-in-six frequency.
• An odds ratio above 1 means the exposed, not the unexposed, have the greater odds.

Definitions and rules of thumb

• Risk factor. A risk factor is a factor associated with a higher chance of developing a disease later. It raises the probability of disease rather than making it certain, so risk factors describe groups, not a guaranteed outcome for one person.
• Who takes part in a risk study. Healthy people take part: a study of risk follows people who do not yet have the disease, because the question is who goes on to develop it, so the outcome of interest is diagnosis.
• Long latency. A long latency between exposure and disease can obscure the association: the disease appears long after the exposure, so the two can look unrelated unless people are followed for a sufficient period of time.
• Relative risk of 1.0. When the risk in the exposed group equals the risk in the unexposed group, the relative risk is 1.0, the null value, and this indicates no association between exposure and disease. For example, 20 of 500 exposed and 16 of 400 unexposed are both 0.04.
• Recall bias. Recall bias arises from participants' difficulty remembering past exposures, which distorts the reported exposure or history; it is a particular weakness of case-control studies.
• Effect modifier. A variable whose presence or absence changes the effect of the exposure is an effect modifier. If an exposure raises the risk much more in older patients than in younger ones, age is an effect modifier, which differs from a confounder that only distorts the association.

8 · Prognosis & Outcomes

Megan B. Finck, MMS, PA-C · Session 7 · Fletcher ch. 5-8

IO 1 — Describe prognosis

Prognosis is the prediction for the course of disease. It is what a clinician tries to estimate once a diagnosis has been made: how the disease will unfold in terms of survival, complications, disability and similar outcomes.

• Prognosis is expressed quantitatively as a rate, such as five-year survival, the case-fatality rate, the response rate, the remission rate and the recurrence rate. Each states how often an outcome occurs.
• Prognosis can also use cure or death as its endpoints.
• The natural history of a disease is what occurs without intervention. Prognosis can often be improved through clinical intervention, which is how clinicians change the natural history to improve the outcome.
• The reason to detect disease at an earlier point is to maximize the effectiveness of treatment. Earlier detection helps only if treatment given sooner actually changes the outcome.

IO 2 — Prognostic factors

Prognostic factors help identify groups of patients who share a similar disease but have different prognoses. They work within one disease, separating the patients who have it into groups that fare better or worse.

• Prognosis studies look for protective or progressing factors — factors that protect against, or promote, progression of the disease. These are different from the factors that raise the chance of getting the disease.
• Multiple factors are a better predictor of outcome than a single data point.
• Clinical prediction rules estimate the probability of an outcome based on a set of patient characteristics, combining several factors into one estimate for an individual patient.
• Adjustment for important prognostic factors addresses whether other factors explain a difference in outcome: are the differences in the prognosis of the groups related to the factor under study or to some other factor?

Example — acute myocardial infarction

In a patient who has had an acute myocardial infarction, the poor prognostic factors are: increasing age, female sex, cigarette smoking, hypotension, anterior infarction, congestive heart failure and ventricular arrhythmia. The outcomes are death, reinfarction and other complications.

Ventricular arrhythmia is a poor prognostic factor that arises after the infarction; it is not a risk factor for developing coronary disease in a healthy person.

IO 3 — Risk factors versus prognostic factors

Risk factors do not necessarily represent a poorer prognosis. Being at risk of getting a disease and faring badly once you have it are separate questions, so the two sets of factors can differ, although some factors appear on both lists.

The coronary disease example

Increasing age and cigarette smoking appear on both lists: they raise the chance of an infarction and worsen the outlook after one. Inactivity, hypertension and the lipid pattern are risk factors for developing coronary disease. Anterior infarction, congestive heart failure, hypotension and ventricular arrhythmia are poor prognostic factors that apply once the infarction has occurred. Hypertension is the blood pressure abnormality on the risk list; hypotension is the one that marks a poor prognosis.

IO 4 — Clinical course and natural history

The natural history of disease is what occurs without intervention. It is the untreated benchmark: as new treatments arise, they are compared with the natural (untreated) outcome of disease.

The five Ds

The course of disease is described and quantified by the five Ds: death, disease, discomfort, disability, dissatisfaction.

Severity of disease matters because it establishes priorities for clinical services and public health programs.

Stages of the clinical course

The first stage is biologic onset. Diagnosis is given immediately after medical care is sought, and treatment is recommended and implemented only after the diagnosis. The outcome comes last.

• The preclinical phase is the period in which the disease is present but signs and symptoms have not yet appeared.
• The clinical phase begins once signs and symptoms of disease are present, and runs through care sought, diagnosis and treatment to the outcome.
• Screening can find disease before signs and symptoms appear, that is, during the preclinical phase. Detecting disease at an earlier point is what screening aims to do.

IO 5 — Rates that describe prognosis

Time under observation is either stated or assumed to be sufficiently long that all events that will occur have been observed.

Outcomes of disease should include the full range of manifestations: consider both biological and clinical outcomes, and also consider quality of life measures.

The arithmetic, with fresh numbers

Case-fatality ratio

The case-fatality ratio is the number of people who die divided by the number of people who have the disease. It is usually used for acute, short-term disease where death occurs shortly after diagnosis, because the deaths can be counted without long follow-up. For chronic disease with survival of decades, survival measures are used instead.

From a risk communication standpoint, be careful with negative framing — presenting the number dying rather than the number surviving. The data are the same, but the framing can sway perception: it changes how the risk is perceived.

Five-year survival

Five-year survival is frequently used in clinical medicine, particularly to describe outcomes of cancer patients.

• Rates can mask important data. Four conditions can have nearly identical five-year survival — each of the four curves ends with roughly 5–10% of patients alive at five years — while the survival curves differ greatly. A single rate does not show when the deaths occur within the five years.

Screening can inflate five-year survival

The five-year time line is problematic when considering screening programs: the disease may be found earlier, but death still occurs at a similar point. Patients simply live longer with a diagnosis, so measured survival rises without anyone living longer. This is a potential bias if disease is detected earlier but the natural history of disease is not impacted favorably by treatment.

Reading the timeline: survival from diagnosis is the year of death minus the year of diagnosis. In the standard timeline used to illustrate this, the disease begins in 1987, is detected by screening in 1989 or diagnosed without screening in 1991, and death occurs in 1995 either way. Survival from diagnosis is therefore 6 years with screening (1989 to 1995) against 4 years without (1991 to 1995), yet death did not come any later. Moving the diagnosis earlier by some number of years adds that many years to the measured survival when the date of death is unchanged.

Another limitation: those with a history of disease duration of less than five years cannot be evaluated by this measure, nor can patients treated with a new therapy introduced within the past five years, because no one has yet been followed for five years.

IO 6 — Survival analysis

Survival analysis follows a cohort over time and records the outcomes at designated time intervals. People can be added at various times in the course of their disease, and each contributes the time they are followed, giving the person-years contributed to the study.

The probability of surviving to any point in time is estimated from the cumulative probability of surviving each of the time intervals that preceded it — the chances of surviving each earlier interval are multiplied together. For example, surviving three successive intervals with probabilities of 0.95, 0.90 and 0.60 gives 0.95 × 0.90 × 0.60 = 0.513, about 51%.

Note on scope. The course material describes the outcome of survival analysis as limited to living versus dying; the same method is also applied to any single event that happens once, such as recurrence or remission.

Kaplan-Meier method

• Rather than using pre-determined intervals, such as death at one year, the exact point in time when each death occurs is noted and a new interval begins. This generates the curve.
• Because each death starts a new interval, the curve is a series of steps: it holds level and then steps down at each death.
• Patients who are censored (no longer followed, and alive as far as is known) leave the number at risk without being counted as deaths. In each interval the probability of surviving is the number who survive divided by the number still at risk, for example 7 of 8 at risk surviving gives 0.875.

Cox proportional hazards survival regression

• Similar to Kaplan-Meier, but it can be used to study the effect of an exposure on outcome after adjusting for confounders.
• It can also be used to find prognostic indicators for survival in a given disease, taking into account the effects of multiple predictors on the shape of the survival curve — for example treatment differences, age differences and differences in drug dosages.
• The hazard ratio that results can be interpreted as a relative risk (risk ratio). A hazard ratio of 2 means the group with the factor has twice the risk of the outcome (dying) as the group without it. A ratio of less than 1 is considered protective; a ratio of 1 means the same risk in both groups.
• A hazard ratio of 0.25 means the group has 25% of the risk of the comparison group, that is, 75% lower.

Reading a survival curve

• Survival curves estimate the survival for a hypothetical, rather than actual, cohort.
• Precision is greater earlier, on the left side of the curve, because the population still being followed is larger.
• The shape of the curve can be deceptive: it tends to flatten or plateau when the event of interest is occurring to fewer numbers of participants being followed, so the flat tail rests on few participants and does not mean deaths have stopped.

Other survival measures

The median is the middle value when survival times are put in order: for seven ordered times of 2, 4, 7, 9, 13, 18 and 30 months it is 9 months, whereas the mean (11.9 months) is pulled upward by the longest survivor. On a curve, the median survival is read where survival crosses 50%. For relative survival, an observed survival of 63% against an expected 90% gives 63 ÷ 90 = 70%.

IO 7 — Bias in cohort studies

Bias in survival cohort studies may falsely create or obscure differences that do exist.

Inception cohorts and zero time

• Cohorts should all be identified at a similar point in the disease course, called zero time — a shared point in the disease course.
• Inception cohorts are assembled at onset of disease, when it first becomes clinically apparent — as early as possible and as uniform as possible. Patients assembled at different points in the course cannot be compared fairly.
• Specific inclusion criteria need to be specified.
• Consider the referral pattern of the patients in the study: patients at an academic medical center may differ from those in the community.
• Sources of selection bias include convenience samples, volunteer samples, declines from those selected for study, and dropouts.

The named biases

The question to ask about selection bias: are the differences in the prognosis of the groups related to the factor under study or to some other factor(s)?

Matching, restriction and stratification handle only the factors chosen for consideration. Randomization is the only method of dealing with differences between groups whose benefits are not limited to the factors selected for consideration, so it also handles factors nobody measured. Studies typically use several methods to control bias. Blinding, matching, restriction and stratification each address a different bias: blinding addresses measurement bias, while the others address selection bias.

Generalizability is often limited because the patients in the study may not be representative of other populations or of individual patients.

What to look for in a prognosis study

1. Was an inception cohort assembled? — so that patients start at a similar point in their disease.
2. Were the entry criteria objective and reasonable?
3. Was the follow-up of subjects adequate?
4. Are the study patients similar to yours?
5. Was there adjustment for important prognostic factors?
6. Were outcomes assessed objectively and blindly?

Definitions and formulas

• Prognostic factors. Prognostic factors identify patients with the same disease who have different prognoses. They separate patients who share a diagnosis into groups that fare differently.
• Risk factors and prognosis. Risk factors do not necessarily represent a poorer prognosis, so being at risk of developing a disease and faring badly with it are separate questions; a risk factor need not worsen prognosis.
• Case-fatality ratio. Case-fatality ratio = number of deaths from a disease ÷ number of people with the disease. For example, 30 deaths among 400 people with the disease is 30/400 = 0.075, or 7.5 percent. It is most useful for acute, short-term disease where death follows quickly after diagnosis, because the deaths can be counted without long follow-up.
• Five-year survival rate. Five-year survival rate = number of patients alive at five years ÷ number of patients in the cohort when no one is lost to follow-up. For example, 90 of 150 patients alive is 90/150 = 0.60, or 60 percent.
• New treatments. As new treatments arise, they are compared with the natural (untreated) outcome of the disease, which is the baseline that shows whether a treatment changes what would otherwise happen.

9 · Prevention & Screening

Megan B. Finck, MMS, PA-C · Session 8 · Fletcher ch. 9–10

Objective 1 — The types of clinical prevention

Prevention is defined as interventions for people without complaints, to identify and modify risk factors to avoid the onset of disease, or to find disease early in its course. Preventive care is built into clinical practice in four ways.

Objective 2 — The levels of prevention

The timeline runs: no disease → onset → asymptomatic disease → clinical diagnosis → clinical course. Primary prevention is labeled “remove risk factors”, secondary “early detection and treatment”, and tertiary “reduce complications”. The three levels are primary, secondary and tertiary; the four types of Objective 1 are different things and are not levels.

Objective 3 — Intervention in terms of prevention

When should a condition be included in preventive care?

Three questions decide whether a condition belongs in preventive care.

Each component belongs to only one criterion: disability, death and hospitalizations describe burden; sensitivity, specificity, simplicity and acceptability describe the test; effectiveness and cost-effectiveness of therapy describe the treatment.

How each level is studied

Cohort design. A cohort study starts with a defined population and divides it, without randomization, into an exposed and a not exposed group, then follows each forward to see who develops disease and who does not. The diagram shows the retrospective form, in which the groups are defined in the past (1980) and followed to the present (2000). Groups are defined by exposure and followed for the outcome; a case-control study does the reverse, starting from the outcome.

Objective 4 — Advantages and disadvantages of screening exams

The advantage

The advantage of screening is that disease can be found early, while it is asymptomatic, and so treated earlier in its course. It pays off when outcomes are better with early detection and treatment than when disease is found later.

Prevalence screens and incidence screens

Three biases in observational studies of screening

The three biases are lead-time bias, length-time bias and compliance bias. Each can make screening look better than it is.

Lead-time bias

Lead time is the time by which screening advances diagnosis: the time between detection from screening and when the condition would have been diagnosed from symptoms. The apparent improvement in survival time is not due to improved survival; it is due to increased time knowing the diagnosis.

Survival from diagnosis is the age at death minus the age at diagnosis. If a cancer is found by screening at age 60 and would have been diagnosed from symptoms at age 64, and the patient dies at age 72 either way, survival from diagnosis is 12 years with screening and 8 years without. The apparent gain is 4 years, which equals the lead time, and the patient has not lived a day longer. Earlier diagnosis happens in every screened group; a longer survival is a real improvement only when early treatment is effective and delays death.

Length-time bias

Outcomes appear better with screening because more slow-developing conditions with better prognoses are detected. Slow-developing conditions are present for longer, so a screen is likely to catch them, and they have better prognoses than fast-developing conditions. Screening works best for conditions that develop slowly. Rapidly developing conditions usually cause symptoms before screening is done, so they tend to escape it and are diagnosed clinically instead. In a diagram of tumor growth against time, the rapidly growing tumors reach the point of diagnosis after symptoms between screens, while a single screen mostly catches the slower-growing ones.

Compliance bias

The outcome in the screened group appears better because those people are more compliant. Compliant patients are more invested in their health and have better outcomes for reasons unrelated to the screening itself. For example, volunteers who attend screening may have lower death rates than people who decline even if screening offers no benefit.

Properties of tests used in screening

• High sensitivity early in the disease, so the test does not miss disease
• High specificity, to reduce false positives. A screened population is mostly healthy, so specificity is what keeps healthy people from being wrongly flagged. (Missing disease is a false negative, which high sensitivity reduces.)
• Consider positive and negative predictive values because disease prevalence is low
• Simple to administer and inexpensive
• Safe to administer

The disadvantages: unintended consequences of screening

A false-positive result flags someone who is healthy; a repeat-screen finding is an incident case, and neither is an incidentaloma. The biases above (lead-time, length-time, compliance) are further disadvantages, because they make screening appear better than it is.

What to look for in prevention studies

1. The rationale for the prevention: the burden, the quality of the screening test and the effectiveness of the intervention
2. Identification of the level of prevention
3. The properties of the tests used
4. Strategies to reduce bias

Objective 5 — Principles of community screening programs

A community screening program takes place in a community setting. Its objective is to test large numbers of individuals at little or no cost to the participants.

Possible objectives of a program

Establishing a screening program

• The condition should be an important or serious public health problem
• There should be effective treatment available
• The natural history of the disease should be well understood
• The disease must be prevalent but not found in everyone
• Those with positive screens must have access to care
• It must be clear who is responsible for following up the results of the screening: what counts as a positive test, how findings become part of a participant’s medical record, and so on

Before a program starts, the benefits of being screened must outweigh the costs and harms. The test should be safe with minimal side effects, quick and easy to administer, inexpensive, and acceptable to the public and to their health care providers. Its sensitivity, specificity, positive predictive value and negative predictive value must be known and be acceptable; the predictive values must be known as well because they depend on how common the disease is in the people screened.

Low prevalence: why high-risk groups are screened

With low prevalence, even highly specific tests generate more false positives than true positive results, because healthy people far outnumber cases: a small error rate applied to a very large healthy group produces more wrong positives than the few true cases produce right ones. Specificity and sensitivity themselves do not change with prevalence; the predictive values do. Therefore programs screen “higher risk” populations, where prevalence is higher. In a higher-risk group the positive predictive value rises (a larger share of positive results are true), while screening only people without risk factors would lower prevalence further and make the ratio of false to true positives worse. A test with lower specificity would generate still more false positives. The positive and negative predictive values can be calculated.

The arithmetic

Build the counts from the number screened, the prevalence, the sensitivity and the specificity (the same two-by-two table as in Session 4):

• People with disease = number screened × prevalence. True positives = people with disease × sensitivity; false negatives = the rest of them.
• Healthy people = number screened minus people with disease. False positives = healthy people × (1 − specificity); only healthy people can be false positives.
• Positive predictive value = true positives ÷ (true positives + false positives).

Example of the method with fresh figures: 5,000 people screened, prevalence 2 percent, sensitivity 80 percent, specificity 90 percent. People with disease = 100, true positives = 80. Healthy = 4,900, false positives = 490. Positive predictive value = 80 ÷ 570, about 14 percent. False positives outnumber true positives about six to one although the test is 90 percent specific.

Objective 6 — Number needed to treat and number needed to harm

Working with the numbers

• The standard calculation is not printed on the slides, which define the ideas. Both numbers are the reciprocal of an absolute change in risk, expressed as a decimal: an absolute risk reduction of 5 percentage points is 0.05, so the number needed to treat is 1 ÷ 0.05 = 20. A risk that falls from 10 percent to 6 percent is a reduction of 4 points (0.04), giving a number needed to treat of 25. A side effect that rises from 1 percent to 4 percent is an increase of 0.03, giving a number needed to harm of about 33.
• Use the difference between the two groups’ risks, not either group’s risk alone, and not the difference in event counts: when 50 of 1,000 untreated and 30 of 1,000 treated patients have an event, the risk falls from 0.05 to 0.03, an absolute reduction of 0.02, and the number needed to treat is 50, not 20.
• Comparing treatments for the same outcome over the same period: the smaller number needed to treat is the more effective treatment. For harm, the larger number needed to harm is the safer.
• Expected effect in a group = number of patients ÷ the number needed to treat (cases prevented) or ÷ the number needed to harm (patients harmed). With a number needed to treat of 50, treating 500 patients prevents 10 cases; with a number needed to harm of 100, treating 500 patients harms 5. Dividing is correct; multiplying is not.
• Comparing benefit and harm: one patient in every number-needed-to-treat benefits, and one in every number-needed-to-harm is harmed. A number needed to harm that is smaller than the number needed to treat means harm is more frequent than benefit (for example a number needed to treat of 30 and a number needed to harm of 20 for equally serious outcomes).
• Baseline risk matters. The number needed to treat depends on the baseline risk as well as on how much the treatment cuts it, so a treatment that halves risk gives a larger number in a low-risk group: halving a risk of 4 percent leaves 2 percent (a reduction of 0.02, number needed to treat 50), whereas halving a risk of 1 percent leaves 0.5 percent (a reduction of 0.005, number needed to treat 200). A large number therefore does not always mean a poor drug; it can also reflect a low-risk group.

Definitions and formulas

• Practical features of a screening test. A screening test is applied to many healthy people, so it must be simple to give, safe and inexpensive.
• Expected false positives. Expected false positives = (1 − specificity) × number of people without the disease; true positives = sensitivity × number with the disease. For example, with specificity 90 percent among 5,000 healthy people, 10 percent test positive, giving 500 false positives.
• Positive predictive value in screening. Positive predictive value = true positives ÷ (true positives + false positives). When prevalence is low, false positives outnumber true positives even for an accurate test, so the positive predictive value is low; for example 50 true positives and 450 false positives give 50/500, or 10 percent.
• Length-time bias in a program. Screening catches more slow, favorable conditions, so screened patients look better even if screening changed nothing. A program that finds mostly slow-growing cancers, with screened patients apparently outliving unscreened patients whose cancers were found by symptoms, illustrates length-time bias.

10 · Research & Trials

Megan B. Finck, MMS, PA-C · Session 8 · Fletcher ch. 9

IO 1 — Testing ideas: coincidence, bias and wishful thinking

Decisions about treatment need to avoid resting on coincidence, biased comparisons and wishful thinking. The better basis is evidence-based information on the relationships between treatments and outcomes.

IO 1 — Treatment and treatment hypotheses

A treatment is an intervention to improve the course of disease. It may be applied at any point in the natural course of disease, and it may be a medication, surgery, a lifestyle change and so on.

A treatment hypothesis may develop from any of four sources:

• the mechanism or etiology of the disease
• clinical observation
• epidemiologic studies
• trial and error

However a hypothesis arises, most require testing to evaluate their usefulness and outcomes. A plausible mechanism, wide use or a clinician's approval does not replace that testing.

IO 1 — Comparative effectiveness

Comparative effectiveness is the direct comparison of two or more interventions believed to improve outcomes. The comparison identifies any harms and benefits and compares the end results.

Treatments can be studied with observational and experimental designs. Cohort studies are limited by systematic differences in the groups other than the treatment factor. The randomized controlled trial is the gold standard for evaluating treatment efficacy.

IO 1 — Sampling

Sampling is the process of selecting participants for a study, trying to represent the larger population. An entire population is rarely studied, which is why a sample is used. The population has to be defined, and defining it too wide or too narrow causes problems.

Mirroring the population. If a clinic population is 60 percent women and a stratified sample of 100 is meant to mirror it, the sample holds 0.60 × 100 = 60 women.

IO 1 — Inclusion and exclusion criteria

• Inclusion criteria are the characteristics participants must have.
• Exclusion criteria are the factors that prevent inclusion.

The same logic applies to any drug: having the disease being studied, or poor control of it, is an inclusion criterion, while a contraindication to the treatment under evaluation (an allergy to an ingredient, for example) is an exclusion criterion.

Each additional set of criteria shrinks the eligible pool. A published chart of patients with chronic obstructive pulmonary disease shows the eligible share falling step by step, as the percentage of all patients still eligible after each set of criteria:

The pool shrinks with every added set of criteria, sometimes to a small fraction of all patients with the condition. To work out how many remain, multiply through the steps in sequence. For example, 1,000 patients with 50 percent meeting a first criterion and 40 percent of those meeting a second leave 1,000 × 0.5 × 0.4 = 200.

IO 1 — Comparison groups, placebo and the Hawthorne effect

Comparison groups are the standard in trials. They may be treatment versus placebo, treatment versus treatment, or treatment versus non-treatment.

• A placebo is indistinguishable from the treatment but has no specific mechanism of action. It helps blinding.
• The placebo effect is the change in condition experienced by individuals given a placebo because they expect a change.
• The Hawthorne effect is the tendency of participants to change their behavior when given special interest or attention during a study.

IO 1 — Randomization

Random sampling and random assignment are different steps. Sampling picks who enters the study from the population. Assignment picks which group each of those subjects joins. Random assignment is preferred for testing a treatment because it limits systematic differences between the groups other than the treatment factor.

Stratified randomization

In a stratified randomized trial the eligible patients are first divided into strata (for example strata 1, 2 and 3). Randomization to treatment or control takes place separately within each stratum. The final study groups are then formed by combining the treatment patients from every stratum into one group and the control patients into the other.

IO 1 — Compliance

Compliance is the degree to which participants act in accordance with instructions (or patients with medical advice). Noncompliance is not always intentional; unintentional causes include misunderstandings, lack of transportation and forgetfulness. Deliberately skipping doses or stopping a drug by choice is intentional noncompliance.

IO 1 — Blinding

Blinding (also called masking) means making participants unaware of which treatment is being received.

IO 1 — Efficacy and effectiveness

A drug can therefore have high efficacy but low effectiveness: it works when everyone takes every dose under close monitoring, but poorly in clinics where doses are missed.

• Internal validity — the degree to which results are correct for the sample of patients studied.
• Generalizability — the degree to which results can be applied to other settings.

IO 2 — Design of a randomized controlled trial

A randomized controlled trial is built in this order:

1. Start with a research question about a population.
2. Draw a sample of people (subjects) from the population to be in the study.
3. Randomly assign the subjects to treatment groups.
4. Follow both groups forward in time and measure the outcome of interest.
5. Calculate the risk of the outcome in each of the treatment groups.
6. Compare risk between groups using the relative risk, the risk difference or the hazard ratio.

Comparing risk between groups

Example (invented numbers). If 15 of 100 treated patients and 30 of 100 comparison patients had the outcome, the risks are 15 percent and 30 percent, the relative risk is 0.15 ÷ 0.30 = 0.5, and the risk difference is 15 percentage points.

IO 2 — Superiority and non-inferiority trials

• Superiority trials determine whether one treatment is better than another.
• Non-inferiority trials determine whether a new treatment is unlikely to be less effective than an established treatment. The new treatment may be cheaper, safer or easier.
• Non-inferiority trials usually require a larger sample size than superiority trials, because the differences may be small.

IO 2 — Advantages and limitations of randomized controlled trials

Advantage: random assignment limits systematic differences between groups, which is why the randomized controlled trial is the gold standard for evaluating treatment efficacy.

Limitations:

• Confounding may exist in smaller randomized trials, because chance imbalances between the groups are more likely when few participants are enrolled.
• They are expensive.
• They raise logistical issues: sample selection, randomization, blinding, consistency with all participants, and time.

IO 3 — Is a trial ethical?

• Randomization is ethical only when no conclusive evidence favors one treatment: there is no reason to believe the treatments under evaluation are unequal. Once conclusive evidence exists that a treatment prevents death, for example, there is no real uncertainty left and assigning patients to placebo would be unethical.
• The primary outcome must be benefits, not harms.
• Participants must give consent.
• If there is evidence of harm, futility or effectiveness, the study is ended.

IO 3 — Methodological issues and possible bias

Five issues can bias a controlled trial:

• Participants without the disease in question being enrolled.
• Compliance problems.
• Cross-over: switching from one randomly assigned treatment to another, so participants no longer stay in the group to which they were assigned.
• Unacknowledged co-interventions: an extra treatment that goes unrecorded and can affect the outcome.
• The assessment of the outcome may be biased.

Even with blinding it may be possible to determine the treatment group, because side effects (and similar clues) can reveal it. Biased assessment of outcomes is avoided by using specific objective criteria and blinding.

The four possible outcomes of a study

Two of the four pairings are correct and two are errors. A conclusion is correct only when it matches reality; blinding does not turn a wrong conclusion into a correct one.

IO 3 — Trials of N = 1 (single subject design)

A rare type of clinical trial in which one patient is given one treatment at a time. The provider and the patient are blinded to which treatment is given when, and the outcomes are assessed.

IO 3 — Phases of clinical trials

A side effect that occurs in about 1 in 20,000 patients is far too rare to show up in the small numbers of phases I and II or usually even in phase III; it is postmarketing surveillance of large numbers that detects it.

IO 3 — Evaluating a treatment study

1. Is it a controlled trial with random assignment?
2. Are the study patients similar to yours? (This tells you whether the results apply to your patient.)
3. Was there complete follow-up? Look at best- and worst-case scenarios for dropouts.
4. Was blinding used?
5. Are the intervention and control groups similar?
6. Is there clinical significance as well as statistical significance?
7. Are other factors affecting the outcome considered?

Best- and worst-case scenarios for dropouts. The analysis assumes the best and worst possible outcomes for the patients lost to follow-up and checks whether the study’s conclusion would change. It shows whether the conclusions survive the lost patients.

Example (invented numbers). In a treated group of 100 patients, 12 were lost and 25 of the 88 who finished had the outcome. If every lost patient is assumed to have had the outcome, the treated group’s risk is (25 + 12) ÷ 100 = 37 percent, not 25 ÷ 88 = 28 percent.

IO 4 — Observational studies: advantages and limitations

Observational studies of treatment collect information on patients currently being treated for a disorder, instead of assigning the treatment.

• Advantage: they include real-world patients, which brings both a benefit and drawbacks.
• Limitation: it is difficult to avoid confounding and effect modification.
• Their findings often agree with the findings of randomized controlled trials, but can be dramatically different.
• It is difficult to draw conclusions from uncontrolled trials, because there is nothing to compare the treated patients against.

Definitions and formulas

• Efficacy. Efficacy is concerned with results under ideal trial circumstances, whereas effectiveness reflects ordinary, real circumstances.
• Compliance. Compliance is the degree to which participants act in accordance with instructions, or patients with medical advice.
• Unintentional noncompliance. Noncompliance is not always intentional: misunderstandings, lack of transportation to visits and forgetfulness are unintentional causes.
• Sources of treatment hypotheses. Treatment hypotheses come from the mechanism or etiology of the disease, clinical observation, epidemiologic studies, and trial and error.
• Consent. Participants must give consent to take part in a trial; consent is one of the conditions for an ethical trial, alongside genuine uncertainty about which treatment is better.
• Basis of treatment decisions. Treatment decisions should rest on evidence about the relationship between a treatment and its outcomes, rather than on coincidence, biased comparisons and wishful thinking.
• Limitations of randomized controlled trials. They are expensive, and they raise logistical issues in sample selection, randomization, blinding and consistency across participants and time.
• Risk in a group. Risk = number of patients with the outcome ÷ number of patients in the group. For example, 30 of 200 is 0.15, or 15 percent.
• Relative risk. Relative risk = risk in the treated group ÷ risk in the comparison group. For example, risks of 0.05 and 0.20 give a relative risk of 0.25.
• Risk difference. Risk difference = risk in the comparison group − risk in the treated group, in percentage points. For example, risks of 30 percent and 18 percent give a difference of 12 percentage points.
• Eligibility criteria shrink the pool. Each eligibility criterion removes patients, so the pool shrinks step by step: multiply the number screened by the proportion meeting each criterion in turn. For example, 1,000 screened, with 50 percent meeting the first criterion and 40 percent of those meeting the second, leaves 200.

11 · Statistics, Correlation & Causation

Megan B. Finck, MMS, PA-C · Session 9 · Fletcher ch. 11-12

Objective 1 — Review of probability

Probability is the likelihood that any one event will occur, given all the possible outcomes. The probability (P) that an event (E) will occur is estimated by:

The denominator is every opportunity for the event, including the ones where it occurred. Dividing by the number of times the event does not occur gives the odds, which is a different quantity, and inverting the fraction gives a value above one, which cannot be a probability.

• Probability must lie between 0 and 1. A value of 0 means the event cannot occur; a value of 1 means it will definitely occur.
• A value of 0.5 means an equal probability of occurring or not occurring, so the event is as likely to occur as not.
• The sum of all probabilities of all events in the sample must equal 1. So the probability that an event does not occur is 1 minus the probability that it does: if the probability of having a condition is 0.35, the probability of not having it is 0.65.
• Counting works the same way: if 36 of 240 patients in a clinic have a condition, the probability that a randomly chosen patient has it is 36 ÷ 240 = 0.15, and the probability of not having it is 204 ÷ 240 = 0.85.

Objective 1 — Significance and the p value

Two kinds of significance matter when reading a study: statistical significance and clinical significance.

Statistical significance describes how likely it is for the result to have happened by chance, so it describes how unlikely chance is as the explanation. More statistically significant = less likely to be chance. Less statistically significant = more likely to be chance. It does not describe how much the treatment helps an individual patient, how carefully the study was designed, or whether the result would change practice.

The p value is the probability of an observed difference happening by chance. Read precisely, it is calculated on the assumption that no true difference exists: it is the probability of a difference at least this large arising by chance alone. It is not the probability that the null hypothesis is true, not the probability that the treatment truly works, not the proportion of patients who respond, not the size of the difference between groups (the effect size), and not the probability that the study is biased. Bias is systematic error, and a p value addresses random chance only.

• Reading a p value of 0.03 (new drug against placebo): 0.03 is below 0.05, so the difference is statistically significant. It does not mean there is a 3 percent chance the drug does not work, and it does not mean the drug works in 97 percent of patients. Statistical significance also does not imply that the difference is large enough to change practice.
• Reading a p value of 0.20: 0.20 is above 0.05, so the result is not statistically significant and chance remains a reasonable explanation. It does not show that the treatment has no effect (the study may lack power, see below), it is not the proportion of patients who improved, and it is nowhere near very highly significant.
• A lower p value shows only that chance is less likely as an explanation. It does not show that the difference is larger or more important.

Objective 1 — The null hypothesis

The null hypothesis is the hypothesis that there is no difference. It is usually the opposite of what the researchers are interested in finding, and it is used in statistical analysis: the test asks how well the data fit it. The hypothesis that a difference is present is the alternative to the null hypothesis. The null hypothesis is not a statement about bias or about whether the sample is representative.

• Example. Researchers hope to show that a new drug lowers blood pressure more than placebo. The null hypothesis is that the drug and placebo lower pressure equally. Drug better than placebo is the researchers' hoped-for alternative, and placebo better than drug states a difference in the wrong direction.
• If there is not evidence to support the null hypothesis, there is more support for a difference being present. Evidence against the null hypothesis, which states no difference, is what lends support to a difference. Identical groups is what the null hypothesis itself states, so weak support for it points away from that idea.

Objective 1 — Type I and type II error (four possible statistical conclusions)

• Type I error (α error, also called an alpha error) = false POSITIVE — stating there is a difference, or concluding that a relationship exists, when there is not. Most medical research is concerned with the possibility of type I error, which is expressed by the p value (judged against the chosen threshold, usually 0.05).
• Type II error (β error, also called a beta error) = false NEGATIVE — stating there is no difference, or concluding that no relationship exists, when there is one. The real difference is missed and the statistic appears not to be significant. (The precise wording is that the test failed to detect a real difference.)
• Type II error sometimes occurs as a result of study limitations, such as length of follow-up and subject compliance: too brief a follow-up, or subjects not adhering to treatment, can hide a real difference. A p value below 0.001 is a significant result, so it cannot be the cause of a false negative, and random assignment and blinding are strengths of design rather than causes of type II error.
• When no true difference exists, the correct conclusion is that the result is not significant. A significant result would be a type I error, the null hypothesis is true in that case (it should not be rejected), and power is defined only for the case where a difference does exist.
• Examples. A trial concludes that a supplement lowers cholesterol but in truth it has no effect: type I error (a false positive). A short trial finds no significant benefit from a drug that truly works: type II error (a false negative). Measuring the outcome incorrectly is measurement bias and choosing the wrong test is a different problem; neither is what defines type I or type II error.

Objective 1 — Common statistical tests

Means are averages of continuous values, medians are middle values, and proportions are shares of a group. The Fisher exact test takes over from the chi-square test when the number of observations is small.

Objective 1 — Confidence intervals

A confidence interval expresses statistical precision. It is a range of scores with boundaries, or confidence limits, that should contain the true value. A 95% confidence interval means that there is a 95% chance that the confidence interval contains the true value; put another way, if the study were repeated many times, about 95 percent of such intervals would contain the true value.

• A narrow interval means greater statistical precision; a wide interval means a less precise one. Two studies can report the same point estimate with different precision: a relative risk of 0.6 with an interval of 0.5 to 0.7 is more precise than the same 0.6 with an interval of 0.2 to 1.8.
• An interval is not the range that holds 95 percent of patients' individual values, it is not a range for the p value, it does not measure or correct bias (an interval can be narrow and still be wrong if the study is biased), and it is not the same as clinical importance or statistical power.

Reading an interval around a relative risk or a difference

For a relative risk, 1 means no difference, and values below 1 mean the risk is lower in the treated or exposed group. In a plot of relative risk the outcomes with intervals lying entirely on one side of 1 are statistically significant, and an interval that spans 1 is not. For a difference (for example in mean weight loss) the value of no effect is 0.

A 95% confidence interval that excludes the value of no effect corresponds to a p value below 0.05 for the same comparison, and an interval that contains it corresponds to a p value above 0.05. The interval adds what the p value alone cannot: the range of plausible effect sizes.

Worked graph (systematic review of corticosteroids for pneumonia). Each outcome is drawn as a point (the relative risk) with a line (its confidence interval) on a scale from 0 to 2, with 1 as the line of no effect. Mortality and mechanical ventilation lie to the left of 1 with intervals that do not reach 1 (corticosteroids benefit). Hyperglycemia lies to the right of 1 with an interval that does not reach 1 (corticosteroids harm). Gastrointestinal bleed has a point just under 1 and a very wide interval that crosses 1, so it is imprecise and not significant.

Objective 1 — Statistical power

Statistical power is the probability that a study will find a statistically significant difference when the difference really exists. It is calculated in terms of type II error:

• Statistical power is high when studies have a high probability of detecting true differences. Power is the complement of the type II error probability, not the type I error probability, not the p value, not the size of the difference and not the chance that the null hypothesis is true.
• Calculation. A study with a 20 percent (0.20) chance of a type II error has power 1 − 0.20 = 0.80, or 80 percent. A study with power 0.90 has a type II error probability of 1 − 0.90 = 0.10. Ninety-five percent is the level used for confidence intervals and 5 percent (0.05) is the usual threshold for type I error; neither is derived from the type II error probability.
• Low power means a high probability of type II error, so a real difference is likely to be missed. A small trial of a drug that truly works can therefore report a non-significant p value (for example 0.30) simply because it had too little power. A non-significant result does not prove there is no effect and does not prove the null hypothesis true; it fails to reject it. That is a type II error, not a type I error.
• Larger samples give more power, which is why pooling data across studies in a meta-analysis usually helps.

Objective 1 — Multiple comparisons

The strength of statistical tests depends on the number of research questions in a study and how focused those questions are. It is not always possible to know how many comparisons were made. Selecting more interesting comparisons after the analysis is complete can distort the data. When many comparisons are made in one study, some may look significant by chance alone; more comparisons do not make any p value more accurate, do not make type II error impossible and do not narrow confidence intervals.

• Expected chance findings. A p value of 0.05 corresponds to a chance of 1 in 20. If a study makes 60 comparisons and none reflects a true difference, about 60 ÷ 20 = 3 will appear significant by chance. At a p value of 0.01 (1 in 100), 200 comparisons with no true difference give about 200 ÷ 100 = 2 chance findings. Using the 0.05 threshold instead of 0.01 would predict 10, which is the wrong rate.

The practice that misleads is picking interesting results after analysis and hiding how many were examined. Naming the primary question beforehand, reporting every comparison that was made, and limiting the study to a few focused questions all protect against the distortion.

Objective 1 — Multivariable modeling

Multivariable modeling examines the effect of two or more variables together, as opposed to a single variable in isolation. It is a mathematical construct (an equation), and it is used in two ways:

• To look at the effect of one variable on the outcome while taking the effect of other variables into account (confounding or effect modifiers), which allows one variable's effect to be isolated.
• To look at the combined effect of multiple variables acting together to predict a clinical event.

Process

1. Identify variables related to the outcome (confounding, effect modifiers).
2. Select variables for inclusion: consider those with the strongest association with the outcome first (not those least associated, easiest to measure or with the most missing data).
3. Identify assumptions. They commonly involve probability distributions, with underlying assumptions such as normality, homoscedasticity and independence of errors.
4. Create the model.

Limitations

• Validity is based on assumptions about the data: a model is valid only if its assumptions hold. The number of patients recruited affects precision and power, and adding more variables or requiring every variable to be significant does not secure validity.
• Poor at identifying effect modification. (Taking confounders into account is what a model is designed to do.)
• May omit rare variables with a strong relationship with the outcome.
• A model cannot prove that the outcome is caused by a variable, cannot remove random error (which is estimated statistically and never eliminated) and does not replace the need for a comparison group.

Objective 2 — Statistical significance versus clinical importance

Statistical significance asks how likely chance is as an explanation; clinical significance (importance) asks whether the difference is relevant to practice and to patients. A short contrast: chance versus relevance to patients. (Random error versus systematic error, precision versus accuracy, and validity versus reliability are different pairs and do not contrast the two kinds of significance.)

• Statistical significance does not imply clinical significance. Differences can be present with a p value below 0.0001 and still not have any relevance to clinical practice. So a very small p value tells you nothing about relevance: it does not make a difference relevant, and it does not make it irrelevant either.
• Larger differences tend to be more clinically significant, but small differences can still be statistically significant (especially in very large samples). A lower p value is not the same as a larger difference, and sample size affects statistical significance and precision rather than how much a difference matters clinically.
• The reverse also happens: an important difference can fail to reach statistical significance, for example in a small study with low power.
• Question test. “Would it matter to the patient?” is a clinical question. “Is the p value below 0.05?”, “Does the interval exclude no effect?” and “Was the null hypothesis rejected?” are all statistical questions.

Objective 3 — The study of causation and the types of cause

Causality exists if a factor produces or contributes to the production of a specified outcome. It is not the same as association (any factor associated with an outcome), nor as merely occurring before an outcome, nor as having a significant p value.

• Single causes, multiple causes, and the web of causation (many factors acting together).
• Koch's postulates for infectious disease are the example of a single cause: the organism must be present in every case of disease; must be isolated and grown in pure culture; must cause a specific disease; and must be recovered from an inoculated animal and be identified.

Applying the definitions. If disease X never occurs without exposure Y but many exposed people stay healthy, Y is a necessary but not sufficient cause. Y would have to always produce disease when present to be sufficient.

From association to cause

Before concluding that a statistical association is causal, three alternative explanations must be excluded, taken in this order: bias in selection or measurement first, then chance, then confounding. If bias is likely, or chance is likely, or confounding is present, the association is not causal; only when none of them explains it is a cause likely. (Dose-response, reversibility, specificity, temporality, strength and analogy are criteria that support cause, not alternative explanations to be excluded, and precision, power and validity are properties of a study.) It is not possible to prove a causal relationship beyond all doubt; what is possible is to increase the evidence.

Objective 3 — Criteria for causality

The criteria applied: Helicobacter pylori and duodenal ulcer

• Temporal: Helicobacter pylori is linked to chronic gastritis, which is associated with the development of duodenal ulcer.
• Strength: H. pylori is found in 90% of patients with duodenal ulcer. Dose-response: its concentration is higher in the mucosa of patients with duodenal ulcer than in ulcer-free patients.
• Replication: the observations have been repeatedly replicated. Biologic plausibility: there are binding sites for H. pylori on antral cells, and it can follow these cells into the duodenum.
• Alternative explanations: smoking increases risk but is not a risk factor in patients without H. pylori. Cessation of exposure: eradication of H. pylori results in healed ulcers.
• Specificity: prevalence in patients with duodenal ulcer is 90-100%, but the organism is also found in some asymptomatic patients and those with gastric ulcers, so specificity is only partly met.
• Consistency with other knowledge: prevalence is the same in men as in women, and duodenal ulcer prevalence peaked in the late 19th century when H. pylori was more prevalent because of poor living conditions.

Objective 3 — Pitfalls in causal research

Objective 4 — The impact of study design on determining causation

Evidence for or against cause depends both on the design of the study and on the findings. The hierarchy of designs for causal evidence runs from strongest to weakest:

1. Systematic review
2. Randomized controlled trial
3. Multiple time series
4. Non-randomized trial
5. Cohort
6. Case-control
7. Time series
8. Cross-sectional
9. Case series
10. Case report (the weakest, at the bottom of the hierarchy)

• The randomized controlled trial gives the strongest single-study evidence for causation (systematic reviews, which pool trials, sit at the very top with it). Random assignment balances confounders between the groups (it spreads them evenly), which is why it strengthens causal conclusions; it does not remove random error (still accounted for through p values), does not guarantee a significant result and does not make measurements more reliable.
• A cohort study ranks above a case-control study, which looks backward from the outcome. Cross-sectional studies, case series and a case report rank below it; a case series describes patients without a comparison group.
• A cross-sectional study is weak evidence for causation because exposure and outcome are measured together at a single point in time, so it cannot show which came first; the temporal sequence, an essential criterion for cause, cannot be established. It can still detect an association and does measure the outcome.
• Strong design plus a finding. A well-designed randomized trial that finds no effect of an exposure on an outcome provides strong evidence against a causal link (provided the trial had adequate power). A design at the top of the hierarchy makes the evidence strong rather than weak, and a finding of no effect argues against cause, however strong the design.

Definitions

• Biologic plausibility. Biologic plausibility is met when the relationship makes sense biologically, for example when a known mechanism explains how an exposure could injure tissue; it strengthens the case that the relationship is causal.
• Evidence for causation. The more criteria for causation that are met, the stronger the case for a cause grows, though causation cannot be proved beyond all doubt.

12 · Clinical Questions & Reviews

Megan B. Finck, MMS, PA-C · Session 10 · Fletcher ch. 13–14

IO 1 — Narrative reviews

A narrative review is an expert summary of the evidence and recommendations. It is written by an author who chooses what to cite and how to describe it, with no set method for finding, choosing or appraising the studies.

Where it fits. Because a narrative review is broad and unstructured, it suits an overview of many aspects of a disease. A question with a defined population, intervention and outcome — whether one drug lowers mortality more than another, whether a test detects a disease more accurately, or whether screening lowers deaths in one age group — is a focused question, and answering it is the job of a systematic review.

IO 2 — Systematic reviews: the question, the studies and how to appraise one

A systematic review is a review of evidence with the goal of answering a specific, predetermined clinical question. That is the contrast with a narrative review, which is broad and open ended: the question of a systematic review is specific and is set in advance, before the studies are found, so the answer cannot be shaped around what the reviewers happen to find.

Defining the question

The first task is to define a focused question. The considerations are:

• Are strong studies available?
• How many studies are available?
• Do all the studies agree?

Put together, whether a review question can be answered depends on whether strong studies exist and whether they agree. Wanting a particular answer, sharing a publisher, or the existence of earlier expert commentary does not show that answerable studies exist.

Summarizing the studies

Once studies are found, the review asks how many quality studies were found and what the results were. The results are summarized as effect size, precision and validity — both internal validity (are the conclusions valid for the sample studied) and external validity (do they hold in other settings).

Evaluating a systematic review article

A reader appraising a systematic review asks five things:

1. How were the studies located?
2. What inclusion and exclusion criteria were used?
3. How valid were the original studies?
4. What biases existed in the original studies?
5. What were the results of the original studies, and was there variation among them?

The search and the criteria are read together with the validity of the original studies. The layout of tables, the length of the reference list and who drafted a section are not appraisal questions.

IO 3 — The four-part clinical question (PICO)

A focused question is organized into four components, abbreviated PICO: Patients, Intervention, Comparison and Outcomes.

The intervention element is wider than a treatment. It can be a prognostic factor or an exposure, so the same format organizes risk and prognosis questions as well as treatment questions. The comparison is the alternative set against the intervention — another treatment, or none. The outcome is what is measured, and the patient element sets the group to which the answer will apply.

• A question comparing two diagnostic tests can name the accuracy of each test — sensitivity and specificity — as its outcome, as in the virtual colonoscopy row above.
• A question that is too broad needs refining. If a question is asked “for all patients”, the patient element is the one to make more specific; a more specific population may need to be defined.
• A complete question has all four parts. A question that omits the comparison does not say what the intervention is being measured against; one that names no group, or no measurable outcome (“better in some way”), cannot be answered from studies. A well-built example: in adults with migraine, does a daily preventive drug, compared with placebo, reduce the number of headache days?

IO 4 — Methods for conducting a systematic review

Read the steps as an order: the question is defined first, the studies are then selected, described and summarized, pooling comes after the summary and only if it is justified, and the exploration of heterogeneity is last.

• Protocol. Writing the question and the planned analysis down before searching stops the reviewers from shaping either around the results they find. A forest plot is a figure made after analysis; a funnel plot checks for publication and reporting bias; sensitivity analysis explores differences after the results are in.
• Consistency. A single reviewer, the lead author alone, or the journal editor cannot show that the criteria were applied consistently; at least two reviewers do, both when selecting studies and when assessing each study's risk of bias.
• Excluded studies. The reason each was excluded is listed, so a reader can judge whether the criteria were applied fairly.
• Forest plots and tables summarize the studies. Funnel plots belong to the selection step, where publication and reporting bias are assessed. A nomogram and a receiver operating characteristic curve are tools for interpreting diagnostic tests, not for summarizing or checking a review.
• Methods that explore differences among pooled studies are subgroup analysis, sensitivity analysis and metaregression. Blinding and randomization protect a single trial from bias and do not explore differences between studies.
• Heterogeneity tests. Cochran's Q and the I-squared statistic quantify how much study results differ; survival-analysis tests, correlation coefficients and rank tests serve other purposes.

Finding studies

To sample the best studies, the reviewers ask:

• How strong was the study design?
• Was the sample size sufficient? (A trial with only a handful of participants is weighed against the others for exactly this reason.)
• What kind(s) of bias was present, and are there concerns for publication bias?
• What were the limitations?
• Was the research peer-reviewed?
• Is the information still current, or have more recent studies been published?

The reviewers also develop inclusion and exclusion criteria; the criteria are explicit and based on the study question and design.

IO 5 — Publication bias

Publication bias: research without statistically significant results is less likely to be published. Two causes are named: journal editor preference, and a sponsoring organization or company being less likely to support research that is unfavorable to them or their product. Research with significant results is more likely to appear in print, whatever the rigor or sample size of the studies.

Reporting bias is different: only a subset of all relevant data is made available. Publication bias concerns whole studies failing to reach print; reporting bias concerns a study that appears but shows only part of its data.

• It should always be considered. A review can include only what has been published, so studies that were never published leave the review looking at a skewed set. It threatens every kind of review that relies on published studies, systematic reviews and meta-analyses included.
• The direction of the distortion. If trials showing no benefit of a drug are never published, a meta-analysis of the journal articles sees only the trials favoring the drug, so the pooled benefit may be overstated.
• What does not fix it. A statistical model, such as a random effects model, cannot restore studies that were never published. Two reviewers selecting studies makes the selection consistent but cannot include studies that do not exist in print. Pooling does not reveal them either.
• How it is assessed. Assessing for publication and reporting bias (for example, with funnel plots) is part of selecting studies for a systematic review. A reader of a meta-analysis asks whether publication bias is possible for the selected topic.

IO 6 — Principles of meta-analysis

A meta-analysis combines the results of similar studies to examine effect size quantitatively. Its principles:

• Weighted analysis of numerical results, proportionally to study size. A larger study counts for more. If weight is in proportion to size, each study's share of the weight is its participants divided by all the participants pooled: with trials of 200 and 800 participants, the smaller carries 200 of 1,000, which is 20 percent, and the larger 80 percent. Equal weighting would give each 50 percent, and picking only the largest study or counting positive studies is not pooling.
• Pooling gives a more precise estimate of effect size than any single study.
• Inclusion and exclusion criteria are very important, because they decide which studies are pooled.
• The studies must be similar. A statistical test for heterogeneity asks how different the reported effect sizes are.

Heterogeneity

Heterogeneity is the extent to which trial results are different from one another. A statistical test for it takes as its null hypothesis that there is no difference between the study results, and looks for statistically significant differences among them. This is not the null hypothesis of the pooled effect (no effect of the intervention), and it is not about bias or agreement between reviewers. The tests named are Cochran's Q and the I-squared statistic.

Fixed effect and random effects models

Both models calculate the summary effect and its confidence interval. So a statistically significant test of heterogeneity across pooled trials favors the random effects model, while a fixed effect model suits pooled trials with no significant heterogeneity. A model that assumes a different effect in every study would not justify pooling to one fixed magnitude, and counting positive studies is not a model.

IO 7 — Meta-analysis: advantages and limitations

• Only as good as the studies. Combining poor studies produces a larger, more confident poor estimate; the flaws of the included studies are carried into the pooled result. A meta-analysis is not immune to them, cannot escape publication bias, and still needs inclusion and exclusion criteria.
• Pooling adds participants together, so the pooled sample is larger than any single study's; it does not reduce sample size. Combining numerical results is exactly what it does, and it is not restricted to rare complications: it can examine benefits, harms, complications and subgroups.
• When it misleads. A meta-analysis may be misleading if the studies combined are not similar. Similar studies, for example those measuring the same outcome, are what pooling calls for, and a larger pooled sample is an advantage, not a hazard.
• What pooling cannot do. It cannot find studies that were never published, and it does not remove bias inside the individual studies.

Evaluating a meta-analysis article

A reader appraising a meta-analysis asks:

1. What is the quality of the included studies — are they all at the same level? (A meta-analysis that combines one high-quality trial with several seriously flawed studies raises exactly this question.)
2. Is there a possibility of publication bias for the selected topic?
3. What differences existed between the included studies, and were they statistically significant?

IO 8 — Grading recommendations for treatment

Grading evidence is a quick way to assess clinical significance. It is most often applied to treatment and screening. It still involves some degree of value judgment, because the pros and cons must be weighed. The scheme is based on the Grading of Recommendations Assessment, Development and Evaluation (GRADE) guidelines.

• 1A versus 1B. Both are strong recommendations, in which benefits clearly outweigh risks and burdens. 1B is graded lower because its randomized trials have important limitations, so further research may change confidence.
• What 1C rests on. Observational studies, unsystematic clinical experience, or seriously flawed trials. 1A rests on consistent well-performed trials; 1B on trials with important limitations.
• What makes a recommendation weak. Benefits closely balanced with risks and burdens, or uncertainty in the estimates. Risks clearly outweighing benefits is “or vice versa” and gives a strong recommendation (against the treatment).
• Acting on a grade. Follow a strong recommendation unless there is a compelling rationale for an alternative; the choice with a weak recommendation varies with circumstances and values, as 2A shows. A grade of 2A has high-quality evidence but is weak because benefits and risks are closely balanced.

Clinical practice guidelines and their standards

Good guidelines give the recommendations along with the supporting evidence and rationale. Apply your best clinical judgment when deciding to use guidelines in your practice; a guideline supports judgment and does not replace it.

Tell the standards apart by what each one checks. Transparency is about how the guideline was developed and funded; conflict of interest is about disclosure of members' interests; group composition is about who sat on the panel that wrote it; external review is about stakeholders assessing the finished guideline; updating is about publication dates and plans to revise it.

IO 9 — Conflict of interest

Accepting a recommendation means asking whether the people behind it stand to gain. Conflict of interest is when financial or personal considerations affect objectivity; its impact may be conscious or unconscious. There are three kinds:

• Why acknowledge it. It is important to acknowledge conflicts of interest so readers can determine whether bias (unintended or otherwise) is present. Disclosure does not exclude a study or change its data; it informs the reader.
• How it manifests: selective reporting of results; study selection for systematic reviews and meta-analyses; exaggeration or fabrication of results. Random assignment, confidence intervals and blinding of assessors are safeguards or statistical tools, not signs of conflict.
• Peer review helps, in addition to disclosure; neither replaces the other.

The peer review process

1. The manuscript is submitted.
2. It is screened by the editors.
3. It is forwarded to peer reviewers, who reject, recommend revision or accept for publication.
4. The editors review language, grammar and so on.
5. Publication.

So the editors screen a manuscript first, and peer reviewers see it only after that; after acceptance the editors still review language and grammar before publication, so an accepted article is neither published with no further review nor corrected for grammar by the peer reviewers.

IO 10 — Resources for answering clinical questions

How to keep current

Suggested ways to keep current: professional organizations, journal articles, colleagues and patients. Sales representatives, advertising and press releases are not among them; they are promotion, not appraised evidence. Systematic reviews, meta-analyses and clinical practice guidelines (IO 2, 6 and 8) are the resources that summarize the evidence for a clinical question.

Screening an article

1. Is the title related to the topic, and does it have the keywords you have in mind? If no, skip the article and go to the next.
2. If yes, read the abstract, summary and conclusion. Look for clear-cut aims and objectives, a well-defined research hypothesis, and precise conclusions.
3. Is that useful or relevant to what you are looking for? If no, skip the article and go to the next. If yes, read the entire article.

Reading an abstract, and beyond it

Where to look: cursory and in-depth review

The figure shows an option to stop after each step, so the reader need not go on to the in-depth review. The size of the effect is read from the figures and tables, whereas the methods show how strong the study was, the introduction shows the importance of the question, and the discussion gives context.

Definitions and questions to ask

• Publication bias always matters. Publication bias should always be considered when reading a review: a review can only include what has been published, so unpublished null results leave it looking at a skewed set of studies, and missing studies can distort the evidence.
• Quick reading of an abstract. Quick reading asks what the study is about, whether the conclusions are valid, who it applies to, and whether it can and should be applied to your patients.
• Appraising the included studies. When a meta-analysis combines studies, ask whether the studies are all of equal quality: the quality of the included studies should be examined, and mixing very different levels of quality weakens the pooled result.
• Missing studies in a review. A review that searches only published articles is most likely to be missing studies without significant results, because research without significant results is less likely to be published.
• Weighting in a meta-analysis. A meta-analysis gives larger, more precise studies more weight, and the pooled result is a more precise estimate of the effect than any single study gives.

★ Quick-Reference Tables

Bias at a glance

Bias versus chance

Which design answers which question

Which property does not move with prevalence

Named examples worth remembering