• AKA: overview

• Is a Rigorous summary of all research evidence that relates to a specific question (r/t causation, Dx, Prognosis)

• Addresses the effectiveness of an intervention

• Every systematic review should grow from a focused Question

- leads to development of inclusion criteria (glossary)

- and sensitive & specific search strategy - use strict inclusion/exclusion criteria

• Overcome possible biases at all stages by following a rigorous methodology for :

- searching (comprehensive)

- research retrieval 

- incl review of study relevance

- data extraction

- data synthesis

- interpretation

- pre-set inclusion criteria r/t relevance & validity (quality)

• bias reduced if

- explicit, preset inclusion criteria (based on relevance and validity)

- 2 people independently select studies and discuss discrepancies

• meta-analys is used in some, but not every systematic review

• summary of research-based knowledge on topic from valid primary studies

Meta-Analysis

(G) (Text, p.139)

• A quantitative combination of the results of similar studies

• can assign diff weights to individual study estimates - now, those with greater precision, or higher quality, will make greater contribution to the summary estimate 

• use meta-analysis to find the RR of treatment failure (relapse) in the Ss group vs. the control group (Ciliska 2001 p139

• Gives an overall summary statistic of the variables effect across different studies.

• Combines samples of diff studies > create on BIG study

• Now: overall summary statistic = more precise

This process involves the analysis and synthesis of qualitative findings, methods, and theories or frameworks from different studies to develop overarching or more conclusive ways of thinking about phenomena

Relative Risk (RR)

U6

(statistics summary sheet)

(text, p. 76-7, 78, 79, 111, 180-1, 200, 198-9, 139-41, glossary)

• AKA: probability (of a good or bad result) (Cullum, 2000 p111)

• A discrete measure

• Term used when referring o the probability of either a good or bad event

• The risk of an event in the intervention group (Ri) is simply the proportion of people in that group who experience the event.

• The strength of the association between the intervention and the outcome is usually represented y the relative risks or adds ratios

• Relative Risk or Risk Ratio (RR) is the proportion of the original risk that is still present when patients receive the intervention

• The proportion of risk that is still present when patients receive the intervention or exposure.

• Studies provide estimates of the true risk of an outcome

• An RR provides and estimate of the risk of a given outcome

• tells the relative benefit of treatment, NOT the “actual” benefit

• not consider # Ss who would developed outcome anyways

• The risk of patients in the intervention or exposed group experiencing the outcome divided by the risk of patients in the control or unexposed group experiencing the outcome

• Proportion of patients experiencing an outcome in the treated (or exposed) group divided by the proportion experiencing the outcome in the control (or unexposed) group (text, glossary)

• Risk of Ri group divided by risk Rc group (Sheldon, 2000 p76)

• Equations [a/(a+b)] / [c/(c+d)] Also… Ri / Rc Also… EER / CER

• Calculate the risk of readmission in the Intervention Group: a / (a+b) = 12 / 84 = 0.143 or 14.3%

• Calculate the risk of readmission in the Control Group: c / (c+d) = 26 / 81 = 0.321 or 32.1%

• Now, compare the two by dividing Intervention by Control = 0.143 / 0.321 = 0.445 or 44.5%

• (If RR=1.0 then there is no difference between groups; the risk of event/outcome is the same in both groups).

• If RR is on the right side of vertical line (RR >1) would favour the CTRL condition

• RR on the left (RR <1) favours the treatment.

• If the risk of a bad outcome is reduced by the intervention compared to the control group, then RR < 1.0.

• If risk of a bad outcome is increased by the intervention compare to the control group, then RR > 1.0.

• The further away the RR is from 1.0, the greater the strength of the association between the intervention and outcome

• Statement: The relative risk of readmission of the Intervention group is 44.5% compared to the Controls

• Statement: over 7 years, the risk of dying after admission to a stroke unit was 0.67 or 67% of the risk of dying after admission to a medical ward.

Odds Ratio (OR)

[(a/b) - (c/d)]

U6

(text p.77-8, 79, 180-2, 189-90, 198-200, 174)

(ppt, p. 5-2)

• Describes the odds of a patient in the Ri group having an event divided by the odds of a patient is the Rc having the event

• Or the odds that a patient was exposed to a given risk factor divided by the odds of that a control patient was exposed to the risk factor an estimate of the odds of having an event versus not having an event

• The odds of an event in the intervention or exposed group divided by the odds of the event in the control or unexposed group

• If (OR = 1.0) then there is no difference between groups; the odds of the event is the same in both groups.

• If (OR < 1.0) then the outcome/event is less likely in the intervention or exposed group compared to the control or unexposed group

• The odds of the outcome (event) in the Ri group divided by the odds of the Rc group

• First, calculate the odds of readmission for the Intervention = a / b = 12 / 72 = 0.167

• Then, calculate the odds of readmission for the Control = c / d = 26 / 55 = 0.473

• Finally, divide the odds of the Ri by the Rc = (a / b) / (c / d) = 0.167 / 0.473 = 0.353

• So, the odds of readmission for the intervention group are 0.353 times that of the control group (or 35.3% of the odds of the control group)

• (a/b) / (c/d)

• Also… (ad) / (bc)

• a measure of the strength of association between the Tx and the outcome

• a measure of effect in studies of causation

• The odds of an event in the intervention/exposed group divided by the odds of the event in the control/unexposed group

• the odds that a Ss was exposed to a given risk factor divided by the odds that a Ctrl was exposed to the risk factor.

• the outcome expressed as the ____ of the event (a/b)

• The ____ ____ is the odds of the event in the Tx group (a/b) devided by the odds of the event in the Ctrl group (c/d)

• an ____ ____ of 1.0 means there is no difference between groups. the odds of an event are the same

• an ____ ____ <1.0 means the event is less likely in the Tx group than the control group

• When the event being measured is quite rare, the OR and RR are numerically similar because the values of a and c are insignificantly small

• odds of death in Tx group

• odds of death in Ctrl group

• Statement: "the odds of dying in the Tx group are 48% of the odds of dying in the control group

• in case-control studies > we use the ____ ____ as the measure of the size of the effect of the exposure on the outcome

• **the calculation of ____ ____ is not confined to case-control studies

• the odds are calculated by dividing the number of ppl with the condition by those without the ratio are obtained by dividing the odds of having an event following an exposure by odds of have the same event who were unexposed

• the OR and RR are similar when freq of outcome is low

• but as outcome increases frequency > OR and RR diverge

• If the 95% CI does NOT include 1 (the null), there is evidence to reject the null hypothesis

• in Case-Control studies, the proportion of Ss with the adverse event (cases) is determined a priori by investigator

• so, the strength of association in case control studies is represented as ___ ___ • OR's of 1 indicate the Tx and Ctrl did NOT diff for the adverse event

• OR's >1 indicate increased risk of the adverse event among those in the Tx group,

• OR's < 1 indicate a decreased risk of the event in the Tx group

• The furthar from 1 > greater strength of an association between the Tx and outcome

• estimate the odds of having an event versus not having an event

• This is called the Odds Ratio (OR)

• First, calculate the odds of readmission for the Intervention = a / b = 12 / 72 = 0.167

• Then, calculate the odds of readmission for the Control = c / d = 26 / 55 = 0.473

• Finally, divide the odds of the Intervention by the Control = (a / b) / (c / d) = 0.167 / 0.473 = 0.353

• Statement: the odds of readmission for the intervention group are 0.353 times that of the control group (or 35.3% of the odds of the control group)

Precision (of the estimate)

(text p104, 142, ) (ppt, p9-2)

• How precise is the estimate of treatment effect?

• A large number of subjects increases precision

• CI’s around the RR and the RRR indicate

- The precision of the estimate of true treatment effect

- Wide CIs = less precision in the estimate

• Precision increases with larger sample size

• Precision of study findings is indicated by the width of the CI:

- The wider the CI, the less precise our study finding

- The narrower the CI, the more precise our study finding

• Precision of study findings depends on the Sample Size of the study:

- The smaller the Sample Size, the wider the CI and the less precise the study finding

- The larger the Sample Size, the narrower the CI and the more precise the study finding

Steps of a Systematic Overview

(ppt, p 2-1)

Critical Appraisal Q for 

Systemic Reviews of

Tx/Prevention Interventions

(U6-Chapt 19 notes)

• Evaluation of Systemic Reviews of Treatment or Prevention Interventions

• Are the Results of the

Systemic Review Valid?

- IS THIS A SYSTEMATIC REVIEW OF RANDOMIZED TRIALS?

(U6-Chapt 19 notes)

• strong Systematic Reviews incl studies that use the most appropriate research design to answer the question

• Qn r/t effectiveness of Tx or Prev'tn best answered by RCT

• Qn r/t harm or prognosis are best answered by cohort studies

• did the Systematic Review address a focused question

- all components of PICO should be in title/text

• was a meta-analysis done?

• Evaluation of Systemic Reviews of Treatment or Prevention Interventions

• Are the Results of the

Systemic Review Valid?

• A thorough search strategy includes (Ciliska, 2001).
  • several databases
    • electronic databases, 
    • not just MEDLINE & CINAHL
  • checked references of relevant articles
  • Searched by hand (Ciliska, 2001 - )
    • hand searching adds to completeness of retrieval
    • MEDLINE users may use wrong keywords, miss article, or miss entire journal issues
  •  Key informants contacted
  • Method to find unpublished studies

    ---

• Inclusion / Exclusion criteria for selection of articles for the review need to be identified
  • what were the incl/excl criteria?
    • think "PICO
    • methodological  standards (only RCT, English... better not to exclude by language)
• Inclusion and exclusion criteria should be r/t the Research Question
  • keeps research questions
  • interpreting the findings
  • defining methodological quality 

• Is there a description of the process of how the validity of each study was assessed?
  • quality rating scales are often used - usually look at components such as 
    • randomization & concealment
    • study design, 
    • blinding (of outcome assessors)
    • follow-up (loss to)
  • checklist ensures that 2 or more independ viewers are consistent and thorough in their appraisal
  • process to reach agreement identified (consensus)
    • Kappa statistic (of agreement) may be reported (>0.80 preferred)
• differences in study quality might explain differences in results because poorer quality tend to overestimate the effectiveness of interventions.
  • Give stronger weight to stronger studies
    • using quality rating scales
• Use of pre-specified quality checklist
• Ensure study reviews are appraised consistently and thoroughly
  • minimizes bias
• Have 2 or more raters
  • reduces bias
  • reduces mistakes
  • increases confidence in the systematic review

• authors may request individual patient data from the investigators of individual studies
  • individual patient data may be combined across studies
  • compares outcomes for subgroups, such as:
    • age
    • severity of illness.

• more confident if results from independent studies are:
  • Qualitatively similar (all showing positive effect, or no effect)
• Systemic reviews identify variation between
  • sample groups
  • timing
  • duration
  • intensity of intervention
• statistical approaches include
  • Fixed effects models (Ciliska, 2001 p139-2)
  • Random effects models (Ciliska, 2001 p139-2)
• Evaluating if study findings are similar across studies in decided using a forest plot figure, which compares:
  • how similar the points estimates (RR, OR, mean diff) are across studies
  • how much overlap between the CI's
  • ** a high degree of overlap of CI's indicates that studies are generally finding the same magnitude of point estimate (RR, OR, mean difference)

    ---

• Was a test of heterogeneity done? (homogeneity)
  • this evaluates variability in results of primary studies
  • if significant, means hi variation
  • if non-signif, means a fixed effects model can be used

    ---

• Questions to asses if appropriate to group study findings statistically
  • similar populations?
  • similar interventions?
  • Similar study method
  • similar outcomes

• Meta-analysis > assign different weight to studies with greater precision and higher quality > greater contribution to summary estimate (Ciliska, 2001 p139)
• examine the RR and OR for dichotomous variables (pooled estimate)
• examine  mean difference score for continuous variables
• Need to examine the CI's
  • the CI's are important when considering the intervention - must consider te upper & lower limits
• Graphic Display of Meta-Analysis results (in Cochrane reviews)
  • box indicates the summary stat (for ex, the RR)
  • horizontal line = the 95% confidence interval around the RR
    • RR of 1 means > no diff in event rates of the tx and control groups for tx failure or relaps; the risks are the same
  • RR on L side of vertical line = fewer relapses in Tx group
  • RR on R side of vertical line = fewer relaps in ctrl group
  • If boxes on both sides of 1 and non-overlapping CI’s >
    • Less confidence apply results of review in practice
  • Because the CI does not cross the verticle line (not incl “1”) >
  • This difference is clinically significant
  • Combining all studies = “overall summary statistic”
    • Noted as a ‘diamond’ (also encompasses the 95% CI)
    • If edges of diamond do NOT cross or touch ‘1’ >
      • So, clinically significant diff in outcome
    • When outcomes are dichotomous (dry or wet) >
      • Meta-analysis uses RR or odds rations as summary stats
    • When outcomes are continuous (eg, weight)
      •  Mean effect size OR mean difference is the summary statistic
    • Each statistic may be weighted or unweighted
    • When mean effect size or mean difference is reported>
      • The vertical line of no difference = ‘0’ (not 1)
  • Treatment n/N column
    • Gives the # in Tx group who had outcome of interest (treatment failure or relapse) (n) out of the total # in the treatment group (alarm) group (N)
  • Control n/N column
    • Gives same info as above, but for control group
  • Weight column
    • Tells how much the study contributed to the overall summary statistic (more weight to greater precision)
  • Relative Risk (fixed) 95% CI
    • Tells the RRs and accompanying 95% CIs for each individual study corresponding to the box and horizontal line for that study
  • Use estimates to calculate other summary statistics:
    • Relative risk reduction (RRR)
    • Number needed to treat 

** results may also be described as a narrative (non statistical summary)

• CI's need to be examined to determine the precision of the treatment effect
• CI’s around the RR and the RRR indicate
  • The precision of the estimate of true treatment effect
  • Wide CIs = less precision in the estimate
• The 95% CI = range within we are 95% sure the true value is
• The CI for the summary RR is fairly narrow (0.46 to 0.68)
  • CI helps decision making because
    • We can look at the limit closest to 1 (no effect)
    • And ask “if effect this small > is worthwhile?
• Do you want to impliment, considering:
  • Cost?
  • Inconvenience?
  • Side effects?

• Determine characteristics of Ss in a specific trial
• Why might the results not be applicable to your patients?
• SR should describe the intervention so reader can recreate experiment

• Is it clinically feasible r/t
  • Ability of care providers to recommend equipment
  • Ability of parents to either buy the equipment or have it provided to them?
• Would I make the same decision about implementing a program at either end of the CI?
  • more important that thinking only about width of CI

• Considerations
  • Mortality
  • Morbidity
  • Quality of life
  • Patient satisfaction
  • Cost-effectiveness

• Are benefits of Tx worth cost and potential s/e?
• Cost-benefit considerations will figure into decision-making