Definition of Quality of a Trial (or Study) The confidence that the trial design, conduct, and analysis has minimized or avoided biases in its treatme

5610 I Critical Appraisal 1

Definition of Quality of a Trial (or Study) The confidence that the trial design, conduct, and analysis has minimized or avoided biases in its treatment comparison. Moher et al Controlled Clin Trials 1995;16:62-73 2

Assessing the quality of controlled clinical trials (BMJ 2001;323:42-6) Internal validity : extent to which systematic error (bias) is minimised in clinical trials Selection bias : : biased allocation to comparison groups Performance bias : : unequal provision of care apart from treatment under evaluation Detection bias : : biased assessment of outcome Attrition bias : : biased occurrence and handling of deviations from protocol and loss to follow up External validity: extent to which results of trials provide a correct basis for generalisation to other circumstances Patients: age, sex, severity of disease and risk factors, comorbidity Treatment regimens: dosage, timing and route of administration, type of treatment within a class of treatments, concomitant treatments Settings: level of care (primary to tertiary) and experience and specialisation of care provider Modalities of outcomes: type or definition of outcomes and duration 3 of follow up

(validity) 4

Summary points (BMJ 2001;323:42-6) Empirical studies show that inadequate quality of trials may distort the results from systematic reviews and meta-analyses The influence of the quality of included studies should routinely be examined in systematic reviews and meta- analyses The use of summary scores from quality scales is problematic it is preferable to examine the influence of key components of methodological quality individually Based on empirical evidence and theoretical considerations, the generation and concealment of the allocation sequence, blinding,, and handling of patient attrition in the analysis should always be assessed 5

6 Jadad Jadad Control Control Clin Clin Trials. 1996;17:1-12. Trials. 1996;17:1-12.

(observational study) (cross-sectional study) (case-control study) (cohort study) (intervention study) (clinical trial) (field trial) Population/Community-based intervention trial 7

CASP (Critical Appraisal Skills Programme,, UK) http://caspjp.umin.ac.jp/pages/home.html, http://www.phru.nhs.uk/casp/casp.htm 8

9 Intention-to-treat analysis Intention-to-treat analysis(1) (1) 200 200 A B

Intention-to-treat analysis(2) A75 B 10

Intention-to-treat analysis(3) ITT ITT Principle 1. AB 2. A 3. 11

ITT Full Analysis Set FAS Intention-to-treat 1047 10 101130 12

CONSORT Standard of Reporting Trials (SORT) 993.3 Asilomar Working Group on Recommendations for Reporting of Clinical Trials in the Biomedical Literature 1994.3 995.9 CONSORT statement 1996 001 001 Lancet 2001;357:1191-1194, Annals of Internal Medicine 2001;134:657-662, JAMA 2001;285:1987-1991 http://www.consort-statement.org/ http://www.sphere.ad.jp/cont/consort_statement/menu.html 13

Participants health care providers data collectors judicial assessors of outcomes data analysts personnel who write the article. Etc. ( Devereaux PJ, et al. Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials. JAMA. 2001;285(15):2000-3. ) 14

Blind Detection bias Performance bias co-intervention confounding factor 15

16

PROBE Prospectively Randomized, Open, Blinded Endpoint PROBE PROBE Detection bias detect detect report 17

Composite outcomes Freemantle et al. Composite outcomes in randomized trials: greater precision but with greater uncertainty? JAMA. 2003;289(19):2554-9. 18

Blinding / Masking Concealment Blinding / / Masking (observation / measurement bias) Concealment 19

Concealment 20

Clinical Relevance 21

1. Total mortality (or overall survival from a defined point in time) 2. Cause-specific mortality (or cause-specific mortality from a defined point in time) 3. Carefully assessed quality of life 4. Indirect surrogates Disease-free survival, Progression-free survival, Tumor response rate http://www.cancer.gov/cancerinfo/pdq/levels-evidence-adult-treatment 22

Barnes DE, Bero LA. Why review articles on the health effects of passive smoking reach different conclusions. JAMA. 1998;279:1566-1570 106 106 39 39 29 74% 74% Grant / Sponsorship bias 23

1. Selection bias random allocation (assignment) 2. ObservationMeasurement Measurementbias blinding 3. Analysis bias intention-to-treat analysis / full analysis set (FAS) analysis 4. Publication bias registry of clinical trials (EBM2001) 24