NPC Archive Item: NPC rapid reviews now indicate the level of evidence

NOTE – This is an archive post from the NPC and has not been updated since first publication. Therefore, some hyperlinks may no longer be working.

1st June 2009

From 1st June 2009 MeReC Rapid Reviews will indicate the level of evidence of the clinical study we report using the SORT criteria. We hope that this will enhance readers’ assessment of the usefulness of studies.

A number of grading systems for clinical evidence are in use, each with their own strengths and weaknesses. We have chosen to use the SORT system (Strength Of Recommendation Taxonomy) because this is highly patient centred, simple to understand and embodies the principles of evidence-informed decision making.

The SORT classification
The SORT system grades the level of a piece of evidence as 1, 2 or 3, with 1 being the highest level. The grading depends on the quality of the evidence, taking into account not only the design and methodological quality but also the nature of the research question and the type of outcome studied.

However well conducted the study is, all studies which consider Disease Oriented Outcomes (DOOs) are automatically graded as level 3. DOOs include physiological or surrogate outcomes, such as changes in blood pressure, blood lipid levels, peak expiratory flow rates, etc. Although DOOs can be useful indicators and are important in drug development, they can also mislead and do not always relate to Patient Oriented Outcomes (POOs)

POOs are outcomes that matter directly to patients: direct evidence as to whether or not the intervention is likely to help them live longer or live better. Thus POOs are morbidity, mortality, symptom improvement, cost reduction and/or quality of life. The differences between POOs and DOOs are explored in more detail on the evidence-informed decision making section of the NPC website.

The table below shows examples of how DOOs may not always relate to POOs and is taken from the paper describing the SORT classification.

Table 1
Examples of Inconsistency Between Disease-Oriented and Patient-Oriented Outcomes
Disease or condition	Disease-oriented outcome	Patient-oriented outcome
Doxazosin for blood pressure¹²	Reduces blood pressure in blacks	Increases mortality
Lidocaine for arrhythmia following acute myocardial infarction¹³	Suppresses arrhythmias	Increases mortality
Finasteride for benign prostatic hypertrophy¹⁴	Improves urinary flow rate	No clinically important change in symptom scores
Arthroscopic surgery for osteoarthritis of the knee¹⁵	Improves appearance of cartilage after débridement	No change in function or symptoms at one year
Sleeping infants on their stomach or side¹⁶	Knowledge of anatomy and physiology suggests that this will decrease the risk of aspiration	Increases risk of sudden infant death syndrome
Vitamin E for heart disease¹⁷	Reduces levels of free radicals	No change in mortality
Histamine antagonists and proton-pump inhibitors for nonulcer dyspepsia¹⁸	Significantly reduce gastric pH levels	Little or no improvement in symptoms in patients with nongastroesophageal reflux disease, nonulcer dyspepsia
Hormone therapy¹⁹	Reduces low-density lipoprotein cholesterol levels, increases high-density lipoprotein cholesterol	No decrease in cardiovascular or all-cause mortality and an increase in cardiovascular events in women older than 60 years (Women’s Health Initiative) with combined hormone therapy
Insulin therapy in type 2 diabetes	Keeps blood glucose levels below mellitus²⁰120 mg per dL (6.7 mmol per L)	Does not reduce overall mortality
Sodium fluoride for fracture prevention²¹	Increases bone density	Does not reduce fracture rate
Lidocaine prophylaxis following acute myocardial infarction²²	Suppresses arrhythmias	Increases mortality
Clofibrate for hyperlipidemia²³	Reduces lipid levels	Does not reduce mortality
Beta blockers for heart failure²⁴	Reduce cardiac output	Reduce mortality in moderate to severe disease
Note: see paper for references for these data

For studies considering treatments for prevention, treatment or screening (the majority of studies we rapid review), the SORT criteria are:

Level 1: good quality patient-oriented evidence

Systematic reviews (SRs) or meta-analyses (MAs) of randomised controlled trials (RCTs) with consistent findings* related to POOs
High quality** individual RCTs which examine POOs
All or none studies***

Level 2: limited quality patient-oriented evidence

SRs or MAs of lower-quality studies related to POOs or of RCTs with inconsistent findings which examine POOs
Lower-quality clinical trials which examine POOs
Cohort studies which examine POOs
Case-control studies which examine POOs

Level 3: other evidence

Any DOO study
Case series
Opinion
Consensus guidelines
Modelling from bench research

*Consistency is seen when most studies in the SR or MA found similar or at least coherent conclusions; coherence means that differences are explainable

**To be categorised as high quality, an RCT must meet the following criteria: allocation concealment, blinding if possible, intention-to-treat analysis, adequate statistical power, adequate follow-up (greater than 80 percent).

***All or none studies are defined as being when the treatment causes a dramatic change in outcomes, such as antibiotics for meningitis or surgery for appendicitis, which precludes study in a controlled trial.

Using the level of evidence
It is important to apply this hierarchy of quality sensibly. One should make decisions on the best level of evidence available, so for example, the several level 1 RCTs which showed limited benefit or even harm from tight glucose control in type 2 diabetes (VADT, ACCORD and ADVANCE) carry more weight than the level 2 observational follow-up study to UKPDS.

However, level 1 evidence is not always available. In its absence, we may well have to look at level 2 or even level 3 evidence. In doing so we should be progressively more circumspect in applying it, recognising the degree to which it may be partial or even erroneous. There may be good reasons why RCT data are not available: for example there are no RCT data that smoking substantially increases the risk of lung cancer yet it would clearly be unethical to conduct such an RCT. By contrast, early level 2 and level 3 data suggested cardiovascular benefits from hormone replacement therapy (HRT). Later RCTs showed that this was erroneous.

More information on assessing clinical studies can be found on the evidence-informed decision making section of the NPC website.

Feedback
Please comment on this rapid review using our feedback form.

Make sure you are signed up to NPC Email updates — the free email alerting system that keeps you up to date with the NPC news and outputs relevant to you.