NPC Archive Item: NPC rapid reviews now indicate the level of evidence

NOTE – This is an archive post from the NPC and has not been updated since first publication. Therefore, some hyperlinks may no longer be working.
merec_stop_press NPC Logo

1st June 2009

From 1st June 2009 MeReC Rapid Reviews will indicate the level of evidence of the clinical study we report using the SORT criteria. We hope that this will enhance readers’ assessment of the usefulness of studies.

A number of grading systems for clinical evidence are in use, each with their own strengths and weaknesses. We have chosen to use the SORT system (Strength Of Recommendation Taxonomy) because this is highly patient centred, simple to understand and embodies the principles of evidence-informed decision making.

The SORT classification
The SORT system grades the level of a piece of evidence as 1, 2 or 3, with 1 being the highest level. The grading depends on the quality of the evidence, taking into account not only the design and methodological quality but also the nature of the research question and the type of outcome studied.

However well conducted the study is, all studies which consider Disease Oriented Outcomes (DOOs) are automatically graded as level 3. DOOs include physiological or surrogate outcomes, such as changes in blood pressure, blood lipid levels, peak expiratory flow rates, etc. Although DOOs can be useful indicators and are important in drug development, they can also mislead and do not always relate to Patient Oriented Outcomes (POOs)

POOs are outcomes that matter directly to patients: direct evidence as to whether or not the intervention is likely to help them live longer or live better. Thus POOs are morbidity, mortality, symptom improvement, cost reduction and/or quality of life. The differences between POOs and DOOs are explored in more detail on the evidence-informed decision making section of the NPC website.

The table below shows examples of how DOOs may not always relate to POOs and is taken from the paper describing the SORT classification.

Table 1
Examples of Inconsistency Between Disease-Oriented and Patient-Oriented Outcomes

Disease or condition

Disease-oriented outcome

Patient-oriented outcome

Doxazosin for blood pressure12

Reduces blood pressure in blacks

Increases mortality

Lidocaine for arrhythmia following acute myocardial infarction13

Suppresses arrhythmias

Increases mortality

Finasteride for benign prostatic hypertrophy14

Improves urinary flow rate

No clinically important change in symptom scores

Arthroscopic surgery for osteoarthritis of the knee15

Improves appearance of cartilage after débridement

No change in function or symptoms at one year

Sleeping infants on their stomach or side16

Knowledge of anatomy and physiology suggests that this will decrease the risk of aspiration

Increases risk of sudden infant death syndrome

Vitamin E for heart disease17

Reduces levels of free radicals

No change in mortality

Histamine antagonists and proton-pump inhibitors for nonulcer dyspepsia18

Significantly reduce gastric pH levels

Little or no improvement in symptoms in patients with nongastroesophageal reflux disease, nonulcer dyspepsia

Hormone therapy19

Reduces low-density lipoprotein cholesterol levels, increases high-density lipoprotein cholesterol

No decrease in cardiovascular or all-cause mortality and an increase in cardiovascular events in women older than 60 years (Women’s Health Initiative) with combined hormone therapy

Insulin therapy in type 2 diabetes

Keeps blood glucose levels below mellitus20120 mg per dL (6.7 mmol per L)

Does not reduce overall mortality

Sodium fluoride for fracture prevention21

Increases bone density

Does not reduce fracture rate

Lidocaine prophylaxis following acute myocardial infarction22

Suppresses arrhythmias

Increases mortality

Clofibrate for hyperlipidemia23

Reduces lipid levels

Does not reduce mortality

Beta blockers for heart failure24

Reduce cardiac output

Reduce mortality in moderate to severe disease

Note: see paper for references for these data

For studies considering treatments for prevention, treatment or screening (the majority of studies we rapid review), the SORT criteria are:

Level 1: good quality patient-oriented evidence

Level 2: limited quality patient-oriented evidence

  • SRs or MAs of lower-quality studies related to POOs or of RCTs with inconsistent findings which examine POOs
  • Lower-quality clinical trials which examine POOs
  • Cohort studies which examine POOs
  • Case-control studies which examine POOs

Level 3: other evidence

  • Any DOO study
  • Case series
  • Opinion
  • Consensus guidelines
  • Modelling from bench research

*Consistency is seen when most studies in the SR or MA found similar or at least coherent conclusions; coherence means that differences are explainable

**To be categorised as high quality, an RCT must meet the following criteria: allocation concealment, blinding if possible, intention-to-treat analysis, adequate statistical power, adequate follow-up (greater than 80 percent).

***All or none studies are defined as being when the treatment causes a dramatic change in outcomes, such as antibiotics for meningitis or surgery for appendicitis, which precludes study in a controlled trial.

Using the level of evidence
It is important to apply this hierarchy of quality sensibly. One should make decisions on the best level of evidence available, so for example, the several level 1 RCTs which showed limited benefit or even harm from tight glucose control in type 2 diabetes (VADT, ACCORD and ADVANCE) carry more weight than the level 2 observational follow-up study to UKPDS.

However, level 1 evidence is not always available. In its absence, we may well have to look at level 2 or even level 3 evidence. In doing so we should be progressively more circumspect in applying it, recognising the degree to which it may be partial or even erroneous. There may be good reasons why RCT data are not available: for example there are no RCT data that smoking substantially increases the risk of lung cancer yet it would clearly be unethical to conduct such an RCT. By contrast, early level 2 and level 3 data suggested cardiovascular benefits from hormone replacement therapy (HRT). Later RCTs showed that this was erroneous.

More information on assessing clinical studies can be found on the evidence-informed decision making section of the NPC website.

Please comment on this rapid review using our feedback form.

Make sure you are signed up to NPC Email updates — the free email alerting system that keeps you up to date with the NPC news and outputs relevant to you.