Do the findings of case series studies vary significantly according to methodological characteristics?

OBJECTIVES: To review the use of case series in National Institute for Clinical Excellence (NICE) Health Technology Assessment (HTA) reports, to review systematically the methodological literature for papers relating to the validity of aspects of case series design, and to investigate characteristics and findings of case series using examples from the UK's Health Technology Assessment programme.

DATA SOURCES: Electronic databases. NICE website. Reports produced as part of the UK's HTA programme.

REVIEW METHODS: NICE HTAs that used information from case series studies were obtained from the NICE website and a range of quality criteria applied. Searches of electronic databases, handsearched journals and the bibliographies of papers were made in order to find studies that assessed aspects of case series design, analysis or quality in relation to study validity. Hypotheses relating to the design of case series studies were developed and empirically investigated using four case examples from existing reports produced as part of the UK's HTA programme (functional endoscopic sinus surgery for nasal polyps, spinal cord stimulation for chronic back pain, percutaneous transluminal coronary angioplasty and coronary artery bypass grafting for chronic angina). Analysis was undertaken comparing studies within each review.

RESULTS: There was no consensus on which case series to include in HTAs, how to use them or how to assess their quality, despite them being used in 30% of NICE HTAs. No previous studies empirically investigating methodological characteristics of case series were found. However, it is possible that the search strategy failed to find relevant studies. Poor reporting of case series characteristics severely constrained analysis and there were insufficient data to investigate all the hypotheses. Findings were not consistent across the different topics and were subject to considerable uncertainty. All the examples in our analysis were surgical interventions, which are prone to additional confounding factors due to difficulties of standardisation compared with drug treatment. Our findings may not be generalisable outside the interventions studied. The case series reports included generally exhibited poor reporting of methodological characteristics. This constrained our analysis. The use of several methods of analysis has led to apparently discrepant results. Given the number of analysis performed, the usual level of significance (p = 0.05) should be viewed with caution. The most important limitation of this study is the small number of cases on which the findings are based. The results are therefore tentative and should be viewed with caution.

CONCLUSIONS: Case series are incorporated in a significant proportion of health technology assessments. Quality criteria have been used to appraise the quality of case series and decide on their inclusion in reviews of studies using this design. In this small series of case studies drawn from HTAs carried out for the NHS HTA programme, little evidence was found to support the use of many of the factors included in quality assessment tools. Importantly, no relationship was found between study size and outcome across the four examples studied. Isolated examples of a potentially important relationship between other methodological factors and outcome were shown, such as blinding of outcome measurement, but these were not shown consistently across the small number of examples studied. This study is based on a very small sample of studies and should therefore be considered as exploratory. Further investigation of the relationship between methodological features and outcome is justified given the frequency of use of case series in health technology assessments. Further research into the methodological features of case series and their outcome is justified in a wider sample of technologies and larger sets of case series. Value of information analyses including case series could be explored. Further exploration of the differences between case series and randomised controlled trial results, preferably using registry or comprehensive case series data, would be valuable.