A comparison of the performance of a model based on administrative data and a model based on clinical data: effect of severity of illness on standardized mortality ratios of intensive care units.

OBJECTIVES: It has been postulated that prognostic models based on administrative data can provide valid adjusted mortality rates in specific patient populations. In this study we compared the performance and robustness of a model based on administrative data (customized hospital standardized mortality ratio) and a model based on clinical data (customized Simplified Acute Physiology Score II) in the Dutch intensive care unit population.

DESIGN: Cohort study of intensive care unit records from a national intensive care unit quality registry linked to administrative records from the Dutch National Medical Registration. The hospital standardized mortality ratio and Simplified Acute Physiology Score II models were first-level customized on the intensive care unit population.

SETTING: Fifty-five Dutch intensive care units.

PATIENTS: A total of 66,564 intensive care unit patients admitted from 2005 to 2008.

INTERVENTIONS: None.

MEASUREMENTS AND MAIN RESULTS: Performance expressed by measures of discrimination, accuracy, and calibration (area under the receiver operating characteristic curve, Brier score, Hosmer-Lemeshow Ĉ-statistic, and calibration plots). Additionally, the robustness of the models was assessed by simulating changes in the population's severity of illness and analyzing the effect on the intensive care units' standardized mortality ratios.The area under the receiver operating characteristic curve and Brier score of the customized Simplified Acute Physiology Score II were significantly superior to that of the customized hospital standardized mortality ratio (0.85 and 0.11 vs. 0.77 and 0.13, respectively). Calibration plots showed good agreement between observed and predicted mortality for low-risk patients in both models, with more discrepancy in the high-risk patients when using the customized hospital standardized mortality ratio. Severity of illness had influence on the intensive care units' standardized mortality ratios in both models, but the customized Simplified Acute Physiology Score II showed more robustness.

CONCLUSIONS: The customized Simplified Acute Physiology Score II outperforms the customized hospital standardized mortality ratio in the Dutch intensive care unit population. Comparing institutions based on standardized mortality ratios can be unfavorable for those with a more severely ill intensive care unit population, especially when using the customized hospital standardized mortality ratio.