Evaluating implementation of the emergency severity index in a Belgian hospital.

INTRODUCTION: Triage aims to categorize patients based on their clinical need and the available departmental resources. To accomplish this goal, one needs to ensure that the implemented triage system is reliable and that staff use it correctly. Therefore this study assessed the ability of Belgium nurses to apply the Emergency Severity Index (ESI), version 4, to hypothetical case scenarios after an educational intervention.

METHODS: An ESI educational intervention was implemented in accordance with the ESI manual. Using paper case scenarios, nurses' interrater agreement was assessed by comparing triage nurse ESI levels with the reference answers noted in the implementation manual. Interrater agreement was measured by the percentage of agreement and Cohen's κ coefficient using different weighting schemes.

RESULTS: Overall, 77.5% of the scenario cases were coded according the ESI guidelines, resulting in a good interrater agreement (κ = 0.72, linear weighted κ = 0.84, quadratic weighted κ = 0.92, and triage-weighted scheme = 0.79). Interrater agreement varied when evaluating each ESI level separately. Undertriage was more common than overtriage. The highest misclassification range (37.8%) occurred in ESI level 2 scenarios, with 99.2% of the misclassifications being undertriaged.

DISCUSSION: Implementation of the ESI into a novel setting guided by a locally developed training program resulted in suboptimal interrater agreement. Existing weighted κ schemes overestimated the interrater agreement between the triage nurse-assigned ESI level and the reference standard. By providing an aggregated measure of agreement, which allows partial agreement, clinically significant misclassification was masked by a misleading "good" interrater agreement.