Reliability of faculty clinical evaluations of non-emergency medicine residents during emergency department rotations.

OBJECTIVES: To assess the reliability of faculty evaluations of non-emergency medicine (non-EM) residents during clinical ED rotations and to determine the effect that the "leniency" of grading by these evaluators had on the residents' final evaluations.

METHODS: A prospective, observational study of the evaluation patterns of EM faculty was performed in an academic ED (50,000 visits yearly census). Each resident was evaluated on a daily basis by a board-certified or board-prepared emergency physician. The evaluation form rated 7 characteristics, but only the rating for overall clinical competence was used for data analysis. If an attending evaluated the same resident more than once, only the first evaluation was used to avoid bias from prior exposure. The scoring patterns of the evaluators, both individually and in groups, were analyzed using 1-way analysis of variance. Evaluator leniency was estimated using the mean evaluator score across all residents. Since each resident was evaluated by a different combination of evaluators, evaluator leniency for each resident was estimated from the mean leniency of the evaluators who specifically assessed that resident.

RESULTS: During the period of the study, 66 residents rotated through the ED, yielding a total of 401 evaluations. When the scoring patterns of individual evaluators were analyzed, a high degree of variability was found in the mean scores (range 5.23-8.09) and SDs (range 0.45-1.55) across evaluators; p = 0.0001. There was a moderate correlation between the mean overall competence score received by each resident and that resident's evaluators' leniency, r = 0.52; p = 0.0001.

CONCLUSIONS: There is significant variability in the scoring patterns of individual evaluators. The evaluators in this study showed large variations in both leniency (as measured by their mean score) and range restriction (as measured by their SD). The differences in evaluator scoring leniency have a moderate correlation with the overall score received by the resident.