Is the script concordance test a valid instrument for assessment of intraoperative decision-making skills?

BACKGROUND: Intraoperative decision making requires both knowledge and experience. The script concordance test (SCT), based on cognitive psychology script theory, is a new tool of clinical-reasoning assessment that may be used to evaluate a candidate's approach to ill-defined problems encountered in the operating room.

METHODS: To develop and validate an SCT for assessment of intraoperative decision making. One hundred questions were prepared based on the objectives for residency training of the American Board of Surgery. These questions were reviewed for face and content validity by 3 board-certified general surgeons. The SCT was administered to 36 general surgical residents ranging from R1 to R5. The scoring grid was obtained by giving the test to 10 board-certified general surgeons who completed the test independently. Aggregate scoring was used. After question optimization, the final test used for statistical analysis was composed of 62 questions.

RESULTS: The test had excellent reliability (Cronbach alpha, .85). Scores increased with higher levels of training except for a small decrease in the R5 scores (R1, 52.5 +/- 9.9; R2, 62.4 +/- 5.1; R3, 68.3 +/- 9.2; R4, 75.7 +/- 9.6; R5, 68 +/- 6.4) There was a significant difference in scores between the junior (R1 + R2) and senior (R3 + R4 + R5) residents: 56.8 +/- 9.5 versus 70.2 +/- 8.8 (P < .0001).

CONCLUSIONS: SCT, applied to the assessment of intraoperative clinical judgment, can discriminate successfully between junior and senior residents. Results from an SCT test must be compared with the present gold standard, the oral examination, to better determine its place as an assessment tool.