Training to Increase Rater Reliability When Assessing the Quality of Ethics Consultation Records with the Ethics Consultation Quality Assessment Tool (ECQAT)

Robert Allan Pearlman, David Alfandre, Barbara L Chanko, Mary Beth Foglia, Kenneth A Berkowitz
Journal of Clinical Ethics 2018, 29 (4): 276-284
The Ethics Consultation Quality Assessment Tool (ECQAT) establishes standards by which the quality of ethics consultation records (ECRs) can be assessed. These standards relate to the ethics question, consultation-specific information, ethical analysis, and recommendations and/or conclusions, and result in a score associated with one of four levels of ethics consultation quality. For the ECQAT to be useful in assessing and improving the quality of healthcare ethics consultations, individuals who rate the quality of ECRs need to be able to reliably use the tool. We developed a short course to train ethics consultants in using the ECQAT, and evaluated whether the participants (1) achieved an acceptable level of calibration in matching expert-established quality scores for a set of ethics consultations, and (2) were satisfied with the course. We recruited 28 ethics consultants to participate in a virtual, six-session course. At each session participants and faculty reviewed, rated, and discussed one to two ECRs. The participants' calibration in matching expert-established quality scores improved with repeated exposure at all levels of ethics consultation quality. Participants were generally more accurate when assessing consultation quality at the dichotomous level of "acceptable" (scores of three or four) versus "unacceptable" (scores of one or two) than they were with more a specific score. Participants had higher rates of accuracy with the extreme ratings of "strong" (level four) or "poor" (level one). Although participants were highly satisfied with the course, only a minority of participants achieved the prespecified acceptable level of calibration (that is, 80 percent or greater accuracy between their score and expert-established scores). These results suggest that ECQAT training may require more sessions or need modification in the protocol to achieve higher reliability in scoring. Such trainings are an important next step in ensuring that the ECQAT is a tool that can be used to promote improvement in ethics consultation quality.

Full Text Links

Find Full Text Links for this Article


You are not logged in. Sign Up or Log In to join the discussion.

Related Papers

Available on the App Store

Available on the Play Store
Remove bar
Read by QxMD icon Read

Search Tips

Use Boolean operators: AND/OR

diabetic AND foot
diabetes OR diabetic

Exclude a word using the 'minus' sign

Virchow -triad

Use Parentheses

water AND (cup OR glass)

Add an asterisk (*) at end of a word to include word stems

Neuro* will search for Neurology, Neuroscientist, Neurological, and so on

Use quotes to search for an exact phrase

"primary prevention of cancer"
(heart or cardiac or cardio*) AND arrest -"American Heart Association"