The consistency of panelists' appropriateness ratings: do experts produce clinically logical scores for rectal cancer treatment?

David C Hodgson, James D Brierley, Geta Cernat, Susan Bondy, Pamela M Slaughter, S Patricia Pinfold, Lawrence F Paszat
Health Policy 2005, 71 (1): 57-65

OBJECTIVE: To quantify the clinical consistency of expert panelists' ratings of appropriateness of pre-operative and post-operative chemotherapy plus radiation for rectal cancer.

METHODS: A panel of nine physicians (two surgeons, four medical oncologists, three radiation oncologists) rated the appropriateness of providing pre-operative and post-operative treatments for rectal cancer, utilizing a modified-Delphi (RAND/UCLA) approach. Clinical scenarios were paired so that each component of a pair differed by only one clinical feature (e.g. tumor stage). A pair of appropriateness ratings was defined as inconsistent when the clinical scenario that should have had the higher (or at least equal) appropriateness rating was given a lower rating. The rate of inconsistency was analyzed for panelists' ratings of pre- and post-operative chemotherapy plus radiation.

RESULTS: The final panel rating was inconsistent for 1.19% of pre-operative scenario pairs, and 0.77% of post-operative scenario pairs. Using the conventional RAND/UCLA definition of appropriateness, the magnitude of the inconsistency would produce inconsistent appropriateness ratings in 0.43% of pre-operative and 0.11% of post-operative scenario pairs. There was significant variation in the rate of inconsistency among individual panelists' final ratings of both pre-operative (range: 0.43-5.17%, P < 0.001) and post-operative (range: 0.51-2.34%, P < 0.001) scenarios. Panelists' overall average rate of inconsistency improved significantly after the panel meeting and discussion (from 5.62 to 2.25% for pre-operative scenarios, and from 1.47 to 1.24% for post-operative scenarios, both P < 0.05). There was no clear difference between specialty groups. Inconsistency was related to the structure of the rating manual: in the second round there were no inconsistent ratings when scenario pairs occurred on the same page of the manual.

CONCLUSIONS: The RAND/UCLA appropriateness method can produce ratings for cancer treatment that are highly clinically consistent. Modifications to the structure of rating manuals to facilitate direct assessment of consistency at the time of rating may reduce inconsistency further.

Full Text Links

Find Full Text Links for this Article


You are not logged in. Sign Up or Log In to join the discussion.

Related Papers

Remove bar
Read by QxMD icon Read

Save your favorite articles in one place with a free QxMD account.


Search Tips

Use Boolean operators: AND/OR

diabetic AND foot
diabetes OR diabetic

Exclude a word using the 'minus' sign

Virchow -triad

Use Parentheses

water AND (cup OR glass)

Add an asterisk (*) at end of a word to include word stems

Neuro* will search for Neurology, Neuroscientist, Neurological, and so on

Use quotes to search for an exact phrase

"primary prevention of cancer"
(heart or cardiac or cardio*) AND arrest -"American Heart Association"