The consistency of panelists' appropriateness ratings: do experts produce clinically logical scores for rectal cancer treatment?

OBJECTIVE: To quantify the clinical consistency of expert panelists' ratings of appropriateness of pre-operative and post-operative chemotherapy plus radiation for rectal cancer.

METHODS: A panel of nine physicians (two surgeons, four medical oncologists, three radiation oncologists) rated the appropriateness of providing pre-operative and post-operative treatments for rectal cancer, utilizing a modified-Delphi (RAND/UCLA) approach. Clinical scenarios were paired so that each component of a pair differed by only one clinical feature (e.g. tumor stage). A pair of appropriateness ratings was defined as inconsistent when the clinical scenario that should have had the higher (or at least equal) appropriateness rating was given a lower rating. The rate of inconsistency was analyzed for panelists' ratings of pre- and post-operative chemotherapy plus radiation.

RESULTS: The final panel rating was inconsistent for 1.19% of pre-operative scenario pairs, and 0.77% of post-operative scenario pairs. Using the conventional RAND/UCLA definition of appropriateness, the magnitude of the inconsistency would produce inconsistent appropriateness ratings in 0.43% of pre-operative and 0.11% of post-operative scenario pairs. There was significant variation in the rate of inconsistency among individual panelists' final ratings of both pre-operative (range: 0.43-5.17%, P < 0.001) and post-operative (range: 0.51-2.34%, P < 0.001) scenarios. Panelists' overall average rate of inconsistency improved significantly after the panel meeting and discussion (from 5.62 to 2.25% for pre-operative scenarios, and from 1.47 to 1.24% for post-operative scenarios, both P < 0.05). There was no clear difference between specialty groups. Inconsistency was related to the structure of the rating manual: in the second round there were no inconsistent ratings when scenario pairs occurred on the same page of the manual.

CONCLUSIONS: The RAND/UCLA appropriateness method can produce ratings for cancer treatment that are highly clinically consistent. Modifications to the structure of rating manuals to facilitate direct assessment of consistency at the time of rating may reduce inconsistency further.