Variation observed in consensus judgements between pairs of reviewers when assessing the risk of bias due to missing evidence in a sample of published meta-analyses of nutrition research.

OBJECTIVE: To evaluate the risk of bias due to missing evidence in a sample of published meta-analyses of nutrition research using the ROB-ME tool and determine inter-rater agreement in assessments.

STUDY DESIGN AND SETTING: We assembled a random sample of 42 meta-analyses of nutrition research. Eight assessors were randomly assigned to one of four pairs. Each pair assessed 21 randomly assigned meta-analyses, and each meta-analysis was assessed by two pairs. We calculated raw percentage agreement and chance corrected agreement using Gwet's Agreement Coefficient (AC) in consensus judgements between pairs.

RESULTS: Across the eight signalling questions in the ROB-ME tool, raw percentage agreement ranged from 52% to 100%, and Gwet's AC ranged from 0.39 to 0.76. For the risk-of-bias judgement, the raw percentage agreement was 76% (95% CI 60% to 92%) and Gwet's AC was 0.47 (95% CI 0.14 to 0.80). In seven (17%) meta-analyses, either one or both pairs judged the risk of bias due to missing evidence as "low risk".

CONCLUSIONS: Our findings indicated substantial variation in assessments in consensus judgements between pairs for the signalling questions and overall risk-of-bias judgements. More tutorials and training are needed to help researchers apply the ROB-ME tool more consistently.