Assessing keel bone damage in laying hens by palpation: effects of assessor experience on accuracy, inter-rater agreement and intra-rater consistency

Accurate assessment is essential when evaluating keel bone damage. Palpation is commonly used to assess keel bone damage in living hens. However, there is little information on the accuracy of assessment of deviations and fractures on different parts of the keel, and on the consistency within, and agreement between, assessors. Crucially, although the importance of experience is commonly emphasized, knowledge on its effect is scarce. Ten assessors with or without prior experience palpated the same 50 75-wk-old hens for deviations, medial fractures, and caudal fractures (scored as present/absent). Accuracy, sensitivity, specificity, precision, and negative predictive value were determined by comparing palpation scores to post-dissection assessment, and then compared between experienced and inexperienced assessors. To determine the effect of the experience gained during the experiment, hens were subsequently re-assessed. Consistency within, and agreement between, assessors were also determined. Assessors with prior experience were more accurate (proportion of accurately assessed deviations: experienced 0.83 vs. inexperienced 0.79±0.01, P = 0.04; medial fractures: 0.82 vs. 0.68±0.03 in session 1 only, P = 0.04; caudal fractures: 0.41 vs. 0.29±0.03, P = 0.03), and inexperienced assessors classified medial fractures more accurately in session 2 (session 1: 0.68 vs. session 2: 0.77±0.04, P = 0.04). However, effect sizes were small for deviations and even experienced assessors lacked accuracy when assessing caudal fractures. Unexpectedly, deviations tended to be assessed more accurately in session 1 than in session 2, regardless of assessor status (1: 0.83 vs. 2: 0.79±0.01, P = 0.06), suggesting that prolonged assessment contributes to errors. Prior experience decreased specificity and precision of fracture assessment (more unfractured keels were classified as fractured) even though overall accuracy was greater. Intra-rater consistency was fair to good (0.55 to 0.67) for deviations and medial fractures, but poor to fair (0.36 to 0.44) for caudal fractures, and unaffected by prior experience (P = 0.49 to 0.89). In conclusion, experience improves accuracy to a limited extent but does not guarantee high accuracy for all types of damage. Future research should determine if other training methods (e.g., comparison to post-dissection scores or to radiographs) improve accuracy.