Speaker
Description
Classification tasks where raters have to classify objects into predefined classes have been gaining popularity in science. Therefore, accuracy and reliability of classification are two necessary conditions. Any classification process is partly based on an evidence-based evaluation and partly on the subjective evaluation of the rater. This last component is crucial for the reliability of the classification. Traditional indexes used to estimate inter-rater agreement typically count the number of observed agreements and correct them by removing chance agreements. In a recent paper Nucci and colleagues introduced a methodology for improving the estimation of interrater agreement in a classification task in the case of two independent raters. Their contribution is based on the definition the Belonging Measure, and the Belonging Threshold of the raters. In this research we study, by means of a number of Monte Carlo simulation studies, the performance of the new methodology in estimating the agreement among multiple raters. Results confirm the accuracy of the new methodology even in the case of multiple raters. Moreover, it is shown that when differences among the Belonging Thresholds increase, traditional inter-rater agreement indexes are biased, while this is not the case for the indexes computed using the new methodology. Future research perspectives are then discussed.
