Interrater reliability: the kappa statistic

A literature search identified 62 studies published by August 1992 comparing Papanicolaou (Pap) test results with histology. Critical appraisal revealed that 82% of these had potential for verification bias and that only 37% stated that cytology and histology were independently assessed. Estimates of sensitivity and specificity ranged from 11 to 99% and 14 to 97%, respectively, and were highly negatively correlated (r = -0.63). Meta-analysis was used to combine data from 59 studies to estimate the accuracy of the Pap test using a summary receiver operating characteristic curve and to examine the effect of study quality. The summary receiver operating characteristic curve suggests that the Pap test may be unable to achieve concurrently high sensitivity and specificity. For example, specificity in the 90-95% range corresponds to sensitivity in the 20-35% range. Pap test accuracy was not associated with reported study characteristics or dimensions of quality. Future primary studies should pay more attention to methodologic standards for the conduct and reporting of diagnostic test evaluations.