69
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: not found

      Assessing Agreement between Multiple Raters with Missing Rating Information, Applied to Breast Cancer Tumour Grading

      research-article

      Read this article at

      ScienceOpenPublisherPMC
      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Background

          We consider the problem of assessing inter-rater agreement when there are missing data and a large number of raters. Previous studies have shown only ‘moderate’ agreement between pathologists in grading breast cancer tumour specimens. We analyse a large but incomplete data-set consisting of 24177 grades, on a discrete 1–3 scale, provided by 732 pathologists for 52 samples.

          Methodology/Principal Findings

          We review existing methods for analysing inter-rater agreement for multiple raters and demonstrate two further methods. Firstly, we examine a simple non-chance-corrected agreement score based on the observed proportion of agreements with the consensus for each sample, which makes no allowance for missing data. Secondly, treating grades as lying on a continuous scale representing tumour severity, we use a Bayesian latent trait method to model cumulative probabilities of assigning grade values as functions of the severity and clarity of the tumour and of rater-specific parameters representing boundaries between grades 1–2 and 2–3. We simulate from the fitted model to estimate, for each rater, the probability of agreement with the majority. Both methods suggest that there are differences between raters in terms of rating behaviour, most often caused by consistent over- or under-estimation of the grade boundaries, and also considerable variability in the distribution of grades assigned to many individual samples. The Bayesian model addresses the tendency of the agreement score to be biased upwards for raters who, by chance, see a relatively ‘easy’ set of samples.

          Conclusions/Significance

          Latent trait models can be adapted to provide novel information about the nature of inter-rater agreement when the number of raters is large and there are missing data. In this large study there is substantial variability between pathologists and uncertainty in the identity of the ‘true’ grade of many of the breast cancer tumours, a fact often ignored in clinical studies.

          Related collections

          Most cited references33

          • Record: found
          • Abstract: not found
          • Book: not found

          Categorical Data Analysis

            Bookmark
            • Record: found
            • Abstract: found
            • Article: not found

            Pathological prognostic factors in breast cancer. I. The value of histological grade in breast cancer: experience from a large study with long-term follow-up.

            Morphological assessment of the degree of differentiation has been shown in numerous studies to provide useful prognostic information in breast cancer, but until recently histological grading has not been accepted as a routine procedure, mainly because of perceived problems with reproducibility and consistency. In the Nottingham/Tenovus Primary Breast Cancer Study the most commonly used method, described by Bloom & Richardson, has been modified in order to make the criteria more objective. The revised technique involves semiquantitative evaluation of three morphological features--the percentage of tubule formation, the degree of nuclear pleomorphism and an accurate mitotic count using a defined field area. A numerical scoring system is used and the overall grade is derived from a summation of individual scores for the three variables: three grades of differentiation are used. Since 1973, over 2200 patients with primary operable breast cancer have been entered into a study of multiple prognostic factors. Histological grade, assessed in 1831 patients, shows a very strong correlation with prognosis; patients with grade I tumours have a significantly better survival than those with grade II and III tumours (P less than 0.0001). These results demonstrate that this method for histological grading provides important prognostic information and, if the grading protocol is followed consistently, reproducible results can be obtained. Histological grade forms part of the multifactorial Nottingham prognostic index, together with tumour size and lymph node stage, which is used to stratify individual patients for appropriate therapy.
              Bookmark
              • Record: found
              • Abstract: not found
              • Book: not found

              R: A language and environment for statistical computing

                Bookmark

                Author and article information

                Contributors
                Role: Editor
                Journal
                PLoS ONE
                plos
                plosone
                PLoS ONE
                Public Library of Science (San Francisco, USA )
                1932-6203
                2008
                13 August 2008
                : 3
                : 8
                : e2925
                Affiliations
                [1 ]Department of Medicine, Lancaster University, Lancaster, United Kingdom
                [2 ]Department of Oncology, Li Ka Shing Centre, University of Cambridge, Cambridge Research Institute, Cambridge, United Kingdom
                [3 ]University of Nottingham, Nottingham, United Kingdom
                [4 ]Division of Pathology, School of Molecular Medical Sciences, University of Nottingham, Nottingham, United Kingdom
                [5 ]Wolfson College, University of Cambridge, Cambridge, United Kingdom
                University of East Piedmont, Italy
                Author notes

                Conceived and designed the experiments: IOE ARG RH. Performed the experiments: IOE ARG RH. Analyzed the data: TF AGL. Wrote the paper: TF AGL.

                Article
                08-PONE-RA-04980R1
                10.1371/journal.pone.0002925
                2488396
                18698346
                009d979c-4169-4b4e-8435-a170facb77dc
                Fanshawe et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
                History
                : 4 June 2008
                : 18 July 2008
                Page count
                Pages: 12
                Categories
                Research Article
                Pathology
                Mathematics/Statistics
                Oncology/Breast Cancer
                Pathology/Histopathology

                Uncategorized
                Uncategorized

                Comments

                Comment on this article

                scite_
                0
                0
                0
                0
                Smart Citations
                0
                0
                0
                0
                Citing PublicationsSupportingMentioningContrasting
                View Citations

                See how this article has been cited at scite.ai

                scite shows how a scientific paper has been cited by providing the context of the citation, a classification describing whether it supports, mentions, or contrasts the cited claim, and a label indicating in which section the citation was made.

                Similar content116

                Cited by8

                Most referenced authors479