12
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Continual updating and monitoring of clinical prediction models: time for dynamic prediction systems?

      letter

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Clinical prediction models (CPMs) have become fundamental for risk stratification across healthcare. The CPM pipeline (development, validation, deployment, and impact assessment) is commonly viewed as a one-time activity, with model updating rarely considered and done in a somewhat ad hoc manner. This fails to address the fact that the performance of a CPM worsens over time as natural changes in populations and care pathways occur. CPMs need constant surveillance to maintain adequate predictive performance. Rather than reactively updating a developed CPM once evidence of deteriorated performance accumulates, it is possible to proactively adapt CPMs whenever new data becomes available. Approaches for validation then need to be changed accordingly, making validation a continuous rather than a discrete effort. As such, “living” (dynamic) CPMs represent a paradigm shift, where the analytical methods dynamically generate updated versions of a model through time; one then needs to validate the system rather than each subsequent model revision.

          Related collections

          Most cited references33

          • Record: found
          • Abstract: found
          • Article: found
          Is Open Access

          Development and validation of QRISK3 risk prediction algorithms to estimate future risk of cardiovascular disease: prospective cohort study

          Objectives To develop and validate updated QRISK3 prediction algorithms to estimate the 10 year risk of cardiovascular disease in women and men accounting for potential new risk factors. Design Prospective open cohort study. Setting General practices in England providing data for the QResearch database. Participants 1309 QResearch general practices in England: 981 practices were used to develop the scores and a separate set of 328 practices were used to validate the scores. 7.89 million patients aged 25-84 years were in the derivation cohort and 2.67 million patients in the validation cohort. Patients were free of cardiovascular disease and not prescribed statins at baseline. Methods Cox proportional hazards models in the derivation cohort to derive separate risk equations in men and women for evaluation at 10 years. Risk factors considered included those already in QRISK2 (age, ethnicity, deprivation, systolic blood pressure, body mass index, total cholesterol: high density lipoprotein cholesterol ratio, smoking, family history of coronary heart disease in a first degree relative aged less than 60 years, type 1 diabetes, type 2 diabetes, treated hypertension, rheumatoid arthritis, atrial fibrillation, chronic kidney disease (stage 4 or 5)) and new risk factors (chronic kidney disease (stage 3, 4, or 5), a measure of systolic blood pressure variability (standard deviation of repeated measures), migraine, corticosteroids, systemic lupus erythematosus (SLE), atypical antipsychotics, severe mental illness, and HIV/AIDs). We also considered erectile dysfunction diagnosis or treatment in men. Measures of calibration and discrimination were determined in the validation cohort for men and women separately and for individual subgroups by age group, ethnicity, and baseline disease status. Main outcome measures Incident cardiovascular disease recorded on any of the following three linked data sources: general practice, mortality, or hospital admission records. Results 363 565 incident cases of cardiovascular disease were identified in the derivation cohort during follow-up arising from 50.8 million person years of observation. All new risk factors considered met the model inclusion criteria except for HIV/AIDS, which was not statistically significant. The models had good calibration and high levels of explained variation and discrimination. In women, the algorithm explained 59.6% of the variation in time to diagnosis of cardiovascular disease (R2, with higher values indicating more variation), and the D statistic was 2.48 and Harrell’s C statistic was 0.88 (both measures of discrimination, with higher values indicating better discrimination). The corresponding values for men were 54.8%, 2.26, and 0.86. Overall performance of the updated QRISK3 algorithms was similar to the QRISK2 algorithms. Conclusion Updated QRISK3 risk prediction models were developed and validated. The inclusion of additional clinical variables in QRISK3 (chronic kidney disease, a measure of systolic blood pressure variability (standard deviation of repeated measures), migraine, corticosteroids, SLE, atypical antipsychotics, severe mental illness, and erectile dysfunction) can help enable doctors to identify those at most risk of heart disease and stroke.
            Bookmark
            • Record: found
            • Abstract: found
            • Article: found
            Is Open Access

            Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research

            In this article, the third in the PROGRESS series on prognostic factor research, Sara Schroter and colleagues review how prognostic models are developed and validated, and then address how prognostic models are assessed for their impact on practice and patient outcomes, illustrating these ideas with examples.
              Bookmark
              • Record: found
              • Abstract: found
              • Article: not found

              Predicting cardiovascular risk in England and Wales: prospective derivation and validation of QRISK2.

              To develop and validate version two of the QRISK cardiovascular disease risk algorithm (QRISK2) to provide accurate estimates of cardiovascular risk in patients from different ethnic groups in England and Wales and to compare its performance with the modified version of Framingham score recommended by the National Institute for Health and Clinical Excellence (NICE). Prospective open cohort study with routinely collected data from general practice, 1 January 1993 to 31 March 2008. 531 practices in England and Wales contributing to the national QRESEARCH database. 2.3 million patients aged 35-74 (over 16 million person years) with 140,000 cardiovascular events. Overall population (derivation and validation cohorts) comprised 2.22 million people who were white or whose ethnic group was not recorded, 22,013 south Asian, 11,595 black African, 10,402 black Caribbean, and 19,792 from Chinese or other Asian or other ethnic groups. First (incident) diagnosis of cardiovascular disease (coronary heart disease, stroke, and transient ischaemic attack) recorded in general practice records or linked Office for National Statistics death certificates. Risk factors included self assigned ethnicity, age, sex, smoking status, systolic blood pressure, ratio of total serum cholesterol:high density lipoprotein cholesterol, body mass index, family history of coronary heart disease in first degree relative under 60 years, Townsend deprivation score, treated hypertension, type 2 diabetes, renal disease, atrial fibrillation, and rheumatoid arthritis. The validation statistics indicated that QRISK2 had improved discrimination and calibration compared with the modified Framingham score. The QRISK2 algorithm explained 43% of the variation in women and 38% in men compared with 39% and 35%, respectively, by the modified Framingham score. Of the 112,156 patients classified as high risk (that is, >or=20% risk over 10 years) by the modified Framingham score, 46,094 (41.1%) would be reclassified at low risk with QRISK2. The 10 year observed risk among these reclassified patients was 16.6% (95% confidence interval 16.1% to 17.0%)-that is, below the 20% treatment threshold. Of the 78 024 patients classified at high risk on QRISK2, 11,962 (15.3%) would be reclassified at low risk by the modified Framingham score. The 10 year observed risk among these patients was 23.3% (22.2% to 24.4%)-that is, above the 20% threshold. In the validation cohort, the annual incidence rate of cardiovascular events among those with a QRISK2 score of >or=20% was 30.6 per 1000 person years (29.8 to 31.5) for women and 32.5 per 1000 person years (31.9 to 33.1) for men. The corresponding figures for the modified Framingham equation were 25.7 per 1000 person years (25.0 to 26.3) for women and 26.4 (26.0 to 26.8) for men). At the 20% threshold, the population identified by QRISK2 was at higher risk of a CV event than the population identified by the Framingham score. Incorporating ethnicity, deprivation, and other clinical conditions into the QRISK2 algorithm for risk of cardiovascular disease improves the accuracy of identification of those at high risk in a nationally representative population. At the 20% threshold, QRISK2 is likely to be a more efficient and equitable tool for treatment decisions for the primary prevention of cardiovascular disease. As the validation was performed in a similar population to the population from which the algorithm was derived, it potentially has a "home advantage." Further validation in other populations is therefore advised.
                Bookmark

                Author and article information

                Contributors
                david.jenkins-5@manchester.ac.uk
                Journal
                Diagn Progn Res
                Diagn Progn Res
                Diagnostic and Prognostic Research
                BioMed Central (London )
                2397-7523
                11 January 2021
                11 January 2021
                2021
                : 5
                : 1
                Affiliations
                [1 ]GRID grid.5379.8, ISNI 0000000121662407, Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, , The University of Manchester, Manchester Academic Health Science Centre, ; Manchester, UK
                [2 ]GRID grid.5379.8, ISNI 0000000121662407, NIHR Greater Manchester Patient Safety Translational Research Centre, , The University of Manchester, ; Manchester, UK
                [3 ]GRID grid.9757.c, ISNI 0000 0004 0415 6205, Centre for Prognosis Research, School of Primary, Community and Social Care, , Keele University, ; Staffordshire, UK
                [4 ]GRID grid.5477.1, ISNI 0000000120346234, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, , Utrecht University, ; Utrecht, The Netherlands
                [5 ]GRID grid.4991.5, ISNI 0000 0004 1936 8948, Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, , University of Oxford, ; Oxford, UK
                [6 ]GRID grid.5379.8, ISNI 0000000121662407, NIHR Manchester Biomedical Research Centre, , The University of Manchester, Manchester Academic Health Science Centre, ; Manchester, UK
                Article
                90
                10.1186/s41512-020-00090-3
                7797885
                33431065
                c160c635-c61a-40a4-b97a-34e64ba7ddb8
                © The Author(s) 2021

                Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

                History
                : 14 May 2020
                : 8 December 2020
                Funding
                Funded by: FundRef http://dx.doi.org/10.13039/501100013235, NIHR Greater Manchester Patient Safety Translational Research Centre;
                Funded by: FundRef http://dx.doi.org/10.13039/100014653, Manchester Biomedical Research Centre;
                Funded by: FundRef http://dx.doi.org/10.13039/501100013373, NIHR Oxford Biomedical Research Centre;
                Funded by: Nederlandse Organisatie voor Wetenschappelijk Onderzoek (NL)
                Award ID: 91617050
                Funded by: FundRef http://dx.doi.org/10.13039/501100000289, Cancer Research UK;
                Award ID: C49297/A27294
                Funded by: Horizon 2020 ()
                Award ID: 825746
                Categories
                Commentary
                Custom metadata
                © The Author(s) 2021

                clinical prediction models,dynamic model,validation,model updating,model development,learning health system

                Comments

                Comment on this article