Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study

There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

Abstract

Objectives

To empirically assess the relation between study characteristics and prognostic model performance in external validation studies of multivariable prognostic models.

Design

Meta-epidemiological study.

Data sources and study selection

On 16 October 2018, we searched electronic databases for systematic reviews of prognostic models. Reviews from non-overlapping clinical fields were selected if they reported common performance measures (either the concordance (c)-statistic or the ratio of observed over expected number of events (OE ratio)) from 10 or more validations of the same prognostic model.

Data extraction and analyses

Study design features, population characteristics, methods of predictor and outcome assessment, and the aforementioned performance measures were extracted from the included external validation studies. Random effects meta-regression was used to quantify the association between the study characteristics and model performance.

Results

We included 10 systematic reviews, describing a total of 224 external validations, of which 221 reported c-statistics and 124 OE ratios. Associations between study characteristics and model performance were heterogeneous across systematic reviews. C-statistics were most associated with variation in population characteristics, outcome definitions and measurement and predictor substitution. For example, validations with eligibility criteria comparable to the development study were associated with higher c-statistics compared with narrower criteria (difference in logit c-statistic 0.21(95% CI 0.07 to 0.35), similar to an increase from 0.70 to 0.74). Using a case-control design was associated with higher OE ratios, compared with using data from a cohort (difference in log OE ratio 0.97(95% CI 0.38 to 1.55), similar to an increase in OE ratio from 1.00 to 2.63).

Conclusions

Variation in performance of prognostic models across studies is mainly associated with variation in case-mix, study designs, outcome definitions and measurement methods and predictor substitution. Researchers developing and validating prognostic models should realise the potential influence of these study characteristics on the predictive performance of prognostic models.

Related collections

Most cited references 26

Record: found
Abstract: found
Article: not found

SAPS 3—From evaluation of the patient to evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital mortality at ICU admission

Rui P Moreno, Philipp G H Metnitz, Eduardo Almeida … (2005)

Objective To develop a model to assess severity of illness and predict vital status at hospital discharge based on ICU admission data. Design Prospective multicentre, multinational cohort study. Patients and setting A total of 16,784 patients consecutively admitted to 303 intensive care units from 14 October to 15 December 2002. Measurements and results ICU admission data (recorded within ±1 h) were used, describing: prior chronic conditions and diseases; circumstances related to and physiologic derangement at ICU admission. Selection of variables for inclusion into the model used different complementary strategies. For cross-validation, the model-building procedure was run five times, using randomly selected four fifths of the sample as a development- and the remaining fifth as validation-set. Logistic regression methods were then used to reduce complexity of the model. Final estimates of regression coefficients were determined by use of multilevel logistic regression. Variables selection and weighting were further checked by bootstraping (at patient level and at ICU level). Twenty variables were selected for the final model, which exhibited good discrimination (aROC curve 0.848), without major differences across patient typologies. Calibration was also satisfactory (Hosmer-Lemeshow goodness-of-fit test Ĥ=10.56, p=0.39, Ĉ=14.29, p=0.16). Customised equations for major areas of the world were computed and demonstrate a good overall goodness-of-fit. Conclusions The SAPS 3 admission score is able to predict vital status at hospital discharge with use of data recorded at ICU admission. Furthermore, SAPS 3 conceptually dissociates evaluation of the individual patient from evaluation of the ICU and thus allows them to be assessed at their respective reference levels. Electronic Supplementary Material Electronic supplementary material is included in the online fulltext version of this article and accessible for authorised users: http://dx.doi.org/10.1007/s00134-005-2763-5

0 comments Cited 460 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

European system for cardiac operative risk evaluation (EuroSCORE).

S Nashef, F. Roques, P. Michel … (1999)

To construct a scoring system for the prediction of early mortality in cardiac surgical patients in Europe on the basis of objective risk factors. The EuroSCORE database was divided into developmental and validation subsets. In the former, risk factors deemed to be objective, credible, obtainable and difficult to falsify were weighted on the basis of regression analysis. An additive score of predicted mortality was constructed. Its calibration and discrimination characteristics were assessed in the validation dataset. Thresholds were defined to distinguish low, moderate and high risk groups. The developmental dataset had 13,302 patients, calibration by Hosmer Lemeshow Chi square was (8) = 8.26 (P 200 micromol/l (2), active endocarditis (3) and critical preoperative state (3). Cardiac factors were unstable angina on intravenous nitrates (2), reduced left ventricular ejection fraction (30-50%: 1, 60 mmHg (2). Operation-related factors were emergency (2), other than isolated coronary surgery (2), thoracic aorta surgery (3) and surgery for postinfarct septal rupture (4). The scoring system was then applied to three risk groups. The low risk group (EuroSCORE 1-2) had 4529 patients with 36 deaths (0.8%), 95% confidence limits for observed mortality (0.56-1.10) and for expected mortality (1.27-1.29). The medium risk group (EuroSCORE 3-5) had 5977 patients with 182 deaths (3%), observed mortality (2.62-3.51), predicted (2.90-2.94). The high risk group (EuroSCORE 6 plus) had 4293 patients with 480 deaths (11.2%) observed mortality (10.25-12.16), predicted (10.93-11.54). Overall, there were 698 deaths in 14,799 patients (4.7%), observed mortality (4.37-5.06), predicted (4.72-4.95). EuroSCORE is a simple, objective and up-to-date system for assessing heart surgery, soundly based on one of the largest, most complete and accurate databases in European cardiac surgical history. We recommend its widespread use.

0 comments Cited 269 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Systematic review: prediction of perioperative cardiac complications and mortality by the revised cardiac risk index.

Meredith Ford, W Scott Beattie, Duminda N. Wijeysundera (2010)

The Revised Cardiac Risk Index (RCRI) is widely used to predict perioperative cardiac complications. To evaluate the ability of the RCRI to predict cardiac complications and death after noncardiac surgery. MEDLINE, EMBASE, and ISI Web of Science (1966 to 31 December 2008). Cohort studies that reported the association of the RCRI with major cardiac complications (cardiac death, myocardial infarction, and nonfatal cardiac arrest) or death in the hospital or within 30 days of surgery. Two reviewers independently extracted study characteristics, documented outcome data, and evaluated study quality. Of 24 studies (792 740 patients), 18 reported cardiac complications; 6 of the 18 studies were prospective and had uniform outcome surveillance and blinded outcome adjudication. The RCRI discriminated moderately well between patients at low versus high risk for cardiac events after mixed noncardiac surgery (area under the receiver-operating characteristic curve [AUC], 0.75 [95% CI, 0.72 to 0.79]); sensitivity, 0.65 [CI, 0.46 to 0.81]; specificity, 0.76 [CI, 0.58 to 0.88]; positive likelihood ratio, 2.78 [CI, 1.74 to 4.45]; negative likelihood ratio, 0.45 [CI, 0.31 to 0.67]). Prediction of cardiac events after vascular noncardiac surgery was less accurate (AUC, 0.64 [CI, 0.61 to 0.66]; sensitivity, 0.70 [CI, 0.53 to 0.82]; specificity, 0.55 [CI, 0.45 to 0.66]; positive likelihood ratio, 1.56 [CI, 1.42 to 1.73]; negative likelihood ratio, 0.55 [CI, 0.40 to 0.76]). Six studies reported death, with a median AUC of 0.62 (range, 0.54 to 0.78). A pooled AUC for predicting death could not be calculated because of very high heterogeneity (I(2) = 95%). Studies generally were of low methodological quality, had varied definitions of cardiac events, and were statistically and clinically heterogeneous. The RCRI discriminated moderately well between patients at low versus high risk for cardiac events after mixed noncardiac surgery. It did not perform well at predicting cardiac events after vascular noncardiac surgery or at predicting death. High-quality research is needed in this area of perioperative medicine.

0 comments Cited 119 times – based on 0 reviews      Review now

Bookmark

All references

Author and article information

Journal

Journal ID (nlm-ta): BMJ Open

Journal ID (iso-abbrev): BMJ Open

Journal ID (hwp): bmjopen

Journal ID (publisher-id): bmjopen

Title: BMJ Open

Publisher: BMJ Publishing Group (BMA House, Tavistock Square, London, WC1H 9JR )

ISSN (Electronic): 2044-6055

Publication date Collection: 2019

Publication date (Electronic): 1 April 2019

Volume: 9

Issue: 4

Electronic Location Identifier: e026160

Affiliations

[1 ] Cochrane Netherlands, University Medical Center Utrecht, Utrecht University , Utrecht, The Netherlands

[2 ] Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University , Utrecht, The Netherlands

Author notes

[Correspondence to ] Dr Johanna A A G Damen; j.a.a.damen@ 123456umcutrecht.nl

Article

Publisher ID: bmjopen-2018-026160

DOI: 10.1136/bmjopen-2018-026160

PMC ID: 6500242

PubMed ID: 30940759

SO-VID: 5894ec6a-19f0-46f7-8131-56704be7fc1b

License:

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

History

Date received : 22 August 2018

Date revision received : 05 November 2018

Date accepted : 04 February 2019

Funding

Funded by: Netherlands Organization for Health Research and Development;

Custom metadata

special-feature unlocked

ScienceOpen disciplines: Medicine

Keywords: meta-epidemiology,prognosis,prognostic models,bias,prediction

Data availability:

ScienceOpen disciplines: Medicine

Keywords: meta-epidemiology, prognosis, prognostic models, bias, prediction

Comments

Comment on this article

scite_

Cited by 13

See all cited by

Most referenced authors 1,749

See all reference authors

Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study

Read this article at

Abstract

Objectives

Design

Data sources and study selection

Data extraction and analyses

Results

Conclusions

Related collections

Karger: Oncology

Most cited references 26

SAPS 3—From evaluation of the patient to evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital mortality at ICU admission

European system for cardiac operative risk evaluation (EuroSCORE).

Systematic review: prediction of perioperative cardiac complications and mortality by the revised cardiac risk index.

Author and article information

Journal

Affiliations

Author notes

Article

History

Funding

Categories

Custom metadata

Comments

Comment on this article

Similar content 78

Cited by 13

Most referenced authors 1,749