Derivation, validation and assessment of a novel nomogram-based risk assessment model for venous thromboembolism in hospitalized patients with lung cancer: A retrospective case control study

Diagnostic and prognostic models are typically evaluated with measures of accuracy that do not address clinical consequences. Decision-analytic techniques allow assessment of clinical outcomes but often require collection of additional information and may be cumbersome to apply to models that yield a continuous result. The authors sought a method for evaluating and comparing prediction models that incorporates clinical consequences,requires only the data set on which the models are tested,and can be applied to models that have either continuous or dichotomous results. The authors describe decision curve analysis, a simple, novel method of evaluating predictive models. They start by assuming that the threshold probability of a disease or event at which a patient would opt for treatment is informative of how the patient weighs the relative harms of a false-positive and a false-negative prediction. This theoretical relationship is then used to derive the net benefit of the model across different threshold probabilities. Plotting net benefit against threshold probability yields the "decision curve." The authors apply the method to models for the prediction of seminal vesicle invasion in prostate cancer patients. Decision curve analysis identified the range of threshold probabilities in which a model was of value, the magnitude of benefit, and which of several models was optimal. Decision curve analysis is a suitable method for evaluating alternative diagnostic and prognostic strategies that has advantages over other commonly used measures and techniques.

Nomograms are widely used as prognostic devices in oncology and medicine. With the ability to generate an individual probability of a clinical event by integrating diverse prognostic and determinant variables, nomograms meet our desire for biologically and clinically integrated models and fulfill our drive towards personalised medicine. Rapid computation through user-friendly digital interfaces, together with increased accuracy, and more easily understood prognoses compared with conventional staging, allow for seamless incorporation of nomogram-derived prognosis to aid clinical decision making. This has led to the appearance of many nomograms on the internet and in medical journals, and an increase in nomogram use by patients and physicians alike. However, the statistical foundations of nomogram construction, their precise interpretation, and evidence supporting their use are generally misunderstood. This issue is leading to an under-appreciation of the inherent uncertainties regarding nomogram use. We provide a systematic, practical approach to evaluating and comprehending nomogram-derived prognoses, with particular emphasis on clarifying common misconceptions and highlighting limitations.

Clinical prediction models combine multiple predictors to estimate risk for the presence of a particular condition (diagnostic models) or the occurrence of a certain event in the future (prognostic models). PROBAST (Prediction model Risk Of Bias ASsessment Tool), a tool for assessing the risk of bias (ROB) and applicability of diagnostic and prognostic prediction model studies, was developed by a steering group that considered existing ROB tools and reporting guidelines. The tool was informed by a Delphi procedure involving 38 experts and was refined through piloting. PROBAST is organized into the following 4 domains: participants, predictors, outcome, and analysis. These domains contain a total of 20 signaling questions to facilitate structured judgment of ROB, which was defined to occur when shortcomings in study design, conduct, or analysis lead to systematically distorted estimates of model predictive performance. PROBAST enables a focused and transparent approach to assessing the ROB and applicability of studies that develop, validate, or update prediction models for individualized predictions. Although PROBAST was designed for systematic reviews, it can be used more generally in critical appraisal of prediction model studies. Potential users include organizations supporting decision making, researchers and clinicians who are interested in evidence-based medicine or involved in guideline development, journal editors, and manuscript reviewers.