Evaluation of inconsistency in networks of interventions

Abstract

Background

The assumption of consistency, defined as agreement between direct and indirect sources of evidence, underlies the increasingly popular method of network meta-analysis. No evidence exists so far regarding the extent of inconsistency and the factors that control its statistical detection in full networks of interventions.

Methods

In this paper the prevalence of inconsistency is evaluated using 40 published networks of interventions involving 303 loops of evidence. Inconsistency is evaluated in each loop by contrasting direct and indirect estimates and by employing an omnibus test of consistency for the entire network. We explore whether different effect measures for dichotomous outcomes are associated with differences in inconsistency and evaluate whether different ways to estimate heterogeneity impact on the magnitude and detection of inconsistency.

Results

Inconsistency was detected in between 2% and 9% of the tested loops, depending on the effect measure and heterogeneity estimation method. Loops that included comparisons informed by a single study were more likely to show inconsistency. About one eighth of the networks were found to be inconsistent. The proportions of inconsistent loops do not materially change when different effect measures are employed. Important heterogeneity or overestimation of the heterogeneity was associated with a small decrease in the prevalence of statistical inconsistency.

Conclusions

The study suggests that changing effect measure might improve statistical consistency and that a sensitivity analysis to the assumptions and estimator of heterogeneity might be needed before concluding about the absence of statistical inconsistency, particularly in networks with few studies.