Ten simple rules for carrying out and writing meta-analyses – ScienceOpen
49
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Ten simple rules for carrying out and writing meta-analyses

      editorial

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Introduction In the context of evidence-based medicine, meta-analyses provide novel and useful information [1], as they are at the top of the pyramid of evidence and consolidate previous evidence published in multiple previous reports [2]. Meta-analysis is a powerful tool to cumulate and summarize the knowledge in a research field [3]. Because of the significant increase in the published scientific literature in recent years, there has also been an important growth in the number of meta-analyses for a large number of topics [4]. It has been found that meta-analyses are among the types of publications that usually receive a larger number of citations in the biomedical sciences [5,6]. The methods and standards for carrying out meta-analyses have evolved in recent years [7–9]. Although there are several published articles describing comprehensive guidelines for specific types of meta-analyses, there is still the need for an abridged article with general and updated recommendations for researchers interested in the development of meta-analyses. We present here ten simple rules for carrying out and writing meta-analyses. Rule 1: Specify the topic and type of the meta-analysis Considering that a systematic review [10] is fundamental for a meta-analysis, you can use the Population, Intervention, Comparison, Outcome (PICO) model to formulate the research question. It is important to verify that there are no published meta-analyses on the specific topic in order to avoid duplication of efforts [11]. In some cases, an updated meta-analysis in a topic is needed if additional data become available. It is possible to carry out meta-analyses for multiple types of studies, such as epidemiological variables for case-control, cohort, and randomized clinical trials. As observational studies have a larger possibility of having several biases, meta-analyses of these types of designs should take that into account. In addition, there is the possibility to carry out meta-analyses for genetic association studies, gene expression studies, genome-wide association studies (GWASs), or data from animal experiments. It is advisable to preregister the systematic review protocols at the International Prospective Register of Systematic Reviews (PROSPERO; https://www.crd.york.ac.uk/Prospero) database [12]. Keep in mind that an increasing number of journals require registration prior to publication. Rule 2: Follow available guidelines for different types of meta-analyses There are several available general guidelines. The first of such efforts were the Quality of Reports of Meta-analyses of Randomized Controlled Trials (QUORUM) [13] and the Meta-analysis of Observational Studies in Epidemiology (MOOSE) statements [14], but currently, the Preferred Reporting Items for Systematic reviews and Meta-analyses (PRISMA) [15] has been broadly cited and used. In addition, there have been efforts to develop specific guidelines regarding meta-analyses for clinical studies (Cochrane Handbook; https://training.cochrane.org/handbook), genetic association studies [16], genome-wide expression studies [17], GWASs [18], and animal studies [19]. Rule 3: Establish inclusion criteria and define key variables You should establish in advance the inclusion (such as type of study, language of publication, among others) and exclusion (such as minimal sample size, among others) criteria. Keep in mind that the current consensus advises against strict criteria concerning language or sample size. You should clearly define the variables that will be extracted from each primary article. Broad inclusion criteria increase heterogeneity between studies, and narrow inclusion criteria can make it difficult to find studies; therefore, a compromise should be found. Prospective meta-analyses, which usually are carried out by international consortia, have the advantage of the possibility of including individual-level data [20]. Rule 4: Carry out a systematic search in different databases and extract key data You can carry out your systematic search in several bibliographic databases, such as PubMed, Embase, The Cochrane Central Register of Controlled Trials, Scopus, Web of Science, and Google Scholar [21]. Usually, searching in several databases helps to minimize the possibility of failing to identify all published studies [22]. In some specific areas, searching in specialized databases is also worth doing (such as BIOSIS, Cumulative index to Nursing and Allied Health Literature (CINAHL), PsycINFO, Sociological Abstracts, and EconLit, among others). Moreover, in other cases, direct search for the data is also advisable (i.e., Gene Expression Omnibus [GEO] database for gene expression studies) [23]. Usually, the bibliography of review articles might help to identify additional articles and data from other types of documents (such as theses or conference proceedings) that might be included in your meta-analysis. The Web of Science database can be used to identify publications that have cited key articles. Adequate extraction and recording of key data from primary articles are fundamental for carrying out a meta-analysis. Quality assessment of the included studies is also an important issue; it can be used for determining inclusion criteria, sensitivity analysis, or differential weighting of the studies. For example the Jadad scale [24] is frequently used for randomized clinical trials, the Newcastle–Ottawa scale [25] for nonrandomized studies, and QUADAS-2 for the Quality Assessment of Diagnostic Accuracy Studies [26]. It is recommended that these steps be carried out by two researchers in parallel and that discrepancies be resolved by consensus. Nevertheless, the reader must be aware that quality assessment has been criticized, especially when it reduces the studies to a single “quality” score [27,28]. In any case, it is important to avoid the confusion of using guidelines for the reporting of primary studies as scales for the assessment of the quality of included articles [29,30]. Rule 5: Contact authors of primary articles to ask for missing data It is common that key data are not available in the main text or supplementary files of primary articles [31], leading to the need to contact the authors to ask for missing data. However, the rate of response from authors is lower than expected. There are multiple standards that promote the availability of primary data in published articles, such as the minimum information about a microarray experiment (MIAME) [32] and the STrengthening the REporting of Genetic Association Studies (STREGA) [33]. In some areas, such as genetics, in which it was shown that it is possible to identify an individual using the aggregated statistics from a particular study [34], strict criteria are imposed for data sharing, and specialized permissions might be needed. Rule 6: Select the best statistical models for your question For cases in which there is enough primary data of adequate quality for a quantitative summary, there is the option to carry out a meta-analysis. The potential analyst must be warned that in many cases the data are reported in noncompatible forms, so one must be ready to perform various types of transformations. Thankfully, there are methods available for extracting and transforming data regarding continuous variables [35–37], 2 × 2 tables [38,39], or survival data [40]. Frequently, meta-analyses are based on fixed-effects or random-effects statistical models [20]. In addition, models based on combining ranks or p-values are also available and can be used in specific cases [41–44]. For more complex data, multivariate methods for meta-analysis have been proposed [45,46]. Additional statistical examinations involve sensitivity analyses, metaregressions, subgroup analyses, and calculation of heterogeneity metrics, such as Q or I2 [20]. It is fundamental to assess and, if present, explain the possible sources of heterogeneity. Although random-effects models are suitable for cases of between-studies heterogeneity, the sources of between-studies variation should be identified, and their impact on effect size should be quantified using statistical tests, such as subgroup analyses or metaregression. Publication bias is an important aspect to consider [47], since in many cases negative findings have less probability of being published. Other types of bias, such as the so-called “Proteus phenomenon” [48] or “winner’s curse” [49], are common in some scientific fields, such as genetics, and the approach of cumulative meta-analysis is suggested in order to identify them. Rule 7: Use available software to carry metastatistics There are several very user-friendly and freely available programs for carrying out meta-analyses [43,44], either within the framework of a statistical package such as Stata or R or as stand-alone applications. Stata and R [50–52] have dozens of routines, mostly user written, that can handle most meta-analysis tasks, even complex analyses such as network meta-analysis and meta-analyses of GWASs and gene expression studies (https://cran.r-project.org/web/views/MetaAnalysis.html; https://www.stata.com/support/faqs/statistics/meta-analysis). There are also stand-alone packages that can be useful for general applications or for specific areas, such as OpenMetaAnalyst [53], NetworkAnalyst [54], JASP [55], MetaGenyo [56], Cochrane RevMan (https://community.cochrane.org/help/tools-and-software/revman-5), EpiSheet (krothman.org/episheet.xls), GWAR [57], GWAMA [58], and METAL [59]. Some of these programs are web services or stand-alone software. In some cases, certain programs can present issues when they are run because of their dependency on other packages. Rule 8: The records and study report must be complete and transparent Following published guidelines for meta-analyses guarantees that the manuscript will describe the different steps and methods used, facilitating their transparency and replicability [15]. Data such as search and inclusion criteria, numbers of abstracts screened, and included studies are quite useful, in addition to details of meta-analytical strategies used. An assessment of quality of included studies is also useful [60]. A spreadsheet can be constructed in which every step in the selection criteria is recorded; this will be helpful to construct flow charts. In this context, a flow diagram describing the progression between the different steps is quite useful and might enhance the quality of the meta-analysis [61]. Records will be also useful if, in the future, the meta-analysis needs to be updated. Stating the limitations of the analysis is also important [62]. Rule 9: Provide enough data in your manuscript A table with complete information about included studies (such as author, year, details of included subjects, DOIs, or PubMed IDs, among others) is quite useful in an article reporting a meta-analysis; it can be included in the main text of the manuscript or as a supplementary file. Software used for carrying out meta-analyses and to generate key graphs, such as forest plots, should be referenced. Summary effect measures, such as a pooled odds ratios or the counts used to generate them, should be always reported, including confidence intervals. It is also possible to generate figures with information from multiple forest plots [63]. In the case of positive findings, plots from sensitivity analyses are quite informative. In more-complex analyses, it is advisable to include in the supplementary files the scripts used to generate the results [64]. Rule 10: Provide context for your findings and suggest future directions The Discussion section is an important scientific component in a manuscript describing a meta-analysis, as the authors should discuss their current findings in the context of the available scientific literature and existing knowledge [65]. Authors can discuss possible reasons for the positive or negative results of their meta-analysis, provide an interpretation of findings based on available biological or epidemiological evidence, and comment on particular features of individual studies or experimental designs used [66]. As meta-analyses are usually synthesizing the existing evidence from multiple primary studies, which commonly took years and large amounts of funding, authors can recommend key suggestions for conducting and/or reporting future primary studies [67]. As open science is becoming more important around the globe [68,69], adherence to published standards, in addition to the evolution of methods for different meta-analytical applications, will be even more important to carry out meta-analyses of high quality and impact.

          Related collections

          Most cited references40

          • Record: found
          • Abstract: found
          • Article: not found

          Extracting summary statistics to perform meta-analyses of the published literature for survival endpoints.

          Meta-analyses aim to provide a full and comprehensive summary of related studies which have addressed a similar question. When the studies involve time to event (survival-type) data the most appropriate statistics to use are the log hazard ratio and its variance. However, these are not always explicitly presented for each study. In this paper a number of methods of extracting estimates of these statistics in a variety of situations are presented. Use of these methods should improve the efficiency and reliability of meta-analyses of the published literature with survival-type endpoints.
            Bookmark
            • Record: found
            • Abstract: not found
            • Article: not found

            The hazards of scoring the quality of clinical trials for meta-analysis.

            Although it is widely recommended that clinical trials undergo some type of quality review, the number and variety of quality assessment scales that exist make it unclear how to achieve the best assessment. To determine whether the type of quality assessment scale used affects the conclusions of meta-analytic studies. Meta-analysis of 17 trials comparing low-molecular-weight heparin (LMWH) with standard heparin for prevention of postoperative thrombosis using 25 different scales to identify high-quality trials. The association between treatment effect and summary scores and the association with 3 key domains (concealment of treatment allocation, blinding of outcome assessment, and handling of withdrawals) were examined in regression models. Pooled relative risks of deep vein thrombosis with LMWH vs standard heparin in high-quality vs low-quality trials as determined by 25 quality scales. Pooled relative risks from high-quality trials ranged from 0.63 (95% confidence interval [CI], 0.44-0.90) to 0.90 (95% CI, 0.67-1.21) vs 0.52 (95% CI, 0.24-1.09) to 1.13 (95% CI, 0.70-1.82) for low-quality trials. For 6 scales, relative risks of high-quality trials were close to unity, indicating that LMWH was not significantly superior to standard heparin, whereas low-quality trials showed better protection with LMWH (P<.05). Seven scales showed the opposite: high quality trials showed an effect whereas low quality trials did not. For the remaining 12 scales, effect estimates were similar in the 2 quality strata. In regression analysis, summary quality scores were not significantly associated with treatment effects. There was no significant association of treatment effects with allocation concealment and handling of withdrawals. Open outcome assessment, however, influenced effect size with the effect of LMWH, on average, being exaggerated by 35% (95% CI, 1%-57%; P= .046). Our data indicate that the use of summary scores to identify trials of high quality is problematic. Relevant methodological aspects should be assessed individually and their influence on effect sizes explored.
              Bookmark
              • Record: found
              • Abstract: found
              • Article: not found

              Meta-analysis methods for genome-wide association studies and beyond.

              Meta-analysis of genome-wide association studies (GWASs) has become a popular method for discovering genetic risk variants. Here, we overview both widely applied and newer statistical methods for GWAS meta-analysis, including issues of interpretation and assessment of sources of heterogeneity. We also discuss extensions of these meta-analysis methods to complex data. Where possible, we provide guidelines for researchers who are planning to use these methods. Furthermore, we address special issues that may arise for meta-analysis of sequencing data and rare variants. Finally, we discuss challenges and solutions surrounding the goals of making meta-analysis data publicly available and building powerful consortia.
                Bookmark

                Author and article information

                Contributors
                Role: Editor
                Journal
                PLoS Comput Biol
                PLoS Comput. Biol
                plos
                ploscomp
                PLoS Computational Biology
                Public Library of Science (San Francisco, CA USA )
                1553-734X
                1553-7358
                16 May 2019
                May 2019
                : 15
                : 5
                : e1006922
                Affiliations
                [1 ] Laboratory of NeuroPsychiatric Genetics, Biomedical Sciences Research Group, School of Medicine, Universidad Antonio Nariño, Bogotá, Colombia
                [2 ] PhD Program in Health Sciences, School of Medicine, Universidad Antonio Nariño, Bogotá, Colombia
                [3 ] Novartis Pharmaceuticals Corporation, East Hanover, New Jersey, United States of America
                [4 ] Departamento de Nutrición y Bioquímica, Facultad de Ciencias, Pontificia Universidad Javeriana, Bogotá., Colombia
                [5 ] Department of Computer Science and Biomedical Informatics, University of Thessaly, Lamia, Greece
                Dassault Systemes BIOVIA, UNITED STATES
                Author notes

                The authors have declared that no competing interests exist.

                Author information
                http://orcid.org/0000-0001-9175-3363
                http://orcid.org/0000-0003-4935-2325
                Article
                PCOMPBIOL-D-18-02100
                10.1371/journal.pcbi.1006922
                6521986
                31095553
                4e0a2f60-340f-42e7-95d2-ebfec8579ae4
                © 2019 Forero et al

                This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

                History
                Page count
                Figures: 0, Tables: 0, Pages: 7
                Funding
                YG-G is supported by a PhD fellowship from Centro de Estudios Interdisciplinarios Básicos y Aplicados CEIBA (Rodolfo Llinás Program). DAF is supported by research grants from Colciencias and VCTI. PGB is partially supported by ELIXIR-GR, the Greek Research Infrastructure for data management and analysis in the biosciences. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
                Categories
                Editorial
                Research and Analysis Methods
                Mathematical and Statistical Techniques
                Statistical Methods
                Metaanalysis
                Physical Sciences
                Mathematics
                Statistics
                Statistical Methods
                Metaanalysis
                Research and Analysis Methods
                Database and Informatics Methods
                Database Searching
                Computer and Information Sciences
                Computer Software
                Research and Analysis Methods
                Research Assessment
                Systematic Reviews
                Biology and Life Sciences
                Genetics
                Gene Expression
                Medicine and Health Sciences
                Clinical Medicine
                Clinical Trials
                Randomized Controlled Trials
                Medicine and Health Sciences
                Pharmacology
                Drug Research and Development
                Clinical Trials
                Randomized Controlled Trials
                Research and Analysis Methods
                Clinical Trials
                Randomized Controlled Trials
                Biology and Life Sciences
                Genetics
                Human Genetics
                Genetic Association Studies
                Computer and Information Sciences
                Software Engineering
                Software Tools
                Engineering and Technology
                Software Engineering
                Software Tools

                Quantitative & Systems biology
                Quantitative & Systems biology

                Comments

                Comment on this article

                scite_
                0
                0
                0
                0
                Smart Citations
                0
                0
                0
                0
                Citing PublicationsSupportingMentioningContrasting
                View Citations

                See how this article has been cited at scite.ai

                scite shows how a scientific paper has been cited by providing the context of the citation, a classification describing whether it supports, mentions, or contrasts the cited claim, and a label indicating in which section the citation was made.

                Similar content57

                Cited by27

                Most referenced authors1,855