Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension

Author(s): Samantha Cruz Rivera ¹ ^, ² ^, ³ , Xiaoxuan Liu ³ ^, ⁴ ^, ⁵ ^, ⁶ ^, ⁷ , An-Wen Chan ⁸ , Alastair K. Denniston ¹ ^, ³ ^, ⁴ ^, ⁵ ^, ⁶ ^, ⁹ ^, , Melanie J. Calvert ¹ ^, ² ^, ³ ^, ⁶ ^, ¹⁰ ^, ¹¹ ^, ¹² , The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, SPIRIT-AI and CONSORT-AI Consensus Group

Publication date (Electronic): 9 September 2020

Journal: Nature Medicine

Publisher: Nature Publishing Group US

Keywords: Clinical trial design, Technology

Read this article at

ScienceOpenPublisher PMC

Bookmark

There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

Abstract

The SPIRIT 2013 statement aims to improve the completeness of clinical trial protocol reporting by providing evidence-based recommendations for the minimum set of items to be addressed. This guidance has been instrumental in promoting transparent evaluation of new interventions. More recently, there has been a growing recognition that interventions involving artificial intelligence (AI) need to undergo rigorous, prospective evaluation to demonstrate their impact on health outcomes. The SPIRIT-AI (Standard Protocol Items: Recommendations for Interventional Trials–Artificial Intelligence) extension is a new reporting guideline for clinical trial protocols evaluating interventions with an AI component. It was developed in parallel with its companion statement for trial reports: CONSORT-AI (Consolidated Standards of Reporting Trials–Artificial Intelligence). Both guidelines were developed through a staged consensus process involving literature review and expert consultation to generate 26 candidate items, which were consulted upon by an international multi-stakeholder group in a two-stage Delphi survey (103 stakeholders), agreed upon in a consensus meeting (31 stakeholders) and refined through a checklist pilot (34 participants). The SPIRIT-AI extension includes 15 new items that were considered sufficiently important for clinical trial protocols of AI interventions. These new items should be routinely reported in addition to the core SPIRIT 2013 items. SPIRIT-AI recommends that investigators provide clear descriptions of the AI intervention, including instructions and skills required for use, the setting in which the AI intervention will be integrated, considerations for the handling of input and output data, the human–AI interaction and analysis of error cases. SPIRIT-AI will help promote transparency and completeness for clinical trial protocols for AI interventions. Its use will assist editors and peer reviewers, as well as the general readership, to understand, interpret and critically appraise the design and risk of bias for a planned clinical trial.

Abstract

The CONSORT-AI and SPIRIT-AI extensions improve the transparency of clinical trial design and trial protocol reporting for artificial intelligence interventions.

Related collections

Most cited references 48

Record: found
Abstract: found
Article: not found

Dermatologist-level classification of skin cancer with deep neural networks

Andre Esteva, Brett Kuprel, Roberto Chacón-Novoa … (2017)

Skin cancer, the most common human malignancy, is primarily diagnosed visually, beginning with an initial clinical screening and followed potentially by dermoscopic analysis, a biopsy and histopathological examination. Automated classification of skin lesions using images is a challenging task owing to the fine-grained variability in the appearance of skin lesions. Deep convolutional neural networks (CNNs) show potential for general and highly variable tasks across many fine-grained object categories. Here we demonstrate classification of skin lesions using a single CNN, trained end-to-end from images directly, using only pixels and disease labels as inputs. We train a CNN using a dataset of 129,450 clinical images—two orders of magnitude larger than previous datasets—consisting of 2,032 different diseases. We test its performance against 21 board-certified dermatologists on biopsy-proven clinical images with two critical binary classification use cases: keratinocyte carcinomas versus benign seborrheic keratoses; and malignant melanomas versus benign nevi. The first case represents the identification of the most common cancers, the second represents the identification of the deadliest skin cancer. The CNN achieves performance on par with all tested experts across both tasks, demonstrating an artificial intelligence capable of classifying skin cancer with a level of competence comparable to dermatologists. Outfitted with deep neural networks, mobile devices can potentially extend the reach of dermatologists outside of the clinic. It is projected that 6.3 billion smartphone subscriptions will exist by the year 2021 (ref. 13) and can therefore potentially provide low-cost universal access to vital diagnostic care.

0 comments Cited 2363 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

SPIRIT 2013 statement: defining standard protocol items for clinical trials.

An-Wen Chan, Jennifer M Tetzlaff, Douglas G. Altman … (2014)

The protocol of a clinical trial serves as the foundation for study planning, conduct, reporting, and appraisal. However, trial protocols and existing protocol guidelines vary greatly in content and quality. This article describes the systematic development and scope of SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) 2013, a guideline for the minimum content of a clinical trial protocol.The 33-item SPIRIT checklist applies to protocols for all clinical trials and focuses on content rather than format. The checklist recommends a full description of what is planned; it does not prescribe how to design or conduct a trial. By providing guidance for key content, the SPIRIT recommendations aim to facilitate the drafting of high-quality protocols. Adherence to SPIRIT would also enhance the transparency and completeness of trial protocols for the benefit of investigators, trial participants, patients, sponsors, funders, research ethics committees or institutional review boards, peer reviewers, journals, trial registries, policymakers, regulators, and other key stakeholders.

0 comments Cited 2329 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials

An-Wen Chan, Jennifer Tetzlaff, Peter C. Gøtzsche … (2014)

High quality protocols facilitate proper conduct, reporting, and external review of clinical trials. However, the completeness of trial protocols is often inadequate. To help improve the content and quality of protocols, an international group of stakeholders developed the SPIRIT 2013 Statement (Standard Protocol Items: Recommendations for Interventional Trials). The SPIRIT Statement provides guidance in the form of a checklist of recommended items to include in a clinical trial protocol. This SPIRIT 2013 Explanation and Elaboration paper provides important information to promote full understanding of the checklist recommendations. For each checklist item, we provide a rationale and detailed description; a model example from an actual protocol; and relevant references supporting its importance. We strongly recommend that this explanatory paper be used in conjunction with the SPIRIT Statement. A website of resources is also available (www.spirit-statement.org). The SPIRIT 2013 Explanation and Elaboration paper, together with the Statement, should help with the drafting of trial protocols. Complete documentation of key trial elements can facilitate transparency and protocol review for the benefit of all stakeholders.

0 comments Cited 2045 times – based on 0 reviews      Review now

Bookmark

All references

Author and article information

Contributors

Alastair K. Denniston:

ORCID: http://orcid.org/0000-0001-7849-0087

a.denniston@bham.ac.uk

Journal

Journal ID (nlm-ta): Nat Med

Journal ID (iso-abbrev): Nat Med

Title: Nature Medicine

Publisher: Nature Publishing Group US (New York )

ISSN (Print): 1078-8956

ISSN (Electronic): 1546-170X

Publication date (Electronic): 9 September 2020

Publication date PMC-release: 9 September 2020

Publication date (Print): 2020

Volume: 26

Issue: 9

Pages: 1351-1363

Affiliations

[1 ]ISNI 0000 0004 1936 7486, GRID grid.6572.6, Centre for Patient Reported Outcomes Research, , Institute of Applied Health Research, University of Birmingham, ; Birmingham, UK

[2 ]ISNI 0000 0004 1936 7486, GRID grid.6572.6, Institute of Applied Health Research, University of Birmingham, ; Birmingham, UK

[3 ]ISNI 0000 0004 1936 7486, GRID grid.6572.6, Birmingham Health Partners Centre for Regulatory Science and Innovation, , University of Birmingham, ; Birmingham, UK

[4 ]ISNI 0000 0004 1936 7486, GRID grid.6572.6, Academic Unit of Ophthalmology, , Institute of Inflammation and Ageing, University of Birmingham, ; Birmingham, UK

[5 ]ISNI 0000 0004 0376 6589, GRID grid.412563.7, University Hospitals Birmingham NHS Foundation Trust, ; Birmingham, UK

[6 ]GRID grid.507332.0, Health Data Research UK, ; London, UK

[7 ]ISNI 0000 0000 9168 0080, GRID grid.436474.6, Moorfields Eye Hospital NHS Foundation Trust, ; London, UK

[8 ]ISNI 0000 0001 2157 2938, GRID grid.17063.33, Department of Medicine, , Women’s College Research Institute, Women’s College Hospital, University of Toronto, ; Ontario, Canada

[9 ]ISNI 0000000121901201, GRID grid.83440.3b, National Institute of Health Research Biomedical Research Centre for Ophthalmology, Moorfields Hospital London NHS Foundation Trust and University College London, Institute of Ophthalmology, ; London, UK

[10 ]ISNI 0000 0004 0376 6589, GRID grid.412563.7, National Institute of Health Research Birmingham Biomedical Research Centre, University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, ; Birmingham, UK

[11 ]National Institute of Health Research Applied Research Collaborative West Midlands, Coventry, UK

[12 ]ISNI 0000 0004 0376 6589, GRID grid.412563.7, National Institute of Health Research Surgical Reconstruction and Microbiology Centre, , University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, ; Birmingham, UK

[13 ]ISNI 0000 0001 2113 8111, GRID grid.7445.2, Patient Safety Translational Research Centre, , Imperial College London, ; London, UK

[14 ]ISNI 0000 0001 2113 8111, GRID grid.7445.2, Institute of Global Health Innovation, Imperial College London, ; London, UK

[15 ]ISNI 0000 0004 5903 3632, GRID grid.499548.d, Alan Turing Institute, ; London, UK

[16 ]ISNI 0000 0004 1936 8948, GRID grid.4991.5, Department of Statistics and Nuffield Department of Medicine, , University of Oxford, ; Oxford, UK

[17 ]ISNI 0000000121662407, GRID grid.5379.8, University of Manchester, ; Manchester, UK

[18 ]ISNI 0000 0000 9606 5108, GRID grid.412687.e, Centre for Journalology, Clinical Epidemiology Program, , Ottawa Hospital Research Institute, ; Ottawa, Canada

[19 ]ISNI 0000 0001 2182 2255, GRID grid.28046.38, School of Epidemiology and Public Health, Faculty of Medicine, , University of Ottawa, ; Ottawa, Canada

[20 ]ISNI 0000 0000 8587 8621, GRID grid.413354.4, Department of Ophthalmology, , Cantonal Hospital Lucerne, ; Lucerne, Switzerland

[21 ]ISNI 0000 0000 8809 1613, GRID grid.7372.1, University of Warwick, ; Coventry, UK

[22 ]ISNI 0000000122986657, GRID grid.34477.33, Department of Ophthalmology, , University of Washington, ; Seattle, WA USA

[23 ]ISNI 0000 0004 1794 1878, GRID grid.416710.5, The National Institute for Health and Care Excellence, ; London, UK

[24 ]Salesforce Research, San Francisco, CA USA

[25 ]ISNI 000000041936754X, GRID grid.38142.3c, Harvard T.H. Chan School of Public Health, ; Boston, MA USA

[26 ]GRID grid.57981.32, Medicines and Healthcare products Regulatory Agency, ; London, UK

[27 ]ISNI 0000 0001 1034 2272, GRID grid.431129.c, New England Journal of Medicine, ; Waltham, MA USA

[28 ]Google Health, London, UK

[29 ]Annals of Internal Medicine, Philadelphia, PA USA

[30 ]Patient Partner, Birmingham, UK

[31 ]British Medical Journal, London, UK

[32 ]ISNI 0000 0001 2297 5165, GRID grid.94365.3d, National Institutes of Health, ; Bethesda, MD USA

[33 ]Patient Partner, London, UK

[34 ]ISNI 0000 0004 1936 7486, GRID grid.6572.6, Patient Partner, Centre for Patient Reported Outcome Research, , Institute of Applied Health Research, University of Birmingham, ; Birmingham, UK

[35 ]ISNI 0000 0004 1936 8948, GRID grid.4991.5, Centre for Statistics in Medicine, , University of Oxford, ; Oxford, UK

[36 ]Hardian Health, London, UK

[37 ]ISNI 0000 0004 5929 4381, GRID grid.417815.e, AstraZeneca, ; Cambridge, UK

[38 ]Nature Research, New York, NY USA

[39 ]ISNI 0000 0001 2243 3366, GRID grid.417587.8, Food and Drug Administration, ; Silver Spring, MD USA

[40 ]Australian Institute for Machine Learning, North Terrace, Adelaide, Australia

[41 ]ISNI 0000 0004 0473 9646, GRID grid.42327.30, The Hospital for Sick Children, ; Toronto, Canada

[42 ]PinPoint Data Science, Leeds, UK

[43 ]ISNI 0000 0004 4647 675X, GRID grid.413701.0, Journal of the American Medical Association, ; Chicago, IL USA

[44 ]The Lancet Group, London, UK

[45 ]ISNI 0000000122478951, GRID grid.14105.31, Medical Research Council, ; London, UK

Author information

Xiaoxuan Liu http://orcid.org/0000-0002-1286-0038

Alastair K. Denniston http://orcid.org/0000-0001-7849-0087

Melanie J. Calvert http://orcid.org/0000-0002-1856-837X

Article

Publisher ID: 1037

DOI: 10.1038/s41591-020-1037-7

PMC ID: 7598944

PubMed ID: 32908284

SO-VID: 607bf894-cd9b-4fba-9da9-58abc2a2c55b

License:

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

History

Date received : 24 April 2020

Date accepted : 23 July 2020

Funding

Funded by: FundRef https://doi.org/10.13039/100004440, Wellcome Trust (Wellcome);

Funded by: FundRef https://doi.org/10.13039/100012338, Alan Turing Institute;

Custom metadata

ScienceOpen disciplines: Medicine

Keywords: clinical trial design,technology

Data availability:

ScienceOpen disciplines: Medicine

Keywords: clinical trial design, technology

Comments

Comment on this article

scite_

Smart Citations

Citing PublicationsSupportingMentioningContrasting

View Citations

See how this article has been cited at scite.ai

scite shows how a scientific paper has been cited by providing the context of the citation, a classification describing whether it supports, mentions, or contrasts the cited claim, and a label indicating in which section the citation was made.

Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension

Read this article at

Abstract

Abstract

Related collections

UCL: UN SDG 03 Good Health and Well-Being

Most cited references 48

Dermatologist-level classification of skin cancer with deep neural networks

SPIRIT 2013 statement: defining standard protocol items for clinical trials.

SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials

Author and article information

Contributors

Journal

Affiliations

Author information

Article

History

Funding

Categories

Custom metadata

Comments

Comment on this article

Similar content 136

Cited by 165

Most referenced authors 1,149