Disparities in dermatology AI performance on a diverse, curated clinical image set

There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

Abstract

An estimated 3 billion people lack access to dermatological care globally. Artificial intelligence (AI) may aid in triaging skin diseases and identifying malignancies. However, most AI models have not been assessed on images of diverse skin tones or uncommon diseases. Thus, we created the Diverse Dermatology Images (DDI) dataset—the first publicly available, expertly curated, and pathologically confirmed image dataset with diverse skin tones. We show that state-of-the-art dermatology AI models exhibit substantial limitations on the DDI dataset, particularly on dark skin tones and uncommon diseases. We find that dermatologists, who often label AI datasets, also perform worse on images of dark skin tones and uncommon diseases. Fine-tuning AI models on the DDI images closes the performance gap between light and dark skin tones. These findings identify important weaknesses and biases in dermatology AI that should be addressed for reliable application to diverse patients and diseases.

Abstract

A diverse, curated clinical dermatology image set can help address AI in dermatology limitations.

Related collections

Most cited references 46

Record: found
Abstract: found
Article: not found

Dermatologist-level classification of skin cancer with deep neural networks

Andre Esteva, Brett Kuprel, Roberto Chacón-Novoa … (2017)

Skin cancer, the most common human malignancy, is primarily diagnosed visually, beginning with an initial clinical screening and followed potentially by dermoscopic analysis, a biopsy and histopathological examination. Automated classification of skin lesions using images is a challenging task owing to the fine-grained variability in the appearance of skin lesions. Deep convolutional neural networks (CNNs) show potential for general and highly variable tasks across many fine-grained object categories. Here we demonstrate classification of skin lesions using a single CNN, trained end-to-end from images directly, using only pixels and disease labels as inputs. We train a CNN using a dataset of 129,450 clinical images—two orders of magnitude larger than previous datasets—consisting of 2,032 different diseases. We test its performance against 21 board-certified dermatologists on biopsy-proven clinical images with two critical binary classification use cases: keratinocyte carcinomas versus benign seborrheic keratoses; and malignant melanomas versus benign nevi. The first case represents the identification of the most common cancers, the second represents the identification of the deadliest skin cancer. The CNN achieves performance on par with all tested experts across both tasks, demonstrating an artificial intelligence capable of classifying skin cancer with a level of competence comparable to dermatologists. Outfitted with deep neural networks, mobile devices can potentially extend the reach of dermatologists outside of the clinic. It is projected that 6.3 billion smartphone subscriptions will exist by the year 2021 (ref. 13) and can therefore potentially provide low-cost universal access to vital diagnostic care.

0 comments Cited 2095 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: found

Is Open Access

The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions

Philipp Tschandl, Cliff Rosendahl, Harald Kittler (2018)

Training of neural networks for automated diagnosis of pigmented skin lesions is hampered by the small size and lack of diversity of available datasets of dermatoscopic images. We tackle this problem by releasing the HAM10000 (“Human Against Machine with 10000 training images”) dataset. We collected dermatoscopic images from different populations acquired and stored by different modalities. Given this diversity we had to apply different acquisition and cleaning methods and developed semi-automatic workflows utilizing specifically trained neural networks. The final dataset consists of 10015 dermatoscopic images which are released as a training set for academic machine learning purposes and are publicly available through the ISIC archive. This benchmark dataset can be used for machine learning and for comparisons with human experts. Cases include a representative collection of all important diagnostic categories in the realm of pigmented lesions. More than 50% of lesions have been confirmed by pathology, while the ground truth for the rest of the cases was either follow-up, expert consensus, or confirmation by in-vivo confocal microscopy.

0 comments Cited 336 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists

T Buhl, A. Blum, A Kalloo … (2018)

Deep learning convolutional neural networks (CNN) may facilitate melanoma detection, but data comparing a CNN's diagnostic performance to larger groups of dermatologists are lacking.

0 comments Cited 298 times – based on 0 reviews      Review now

Bookmark

All references

Author and article information

Contributors

Roxana Daneshjou:

ORCID: https://orcid.org/0000-0001-7988-9356

Role: ConceptualizationRole: Data curationRole: Formal analysisRole: InvestigationRole: MethodologyRole: Project administrationRole: ResourcesRole: SoftwareRole: SupervisionRole: ValidationRole: VisualizationRole: Writing - original draftRole: Writing - review & editing

Kailas Vodrahalli:

ORCID: https://orcid.org/0000-0002-9849-3055

Role: ConceptualizationRole: Data curationRole: Formal analysisRole: InvestigationRole: MethodologyRole: SoftwareRole: ValidationRole: VisualizationRole: Writing - original draftRole: Writing - review & editing

Roberto A. Novoa: Role: ConceptualizationRole: Funding acquisitionRole: InvestigationRole: MethodologyRole: ResourcesRole: SupervisionRole: ValidationRole: VisualizationRole: Writing - review & editing

Melissa Jenkins: Role: ConceptualizationRole: Data curationRole: InvestigationRole: Project administrationRole: SupervisionRole: Validation

Weixin Liang: Role: ConceptualizationRole: Data curationRole: Formal analysisRole: InvestigationRole: MethodologyRole: SoftwareRole: ValidationRole: Writing - review & editing

Veronica Rotemberg:

ORCID: https://orcid.org/0000-0003-0639-2677

Role: ConceptualizationRole: MethodologyRole: SupervisionRole: ValidationRole: Writing - review & editing

Justin Ko: Role: ConceptualizationRole: Data curationRole: Funding acquisitionRole: MethodologyRole: ValidationRole: Writing - review & editing

Susan M. Swetter:

ORCID: https://orcid.org/0000-0003-2098-2948

Role: Data curationRole: InvestigationRole: Writing - review & editing

Elizabeth E. Bailey: Role: ConceptualizationRole: InvestigationRole: Methodology

Olivier Gevaert:

ORCID: https://orcid.org/0000-0002-9965-5466

Role: ConceptualizationRole: Formal analysisRole: Funding acquisitionRole: MethodologyRole: SupervisionRole: Writing - review & editing

Pritam Mukherjee:

ORCID: https://orcid.org/0000-0002-9975-9994

Role: Formal analysisRole: InvestigationRole: MethodologyRole: SoftwareRole: Validation

Michelle Phung: Role: Project administrationRole: Supervision

Kiana Yekrang:

ORCID: https://orcid.org/0000-0003-0714-5360

Role: Data curationRole: InvestigationRole: Project administrationRole: Resources

Bradley Fong: Role: Formal analysisRole: InvestigationRole: Project administrationRole: ResourcesRole: Visualization

Rachna Sahasrabudhe:

ORCID: https://orcid.org/0000-0003-0019-3554

Role: Formal analysisRole: InvestigationRole: Project administrationRole: ResourcesRole: Visualization

Johan A. C. Allerup:

ORCID: https://orcid.org/0000-0002-1517-2505

Role: Data curation

Utako Okata-Karigane:

ORCID: https://orcid.org/0000-0003-0540-1711

Role: Investigation

James Zou: Role: ConceptualizationRole: Funding acquisitionRole: MethodologyRole: Project administrationRole: SupervisionRole: VisualizationRole: Writing - original draftRole: Writing - review & editing

Albert S. Chiou:

ORCID: https://orcid.org/0000-0002-8436-0008

Role: ConceptualizationRole: Data curationRole: Formal analysisRole: Funding acquisitionRole: InvestigationRole: MethodologyRole: Project administrationRole: ResourcesRole: SupervisionRole: ValidationRole: VisualizationRole: Writing - original draftRole: Writing - review & editing

Journal

Journal ID (nlm-ta): Sci Adv

Journal ID (iso-abbrev): Sci Adv

Journal ID (publisher-id): sciadv

Journal ID (hwp): advances

Title: Science Advances

Publisher: American Association for the Advancement of Science

ISSN (Electronic): 2375-2548

Publication date Collection: August 2022

Publication date (Electronic, pub): 12 August 2022

Volume: 8

Issue: 31

Electronic Location Identifier: eabq6147

Affiliations

[ ¹ ]Department of Dermatology, Stanford School of Medicine, Redwood City, CA, USA.

[ ² ]Department of Biomedical Data Science, Stanford School of Medicine, Stanford, CA, USA.

[ ³ ]Department of Electrical Engineering, Stanford University, Stanford, CA, USA.

[ ⁴ ]Department of Pathology, Stanford School of Medicine, Stanford, CA, USA.

[ ⁵ ]Department of Computer Science, Stanford University, Stanford, CA, USA.

[ ⁶ ]Dermatology Service, Memorial Sloan Kettering Cancer Center, New York, NY, USA.

[ ⁷ ]Department of Dermatology, Keio University School of Medicine, Tokyo, Japan.

[ ⁸ ]Chan-Zuckerberg Biohub, San Francisco, CA, USA.

Author notes

[* ]Corresponding author. Email: jamesz@ 123456stanford.edu (J.Z.); achiou@ 123456stanford.edu (A.S.C.)

[†]

These authors contributed equally to this work.

[‡]

Present address: Department of Radiology and Imaging Sciences, Imaging Biomarkers and Computer-Aided Diagnosis Laboratory, National Institutes of Health Clinical Center, Bethesda, MD, USA.

[§]

Present address: Des Moines University, Des Moines, IA, USA.

Author information

Roxana Daneshjou https://orcid.org/0000-0001-7988-9356

Kailas Vodrahalli https://orcid.org/0000-0002-9849-3055

Veronica Rotemberg https://orcid.org/0000-0003-0639-2677

Susan M. Swetter https://orcid.org/0000-0003-2098-2948

Olivier Gevaert https://orcid.org/0000-0002-9965-5466

Pritam Mukherjee https://orcid.org/0000-0002-9975-9994

Kiana Yekrang https://orcid.org/0000-0003-0714-5360

Rachna Sahasrabudhe https://orcid.org/0000-0003-0019-3554

Johan A. C. Allerup https://orcid.org/0000-0002-1517-2505

Utako Okata-Karigane https://orcid.org/0000-0003-0540-1711

Albert S. Chiou https://orcid.org/0000-0002-8436-0008

Article

Publisher ID: abq6147

DOI: 10.1126/sciadv.abq6147

PMC ID: 9374341

PubMed ID: 35960806

SO-VID: 52c5338f-995d-42b4-ae39-13f1f8e125be

Copyright © Copyright © 2022 The Authors, some rights reserved; exclusive licensee American Association for the Advancement of Science. No claim to original U.S. Government Works. Distributed under a Creative Commons Attribution NonCommercial License 4.0 (CC BY-NC).

License:

This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial license, which permits use, distribution, and reproduction in any medium, so long as the resultant use is not for commercial advantage and provided the original work is properly cited.

History

Date received : 19 April 2022

Date accepted : 30 June 2022

Funding

Funded by: FundRef http://dx.doi.org/10.13039/100000052, NIH Office of the Director;

Award ID: 5T32AR007422-38

Funded by: FundRef http://dx.doi.org/10.13039/100000179, NSF Office of the Director;

Award ID: CAREER 1942926

Funded by: FundRef http://dx.doi.org/10.13039/100000179, NSF Office of the Director;

Award ID: Graduate Research Fellowship