Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Reducing racial bias in AI models for clinical use requires a top-down intervention

Nature Machine Intelligence volume 3, page 460 (2021)Cite this article

Subjects

To the Editor — Artificial intelligence (AI) models built for clinical tasks show clear patterns of discrimination against patients of colour1,2. This discrimination, however, may not show up as an error when training or testing as the data themselves are likely to be error-ridden, in the form of misdiagnosis or prognosis, for minority patients. Many calls to correct racial bias focus on traditional methods of diversifying datasets to include more minority patients, or even consulting social scientists when creating and evaluating datasets and model performance2,3. While these actions may be helpful to some extent, they fail to address the root cause, which is the incorrect clinical data. Currently, AI models are regarded as successful if they effectively mimic what a clinician would do, offering no incentive, let alone avenue, for AI researchers to improve on clinical practices. Researchers are expected to build models that closely emulate clinical data, which inadvertently include the racial bias. To fully address the issue of racial bias in AI models for clinical use, a top-down intervention is required. Regulatory bodies and publishing standards must move away from relying solely on agreement with a clinical dataset for accuracy, and also require evidence of objective accuracy, even if it deviates from clinical data.

This is a preview of subscription content

Access options

Buy article

Get time limited or full article access on ReadCube.

$32.00

Buy

All prices are NET prices.

References

  1. Zou, J. & Schiebinger, L. Nature 559, 324–326 (2018).

    Article  Google Scholar 

  2. Vyas, D. A., Eisenstein, L. G. & Jones, D. S. N. Engl. J. Med. 383, 874–882 (2020).

    Article  Google Scholar 

  3. Owens, K. & Walker, A. Nat. Med. 26, 1327–1328 (2020).

    Article  Google Scholar 

  4. Software as a Medical Device (SAMD): Clinical Evaluation — Guidance for Industry and Food and Drug Administration Staff (FDA, 2017).

  5. Cruz Rivera, S. et al. Nat. Med. 26, 1351–1363 (2020).

    Article  Google Scholar 

  6. Liu, X. et al. Nat. Med. 26, 1364–1374 (2020).

    Article  Google Scholar 

  7. Geiger, H. J. in Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care (Smedley, B. D. et al.) 417–454 (National Academies Press, 2003).

  8. Cone, D. C., Richardson, L. D., Todd, K. H., Betancourt, J. R. & Lowe, R. A. Acad. Emerg. Med. 10, 1176–1183 (2003).

    Article  Google Scholar 

  9. Esteva, A. et al. Nature 542, 115–118 (2017).

    Article  Google Scholar 

  10. Adamson, A. S. & Smith, A. JAMA Dermatol. 154, 1247–1248 (2018).

    Article  Google Scholar 

Download references

Acknowledgements

This work was made possible through funding from the Precision Medicine: Ethics, Politics and Culture Project, Precision Medicine & Society, Columbia University.

Author information

Affiliations

  1. Mailman School of Public Health, Department of Sociomedical Sciences, Columbia University, New York, NY, USA

    Supriya Kapur

Authors
  1. Supriya Kapur
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Supriya Kapur.

Ethics declarations

Competing interests

The author declares no competing interests.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kapur, S. Reducing racial bias in AI models for clinical use requires a top-down intervention. Nat Mach Intell 3, 460 (2021). https://doi.org/10.1038/s42256-021-00362-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-021-00362-7

Further reading

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing