Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Reply to Winter et al: Interpreting weights of multimodal machine learning models—problems and pitfalls

Neuropsychopharmacology volume 46, page 1863 (2021)Cite this article

Subjects

In their correspondence, Winter et al. [1] raised concerns with the application of machine learning to examine associations between brain variables and childhood maltreatment in [2]. The primary concern was that the association between maltreatment and brain variables may have been obscured because the reported model contained non-brain covariates. Specifically, the results may have fallen victim to the Rashomon effect – the possibility that there are numerous combinations of brain variables that yield comparable findings to the reported model due to the inclusion of clinically-relevant covariates. This concern is important given the possible instability of machine learning results [3]. We addressed this concern in two ways. First, the brain regions were selected by aggregating over a set of 500 models, which is consistent with Breiman’s recommendation for combating the Rashomon effect [3]. Second, we evaluated 250,000 competing models, constructed from permuting features from the entire feature set. As indicated in supplemental materials, the reported model (AUC = .90) outperformed all competitor models (AUCMean = .74), which suggested the specific features in the reported model were likely associated with maltreatment.

This is a preview of subscription content

Access options

Buy article

Get time limited or full article access on ReadCube.

$32.00

Buy

All prices are NET prices.

References

  1. Winter NR, Goltermann J, Dannlowski U, Hahn T. Interpreting weights of multimodal machine learning models—problems and pitfalls. Neuropsychopharmacology. 2021:1–2. https://www.nature.com/articles/s41386-021-01030-5#citeas.

  2. Price M, Albaugh M, Hahn S, Juliano AC, Fani N, Brier ZMF, et al. Examination of the association between exposure to childhood maltreatment and brain structure in young adults: a machine learning analysis. Neuropsychopharmacology. 2021:1–7. https://www.nature.com/articles/s41386-021-01030-5#citeas.

  3. Breiman L. Statistical modeling: the two cultures (with comments and a rejoinder by the author). Stat Sci. 2001;16:199–231.

    Article  Google Scholar 

Download references

Funding & Disclosures

The authors report no biomedical financial interests or potential conflicts of interest. This work received support from the following sources: NIMH K08MH107661-A1 and the European Union-funded FP6 Integrated Project IMAGEN (Reinforcement-related behavior in normal brain function and psychopathology) (LSHM-CT- 2007-037286).

Author information

Affiliations

  1. Center for Research on Emotion Stress and Technology, Department of Psychological Science, University of Vermont, Burlington, VT, USA

    Matthew Price

  2. Department of Psychiatry, University of Vermont, Burlington, VT, USA

    Nicholas Allgaier & Hugh Garavan

Authors
  1. Matthew Price
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicholas Allgaier
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hugh Garavan
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

MP: conceptualized the response, conducted follow-up analyses, and drafted the majority of the manuscript. NA: Provided substantial guidance on the analysis and reviewed drafts of the manuscript. HG: Reviewed drafts of the manuscript and provided input on the narrative direction.

Corresponding author

Correspondence to Matthew Price.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Price, M., Allgaier, N. & Garavan, H. Reply to Winter et al: Interpreting weights of multimodal machine learning models—problems and pitfalls. Neuropsychopharmacol. 46, 1863 (2021). https://doi.org/10.1038/s41386-021-01082-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41386-021-01082-7

Search

Advanced search

Quick links