Low responsiveness of machine learning models to critical or deteriorating health conditions

 

• Tanmoy Sarkar Pias, 
• Sharmin Afrose, 
• Moon Das Tuli, 
• Ipsita Hamid Trisha, 
• Xinwei Deng, 
• Charles B. Nemeroff & 
• Danfeng Daphne Yao
  
  

  Abstract

  Background

  Machine learning (ML) based mortality prediction models can be immensely useful in intensive care units. Such a model should generate warnings to alert physicians when a patient’s condition rapidly deteriorates, or their vitals are in highly abnormal ranges. Before clinical deployment, it is important to comprehensively assess a model’s ability to recognize critical patient conditions.

  Methods

  We develop multiple medical ML testing approaches, including a gradient ascent method and neural activation map. We systematically assess these machine learning models’ ability to respond to serious medical conditions using additional test cases, some of which are time series. Guided by medical doctors, our evaluation involves multiple machine learning models, resampling techniques, and four datasets for two clinical prediction tasks.

  Results

  We identify serious deficiencies in the models’ responsiveness, with the models being unable to recognize severely impaired medical conditions or rapidly deteriorating health. For in-hospital mortality prediction, the models tested using our synthesized cases fail to recognize 66% of the injuries. In some instances, the models fail to generate adequate mortality risk scores for all test cases. Our study identifies similar kinds of deficiencies in the responsiveness of 5-year breast and lung cancer prediction models.

  Conclusions

  Using generated test cases, we find that statistical machine-learning models trained solely from patient data are grossly insufficient and have many dangerous blind spots. Most of the ML models tested fail to respond adequately to critically ill patients. How to incorporate medical knowledge into clinical machine learning models is an important future research direction.

  Plain language summary

  Computational models can be used to evaluate a patient’s health condition and predict their risk of dying, for example, in the intensive care unit. These models could be useful to identify patients with worsening health conditions and alert doctors promptly. We test how well several computational models recognize patients with serious or worsening health conditions. We find most of the computational models evaluated cannot recognize critical health events in our tests, which is concerning. Our work highlights the importance of using medical knowledge guided testing to ensure models are suitable, as well as the need to incorporate fundamental medical knowledge into the design of such models.
  
  https://www.nature.com/articles/s43856-025-00775-0