The World's First Multimodal AI Model in Sarcoma

---

What is the main takeaway from this paper?

Better, and individualized, predictions of a sarcoma patient's overall survival and risk of metastases are needed to help us choose optimal treatments and improve survival. Our multimodal neural network (MMNN), capable of analyzing an MRI of a patient's sarcoma as well as clinical variables, has better performance than previous prediction models, and radiomics models. 

Future work should focus on enabling accurate subtype-specific predictions to further individualize sarcoma patient management. The future deep learning AI models that guide the management of our patients are likely to incorporate end-to-end neural networks, gradient blending, prospective curation of high-quality data, the inclusion of genomic data, and the involvement of multiple centers through federated learning. 

How can federated learning help?

In order to provide the best predictions, the ideal MMNN would train on as many patients from as many different institutions as possible. Federated learning allows an algorithm in the cloud, such as our MMNN, to train on patients from multiple institutions - without that data having to be sent or shared outside the institution. This helps preserve patient privacy while easing some barriers to collaboration. 

How does this model compare to current models used in treatment today?

This model surpasses all previous prediction models in sarcoma, such as Sarculator.  The multimodal neural network is able to analyze the MRI of patient's sarcoma, as well as their clinical variables, in order to predict overall survival and risk of metastases. By analyzing individualized data like MRIs, the model can enable precision medicine - matching individual patients to the best treatment specific for them. 

What inspired this project?

This work started when I was a fellow at Memorial Sloan Kettering Cancer Center, and will continue here at the McGill University Health Center. Work is underway to externally validate this published model using federated learning. I am grateful for the opportunity to present this work at the upcoming MSTS and CTOS international sarcoma meetings, and I aim to recruit multiple collaborative centers to increase the training data and generalizability of the model. 

Dr Robert Turcotte established a top tier sarcoma unit at the MUHC. Alongside Dr Ahmed Aoude, Dr James Tsui and Dr Natalia Gorelik, we will continue to advance the use of AI to help our sarcoma patients at the MUHC.

Were there mentors or institutions involved in your research?

I am deeply grateful to the MGH Foundation and Cedars Cancer Foundation for their support, and to my research mentors Dr Michelle Ghert and Dr John Healey for their guidance. 

How will this study improve sarcoma care in the future?

By analyzing individualized data like MRIs, the model can enable precision medicine - matching individual patients to the best treatment specific for them. This may improve their survival and other patient important outcomes. More work is needed and we will continue to strive to improve outcomes with our research in the future.

What are the benefits of my (or any) institution collaborating in future versions of this multimodal AI model?

We have open sourced our model and are seeking to help other institutions be able to firstly perform multimodal AI studies on their own, and secondly collaborate with us on future versions of our model. 

Please click here to learn more about collaboration.

Where is the future of sarcoma headed with your model?

While our published model is currently the best in sarcoma, it can be improved in several ways. 

We hope to one day provide for our sarcoma patients subtype specific predictions that guide their treatment and surveillance decisions on an individual basis, in order to improve their survival and outcomes. Work is underway to collaborate with other institutions to obtain as much training data as possible to enable these subtype specific predictions. Additionally, we hope to include genomic data, in addition to the MRI and clinical variables, which is likely to explain some of the remaining variability in our predictions. 

The Sarcoma AI Hub at the MUHC will be central to the development of future versions of this model.