Opening Keynote Address | Multimodal Foundation Models for Biomedical AI
Biomedical AI is creating new opportunities to connect tissue structure, molecular biology, and the course of disease over time.
In this keynote, I will trace our work from computational pathology to multimodal foundation models for medicine. I will discuss CLAM for data-efficient learning (Nature Biomedical Engineering, 2021), TOAD for predicting the origins of cancers of unknown primary (Nature, 2021), CRANE for assessing cardiac transplant rejection (Nature Medicine, 2022), and Pathomic Fusion and PORPOISE for integrating histology with molecular profiles. Advances in hierarchical learning and image retrieval through HIPT and SISH helped establish the foundation for UNI and CONCH (Nature Medicine, 2024), TITAN (Nature Medicine, 2025), and THREADS, which learn transferable representations from tissue images, language, and molecular data. I will also discuss PathChat (Nature, 2024), which combines image interpretation and dialogue in a multimodal AI assistant for pathology.
Extending this work across biological scales, I will present TriPath for three-dimensional pathology (Cell, 2024), HESTfor connecting histology and spatial transcriptomics (NeurIPS, 2024), and recent advances through VORTEX for predicting three-dimensional gene expression and KRONOS for spatial proteomics. I will then introduce APOLLO, our multimodal, temporal foundation model developed using approximately 25 billion clinical records from more than seven million patients spanning over three decades. APOLLO integrates structured events, clinical text, and medical images into evolving patient representations, supporting forecasting of disease onset, progression, treatment response, and adverse events across medical specialties.
Together, these studies illustrate a path toward AI that connects biological measurements with the longitudinal context of patient care. I will conclude with the challenges of validation, fairness, and clinical translation, alongside opportunities for AI agents and biomedical simulation.
Objectives
Describe the evolution from task-specific computational pathology methods to multimodal foundation models and generative AI assistants.
Explain how integrating imaging, molecular, spatial, and longitudinal clinical data enables disease characterization and prediction of patient outcomes.
Evaluate opportunities and challenges for clinical translation, including model generalizability, rigorous validation, algorithmic fairness, and integration into clinical workflows.
Speakers
Faisal Mahmood, PhD
Associate Professor, Harvard Medical School
Associate Professor, Division of Computational Pathology, Brigham and Women's Hospital