Fusion is not one-size-fits-all: Cross-Modal Representation Alignment for Time-to-Event Modeling

A new framework aligns CT imaging and electronic health record data using foundation models, improving time-to-event prediction for pulmonary embolism and cardiovascular disease across multiple institutions, researchers report. The framework encodes CT scans and longitudinal EHR data independently with domain-specific foundation models, then aligns them in a shared latent space through four fusion strategies: late fusion, contrastive alignment, cross-attention, and co-attention [1][2]. The study evaluated two clinically distinct time-to-event tasks — pulmonary embolism mortality and cardiovascular disease outcomes — using large-scale multi-institutional cohorts. The pulmonary embolism arm included 3,099 training samples, 1,098 internal validation samples, and 435 external samples; the cardiovascular disease arm comprised 2,951 training samples, 837 internal samples, and 682 external samples [1][2]. Fusion consistently improved the concordance index by 1.5 to 5.4 percent over unimodal baselines when both modalities contributed comparably [1][2]. Contrastive multimodal fusion, particularly when paired with CLMBR representations, delivered the most consistent and statistically robust gains, especially for pulmonary embolism mortality prediction [1][2]. For major adverse cardiovascular events, cross-attention with one-hot encoding achieved the highest internal performance, while image-guided co-attention performed best on external data [1][2]. CT scanning, which uses a rotating X-ray tube and detectors to measure tissue attenuation and reconstruct cross-sectional images, has been a staple of medical diagnosis since its development in the 1970s [3]. The technique earned Godfrey Hounsfield and Allan MacLeod Cormack the 1979 Nobel Prize in Physiology or Medicine [3]. Combining such imaging with structured EHR data introduces challenges of modality imbalance and distribution shift, which the new framework explicitly addresses [1][2]. The authors describe the work as the first systematic analysis of fusion behavior under modality imbalance in time-to-event prediction and argue that task-aware multimodal alignment is a necessary design principle for robust generalization and scalable clinical deployment [1][2]. The paper was submitted to arXiv on 13 June 2026 [1].