Deep multimodal fusion of image and non-image data in identification of high-risk carotid atheromatous plaque

Abstract

Cardiovascular diseases remain the leading cause of global mortality, accounting for over 19 million deaths annually according to the World Health Organization. Among these, carotid atherosclerosis, a pathological process characterized by plaque accumulation in the carotid arteries, is a major contributor to ischemic stroke, driven by plaque rupture and thromboembolic events. Traditional diagnostic approaches, which rely on single-modality imaging or clinical data, often fail to capture the complex interplay of morphological, biological, and hemodynamic factors that determine plaque vulnerability. This dissertation addresses this critical gap by developing advanced deep multimodal fusion frameworks to integrate heterogeneous data sources, enabling precise risk stratification of carotid atheromatous plaques. The primary objective of this work is to create an end-to-end trainable system that synergistically combines B-mode carotid ultrasound imaging with non-image clinical data, including biochemical markers, protein biomarkers, and patient demographics. Three fusion strategies were rigorously investigated: (1) Joint Attention-Based Fusion, which dynamically weights imaging and tabular data contributions through learned attention mechanisms; (2) Early Fusion, merging raw inputs at the feature level; and (3) AttentionGated Video Hybrid Fusion, a novel architecture designed to process spatiotemporal ultrasound frame sequences alongside clinical data. These models were trained and validated on a multimodal dataset comprising 96 DICOM ultrasound recordings and 73 curated clinical profiles from a cohort of 73 patients, stratified into high-risk (symptomatic with ≥ 50% stenosis or asymptomatic with ≥ 70% stenosis) and low-risk groups. The Joint Attention-Based Fusion model with an EfficientNet-B0 backbone achieved superior performance, yielding AUC: 86.07%, Balanced Accuracy: 73.28%, F1 Score: 78.42 %, Sensitivity: 81.67%, outperforming unimodal approaches (imaging-only AUC: 84.55%; tabular-only AUC: 64.61%). Attention weights highlighted the dominance of imaging data (69.4% contribution), while clinical biomarkers provided complementary risk context. The proposed Attention-Gated Video Hybrid Fusion framework demonstrated feasibility for dynamic plaque analysis but faced computational constraints in scaling 3D spatiotemporal convolutions. In conclusion, this thesis advances the field of precision vascular medicine by demonstrating that multimodal deep learning can substantially improve the accuracy and reliability of carotid atheromatous plaque risk assessment compared to conventional single-modality approaches. The proposed architectures, particularly the attention-based fusion paradigm, provide a scalable framework for integrating heterogeneous data sources. Additionally, a prototype Clinical Decision Support System (CDSS) was developed as a web-based interface, illustrating the translational potential of these models for future integration into clinical workflows. Together, these contributions highlight the promise of AI-driven methodologies in enhancing stroke prevention strategies and supporting individualized therapeutic decision-making.