Tri-Modal Bone Cancer Intelligence: Late-Fusion and Cross-Modal Attention over Radiographs, WSIs, and Omics

Abstract

Bone cancers have diverse radiologic and histopathologic characteristics, and visual ambiguity frequently constrains single-modality AI. To create a tri-modal system that learns from (i) radiographs in the Bone Cancer Detection (Kaggle) dataset, (ii) whole-slide images (WSIs) with weak labels through multiple-instance learning (MIL), and (iii) RNA-seq±mutation profiles (TARGET-OS) represented by a compact variational/Multi Layer Perceptron (MLP) bottleneck. To align modality latents with Maximum Mean Discrepancy (MMD) and Information Noise-Contrastive Estimation (InfoNCE) contrast, and then to employ reliability-aware late-fusion with optional cross-modal co-attention to combine them. Temperature scaling adjusts the chances. The fused model gets Accuracy of 0.926±0.016, Macro-averaged F1 score of 0.914±0.018, Area Under the Receiver Operating Characteristic Curve (AUROC)of 0.965±0.010, Area Under the Precision–Recall Curve (AUPRC)of 0.958±0.011, Brier score of 0.067±0.008, and Expected Calibration Error (ECE)of 0.018±0.006 on tests that were held out. Single streams do worse (Radiograph AUROC 0.940; WSI-MIL 0.918; Omics 0.902), while two-stream combinations close much of the gap. Ablations show that alignment and co-attention are quite important (for example, taking out MMD lowers AUROC by 0.012 and raises ECE by 0.006). Robustness experiments demonstrate elegant decline in the presence of X-ray blur/jitter, stain jitter, and omics batch shifts; the external-site AUROC remains robust at 0.958 (fused). An adjusted operating point θ* gives Coverage 94.1%, Sensitivity 0.936, Specificity 0.903, Positive Predictive Value (PPV) 0.907, Negative Predictive Value (NPV) 0.933, and the best utility when costs are high for false negatives. This study shows that combining morphologic and molecular signals with an awareness of uncertainty leads to accurate, well-calibrated, and more generalizable predictions of bone cancer. This is true even when only radiographs are available.