← 研踪 PaperTrailarXiv:2605.31504 · v2
When Are Multimodal Predictions Biologically Supported? A Diagnostic Evaluation Framework
首次提交 2026-05-30 00:25
版本更新 2026-11-08 01:33
雷达观测 2026-11-10 11:14
摘要
Multimodal models in oncology can produce accurate predictions, but accurate prediction does not reveal whether the model has learned biology that is shared across modalities, biology confined to one modality, or spurious correlations that reflect confounders rather than genuine biology. We introduce DECAT, a model-agnostic post-hoc evaluation framework that classifies multimodal representations into four diagnostic scenarios (shared, spurious, modality-specific, or no signal) for a given task and modality, using five null-referenced metrics and a rule-based decision procedure. The framework operates on learned representations, requires no knowledge of which specific confounder is present, and returns indeterminate when the evidence is insufficient. We evaluate multimodal architecture properties using DECAT on synthetic data across four multimodal model classes spanning over 2,500 trained representations, and demonstrate its operational use on 8,979 TCGA patients with both multimodal model embeddings and five pretrained pathology foundation models. Entangled models (e.g., CLIP) detect shared signal near-perfectly on synthetic data but also falsely claim it when it is absent, doing so in the majority of such cases on real foundation model embeddings. This false claim rate increases with signal strength and sample size, so that larger cohorts and stronger representations produce more confident but still incorrect diagnoses. Applied to both multimodal TCGA embeddings and five pathology foundation models without paired RNA, DECAT detects confounding invisible to AUROC without requiring the confounder labels, as confirmed by post-hoc stratification.
作者备注：NeurIPS 2026 (Evaluations and Datasets Track). Code: https://github.com/dst53/DECAT
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-08 01:33 · 最新观测版本
When Are Multimodal Predictions Biologically Supported? A Diagnostic Evaluation Framework
Multimodal models in oncology can produce accurate predictions, but accurate prediction does not reveal whether the model has learned biology that is shared across modalities, biology confined to one modality, or spurious correlations that reflect confounders rather than genuine biology. We introduce DECAT, a model-agnostic post-hoc evaluation framework that classifies multimodal representations into four diagnostic scenarios (shared, spurious, modality-specific, or no signal) for a given task and modality, using five null-referenced metrics and a rule-based decision procedure. The framework operates on learned representations, requires no knowledge of which specific confounder is present, and returns indeterminate when the evidence is insufficient. We evaluate multimodal architecture properties using DECAT on synthetic data across four multimodal model classes spanning over 2,500 trained representations, and demonstrate its operational use on 8,979 TCGA patients with both multimodal model embeddings and five pretrained pathology foundation models. Entangled models (e.g., CLIP) detect shared signal near-perfectly on synthetic data but also falsely claim it when it is absent, doing so in the majority of such cases on real foundation model embeddings. This false claim rate increases with signal strength and sample size, so that larger cohorts and stronger representations produce more confident but still incorrect diagnoses. Applied to both multimodal TCGA embeddings and five pathology foundation models without paired RNA, DECAT detects confounding invisible to AUROC without requiring the confounder labels, as confirmed by post-hoc stratification.
查看此版本解读 →查看此版本原文 ↗