Who Gets to Constrain the Decision? Authority-Conditioned Graph Fusion for Clinical Multimodal Learning

Multimodal feature fusion combines heterogeneous evidence for structured decision-making, but clinically motivated scenarios raise a fundamental question: when predictive strength, reliability, and evidential authority conflict, which evidence source should constrain the final output? We formulate this problem as an authority-asymmetric multimodal learning process and propose the Authority-Consistent Fusion (ACF) framework that treats authority as a task-dependent specification. ACF includes three components: a label-free Authority Specification Compiler to derive explicit authority constraints, an Authority-Protected Graph Mixer to prevent unauthorized information propagation, and an Authority-Consistent Projection to enforce admissible outputs under missing, invalid, and conflicting evidence. Under explicit assumptions, we establish relation-preservation, fixed-parameter non-interference, and conditional squared-loss guarantees for the proposed framework. Experiments on the UTSW MRI-to-semantic retrieval benchmark demonstrate that ACF achieves comparable calibration to information-matched learned joint graph fusion, with a clean Brier score difference within 7.0x10-5. Controlled evaluations further show that ACF maintains a zero S1 authority-violation rate under activated relations and incurs Brier increases of only 0.084–0.189 under incorrect authority specifications. Additional matched experiments attribute 6.49 and 2.48 mAP improvements to the evaluated geometric and communication configurations, respectively. ACF establishes evidential authority as an explicit decision variable and provides a principled foundation for reliable and accountable multimodal decision-making when heterogeneous evidence sources disagree.