ReFuse: Widening Multimodal Fusion via Recursive Selective Refinement

Standard multimodal fusion performs a single integration of all modality representations and feeds the fused result for prediction. This one-shot paradigm risks under-fusing modality information, particularly for weak modalities whose task-relevant signals can be overwhelmed by dominant modalities. To this end, we propose ReFuse (Refinement Fusion), a recursive fusion framework that iteratively diagnoses information insufficiency in the current fused representation and selectively re-fuses only modalities that still carry task-relevant information not yet captured. At the core of is anReFuse information sufficiency diagnoser that estimates the conditional mutual information between each modality and label given the current fused representation, driving both modality selection and adaptive early termination. The fusion module is parameter-shared across all recursion steps, so ReFuse widens modality fusion rather than deepening the network. We provide theoretical analysis showing that ReFuse monotonically increases task-relevant information, achieves approximate sufficiency upon termination, and enjoys better parameter efficiency under equal parameter budgets. Experiments on multiple multimodal benchmarks demonstrate that ReFuse consistently outperforms one-shot fusion baselines and, under strict equal-parameter comparison, outperforms deeper fusion networks especially under modality imbalance.