acceptodds
Withdrawn from ICLR 2027

ReFuse: Widening Multimodal Fusion via Recursive Selective Refinement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Standard multimodal fusion performs a single integration of all modality representations and feeds the fused result for prediction. This one-shot paradigm risks under-fusing modality information, particularly for weak modalities whose task-relevant signals can be overwhelmed by dominant modalities. To this end, we propose ReFuse (Refinement Fusion), a recursive fusion framework that iteratively diagnoses information insufficiency in the current fused representation and selectively re-fuses only modalities that still carry task-relevant information not yet captured. At the core of is anReFuse information sufficiency diagnoser that estimates the conditional mutual information between each modality and label given the current fused representation, driving both modality selection and adaptive early termination. The fusion module is parameter-shared across all recursion steps, so ReFuse widens modality fusion rather than deepening the network. We provide theoretical analysis showing that ReFuse monotonically increases task-relevant information, achieves approximate sufficiency upon termination, and enjoys better parameter efficiency under equal parameter budgets. Experiments on multiple multimodal benchmarks demonstrate that ReFuse consistently outperforms one-shot fusion baselines and, under strict equal-parameter comparison, outperforms deeper fusion networks especially under modality imbalance.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Multimodal Fusion, Recrusive Refinement, Multimodal Learning, Imbalance Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EqnpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

To address the problem that a one-shot fusion paradigm tends to result in insufficient multimodal information fusion—namely, one modality suppressing another and leading to suboptimal performance—this paper proposes ReFuse, a recursive fusion framework that iteratively diagnoses information deficiencies in the current fused representation and selectively re-fuses only those modalities that contain task-relevant information not yet captured.

Critical strengths

1、The authors propose a recursive refinement fusion framework, i.e., an iterative "diagnose–select–reject" loop, to address the problems in the one-time fusion paradigm.

2、The authors designed an information sufficiency diagnoser based on conditional mutual information.

3、Experimental results on multiple datasets validate the effectiveness of the proposed method.

Critical weaknesses

1、The authors' statement in the introduction that the one-shot fusion paradigm has only one opportunity to fuse task-relevant information from each modality at the fusion layer is not sufficiently accurate. During training, the model updates its parameters at every batch and every epoch, so there is not merely "one fusion opportunity." The authors' problem definition is not clear enough.

2、The authors repeatedly emphasize that modality suppression during fusion may lead to suboptimal performance. However, they do not report the performance of the prediction when using only a single modality on each task. Could modality suppression stem from insufficient unimodal feature extraction at the outset, thereby resulting in degraded performance? Or is it simply because a certain modality does not contain task-relevant discriminative features, causing the multimodal task to ultimately only select the task-relevant modality?

3、In a one-shot fusion process, the suppressed modality may be alleviated by dynamically adjusting gradients. Can the recursive refinement framework then incorporate a dynamic gradient adjustment strategy? If so, how effective would it be? Moreover, a one-shot fusion process is already likely to introduce a substantial amount of noise. Could adding weak modality features over multiple steps introduce even more noise, causing the integration of noisy information to outweigh the discriminative features of the genuinely weak modality?

4、The code is not available, which compromises reproducibility.

5、The writing could be clearer, and the logical connections between sections could be improved, which makes the paper somewhat difficult to follow.

Questions for authors

All the questions that need to be addressed have been listed in the Weaknesses.

Reviewer_FGeLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ReFuse, which aims to address the issue in traditional one-shot multimodal fusion where weak modal information may only be fused once and thus get suppressed by the dominant modality. ReFuse starts from the initial fused representation and, at each step, determines which modalities still contain task-relevant information not yet captured by the current state hth_t, and re-fuses only those modalities. At the same time, it uses adaptive termination to decide when to stop the recursion. The authors conducted extensive theoretical analysis and experimental validation to demonstrate the effectiveness of this method from various perspectives.

Critical strengths
  1. This manuscript raises a valid point: even if a weak-modal encoder already contains useful information, a single fusion may not sufficiently transfer that information to the final multimodal representation. Therefore, I believe the motivation behind this manuscript is sound.

  2. The authors conducted sufficient experiments to demonstrate the effectiveness of the proposed algorithm.

Critical weaknesses
  1. The author needs to provide further clarification on some of the proofs of the formulas.

  2. As shown in Table 8, this method involves a large number of hyperparameters, which suggests that its performance may be attributable to rigorous hyperparameter tuning.

Questions for authors
  1. In Section 4(1), it is stated that recursive fusion does not destroy task information, and that the information increases monotonically with recursion. However, in Lemma A.1 and Theorem 1, the object that actually exhibits monotonicity is St=(ht,{mi})S_t=(h_t,\{m_i\}), that is, the current fused state plus all previously accessed original modality vectors. Once mim_i is directly stored as a variable in St+1S_{t+1}, I(St+1;y)≥I(St;y)I(S_{t+1};y) \ge I(S_t;y) naturally follows to some extent from the addition of an extra observable variable to the state. However, g(hT)g(h_T) is used in the final prediction rather than g(ST)g(S_T). Therefore, I believe the authors should directly prove that I(ht+1;y)≥I(ht;y)I(h_{t+1};y) \ge I(h_t;y). Since the authors have provided a large number of formula-based proofs, if I have overlooked any, please point them out and explain.

  2. Corollary 1 defines the weak-modality residual as: ri(t)=I(mi;y∣St)r_i(t)=I(m_i;y\mid S_t). Once mim_i is selected, the next St+1S_{t+1} directly includes mim_i, which naturally leads to I(mi;y∣St+1)=0I(m_i;y\mid S_{t+1})=0. Therefore, the authors need to clarify: Does the phrase “modality has been captured” here mean that the information from mim_i has truly been integrated into ht+1h_{t+1}, or is it merely because the analytical state St+1S_{t+1} itself directly preserves mim_i?

  3. The paper describes the theoretical advantage of parameter sharing as follows: under the same budget, the spectral capacity of each layer in a deep model is reduced by a factor of approximately T\sqrt{T}. However, the actual assumption in the theory is that $ \sum_{l=1}^T||W_l||_F^2 = ||W||_F^2$, whereas in the experiments, the matched budget is based on roughly the same parameter count. Since these two conditions are generally not equivalent, I would like the authors to explain why an equal parameter count corresponds to an equal Frobenius-norm budget.

  4. In Theorem 3, the guarantee of (1−1/e)(1-1/e) is obtained by the standard greedy subset selection: ik=arg⁡max⁡iΔ(i∣Sk−1)i_k = \arg\max_i \Delta(i\mid S_{k-1}), where, after each modality is selected, the next marginal gain is recalculated based on the new Sk−1S_{k-1}. However, ReFuse actually uses: It={i:sistudent,t>τ}I_t = \{i: s_i^{\text{student},t} > \tau\}, meaning that all modalities are scored simultaneously based on the same hth_t. Therefore, I would like the authors to clarify: Does the current simultaneous threshold rule itself truly inherit the (1−1/e)(1-1/e) guarantee? For example, two modalities might each have a high I(mi;y∣ht)I(m_i;y\mid h_t), but they are highly redundant. If one is selected first, the true marginal gain of the other might already be very small, yet the threshold rule would still select both simultaneously.

  5. As shown in Table 8, this paper uses a large number of hyperparameters. I’m curious to know how the authors tuned these hyperparameters and whether they followed a specific tuning strategy.

Reviewer_RVp5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReFuse, a recursive fusion framework that replaces one-shot multimodal fusion with a “diagnose–select–refuse” loop, so weak modalities get multiple chances to contribute. A label-free student diagnoser, distilled from a teacher based on a loss difference, estimates how much task-relevant information each modality can still add, and uses this to decide which modalities to re-fuse and when to stop. A single parameter-shared fusion unit is reused at every step, and experiments on six benchmarks show gains over one-shot fusion and over deeper or wider networks with the same parameter budget.

Critical strengths
  • The motivation is reasonable. The paper pins down a concrete problem: in one-shot fusion, weak modalities get drowned out by dominant ones and never get a second chance. Using a diagnostic signal to decide what to re-fuse and when to stop is a natural answer to that problem.
  • The improvement is clear. When the fusion unit is replaced with simple operators like AddSum, ConcatMLP, and GMU, adding the loop still improves results (Section 5.3). This helps separate the effect of the loop from the strength of the fusion operator.
Critical weaknesses
  • The mechanism by which re-fusion “recovers information” is unclear. Each modality is averaged over time into a single vector, and usually only one modality is selected per step. In that case, the cross-attention has a single key, so it reduces to a gated linear projection. ReFuse is therefore essentially re-injecting the same vector into a weight-shared block several times, which is closer to “weight-shared iteration + adaptive halting” methods like Universal Transformer or PonderNet. The paper doesn’t explain why reading the same vector repeatedly recovers information that a single stronger fusion step can’t, and the related work doesn’t discuss this line of methods.
  • The comparison setup and statistics are not rigorous enough. including:
  1. The table caption says results marked with † are taken from the original papers, but no entry in the table is actually marked with †. So it’s impossible to tell which results were rerun and which were copied.
  2. Possibly inconsistent text encoders. ReFuse uses DeBERTa on MOSI and MOSEI, while some baselines use BERT.
Questions for authors
  • On MOSEI (Table 5), the one-shot model gets 54.4 Acc7, which is higher than several looped variants: no diagnoser (53.4), no selection (53.6), and unshared units (53.2). How does this fit with the claim that iteration itself is essential?
  • The conditional projection W′_i only receives gradients from the distillation loss, while the teacher score that depends on it is detached. What stops W′_i from collapsing toward zero, which makes both the teacher and student scores trivially zero? Can you show the actual distribution of teacher scores during training?
  • Each modality is averaged over time into a single vector, and usually only one modality is selected per step. In that case, the cross-attention has a single key, so it reduces to a gated linear projection. Why does re-injecting the same fixed vector several times recover information that one stronger fusion step can’t?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.