- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FINE, an information flow-guided noise intervention framework for mitigating text dominance in early-fusion multimodal large language models. The work aims to increase the contribution of visual information to model predictions. The authors first formulate visual reliance using conditional mutual information and introduce layer-conditioned mutual information perturbed difference as a tractable surrogate. Then they inject Gaussian noise into visual-aligned hidden states and maximize the Jensen-Shannon divergence between clean and corrupted predictions during fine-tuning. They further derive approximation-gap decompositions to analyze the effects of intervention depth and noise level. Experiments on Qwen3-VL models of three scales and five multimodal benchmarks report improvements over matched supervised fine-tuning.
Critical strengths
- The paper studies the important problem of text dominance in early-fusion MLLMs and provides an intuitive information-theoretic formulation.
- The proposed method provides a meaningful extension of MCR from late-fusion models to early-fusion MLLMs by applying controllable Gaussian interventions to intermediate visual representations.
- The paper provides theoretical analysis of the approximation gaps and derives lower- and upper-bound relationships among CMI, LC-MIPD, and the practical JSD objective.
Critical weaknesses
- Assumption 3.1(ii) states that , but its justification is not sufficiently explained. According to Eq. (5), if the prediction depends solely on the intermediate representations, the stronger condition should hold directly. The authors should clarify the theoretical basis of this assumption and explain why is additionally included in the conditioning set.
- The expression for at the beginning of Section 3.4 appears to be incorrect. According to Eq. (7), the correct expression should be$M_k = H(\hat Y \mid \tilde Z_1^{(k)}, Z_2^{(k)}, X_2) - H(\hat Y \mid Z_1^{(k)}, Z_2^{(k)}, X_2).$ The expression currently provided actually corresponds to , rather than . The authors should correct this inconsistency.
- The perturbation controls in Table 2 are not fully matched with FINE. Text-Perturb and Visual-Perturb modify only a sampled subset of tokens, whereas FINE perturbs all visual tokens and therefore has greater total perturbation energy. This confound weakens the claim that the gain is specifically caused by visual intervention.
- FINE doubles the batch and reruns the remaining decoder layers for the corrupted branch, but no training-time, memory, or computational-overhead comparison is reported. An efficiency analysis against matched SFT is needed.
- The manuscript contains several grammatical and typographical errors, such as "Using a the trained model" in Section 4.3, "As what we expect" in Section 4.1, and "We tests whether" in Appendix H. The paper would benefit from careful proofreading.
- The discussion of using mutual information to characterize the effect of noise is incomplete. Related studies have analyzed beneficial noise through mutual information and variational bounds, including MuNG [1]. References: [1]"Explore how to inject beneficial noise in mllms".
Questions for authors
See weaknesses. If the authors can adequately address these concerns, I would consider raising my score.
4 comments
Response to Reviewer r9MY
W4: Training-time, memory, and computational overhead
The analytic decoder-block workload relative to SFT is about , but this ignores the language-model head and the full-vocabulary JSD. We therefore also report measured update time and memory in T6 over the depth grid of Qwen3-VL-2B, restricted to valid decoder depths . The measurements use the same LoRA configuration, optimizer, image/text batch, sequence length of 1024, and global batch of 64 for SFT and FINE. On one RTX 4090, the global batch is reached with 16 accumulation steps of four examples.
T6: Efficiency at matched settings
| Method | Sec/update | Samples/sec | Peak GPU memory | Extra decoder FLOPs | |
|---|---|---|---|---|---|
| SFT | -- | 14.460 0.061 | 4.43 0.02 | 6.72 0.00 GiB | 0 |
| FINE | 4 | 26.177 0.097 | 2.44 0.01 | 7.15 0.00 GiB | |
| FINE | 6 | 25.473 0.294 | 2.51 0.03 | 7.12 0.00 GiB | |
| FINE | 10 | 23.346 0.070 | 2.74 0.01 | 7.06 0.00 GiB | |
| FINE | 14 | 21.414 0.073 | 2.99 0.01 | 6.99 0.00 GiB | |
| FINE | 18 | 19.478 0.052 | 3.29 0.01 | 6.93 0.00 GiB | |
| FINE | 22 | 17.618 0.075 | 3.63 0.02 | 6.87 0.00 GiB | |
| FINE | 26 | 15.658 0.084 | 4.09 0.02 | 6.81 0.00 GiB |
Profiling keeps the image/text batch resident on the GPU, so the reported time covers the forward pass, backward pass, gradient clipping, and optimizer step, but excludes data loading. Warm-up updates are discarded, and the reported mean standard deviation is computed over three complete optimizer updates. Peak GPU memory is the PyTorch allocator peak (torch.cuda.max_memory_allocated). "Extra decoder FLOPs" is the analytic replayed-tail fraction ; the measured time additionally includes both language-model heads and the full-vocabulary JSD.
At the chosen depth , FINE takes the SFT update time ( versus s) and reaches of SFT throughput. Measured peak memory increases from to GiB, and the largest observed increase at any depth is GiB at . This is smaller than a naive doubling because gradient checkpointing bounds activation retention and the response projection is token-chunked. Measured total overhead decreases monotonically with depth, from at to at ; this is why the compute–accuracy default avoids unnecessarily early intervention. The deepest measured point is cheaper because it replays only the final two decoder blocks, not because it is a better fidelity choice. At inference, the trained model is used without the noisy branch, so the extra memory and decoder work are both zero.
W5: Typographical and consistency errors
We will fix the cited examples and complete a full proofreading pass over the text, appendices, tables, captions, and equations. We will also correct the following:
- the leftover label in Figure 1;
- "Section B", which should read Appendix B, and the stray "B" in "calibrated selection B of ";
- the Appendix D reference to "Table 1", which should be Table 3;
- the Table 7 / Figure 3 grid mismatch in noise levels and benchmarks;
- the naming mismatch between Text-/Visual-Perturb (Table 2) and Text-/Visual-Row (Appendix H);
- the description of the objective in Appendix F as the "scaled JSD in (10)", whereas Eq. (10) uses the unscaled JSD;
- the broken sentence in the Introduction beginning "mathematically prove that either the approximation gap…";
- the numerical tolerance for the synthetic validation, which is stated as in Appendix D but as "less than " in Section 4.3, while the largest residual in Table 3 is .
We will also revise the explanation in Section 4.1 of the MCR degradation via Theorem 3.6, which is not directly supported by an experiment, so that it reads as a hypothesis.
Response to Reviewer r9MY
W6: Beneficial noise and MI bounds
Relation to variational-bound analyses of beneficial noise
A complementary line of work analyzes noise through variational bounds on conditional entropy rather than through conditional mutual information (CMI). For instance, MuNG defines the complexity of a multimodal task as its entropy , where is the answer, and calls a noise variable beneficial if . It introduces a tractable decoder and uses the non-negativity of KL divergence to obtain the variational upper bound
\mathbb{E}_{p(A, X_V, X_L, \mathcal{E})}[-\log q(A \mid X_V, X_L, \mathcal{E})] \ge H(\mathcal{T} \mid \mathcal{E}).$$ This bound is then minimized by learning a noise generator through the reparameterization trick. That perspective asks whether noise *reduces uncertainty about the correct answer*, treating noise as a learned perturbation that is also applied at inference. FINE addresses a different question. In early-fusion multimodal large language models (MLLMs), the dominant failure mode is *text dominance*: the model can predict correctly from text alone and underuses visual evidence. FINE therefore regularizes the *reliance* of the prediction on the visual input, measured by the conditional mutual information $\mathrm{CMI} = I(X_1; \hat{Y} \mid X_2)$, and injects calibrated Gaussian noise *during training* to construct the corrupted branch required for the LC-MIPD surrogate. The theoretical analysis in Section 3 characterizes the *fidelity* of this surrogate, specifically the approximation gap $\mathrm{CMI} - J_k$, as a function of the injection depth and the noise level. It does not claim that noise directly reduces $H(\hat{Y} \mid X_1, X_2)$; rather, it establishes where and how intensely to intervene so that the Jensen–Shannon divergence (JSD) objective remains a faithful proxy for CMI. **Conceptual distinctions and complementarity** The two quantities are related but distinct. Because $$\hspace{25em} \mathrm{CMI} = H(\hat{Y} \mid X_2) - H(\hat{Y} \mid X_1, X_2),$$ maximizing CMI reduces $H(\hat{Y} \mid X_1, X_2)$ only if $H(\hat{Y} \mid X_2)$ remains stable. During training, however, that term fluctuates: a model can become more confident overall or more visually sensitive without necessarily improving in absolute accuracy. Conversely, a model can remain confident and accurate while still exhibiting text dominance on simpler instances. Consequently, FINE and MuNG target different failure modes (under-reliance versus uncertainty) and use noise in different operational roles (training regularizer versus learned perturbation generator). We view them as complementary: a promising direction for future work is to combine CMI-based regularization with an entropy-based objective, so that models both make better use of visual evidence and reduce overall answer uncertainty.Thank you for the detailed rebuttal and additional clarifications. The authors have addressed most of my concerns, and I will maintain my original score.
Response to Reviewer r9MY
We thank the reviewer for the careful reading and the constructive questions.
W1: Conditioning set in Assumption 3.1(ii)
We apologize for the unclear presentation. Assumption 3.1(ii) states: "Once we know the intermediate hidden representations (Z1 and Z2) and the original text input (X2), knowing the original image X1 provides no additional information about the model's prediction Y^." Equivalently,
I(X_1; \hat{Y} \mid Z_1, Z_2, X_2) = 0.$$ The role of $X_2$ in the conditioning set is as follows. Eq. (5) implies that $\hat Y$ depends on the inputs only through $(Z_1,Z_2)$, i.e., $\hat Y\perp(X_1,X_2)\mid(Z_1,Z_2)$. This in turn implies the stronger statement the reviewer mentions, $\hat Y\perp X_1\mid(Z_1,Z_2)$. We state the assumption conditional on $X_2$ because the target CMI $I(X_1;\hat Y\mid X_2)$ and LC-MIPD are both defined conditional on $X_2$, and the proof uses exactly this form. This assumption links the original input-level CMI to LC-MIPD and is used to prove Theorem 3.3 in the Appendix (Equation 22). Recall that the original CMI is $$\hspace{25em} \mathrm{CMI}=I(X_1;\hat Y\mid X_2).$$ Since $(Z_1,Z_2)$ is a deterministic function of $(X_1,X_2)$ (Assumption 3.1(i)), the chain rule of mutual information gives $$\hspace{25em} \begin{aligned} I(X_1;\hat Y\mid X_2) &= I(Z_1,Z_2;\hat Y\mid X_2)+I(X_1;\hat Y\mid Z_1,Z_2,X_2) =I(Z_1,Z_2;\hat Y\mid X_2). \end{aligned}$$ This is why Eq. (22) holds. We will make the justification of Assumption 3.1(ii) clearer in the final manuscript. # W2: Expression for $M_k$ We agree with the correction. The expression at the start of Section 3.4 equals $\rho_k$. The correct form, consistent with Eq. (7) and the proof, is $$\hspace{25em} M_k=H(\hat Y\mid\tilde Z_1^{(k)},Z_2^{(k)},X_2)-H(\hat Y\mid Z_1^{(k)},Z_2^{(k)},X_2).$$ We will correct it and check all other uses in the final manuscript. # W3: Perturbation-energy confound We agree with this concern. Text-Perturb and Visual-Perturb are matched to each other in row count and per-coordinate scale, but FINE perturbs every visual row. We will describe the current controls as row-budget matched and add energy-matched and SNR-matched controls. Because the text-state and visual-state scales differ, both matchings are needed. T5 reports the energy-matched text control; the SNR-matched control, in which the text-state noise is scaled to the text-state RMS, will be added in the revision. **T5: Energy-matched location controls** | Condition | Rows | Variance | Expected energy | MMBench | ChartQA | POPE | | --------------------- | ----------------- | ------------------------------------------ | --------------------------------------------- | ------: | ------: | ---: | | SFT | 0 | 0 | 0 | 72.4 | 71.8 | 82.1 | | Text-Perturb | $\min(n_v,n_t)$ | $\sigma_{\mathrm{rel}}^2$ | $\min(n_v,n_t)\,d\,\sigma_{\mathrm{rel}}^2$ | 72.9 | 70.9 | 82.8 | | Visual-Perturb | $\min(n_v,n_t)$ | $\sigma_{\mathrm{rel}}^2$ | $\min(n_v,n_t)\,d\,\sigma_{\mathrm{rel}}^2$ | 74.9 | 72.8 | 84.4 | | Text (energy-matched) | $n_t$ text | $\frac{n_v}{n_t}\sigma_{\mathrm{rel}}^2$ | $n_v\,d\,\sigma_{\mathrm{rel}}^2$ | 73.2 | 71.5 | 82.5 | | Visual (FINE) | $n_v$ visual | $\sigma_{\mathrm{rel}}^2$ | $n_v\,d\,\sigma_{\mathrm{rel}}^2$ | 77.4 | 74.1 | 87.2 |