acceptodds
Under review as a conference paper at ICLR 2027

FINE: Information Flow-Guided Intervention with Noise for Mitigating Text Dominance in Multimodal Large Language Models

Abstract

Multimodal LLMs, typically powered by pretrained LLMs, often suffer from undesirable text dominance, with the non-text modalities underutilized. Most of the existing approaches for modality balancing focus on late-fusion model architectures, with the balancing mechanism applied before the fusion. For the MLLMs adopting early-fusion architectures, information of multimodal inputs flows across modalities over the model layers, making the balancing non-trivial. To this end, we propose an information flow-aware learning framework which alleviates text dominance by injecting noises to the intermediate transformer layers of MLLMs, with reference to the cross-modality information flow. We measure text dominance using conditional mutual information and propose to approximate it using a newly proposed layer-conditional mutual information perturbed difference (LC-MIPD). We show that the approximation gap is characterized by the information flow from visual to textual and the visual information retained given noise injection, which provides a theoretical ground for controlling the noise intervention mechanism. We conduct extensive experiments on Qwen3-VL models across multiple multimodal benchmarks, and demonstrate improvement on overall performance with reduced hallucinations results.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLM, Text Dominance, Mutual Information Perturbed Difference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_r9MYOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FINE, an information flow-guided noise intervention framework for mitigating text dominance in early-fusion multimodal large language models. The work aims to increase the contribution of visual information to model predictions. The authors first formulate visual reliance using conditional mutual information and introduce layer-conditioned mutual information perturbed difference as a tractable surrogate. Then they inject Gaussian noise into visual-aligned hidden states and maximize the Jensen-Shannon divergence between clean and corrupted predictions during fine-tuning. They further derive approximation-gap decompositions to analyze the effects of intervention depth and noise level. Experiments on Qwen3-VL models of three scales and five multimodal benchmarks report improvements over matched supervised fine-tuning.

Critical strengths
  1. The paper studies the important problem of text dominance in early-fusion MLLMs and provides an intuitive information-theoretic formulation.
  2. The proposed method provides a meaningful extension of MCR from late-fusion models to early-fusion MLLMs by applying controllable Gaussian interventions to intermediate visual representations.
  3. The paper provides theoretical analysis of the approximation gaps and derives lower- and upper-bound relationships among CMI, LC-MIPD, and the practical JSD objective.
Critical weaknesses
  1. Assumption 3.1(ii) states that Y^⊥X1∣(Z1(k),Z2(k),X2)\hat Y \perp X_1 \mid (Z_1^{(k)}, Z_2^{(k)}, X_2), but its justification is not sufficiently explained. According to Eq. (5), if the prediction depends solely on the intermediate representations, the stronger condition Y^⊥X1∣(Z1(k),Z2(k))\hat Y \perp X_1 \mid (Z_1^{(k)}, Z_2^{(k)}) should hold directly. The authors should clarify the theoretical basis of this assumption and explain why X2X_2 is additionally included in the conditioning set.
  2. The expression for MkM_k at the beginning of Section 3.4 appears to be incorrect. According to Eq. (7), the correct expression should be$M_k = H(\hat Y \mid \tilde Z_1^{(k)}, Z_2^{(k)}, X_2) - H(\hat Y \mid Z_1^{(k)}, Z_2^{(k)}, X_2).$ The expression currently provided actually corresponds to ρk\rho_k, rather than MkM_k. The authors should correct this inconsistency.
  3. The perturbation controls in Table 2 are not fully matched with FINE. Text-Perturb and Visual-Perturb modify only a sampled subset of tokens, whereas FINE perturbs all visual tokens and therefore has greater total perturbation energy. This confound weakens the claim that the gain is specifically caused by visual intervention.
  4. FINE doubles the batch and reruns the remaining decoder layers for the corrupted branch, but no training-time, memory, or computational-overhead comparison is reported. An efficiency analysis against matched SFT is needed.
  5. The manuscript contains several grammatical and typographical errors, such as "Using a the trained model" in Section 4.3, "As what we expect" in Section 4.1, and "We tests whether" in Appendix H. The paper would benefit from careful proofreading.
  6. The discussion of using mutual information to characterize the effect of noise is incomplete. Related studies have analyzed beneficial noise through mutual information and variational bounds, including MuNG [1]. References: [1]"Explore how to inject beneficial noise in mllms".
Questions for authors

See weaknesses. If the authors can adequately address these concerns, I would consider raising my score.

4 comments
AuthorsOpenReview ↗

Response to Reviewer r9MY

We thank the reviewer for the careful reading and the constructive questions.

W1: Conditioning set in Assumption 3.1(ii)

We apologize for the unclear presentation. Assumption 3.1(ii) states: "Once we know the intermediate hidden representations (Z1Z_1 and Z2Z_2) and the original text input (X2X_2), knowing the original image X1X_1 provides no additional information about the model's prediction Y^\hat{Y}." Equivalently,

I(X_1; \hat{Y} \mid Z_1, Z_2, X_2) = 0.$$ The role of $X_2$ in the conditioning set is as follows. Eq. (5) implies that $\hat Y$ depends on the inputs only through $(Z_1,Z_2)$, i.e., $\hat Y\perp(X_1,X_2)\mid(Z_1,Z_2)$. This in turn implies the stronger statement the reviewer mentions, $\hat Y\perp X_1\mid(Z_1,Z_2)$. We state the assumption conditional on $X_2$ because the target CMI $I(X_1;\hat Y\mid X_2)$ and LC-MIPD are both defined conditional on $X_2$, and the proof uses exactly this form. This assumption links the original input-level CMI to LC-MIPD and is used to prove Theorem 3.3 in the Appendix (Equation 22). Recall that the original CMI is $$\hspace{25em} \mathrm{CMI}=I(X_1;\hat Y\mid X_2).$$ Since $(Z_1,Z_2)$ is a deterministic function of $(X_1,X_2)$ (Assumption 3.1(i)), the chain rule of mutual information gives $$\hspace{25em} \begin{aligned} I(X_1;\hat Y\mid X_2) &= I(Z_1,Z_2;\hat Y\mid X_2)+I(X_1;\hat Y\mid Z_1,Z_2,X_2) =I(Z_1,Z_2;\hat Y\mid X_2). \end{aligned}$$ This is why Eq. (22) holds. We will make the justification of Assumption 3.1(ii) clearer in the final manuscript. # W2: Expression for $M_k$ We agree with the correction. The expression at the start of Section 3.4 equals $\rho_k$. The correct form, consistent with Eq. (7) and the proof, is $$\hspace{25em} M_k=H(\hat Y\mid\tilde Z_1^{(k)},Z_2^{(k)},X_2)-H(\hat Y\mid Z_1^{(k)},Z_2^{(k)},X_2).$$ We will correct it and check all other uses in the final manuscript. # W3: Perturbation-energy confound We agree with this concern. Text-Perturb and Visual-Perturb are matched to each other in row count and per-coordinate scale, but FINE perturbs every visual row. We will describe the current controls as row-budget matched and add energy-matched and SNR-matched controls. Because the text-state and visual-state scales differ, both matchings are needed. T5 reports the energy-matched text control; the SNR-matched control, in which the text-state noise is scaled to the text-state RMS, will be added in the revision. **T5: Energy-matched location controls** | Condition | Rows | Variance | Expected energy | MMBench | ChartQA | POPE | | --------------------- | ----------------- | ------------------------------------------ | --------------------------------------------- | ------: | ------: | ---: | | SFT | 0 | 0 | 0 | 72.4 | 71.8 | 82.1 | | Text-Perturb | $\min(n_v,n_t)$ | $\sigma_{\mathrm{rel}}^2$ | $\min(n_v,n_t)\,d\,\sigma_{\mathrm{rel}}^2$ | 72.9 | 70.9 | 82.8 | | Visual-Perturb | $\min(n_v,n_t)$ | $\sigma_{\mathrm{rel}}^2$ | $\min(n_v,n_t)\,d\,\sigma_{\mathrm{rel}}^2$ | 74.9 | 72.8 | 84.4 | | Text (energy-matched) | $n_t$ text | $\frac{n_v}{n_t}\sigma_{\mathrm{rel}}^2$ | $n_v\,d\,\sigma_{\mathrm{rel}}^2$ | 73.2 | 71.5 | 82.5 | | Visual (FINE) | $n_v$ visual | $\sigma_{\mathrm{rel}}^2$ | $n_v\,d\,\sigma_{\mathrm{rel}}^2$ | 77.4 | 74.1 | 87.2 |
AuthorsOpenReview ↗

Response to Reviewer r9MY

W4: Training-time, memory, and computational overhead

The analytic decoder-block workload relative to SFT is about 1+L−kL1+\frac{L-k}{L}, but this ignores the language-model head and the full-vocabulary JSD. We therefore also report measured update time and memory in T6 over the depth grid of Qwen3-VL-2B, restricted to valid decoder depths k∈{4,6,10,14,18,22,26}k\in\{4,6,10,14,18,22,26\}. The measurements use the same LoRA configuration, optimizer, image/text batch, sequence length of 1024, and global batch of 64 for SFT and FINE. On one RTX 4090, the global batch is reached with 16 accumulation steps of four examples.

T6: Efficiency at matched settings

MethodkkSec/updateSamples/secPeak GPU memoryExtra decoder FLOPs
SFT--14.460 ±\pm 0.0614.43 ±\pm 0.026.72 ±\pm 0.00 GiB0
FINE426.177 ±\pm 0.0972.44 ±\pm 0.017.15 ±\pm 0.00 GiB(L−4)/L=85.7%(L-4)/L = 85.7\%
FINE625.473 ±\pm 0.2942.51 ±\pm 0.037.12 ±\pm 0.00 GiB(L−6)/L=78.6%(L-6)/L = 78.6\%
FINE1023.346 ±\pm 0.0702.74 ±\pm 0.017.06 ±\pm 0.00 GiB(L−10)/L=64.3%(L-10)/L = 64.3\%
FINE1421.414 ±\pm 0.0732.99 ±\pm 0.016.99 ±\pm 0.00 GiB(L−14)/L=50.0%(L-14)/L = 50.0\%
FINE1819.478 ±\pm 0.0523.29 ±\pm 0.016.93 ±\pm 0.00 GiB(L−18)/L=35.7%(L-18)/L = 35.7\%
FINE2217.618 ±\pm 0.0753.63 ±\pm 0.026.87 ±\pm 0.00 GiB(L−22)/L=21.4%(L-22)/L = 21.4\%
FINE2615.658 ±\pm 0.0844.09 ±\pm 0.026.81 ±\pm 0.00 GiB(L−26)/L=7.1%(L-26)/L = 7.1\%

Profiling keeps the image/text batch resident on the GPU, so the reported time covers the forward pass, backward pass, gradient clipping, and optimizer step, but excludes data loading. Warm-up updates are discarded, and the reported mean ±\pm standard deviation is computed over three complete optimizer updates. Peak GPU memory is the PyTorch allocator peak (torch.cuda.max_memory_allocated). "Extra decoder FLOPs" is the analytic replayed-tail fraction (L−k)/L(L-k)/L; the measured time additionally includes both language-model heads and the full-vocabulary JSD.

At the chosen depth k∗=14k^*=14, FINE takes 1.48×1.48\times the SFT update time (21.4121.41 versus 14.4614.46 s) and reaches 67.5%67.5\% of SFT throughput. Measured peak memory increases from 6.726.72 to 6.996.99 GiB, and the largest observed increase at any depth is 0.430.43 GiB at k=4k=4. This is smaller than a naive doubling because gradient checkpointing bounds activation retention and the response projection is token-chunked. Measured total overhead decreases monotonically with depth, from 1.81×1.81\times at k=4k=4 to 1.08×1.08\times at k=26k=26; this is why the compute–accuracy default avoids unnecessarily early intervention. The deepest measured point is cheaper because it replays only the final two decoder blocks, not because it is a better fidelity choice. At inference, the trained model is used without the noisy branch, so the extra memory and decoder work are both zero.

W5: Typographical and consistency errors

We will fix the cited examples and complete a full proofreading pass over the text, appendices, tables, captions, and equations. We will also correct the following:

  • the leftover LTDRL_{TDR} label in Figure 1;
  • "Section B", which should read Appendix B, and the stray "B" in "calibrated selection B of (k,σ)(k,\sigma)";
  • the Appendix D reference to "Table 1", which should be Table 3;
  • the Table 7 / Figure 3 grid mismatch in noise levels and benchmarks;
  • the naming mismatch between Text-/Visual-Perturb (Table 2) and Text-/Visual-Row (Appendix H);
  • the description of the objective in Appendix F as the "scaled JSD in (10)", whereas Eq. (10) uses the unscaled JSD;
  • the broken sentence in the Introduction beginning "mathematically prove that either the approximation gap…";
  • the numerical tolerance for the synthetic validation, which is stated as 10−410^{-4} in Appendix D but as "less than 2×10−42\times10^{-4}" in Section 4.3, while the largest residual in Table 3 is 2×10−42\times10^{-4}.

We will also revise the explanation in Section 4.1 of the MCR degradation via Theorem 3.6, which is not directly supported by an experiment, so that it reads as a hypothesis.

AuthorsOpenReview ↗

Response to Reviewer r9MY

W6: Beneficial noise and MI bounds

Relation to variational-bound analyses of beneficial noise

A complementary line of work analyzes noise through variational bounds on conditional entropy rather than through conditional mutual information (CMI). For instance, MuNG defines the complexity of a multimodal task T\mathcal{T} as its entropy H(T)=E[−log⁡p(A∣XV,XL)]H(\mathcal{T})=\mathbb{E}[-\log p(A \mid X_V, X_L)], where AA is the answer, and calls a noise variable E\mathcal{E} beneficial if I(T;E)=H(T)−H(T∣E)>0I(\mathcal{T}; \mathcal{E}) = H(\mathcal{T}) - H(\mathcal{T} \mid \mathcal{E}) > 0. It introduces a tractable decoder q(A∣XV,XL,E)q(A \mid X_V, X_L, \mathcal{E}) and uses the non-negativity of KL divergence to obtain the variational upper bound

\mathbb{E}_{p(A, X_V, X_L, \mathcal{E})}[-\log q(A \mid X_V, X_L, \mathcal{E})] \ge H(\mathcal{T} \mid \mathcal{E}).$$ This bound is then minimized by learning a noise generator through the reparameterization trick. That perspective asks whether noise *reduces uncertainty about the correct answer*, treating noise as a learned perturbation that is also applied at inference. FINE addresses a different question. In early-fusion multimodal large language models (MLLMs), the dominant failure mode is *text dominance*: the model can predict correctly from text alone and underuses visual evidence. FINE therefore regularizes the *reliance* of the prediction on the visual input, measured by the conditional mutual information $\mathrm{CMI} = I(X_1; \hat{Y} \mid X_2)$, and injects calibrated Gaussian noise *during training* to construct the corrupted branch required for the LC-MIPD surrogate. The theoretical analysis in Section 3 characterizes the *fidelity* of this surrogate, specifically the approximation gap $\mathrm{CMI} - J_k$, as a function of the injection depth and the noise level. It does not claim that noise directly reduces $H(\hat{Y} \mid X_1, X_2)$; rather, it establishes where and how intensely to intervene so that the Jensen–Shannon divergence (JSD) objective remains a faithful proxy for CMI. **Conceptual distinctions and complementarity** The two quantities are related but distinct. Because $$\hspace{25em} \mathrm{CMI} = H(\hat{Y} \mid X_2) - H(\hat{Y} \mid X_1, X_2),$$ maximizing CMI reduces $H(\hat{Y} \mid X_1, X_2)$ only if $H(\hat{Y} \mid X_2)$ remains stable. During training, however, that term fluctuates: a model can become more confident overall or more visually sensitive without necessarily improving in absolute accuracy. Conversely, a model can remain confident and accurate while still exhibiting text dominance on simpler instances. Consequently, FINE and MuNG target different failure modes (under-reliance versus uncertainty) and use noise in different operational roles (training regularizer versus learned perturbation generator). We view them as complementary: a promising direction for future work is to combine CMI-based regularization with an entropy-based objective, so that models both make better use of visual evidence and reduce overall answer uncertainty.

Thank you for the detailed rebuttal and additional clarifications. The authors have addressed most of my concerns, and I will maintain my original score.

Reviewer_mTTpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

FINE aims to mitigate text dominance in multimodal LLMs, by injecting Gaussian noise into visual-token hidden states at an intermediate layer during training and maximizing the JSD between clean and perturbed predictions. The paper analyzes cross-modal information flow and uses it to motivate the choice of intervention depth and noise scale. Experiments on Qwen3-VL-2B/4B/8B report gains over baselines on downstream benchmarks.

Critical strengths
  1. The paper tries to explore an important problem. LLaVA-style MLLMs do exhibit substantial text dominance, and this issue matters as the paradigm has already been scaled. Reducing the damage that text dominance causes to multimodal capability is therefore an important direction.
  2. The authors approach the problem from an information-theoretic perspective, which is extremely hard. They explicitly recognize that LLaVA-style early-fusion MLLMs differ from the late-fusion settings used in much prior work on modality balancing, and develop their approach with this difference in mind rather than directly transferring methods studied in traditional small models.
Critical weaknesses
  1. The theoretical analysis provides limited guidance for the method’s design, and it remains unclear whether optimizing the proposed JSD objective achieves its intended theoretical target. As I understand it, the paper does not show that the chosen procedure minimizes the gap between JSD and LC-MIPD or CMI. Moreover, J_k = M_k + C_k − S_k does not imply that maximizing JSD increases M_k or CMI, since the relative sizes of C_k and S_k are not controlled on the evaluated models.
  2. The experiments do not directly domanstrate that text dominance has been mitigated. The paper reports results on MMBench, ChartQA, MMStar, MME, and POPE, but benchmark improvements alone do not show that the model relies more on non-text evidence. The evaluation would be more convincing with broader coverage of vision-centric benchmarks, and more importantly, additional analyses about the change in how model uses visual and language information.
Questions for authors
  1. Section 4.4 selects the intervention layer using KL changes under visual-state perturbation, while Appendix B/I calibrates noise relative to the visual-state RMS. Can the authors show that these design choices reduce either of the two gaps analyzed in the paper: the gap between CMI and LC-MIPD, or the gap between LC-MIPD and JSD?

  2. Given J_k = M_k + C_k − S_k, what evidence supports that maximizing JSD increases CMI or visual information use on the evaluated models, since the relative sizes of C_k and S_k are not controlled?

  3. Could the authors evaluate FINE on additional vision-centric benchmarks and provide analyses that directly test whether predictions rely on non-text evidence?

4 comments
AuthorsOpenReview ↗

Response to Reviewer mTTp

We thank the reviewer for the suggestion of clarifying the key logical gap between maximizing JkJ_k and increasing CMI.

Q1: Do the depth and noise choices reduce either gap?

Short answer. The depth choice trades one gap (ℓk\ell_k) component against training cost and accuracy, while the relative-noise choice tightens an upper bound on the first gap (Theorem 3.3) but has no monotonic guarantee on the second (Theorem 3.6).

Layer selection

The first gap has the exact decomposition

\mathrm{CMI}-M_k=\ell_k+\rho_k,$$ where $\ell_k=I(Z_2^{(k)};\hat Y\mid X_2)$ measures visual information already present in the text state and $\rho_k=I(\tilde Z_1^{(k)};\hat Y\mid Z_2^{(k)},X_2)$ measures visual information remaining in the corrupted visual state. Theorem 3.3 shows that both terms are non-negative. It further motivates the expectation that $\ell_k$ grows with depth as more visual information flows into the text state. Consequently, an earlier layer would reduce the $\ell_k$ component of the first gap. However, $k$ does not control $\ell_k$ alone: $\rho_k$ and the second-gap terms $C_k$ and $S_k$ are also model- and layer-dependent, and the paper provides no monotonic bound for them in $k$. The second gap satisfies $M_k-J_k=S_k-C_k$. Since $C_k$ and $S_k$ enter with opposite signs, reducing either gap cannot be inferred from changing $k$ without measuring their difference. Thus, we cannot claim that the selected layer reduces both gaps. The actual choice $k^* = \lfloor 0.5L \rfloor$ is a compute–accuracy default, not a minimizer of either gap. It gives $k=14$ for Qwen3-VL-2B and $k=18$ for Qwen3-VL-4B/8B. In the 2B profiling, active visual-to-text flow begins around depth 18, so the default precedes substantial flow while avoiding the cost of replaying nearly the entire decoder. For intervention at layer $k$, the decoder-block workload relative to SFT scales as roughly $1+\frac{L-k}{L}$, excluding the language-model head and full-vocabulary JSD (Reviewer r9MY, W4). Earlier layers therefore reduce the $\ell_k$ component but incur a higher training cost. ## Relative noise Fix the intervention layer $k$. Theorem 3.4 gives $$\hspace{25em} \rho_k\le \frac{m}{2}\log\left(1+\frac{E_k}{m\sigma^2}\right),$$ whose bound decreases monotonically to zero as $\sigma$ grows. The proof also establishes that $\rho_k$ is non-increasing in $\sigma$; hence stronger absolute noise can only tighten the first gap $\mathrm{CMI}-M_k$, while $\ell_k$ is unchanged for a fixed model. With the RMS scaling $\sigma_k=\sigma_{\mathrm{rel}}\sqrt{E_k/m}$, the bound becomes $\frac{m}{2}\log(1+\sigma_{\mathrm{rel}}^{-2})$, so larger $\sigma_{\mathrm{rel}}$ tightens the same bound uniformly across depths. At the adopted value $\sigma_{\mathrm{rel}}=0.2$, this bound is still positive: the theorem establishes the direction of change, not that the default eliminates the gap. This guarantee does not extend to the second gap. Theorem 3.6 relates $J_k$ and $M_k$ through $$\hspace{25em} M_k-J_k=S_k-C_k.$$ Larger noise can increase both the decoder-mismatch term $C_k$ and the JSD-compression term $S_k$. Because their difference is not controlled by the theorem, larger noise may reduce, preserve, or enlarge the LC-MIPD–JSD gap depending on which term grows faster. We therefore do not claim that $J_k$ strictly tracks CMI, nor that the noise choice minimizes both gaps. Instead, the decomposition identifies the competing fidelity effects of the intervention.
AuthorsOpenReview ↗

Response to Reviewer mTTp

Q2: Evidence that maximizing JSD increases CMI or visual information use

We agree that Jk=Mk+Ck−SkJ_k=M_k+C_k-S_k provides no unconditional guarantee that increasing JkJ_k increases CMI. Also, FINE optimizes the combined loss Lpred−βJkL_{pred}-\beta J_k, not JkJ_k in isolation. Therefore, we believe that it is important to see whether training with FINE increases CMI or visual reliance relative to matched SFT. To provide empirical evidence, we conduct two additional experiments.

1. Tracing the values of JSD and CMI during training (T1, shared with Reviewer T2VN Q1).

Experiment setting:

Because computing CMICMI, MkM_k, JkJ_k, CkC_k, and SkS_k for comparison is intractable for full MLLMs, we make use of the toy model used in the original submission and apply SFT and FINE to train it. The clean quantities are computed by exact enumeration over the finite support, and the corrupted quantities are computed based on 16,384 common Monte Carlo draws. T1 reports the results at two levels of depths (k=0k=0 and k=2k=2) computed at two training stages (epochs 10 and 30), each repeated over five seeds. This lets us observe CMI and MkM_k directly, while still computing the same JkJ_k optimized during training.

Results:

For all combinations of depth (kk) and epoch, FINE results in higher CMI, MkM_k, and JkJ_k than those of the matched SFT. Across the four depth–epoch cells, the mean differences range from +0.055+0.055 to +0.077+0.077 nats for CMI, +0.017+0.017 to +0.041+0.041 nats for MkM_k, and +0.071+0.071 to +0.190+0.190 nats for JkJ_k. Thus, the run with the higher JSD term also produces higher CMI and LC-MIPD.

Yet, it has to be noted that T1 does not show that JkJ_k is uniformly close to MkM_k. At k=2k=2, the larger Ck−SkC_k-S_k difference makes JkJ_k overestimate MkM_k, and can occasionally exceed the value of CMI.

2. Comparing FINE with noise added to visual state against adding noise to text state (T5, shared with Reviewer r9MY W3).

Experiment setting:

We further conduct an experiment to compare the performance of visual-state perturbation used by FINE with that of text-state perturbation with expected total noise energy matched to FINE's.

Results:

According to T5, applying text-state perturbation achieves 73.2 / 71.5 / 82.5 on MMBench / ChartQA / POPE, whereas FINE obtains 77.4 / 74.1 / 87.2, resulting in performance gain of 4.2, 2.6, and 4.7 points, respectively. The row-budget-matched control shows the same ordering: Visual-Perturb exceeds Text-Perturb on all three benchmarks (74.9 versus 72.9, 72.8 versus 70.9, and 84.4 versus 82.8). This shows that the gain is due to perturbating specifically the visual-state instead of the other state, which indirectly shows the visual information better use.

AuthorsOpenReview ↗

Response to Reviewer mTTp

Q3: Vision-centric benchmarks and direct tests of reliance

As suggested, we further evaluate the existing matched SFT and FINE checkpoints for Qwen3-VL-2B/4B/8B using additional vision-centric benchmarks. The experiment settings adopted are the same as the main evaluation's, including three training seeds, zero-shot prompting, greedy decoding, and the same image-resolution bounds.

Additional vision-centric benchmarks:

  • GQA [1] and CV-Bench [2] broaden the vision-centric evaluation to compositional and controlled 2D/3D reasoning, but accuracy gains alone do not establish image dependence.
  • MMVP [3] uses 150 visually confusable image pairs and reports both per-image accuracy and pair accuracy (both images correct).
  • HallusionBench [4] adds visually grounded yes/no questions; we use its image subset with an API-free local scorer to avoid conflating our results with its GPT-judged score.
  • VLMs-Are-Biased [5] provides a direct counterfactual test using images that contradict memorized priors. The held-out counterfactual pairs in the main paper provide a related test (Table 2), requiring correct responses to pairs with different visual content and correct answers.
  • We also propose a new benchmark score (called image necessity) which is computed as follows: Reuse the 150 MMVP examples and evaluate each example with its real image and a blank image, while keeping the question, options, and visual-token layout fixed. For each checkpoint and seed, we define $ I=100\left(Acc_{real} - Acc_{blank}\right). $

A larger II means that removing visual content causes a larger accuracy loss, and therefore indicates stronger reliance on the image. These tests probe behavior which cannot be isolated by broad Question-Answering accuracy, including i) correctly answering both members of a visually confusable pair, ii) following an image that conflicts with a prior, and iii) degrading when image content is removed.

Results:

T4 reports the mean ±\pm standard deviation of benchmark scores over the seeds in percentage points.

T4: Vision-centric benchmarks and direct reliance tests

BenchmarkSFT (2B)FINE (2B)SFT (4B)FINE (4B)SFT (8B)FINE (8B)
GQA [1]45.02 ±\pm 0.7846.22 ±\pm 0.4247.02 ±\pm 0.6748.22 ±\pm 0.7248.27 ±\pm 0.5748.72 ±\pm 0.73
CV-Bench [2]78.18 ±\pm 0.2479.13 ±\pm 0.2879.77 ±\pm 0.3479.93 ±\pm 0.2182.01 ±\pm 0.5482.33 ±\pm 0.76
MMVP (per-image) [3]70.67 ±\pm 1.1972.33 ±\pm 0.9873.11 ±\pm 1.7973.33 ±\pm 0.8474.72 ±\pm 0.8974.93 ±\pm 1.08
MMVP (pair accuracy) [3]44.67 ±\pm 1.8945.78 ±\pm 1.6645.77 ±\pm 1.2547.78 ±\pm 1.3149.87 ±\pm 1.1449.78 ±\pm 0.76
HallusionBench [4]55.29 ±\pm 0.7756.47 ±\pm 0.4356.78 ±\pm 0.6358.17 ±\pm 0.1359.79 ±\pm 0.8760.87 ±\pm 0.23
Visual conflict ACC [5]23.16 ±\pm 0.1124.07 ±\pm 0.2724.71 ±\pm 0.3125.07 ±\pm 0.1325.34 ±\pm 0.3425.98 ±\pm 0.35
Image necessity (ours)21.67 ±\pm 1.1922.33 ±\pm 0.9825.17 ±\pm 1.3726.83 ±\pm 0.7828.88 ±\pm 0.5930.63 ±\pm 0.56

As shown in T4, FINE improves over matched SFT in 20 out of 21 comparisons. The sole exception is a 0.09-point decrease for 8B MMVP pair accuracy, which is statistically insignificant. There are 13 comparisons where the gain are statistically significant (beyond the standard devision). We believe that these additional results provides further evidence that FINE can achieve a higher visual reliance.

Benchmark references

[1] D. A. Hudson and C. D. Manning. GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. CVPR 2019.

[2] S. Tong et al. Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. NeurIPS 2024.

[3] S. Tong et al. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. CVPR 2024.

[4] T. Guan et al. HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models. CVPR 2024.

[5] A. Vo et al. Vision Language Models are Biased. arXiv:2505.23941, 2025.

Thank you for the detailed response and additional experiments. They clarify the method and strengthen the empirical evidence. However, the connection between the theoretical analysis and the method remains weak. Overall, I maintain my score.

Reviewer_T2VNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes adding Gaussian noise to visual-token representations at an intermediate layer. It uses contrastive decoding to encourage reliance on visual inputs. The authors introduce LC-MIPD and derive two decompositions relating it to conditional mutual information and a Jensen-Shannon divergence objective.

Critical strengths
  1. The method is a test-time method and does not require a new model component or a separate learned network.
  2. The paper clearly describes its experimental setup and provides depth and noise senstivity analysis.
Critical weaknesses
  1. The contribution appears incremental, as 'Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding' already introduces contrastive decoding with noise. 'Balancing Multimodal Training through Game-Theoretic Regularization ' already use conditional mutual information, MIPD, a divergence-based objective, and latent-space corruption.
  2. The theoretical decomposition appears to apply standard mutual-information chain-rule and data-processing arguments to the intermediate-state setting. The second theoretical analysis does not bound the net difference or establish that the JSD objective reliably tracks LC-MIPD. So, all analyses offer some interpretation of the design, but do not yet provide a strong theoretical guarantee for the training objective.
Questions for authors
  1. What guarantees that maximizing the clean-versus-corrupted JSD increases visual reliance?
  2. How reliable is the proposed layer-selection procedure?
5 comments
AuthorsOpenReview ↗

Response to Reviewer T2VN

We thank the reviewer for the focused questions.

Weakness 1 (Incremental contribution)

We respectfully clarify the distinction of FINE from the two cited prior works.

VCD is a training-free decoding method. It contrasts the logits from the original image and a distorted image with noise. It defines no information-theoretic surrogate and does not choose an intervention layer. FINE is a training-time regularizer. For inference, it adds no noise, and does not require additional network or decoding pass. The trained model itself becomes less text-dominant.

MCR is the closest work. It targets late-fusion models and corrupts the non-text input before fusion via cross-sample permutation. This presumes that the modalities stay separable until fusion, which does not hold for early-fusion MLLMs, where the attention mixes visual and textual states at every layer. According to Table 1 in the original manscript (evaluation on Qwen3-VL-2B), an MCR-style permutation does not help improve over SFT: as its shuffling operation increases the gap between LC-MIPD and JSD according to Theorem 3.6.

In contrast to MCR, the contributions of FINE are as follows:

  1. Layer-conditioned formulation. We define CMIk\mathrm{CMI}_k and MkM_k at layer kk, conditioning on the layer-kk text state Z2(k)Z_2^{(k)}, because visual information has already flowed into the text states and thus should be taken into consideration in the formulation.
  2. Exact gap accounting. Theorem 3.3 gives CMI−Mk=ℓk+ρk\mathrm{CMI}-M_k=\ell_k+\rho_k, and Theorem 3.6 gives CMI−Jk=ℓk+ρk−Ck+Sk\mathrm{CMI}-J_k=\ell_k+\rho_k-C_k+S_k. As the reviewer notes, the proofs are elementary, but effective. Their contribution is to separate the flow term ℓk\ell_k, the corrupted-information term ρk\rho_k, and the decoder-mismatch and compression terms CkC_k and SkS_k. Among these, only ℓk\ell_k is specific to the early-fusion setting; CkC_k and SkS_k also arise in MCR's JSD approximation, but they are not analyzed in the MCR paper.
  3. Calibration guidance. Theorem 3.3 shows that ℓk\ell_k is expected to grow with depth as more visual information will flow into the text state, which implies the principle of applying the intervention before substantial visual-to-text flow happens in order to be effective. In addition, Theorem 3.4 bounds ρk\rho_k, which motivates scaling σ\sigma to the per-element root-mean-square (RMS) of the "visual" state.
AuthorsOpenReview ↗

Response to Reviewer T2VN

Weakness 2 and Q1: What guarantees that maximizing JSD tracks LC-MIPD or increases visual reliance?

Regarding whether JSD tracks LC-MIPD or not, we first refer back to the formula CMI≥Mk=Jk−Ck+Sk,\hspace{25em} \mathrm{CMI} \ge M_k = J_k - C_k + S_k,

where Ck=E[DKL(Qk∥Rk)]C_k = \mathbb{E}[D_{\mathrm{KL}}(Q_k \Vert R_k)] and Sk=2E[DKL(Ak∥Rk)]S_k = 2 \mathbb{E}[D_{\mathrm{KL}}(A_k \Vert R_k)].

In general, Jk=MkJ_k = M_k only when Ck=SkC_k = S_k which does not hold in general. However, we can show that there is a nontrivial regime in which the two terms are balanced to leading order, as presented in the following Proposition 3.7. Direct evidence that the objective improves visual reliance is provided separately by additional experiments T1, T4 (shared with Reviewer mTTp), and T5 (shared with Reviewer r9MY).

Empirical Evidence

As computing JkJ_k, CkC_k, SkS_k and MkM_k are intractable for real MLLMs, we further provide empirical evidence under a toy experiment setting below (T1) to illustrate JkJ_k tracking LC-MIPD during training.

T1: Toy Experiment

kkMethodEpochCMI\mathrm{CMI}MkM_kJkJ_kCkC_kSkS_kℓk\ell_kρk\rho_k
0SFT100.132 ±\pm 0.0750.069 ±\pm 0.0390.111 ±\pm 0.0660.181 ±\pm 0.1080.139 ±\pm 0.0820.000 ±\pm 0.0000.063 ±\pm 0.036
0SFT300.115 ±\pm 0.0730.060 ±\pm 0.0390.099 ±\pm 0.0660.161 ±\pm 0.1090.122 ±\pm 0.0820.000 ±\pm 0.0000.055 ±\pm 0.034
0FINE100.201 ±\pm 0.0600.104 ±\pm 0.0340.182 ±\pm 0.0640.307 ±\pm 0.1110.232 ±\pm 0.0830.000 ±\pm 0.0000.097 ±\pm 0.027
0FINE300.192 ±\pm 0.0560.101 ±\pm 0.0330.175 ±\pm 0.0590.303 ±\pm 0.0960.229 ±\pm 0.0720.000 ±\pm 0.0000.091 ±\pm 0.024
2SFT100.132 ±\pm 0.0750.011 ±\pm 0.0140.053 ±\pm 0.0570.098 ±\pm 0.1050.056 ±\pm 0.0600.087 ±\pm 0.0670.033 ±\pm 0.044
2SFT300.115 ±\pm 0.0730.006 ±\pm 0.0080.059 ±\pm 0.0650.113 ±\pm 0.1240.060 ±\pm 0.0670.092 ±\pm 0.0780.016 ±\pm 0.022
2FINE100.202 ±\pm 0.0350.029 ±\pm 0.0190.244 ±\pm 0.0990.546 ±\pm 0.2700.331 ±\pm 0.1620.102 ±\pm 0.0970.071 ±\pm 0.059
2FINE300.170 ±\pm 0.0800.031 ±\pm 0.0160.242 ±\pm 0.0830.525 ±\pm 0.1890.313 ±\pm 0.1110.062 ±\pm 0.0510.078 ±\pm 0.070

We report separate matched training runs at k=0k=0 and k=2k=2, with σrel=4\sigma_{\mathrm{rel}}=4 and nξ=1n_\xi=1; FINE uses β=1\beta=1, and SFT has no JSD term. One toy epoch denotes 25 optimizer updates, each on a batch of 256 pairs drawn from the 100-pair finite support. Clean quantities are exact finite sums; noisy quantities use 16,384 common Monte Carlo draws. All values are in nats and are reported as mean ±\pm standard deviation over five seeds. For conciseness, we show epochs 10 and 30.

For all the rows with different combinations of depth and epoch, FINE gives higher CMI\mathrm{CMI}, MkM_k, and JkJ_k than matched SFT. At k=0k=0, the differences are respectively +0.069+0.069, +0.036+0.036, and +0.071+0.071 nats at epoch-10, and +0.077+0.077, +0.041+0.041, and +0.076+0.076 nats at epoch-30. At k=2k=2, the corresponding differences are +0.070+0.070, +0.017+0.017, and +0.190+0.190 nats at epoch 10 and +0.055+0.055, +0.025+0.025, and +0.183+0.183 nats at epoch 30. Thus, FINE raises both CMI and MkM_k relative to SFT, although at k=2k=2 the larger Ck−SkC_k-S_k gap makes JkJ_k overestimate MkM_k more strongly; this gap is not monotonic across training. We emphasize that CMI is not directly optimized by the FINE objective. The training objective is LFINE=Lpred−βJkL_{FINE}=L_{pred}-\beta J_k, which encourages larger JkJ_k only insofar as the prediction-loss term permits. Consequently, a CMI drop is simply resulted from an optimization trade-off. T4 response to Reviewer mTTp and T5 response to Reviewer r9MY provide additional evidences that FINE increases visual reliance.

AuthorsOpenReview ↗

Response to Reviewer T2VN

Proposition 3.7 (Matched-variance Gaussian corruption)

Assume that the clean non-text representation Z1(k)Z_1^{(k)} conditioned on (Z2(k),X2)(Z_2^{(k)},X_2) follows a Gaussian distribution with mean μk\mu_k and variance τk2\tau_k^2:

Z_1^{(k)}\mid (Z_2^{(k)},X_2) \sim \mathcal{N}(\mu_k,\tau_k^2 I_m),$$ and that the injected corruption is independent, $$\hspace{25em} \xi\sim\mathcal{N}(0,\sigma_k^2 I_m).$$ Also, we assume that the decoder can be locally approximated with respect to the non-text representation $Z_1^{(k)}$ by an affine mapping with Jacobian $G_k$, and that the clean and corrupted predictive distributions are close enough for the second-order KL approximation to hold. Let $D_{R_k}=\operatorname{diag}(R_k)$ and define $$\hspace{25em} \Gamma_k= \mathbb{E}\left[ \operatorname{tr} \left( D_{R_k}^{-1}G_kG_k^\top \right) \right].$$ Then, we can show that $$\hspace{25em} C_k-S_k \approx \frac{\sigma_k^2(\sigma_k^2-\tau_k^2)} {4(\tau_k^2+\sigma_k^2)} \Gamma_k.$$ *Implication*: When the corruption variance matches the conditional variance of the clean non-text representation, i.e., $\sigma_k^2=\tau_k^2$, the leading-order difference vanishes and therefore $$\hspace{25em} C_k\approx S_k, \qquad J_k\approx M_k.$$ As $\Gamma_k>0$, the leading-order sign is also informative: The JSD objective underestimates LC-MIPD when $\sigma_k^2<\tau_k^2$ and overestimates it when $\sigma_k^2>\tau_k^2$. Thus the JSD-to-LC-MIPD gap is minimized locally at matched variance, rather than shrinking monotonically with larger noise. This provides a precise sense in which JSD tracks LC-MIPD in the matched-variance regime. *Note*: This proposition is deliberately local and it should be noted that: First, it does not remove the CMI-to-LC-MIPD gap: $\mathrm{CMI}-M_k=\ell_k+\rho_k$ still holds. Second, it relies on the Gaussian assumption, independent Gaussian corruption, affine-decoder, common-support, and second-order KL approximations; we do not claim exact equality outside that regime. Third, the RMS calibration $\sigma_k=\sigma_{\mathrm{rel}}\sqrt{E_k/m}$ controls the second-moment scale of the visual state, whereas $\tau_k^2$ is its conditional variance around $\mu_k$. # Derivation of Proposition 3.7 For two sufficiently close distributions $U$ and $R_k$, we assume $$\hspace{25em} D_{\mathrm{KL}}(U\Vert R_k) \approx \frac{1}{2} \sum_y \frac{(U(y)-R_k(y))^2}{R_k(y)}.$$ Applying this approximation to $C_k$ and $S_k$ gives $$\hspace{25em} C_k \approx \frac{1}{2} \mathbb{E} \left[ \lVert Q_k-R_k\rVert_{R_k^{-1}}^2 \right], \qquad S_k \approx \frac{1}{4} \mathbb{E} \left[ \lVert P_k-R_k\rVert_{R_k^{-1}}^2 \right],$$ where $\lVert u\rVert_{R_k^{-1}}^2=\sum_y u(y)^2/R_k(y)$. By the law of total variance, $$\hspace{25em} \begin{aligned} &\mathbb{E}\left[ \lVert P_k-R_k\rVert_{R_k^{-1}}^2 \mid V \right] = \lVert Q_k-R_k\rVert_{R_k^{-1}}^2 + \mathbb{E}\left[ \lVert P_k-Q_k\rVert_{R_k^{-1}}^2 \mid V \right]. \end{aligned}$$ Therefore, $$\hspace{25em} \begin{aligned} C_k-S_k \approx \frac{1}{4}\mathbb{E}\Big[ &\lVert Q_k-R_k\rVert_{R_k^{-1}}^2 - \operatorname{tr} \left( D_{R_k}^{-1}\operatorname{Cov}(P_k\mid V) \right) \Big]. \end{aligned} \tag{18}$$ Under the local affine approximation, $$\hspace{25em} P_k-R_k \approx G_k\left(Z_1^{(k)}-\widetilde Z_1^{(k)}\right),$$ and hence $$\hspace{25em} \begin{aligned} Q_k-R_k &\approx G_k\left( \mathbb{E}[Z_1^{(k)}\mid V]-\widetilde Z_1^{(k)} \right),\\ \operatorname{Cov}(P_k\mid V) \approx G_k\operatorname{Cov}(Z_1^{(k)}\mid V)G_k^\top. \end{aligned}$$ For the assumed Gaussian prior and independent Gaussian corruption, the posterior is $$\hspace{25em} Z_1^{(k)}\mid V \sim \mathcal{N} \left( \frac{\sigma_k^2\mu_k+\tau_k^2\widetilde Z_1^{(k)}} {\tau_k^2+\sigma_k^2}, \frac{\tau_k^2\sigma_k^2} {\tau_k^2+\sigma_k^2}I_m \right).$$ It follows that $$\hspace{25em} \mathbb{E}[Z_1^{(k)}\mid V]-\widetilde Z_1^{(k)}= \frac{\sigma_k^2} {\tau_k^2+\sigma_k^2} \left( \mu_k-\widetilde Z_1^{(k)} \right),$$ and, since $\widetilde Z_1^{(k)}-\mu_k\sim\mathcal{N}(0,(\tau_k^2+\sigma_k^2)I_m)$, $$\hspace{25em} \begin{aligned} &\mathbb{E}\left[ \left(\mathbb{E}[Z_1^{(k)}\mid V]-\widetilde Z_1^{(k)}\right) \left(\mathbb{E}[Z_1^{(k)}\mid V]-\widetilde Z_1^{(k)}\right)^\top \right] = \frac{\sigma_k^4} {\tau_k^2+\sigma_k^2}I_m. \end{aligned}$$ The conditional covariance is $$\hspace{25em} \operatorname{Cov}(Z_1^{(k)}\mid V)= \frac{\tau_k^2\sigma_k^2} {\tau_k^2+\sigma_k^2}I_m.$$ Substituting these posterior moments into Eq. (18) and using the definition of $\Gamma_k$ yields $$\hspace{25em} \begin{aligned} C_k-S_k &\approx \frac{1}{4} \left( \frac{\sigma_k^4} {\tau_k^2+\sigma_k^2} - \frac{\tau_k^2\sigma_k^2} {\tau_k^2+\sigma_k^2} \right) \Gamma_k = \frac{\sigma_k^2(\sigma_k^2-\tau_k^2)} {4(\tau_k^2+\sigma_k^2)} \Gamma_k. \end{aligned}$$ Setting $\sigma_k^2=\tau_k^2$ makes the leading-order difference zero.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal

We sincerely thank the reviewers and the Area Chair for their time and constructive feedback, which has helped us sharpen the paper's claims.

The main concerns were the scope of the theory and its connection to the method (T2VN, mTTp), whether the gains reflect genuine visual reliance rather than generic perturbation (T2VN, mTTp, r9MY), the reliability of layer selection (T2VN), and several technical, efficiency, and presentation issues (r9MY). We addressed these through additional experiments, analyses, and clarifications:

  1. Contribution and theory scope (T2VN, mTTp). FINE is a training-time intervention for early-fusion MLLMs. It differs from decoding-time VCD and from MCR-style late-fusion permutation. The theory provides an exact layer-conditioned accounting of the surrogate error, but we do not claim a monotonic optimization guarantee: maximizing JkJ_k is not assumed to increase CMI or visual reliance in general. The defaults k∗=⌊L/2⌋k^*=\lfloor L/2\rfloor and σrel=0.2\sigma_{\mathrm{rel}}=0.2 are compute–accuracy choices, not theoretical optima.
  2. Evidence for increased visual reliance (T2VN, mTTp, r9MY). We provided matched toy runs with exact CMI and Monte Carlo estimates of the noisy terms (T1), vision-centric and counterfactual benchmarks with an image-necessity test (T4), and energy-matched location controls (T5). FINE improves over matched SFT in 20 of 21 vision-centric and reliance comparisons in T4, and the energy-matched visual intervention outperforms the text intervention by 4.2, 2.6, and 4.7 points on MMBench, ChartQA, and POPE, respectively. We describe these as supporting, rather than conclusive, evidence and do not present them as real-model estimates of CMI.
  3. Layer-selection reliability (T2VN). We profiled frozen Qwen3-VL-4B and 8B and evaluated trained FINE models across the depth-noise grid (T2, T3). KL attenuation and the onset of performance decline align between depths 6 and 10 at both scales. T2VN explicitly confirmed satisfaction with this analysis.
  4. Efficiency and presentation (r9MY). We reported measured training time and memory against matched SFT (T6). On the Qwen3-VL-2B profile, FINE at the default depth takes approximately 1.48 times the SFT update time, with a 0.27 GiB increase in peak memory, and adds no cost at inference. We also committed to a full proofreading pass and to discussing beneficial-noise work such as MuNG.

We appreciate the reviewers' recognition of the importance of text dominance in early-fusion MLLMs and of the strengthened empirical evidence. We will incorporate the additional results and clarifications into the revised manuscript. We believe the paper's primary contribution is a training-time intervention with matched empirical support, together with an explicit analysis of where its information-theoretic surrogate can fail.

We would be happy to clarify any remaining questions from the reviewers or the Area Chair. Thank you for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.