acceptodds
Under review as a conference paper at ICLR 2027

From Abnormal Visual Tokens to Faithful Decoding: Rebalancing Visual Evidence for Hallucination Mitigation in MLLMs

Abstract

Multimodal Large Language Models (MLLMs) remain vulnerable to hallucinations despite their strong visual understanding and generation capabilities. Existing hallucination mitigation methods mainly focus on language-side biases, while abnormal visual behaviors and corresponding theoretical implications for faithful decoding remain largely underexplored. Through extensive and statistically significant investigations into the causes of hallucination, we reveal that unusually high-magnitude visual tokens exhibit persistent off-target responses and increasingly interfere with grounded object prediction in deeper layers, accompanied by a progressive weakening of visual evidence during decoding. Based on these observations, we propose a Visual Evidence Rebalancing and Grounding Enhancement method called VERGE, a training-free framework that combines evidence-guided channel correction, uncertainty-triggered visual retracing, and prior-calibrated contrastive decoding. Furthermore, we provide theoretical analyses that characterize the improvement of faithful-token preference and the benefit of recovering target-relevant visual information. Extensive comparative experiments demonstrate that VERGE effectively suppresses hallucinations while maintaining strong generation capabilities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Model, Abnormal Visual Behavior, Hallucination Mitigation, Theoretical Analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zhCeOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

VERGE is a training-free framework for mitigating hallucination in multimodal language models. It partitions visual tokens by magnitude and uses the resulting prototypes to suppress or enhance FFN channels. It then retrieves visual features under high uncertainty and calibrates decoding against a text-only branch. The paper provides conditional theoretical analyses and evaluates five backbone configurations on hallucination and general multimodal benchmarks.

Critical strengths
  • The paper addresses an important limitation of multimodal models without requiring additional training.
  • The empirical analysis identifies an association between high-magnitude visual tokens and diffuse attention, together with increased alignment of hallucinated object tokens to these representations in later layers. These observations provide a useful basis for investigating failures of visual grounding.
  • Evaluation spans several model families and includes both hallucination and general capability benchmarks.
Critical weaknesses
  1. The motivating intervention does not isolate token magnitude. Replacing a token with the image mean also changes its semantic content and direction. Moreover, the paper provides limited direct evidence that prototype similarity identifies channels responsible for hallucination. The detailed diagnosis is concentrated on LLaVA, so performance gains on other backbones do not by themselves establish the same mechanism.

  2. The contribution beyond existing methods is insufficiently isolated. Visual retracing and language-prior calibration are closely related to MemVR and PAI. The component ablation is useful, but measures full-MME performance, where the incremental effect of channel correction is modest. Hallucination-specific ablations are needed to establish whether this component materially improves the combined method.

  3. The theory provides conditional justification without establishing that the implemented method satisfies its central conditions. The margin result assumes favorable alignment of the correction direction, while the retracing analysis requires sufficient information gain. Neither is verified for VERGE. In particular, the analysis does not establish why gate-weight/prototype similarities produce the required direction or provide an operational choice of correction strength.

  4. The contrastive term and candidate filtering are not independently evaluated. The contrastive coefficient also determines the plausibility threshold, so the reported sweep changes both mechanisms simultaneously. This prevents attribution of the decoding gains to language-prior subtraction. Separate controls under the same decoding rule are needed. POPE answer frequencies and class-specific errors would also help distinguish improved grounding from a shifted response tendency.

  5. Parameter sensitivity is examined, but the selection protocol remains unclear. The method uses backbone-specific coefficients, thresholds, and intervention layers, and the reported sweeps show different responses across models. Without a clearly separated validation procedure, these results do not establish that the selected configuration transfers to unseen evaluation data.

Questions for authors
  1. Can norm-only interventions and measurements on another backbone establish the proposed link between token magnitude, selected FFN channels, and hallucination?
  2. What is the incremental benefit of channel correction on CHAIR or HallusionBench when retracing and prior calibration are already enabled?
  3. How often does the implemented correction satisfy the theorem’s alignment and step-size conditions? Does the theory predict successful and unsuccessful interventions?
  4. Can prior subtraction and plausibility filtering be varied independently under identical decoding settings?
4 comments

We thank the reviewer for asking about magnitude, FFN attribution, and component novelty. Our controls examine prototype-guided channel selection.

W1&Q1: Separating magnitude, direction, and channel effects. Figs. 2–3 link high norms to diffuse attention and hallucinated-object alignment in deeper layers. Such tokens can also encode useful scene context. The motivating Centroid intervention replaces selected embeddings with the image-wide mean, changing both magnitude and semantic direction.

Random rescale attenuates a randomly selected, equally sized token subset using the same strength as high-norm attenuation. Norm-only rescales the selected high-norm embeddings but preserves their feature directions. Direction-only shifts these embeddings toward the centroid while restoring each original L2 norm. Centroid replaces the full embeddings with the global image mean, altering direction and norm together. In contrast, FFN (Full) keeps every original visual token and changes downstream FFN channel responses based on cosine similarity with the visual prototypes.

To isolate the interventions, we hold retracing (R), prior calibration (P), and evaluation fixed. CHAIR-I (CI) tracks hallucinated object mentions, while Recall tracks real-object coverage on LLaVA and Qwen2.5:

SettingLLaVA CI↓LLaVA R↑Qwen CI↓Qwen R↑
R+P14.180.09.569.0
+Random rescale14.079.99.568.9
+Norm-only13.779.89.368.8
+Direction-only13.878.69.468.2
+Centroid13.577.39.267.1
+FFN (Full)12.881.18.969.4

Norm-only lowers CI from 14.1 to 13.7 on LLaVA and from 9.5 to 9.3 on Qwen, outperforming Random rescale. Centroid lowers CI further but reduces Recall to 77.3 and 67.1. Full FFN correction attains the lowest CI and highest Recall on both models: 12.8/81.1 and 8.9/69.4. These cross-backbone results favor magnitude-aware selection and visual preservation in VERGE

Why channel selection differs from token replacement. VERGE forms relevance-weighted prototypes from the high-norm subset and the remaining evidence subset. For each FFN gate-weight row, cosine similarity to the high-norm prototype guides bounded suppression, while similarity to the evidence prototype guides additive enhancement. For intermediate activation aa, the realized channel correction is

d=\tilde a-a=a\odot(m-\mathbf 1)+g.$$ Mask $m$ is lower-bounded by $c_{\min}$ to preserve global cues, while $g$ enhances evidence-aligned channels. Unlike centroid replacement, VERGE retains original tokens and spatial detail, adjusting only their downstream channel influence. **W2&Q2: Isolating the FFN correction benefit.** MemVR retrieves visual memory through the FFN pathway, whereas PAI reweights visual attention and calibrates logits. **VERGE instead uses two visual prototypes to select intermediate FFN channels** for bounded suppression and evidence enhancement. Rather than replacing visual tokens or adjusting attention weights, this directly rebalances how specific FFN features influence downstream predictions. **To test whether this distinction reduces hallucination while preserving object recall**, we keep retracing, prior calibration, intervention layers, and decoding settings fixed, removing only FFN correction to obtain **R+P**. We then compare **Suppression only** ($a\odot m$), **Enhancement only** ($a+g$), and their full combination. To test whether the benefit comes from visually informed channel assignment rather than a generic perturbation, **Random channels** uses a Gaussian vector matched to the realized update $d$ in dimension and L2 norm. **Shuffled channels** permutes the actual correction values across FFN channels, retaining every value and its total norm but breaking the prototype-guided correspondence. Comparing these controls with Full isolates the importance of **where the correction is applied** under matched intervention magnitude. |Configuration|CHAIR-S↓|CHAIR-I↓|Recall↑|POPE↑| |-|-|-|-|-| |R+P|48.4|14.1|80.0|87.0| |+Suppression|46.9|13.3|80.5|87.2| |+Enhancement|47.3|13.5|80.4|87.1| |+Random channels|48.2|14.0|80.1|87.0| |+Shuffled channels|48.1|13.9|80.2|87.0| |+Full correction|**45.8**|**12.8**|**81.1**|**87.4**| CHAIR-S counts descriptions with hallucinated objects, while CHAIR-I counts hallucinated mentions. Full lowers CHAIR-S/I by **2.6/1.3 points** versus R+P and raises Recall/POPE by **1.1/0.4**. Suppression and enhancement help individually, but Full is best; norm-matched Random and Shuffled remain near R+P. These hallucination-specific gains isolate FFN correction beyond full-MME ablations. **In summary,** the separate magnitude and direction controls clarify the high-norm observation, the Qwen comparison extends it beyond LLaVA, and the randomized and shuffled FFN controls isolate the value of visually guided channel correction. Overall, VERGE improves grounding while retaining real-object coverage.

We test how VERGE's implemented interventions relate to Theorems 1 and 2 through the realized FFN update and paired Retracing ON/OFF diagnostics.

W3&Q3: Empirical verification of Theorem 1. Cosine similarity between FFN gate rows and visual prototypes guides bounded suppression and enhancement. The realized update to activation aa is

d=a~−a=a⊙(m−1)+g.d=\tilde a-a=a\odot(m-\mathbf 1)+g.

Here dd includes actual masks and uncertainty scaling. For true object y⋆y^\star and incorrect object yy, define My(a)=ℓy⋆(a)−ℓy(a)M_y(a)=\ell_{y^\star}(a)-\ell_y(a). Its alignment is

Ay=∇aMy(a)⊤d.A_y=\nabla_a M_y(a)^\top d.

Positive AyA_y favors the true object locally. Given smoothness bound LyL_y,

My(a+d)−My(a)≥Ay−Ly2∥d∥22.M_y(a+d)-M_y(a)\ge A_y-\frac{L_y}{2}\|d\|_2^2.

The first term measures direction, the second penalizes strength. Since cosine similarity selects channels but the theorem concerns faithful margins, we test the link through actual downstream-logit gradients, not similarity alone.

To examine both alignment and its prediction effects, Align+ checks positive alignment against the leading wrong object; Top-5+ requires it against five incorrect candidates. Margin gain+ records actual positive logit-margin changes with other modules fixed. Step+ checks the estimated sufficient condition Ay−Ly2∥d∥22>0A_y-\frac{L_y}{2}\|d\|_2^2>0, comparing first-order gain with curvature cost. Together they test direction, finite-step success, and strength.

ModelAlign+Top-5+Margin gain+Step+
LLaVA-7B89.386.788.678.1
LLaVA-13B88.185.087.377.0
Qwen2.5-VL87.084.286.175.4
InternVL385.682.484.572.6
GLM-4V84.480.883.770.9

Across backbones, Align+ reaches 84.4–89.3%, Top-5+ 80.8–86.7%, and realized Margin gain+ 83.7–88.6%. Step+ holds in 70.9–78.1%, selecting contexts where the estimated first-order benefit exceeds the curvature cost. Thus Step+ predicts sufficient-margin improvement, while Margin gain+ separately records actual success. A nonpositive Step+ indicates absence of this sufficient certificate, not necessarily a reduced margin.

To further isolate the role of channel direction, Random uses a Gaussian update matched to dd in L2 norm; Reversed applies −d-d, keeping norm and channel pattern but reversing direction. Image, prefix, layer, and competitor are fixed. Alignment rates (%) are:

DirectionL7L13QwenInternGLM
Random52.351.652.050.650.9
Reversed10.711.913.014.415.6
VERGE89.388.187.085.684.4

Random reaches 50–52% and Reversed only 10.7–15.6%, isolating the prototype-guided direction rather than update norm.

We also examine how the theoretical trade-off guides intervention strength. Theorem 1 expresses the nominal update as a~=a+r(u)d0\tilde a=a+r(u)d_0, where r(u)=(1−η)+ηur(u)=(1-\eta)+\eta u. The lower bound on faithful-margin improvement is

r(u)A⋆−L⋆2r(u)2∥d0∥22.r(u)A_\star-\frac{L_\star}{2}r(u)^2\|d_0\|_2^2.

The linear benefit competes with quadratic cost. With other modules fixed, s=0,0.5,1.0,1.5s=0,0.5,1.0,1.5 yields margin gains 0,+0.19,+0.28,+0.140,+0.19,+0.28,+0.14 and CHAIR-I 14.3,13.2,12.8,13.414.3,13.2,12.8,13.4. Nominal s=1.0s=1.0 also maximizes Recall (81.1). This non-monotonicity offers practical strength guidance and distinguishes effective from over-strong updates.

Theorem 2: Visual-information recovery. At triggered contexts T=1T=1, Theorem 2 compares pre/post-retracing representations S0,S1S_0,S_1 through target-information gain:

GVR=I(S1;Y∣T=1)−I(S0;Y∣T=1).G_{\mathrm{VR}}=I(S_1;Y\mid T=1)-I(S_0;Y\mid T=1).

Theorem 2 predicts improved complexity when GVR>ΔH+ΔΓG_{\mathrm{VR}}>\Delta H+\Delta\Gamma; the full bound also involves residual ρS\rho_S. Direct estimation is difficult: GVRG_{\mathrm{VR}} and ΔH\Delta H require conditional mutual-information/entropy estimates for high-dimensional hidden states, ΔΓ\Delta\Gamma depends on nuisance-factor sensitivity, and ρS\rho_S on surrogate losses. We therefore indirectly assess recovered target evidence through paired prediction outcomes, while VERGE controls added complexity via one-shot retrieval of all original visual tokens, scale matching, and bounded blending.

To isolate retracing, we compare ON/OFF on 1,000 images per backbone (one triggered object event each), holding image, prefix, state, layer, FFN correction, and decoding fixed. LL Gain+ tracks true-object log-probability increases; Useful tracks larger margins against OFF's fixed strongest wrong object; Failed counts nonpositive margin changes.

ModelLL Gain+↑Useful↑Failed↓
LLaVA-7B87.9%89.5%10.5%
LLaVA-13B86.6%87.9%12.1%
Qwen2.5-VL85.2%86.6%13.4%
InternVL383.7%85.0%15.0%
GLM-4V81.9%83.2%16.8%

Retracing improves true-object likelihood in 81.9–87.9% of cases and margins in 83.2–89.5%. LL Gain+ measures target support; Useful tests discrimination against OFF's fixed incorrect competitor. These paired, controlled prediction gains provide strong empirical evidence of recovered visual information, consistent with Theorem 2's mechanism.

We first isolate the decoding components and then test whether a fixed VERGE configuration transfers across model backbones.

W4&Q4: Isolating the decoding contributions. Unlike PAI's visual-attention reweighting and logit calibration, VERGE applies channel correction upstream and combines text-prior subtraction with an image-conditioned plausibility filter. The two decoding operations have different roles: prior subtraction adjusts scores of text-favored candidates, whereas filtering restricts selection to image-supported candidates. We therefore test them independently to identify the source of decoding gains under matched settings.

With image-conditioned and text-only log probabilities ℓ+,ℓ−\ell^+,\ell^-, the score is

ℓ~=(1+δ)ℓ+−δℓ−.\tilde\ell=(1+\delta)\ell^+-\delta\ell^-.

The image-conditioned plausibility filter retains candidates satisfying:

p+(w)≥κmax⁡vp+(v).p^+(w)\ge\kappa\max_v p^+(v).

To isolate their contributions, we decouple the cutoff κ\kappa from δ\delta. Prior-only applies δ\delta without filtering; Filter-only applies κ\kappa without subtraction; Combined activates both at fixed strengths, while Baseline disables both. All configurations share identical inputs and visual-side modules, while the filtering threshold is held fixed to assess the effect of prior subtraction.

Checking grounding beyond answer frequency. To distinguish better object decisions from a general tendency to answer 'no', we also inspect POPE's class-specific behavior. Yes% records positive answers, FPR absent objects answered yes, and FNR present objects answered no. CHAIR-I separately measures hallucinated mentions in free-form descriptions.

PriorFilterCHAIR-I↓POPE↑Yes%FPR↓FNR↓
OffOff14.186.15821.95.9
OnOff13.386.55518.58.5
OffOn13.786.45417.69.6
OnOn12.887.45214.610.6

Prior-only yields CHAIR-I 13.3 and POPE 86.5; Filter-only gives 13.7 and 86.4. Joint decoding achieves 12.8 and 87.4, improving both metrics by 1.3 points over Neither. Yes% shifts from 58% to 52%. FPR drops by 7.3 points (21.9 to 14.6), while FNR rises by 4.7 (5.9 to 10.6). Since POPE is balanced, the larger FPR reduction yields a net +1.3 accuracy points. The accompanying CHAIR-I decrease supports fewer hallucinated object mentions, while both error classes show how the yes/no tendency changes. The 2×2 study isolates each operator and their complementary effect. Prior subtraction counters text-driven scores, while filtering excludes visually implausible candidates, explaining their complementary benefit.

W5: Parameter sensitivity and transfer. We separate two complementary analyses. Seven-point sweeps on full MME vary one FFN coefficient (α,β)(\alpha,\beta), partition ratio ρ\rho, retracing weight λ\lambda, or decoding strength δ\delta while other coefficients, triggers, and layers remain at their model-specific reference values. This characterizes the effect of each choice. To examine reuse rather than retuning, the separate POPE transfer experiment freezes a shared or source-selected parameter vector before evaluating a destination backbone. The benchmark and decoding protocol remain unchanged across conditions. This directly tests performance with fixed source coefficients instead of optimizing anew for each target. The reported procedure characterizes sensitivity and cross-backbone transfer, without specifying an image-disjoint calibration/test split.

To test this distinction on POPE accuracy (%), we compare Greedy (unmodified model), Shared (one common configuration), From L7/From Qwen (complete source-selected vectors transferred without target retuning), and Per-model (individual settings). Each backbone uses the same evaluation across conditions.

SettingL7L13QwenInternGLM
Greedy84.187.087.790.889.3
Shared86.587.988.690.989.9
From L787.487.988.090.789.4
From Qwen85.887.589.190.889.8
Per-model87.488.789.191.090.1

Shared settings consistently outperform Greedy across all five backbones, with POPE accuracy gains of +2.4 (L7), +0.9 (L13), +0.9 (Qwen), +0.1 (Intern), and +0.6 (GLM). Configurations transferred from L7 or Qwen also achieve competitive performance without target-specific retuning, while per-model settings provide further improvements on some backbones. Although sensitivity analyses reveal different responses to individual hyperparameters, these consistent gains demonstrate VERGE's robustness under shared settings, while allowing for different optimal configurations across evaluation tasks.

In summary, the independent decoding experiments confirm the complementary contributions of prior subtraction and plausibility filtering to hallucination mitigation and object-level prediction. Together with the cross-backbone results, these findings highlight VERGE's effectiveness and practical applicability across diverse MLLMs.

We kindly remind you that the rebuttal period is drawing to a close. We sincerely hope to clarify the misunderstandings in our paper and address your concerns, and we truly appreciate your taking the time to read our response.

Reviewer_9drUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the underlying relationships between visual tokens with unusual high norm and hallucination in MLLMs. These tokens tend to lie outside target regions and become increasingly associated with hallucinated-object predictions in deeper layers. Guided by these findings, VERGE is proposed by authors for hallucination suppression, which combines FFN channel suppression and enhancement, uncertainty-triggered visual retracing, and contrast decoding. In addition, the theorems provides mathematical analysis on margin improvement and generalization bound, which further verifies the methods. The decrease of average CHAIR-I from 11.6 to 10.2 across five backbones and the improvement of object recall indicates the effectiveness of VERGE.

Critical strengths

1.Insightful analysis and sufficient experiments on visual-side hallucination with complementary spatial, layer-wise, and intervene evidence. The experimental investigation is extensive, including visual-token norm statistics, alignment patterns across token categories, and high-norm versus low-norm token replacement at multiple intervention ratios. The method is further evaluated across five MLLM backbones and diverse benchmarks, with component ablations and parameter sensitivity analyses providing a broad assessment of its effectiveness. 2.The proposed framework is training-free, and improves faithfulness while preserving useful visual content. For example, CHAIR-I decreases from 15.7 to 12.8 on LLaVA-1.5-7B, while recall increases from 77.4 to 81.1. These results support a favorable balance between suppressing unsupported predictions and retaining grounded information.

Critical weaknesses

1.Concerns remains to the connection between the theoretical guarantees and the implemented algorithms. Theorem 1 requires positive alignment with faithful token margins, but these conditions are not verified directly for the correction that is based on the cosine similarity. Theorem 2 similarity requires information gain to outweigh complexity and residual costs, which lacks the corresponding empirical verification. Therefore, a gap still remains between the mathematical derivations and actual improvement. 2.Incompleteness experiment of the validation of central channel correction module. The paper only reports aggregate MME scores without the results of corresponding CHAIR, or object recall. In addition, high-norm versus low-norm token replacement does not directly validate the proposed FFN channel-selection rule. Comparing the method with a correction that preserves perturbation magnitude but shuffles channel assignments would help establish whether hallucination reduction specifically benefits from visually guided channel selection.

Questions for authors
  1. What empirical evidence supports the assumptions of Theorems 1 and 2 for the implemented method?
  2. Can you please report channel-correction ablations on CHAIR and object recall while keeping the other two modules fixed?
4 comments

We thank the reviewer for asking how Theorem 1 applies to VERGE's cosine-guided FFN correction. Our tests examine the realized update, not an idealized direction.

W1&Q1: Empirical support for Theorem 1. VERGE builds prototypes from high-norm and remaining visual tokens. Cosine similarity with FFN gate-weight rows selects channels for bounded suppression or evidence enhancement. For intermediate activation aa, the update is

d=a~−a=a⊙(m−1)+g,d=\tilde a-a=a\odot(m-\mathbf1)+g,

Here mm is the suppression mask and gg the enhancement term, including actual clipping and uncertainty modulation. Cosine affinities determine which channels change; mm and gg determine how. Unlike direct token replacement or generic hidden-state offsets, their combined effect is evaluated through the update actually passed into the frozen network.

For an annotated correct object token y⋆y^\star and an incorrect object token yy, we define their logit margin and its alignment with the correction:

A_y=\nabla_a M_y(a)^\top d.$$ Both logits use the same frozen downstream network. Positive $A_y$ indicates a locally improving direction. With local smoothness, $$M_y(a+d)-M_y(a)\ge A_y-\frac{L_y}{2}\|d\|_2^2.$$ The first term measures **direction**; the second penalizes excessive **magnitude**. **Matched direction controls.** With image, prefix, layer, hidden state, and competitor fixed, **VERGE** uses the actual update; **Random** uses a Gaussian update rescaled to the same L2 norm; **Reversed** uses $-d$, preserving magnitude but reversing direction. **Alignment+** measures the proportion with $A_y>0$ against the strongest incorrect-object candidate. |Direction|LLaVA-7B|LLaVA-13B|Qwen2.5|InternVL3|GLM-4V| |-|-|-|-|-|-| |Random|52.3|51.6|52.0|50.6|50.9| |Reversed|10.7|11.9|13.0|14.4|15.6| |VERGE|**89.3**|**88.1**|**87.0**|**85.6**|**84.4**| VERGE achieves 84.4-89.3% positive alignment on five different backbones, compared with approximately 51% for magnitude-matched random updates. Reversing the same correction sharply lowers positive alignment to 10.7-15.6%. With norm and context matched, this favors **prototype-guided orientation**, not perturbation size. Random holds intervention magnitude fixed but removes channel guidance. Reversed preserves the channel pattern but changes its sign. Their separation from VERGE supports the role of visually guided orientation. **From local alignment to realized prediction improvement.** We apply stricter diagnostics. **Top-5 Align+** requires positive alignment against *all five* leading incorrect objects. **Margin gain+** measures actual increases in correct-minus-incorrect logit margin after $d$, with other modules fixed. **Step+** tests whether first-order gain exceeds a locally estimated curvature penalty. These respectively test multiple competitors, finite-step improvement, and strength control. |Model|Alignment+|Top-5 Align+|Margin gain+|Step+| |-|-|-|-|-| |LLaVA-7B|89.3|86.7|88.6|78.1| |LLaVA-13B|88.1|85.0|87.3|77.0| |Qwen2.5-VL|87.0|84.2|86.1|75.4| |InternVL3|85.6|82.4|84.5|72.6| |GLM-4V|84.4|80.8|83.7|70.9| Top-5 Align+ reaches 80.8-86.7%, realized margin gain 83.7-88.6%, and curvature-aware Step+ 70.9-78.1%. Together they connect the theorem's directional and step-size conditions to implemented updates. Top-5 Align+ tests whether the favorable direction persists against several competing object labels. Margin gain+ then verifies the finite update in output logits, while Step+ incorporates estimated curvature and strength. Together, they go beyond a positive local dot product. **Correction strength and the predicted trade-off.** Theorem 1 expresses the nominal uncertainty-scaled update as $\tilde a=a+r(u)d_0$, with $r(u)=(1-\eta)+\eta u$. If $A_\star>0$ denotes the uniform alignment lower bound and $L_\star$ the local smoothness bound, the faithful-margin improvement is lower-bounded by $$r(u)A_\star-\frac{L_\star}{2}r(u)^2\|d_0\|_2^2.$$ The bound explains why greater correction need not produce greater gains: its directional benefit grows linearly, whereas the curvature cost grows quadratically. We test this relationship by applying a separate multiplier $s$ to both correction coefficients, $\alpha'=s\alpha$ and $\beta'=s\beta$, while keeping the uncertainty schedule, intervention layers, retracing, and decoding unchanged. |Scale $s$|Margin Gain|CHAIR-I↓|Recall↑| |-|-|-|-| |0|0.00|14.3|79.0| |0.5|+0.19|13.2|80.3| |1.0|**+0.28**|**12.8**|**81.1**| |1.5|+0.14|13.4|80.0| At nominal $s=1.0$, margin gain is largest (+0.28), CHAIR-I lowest (12.8), and object Recall highest (81.1). Weaker and stronger corrections yield smaller gains. Alignment tests direction, Margin gain+ tests output consequences, and Step+ checks the smoothness-adjusted strength regime. The matched controls, realized-margin tests, and strength sweep together connect cosine-guided FFN correction to Theorem 1 at gradient and prediction levels.

We thank the reviewer for asking how Theorem 2 relates to uncertainty-triggered visual retracing. We examine whether retrieving original image features improves support for the true object and discrimination from incorrect alternatives.

W1&Q1: Empirical support for Theorem 2. The theorem compares representations S0S_0 and S1S_1 before and after retracing at triggered steps T=1T=1. Its target-information gain is

GVR=I(S1;Y∣T=1)−I(S0;Y∣T=1),G_{\mathrm{VR}}=I(S_1;Y\mid T=1)-I(S_0;Y\mid T=1),

where YY is the faithful next-token target. The theoretical analysis also tracks the change in representation complexity ΔH\Delta H, sensitivity-related cost ΔΓ\Delta\Gamma, and the surrogate residual ρS\rho_S. The representation-dependent complexity term improves when GVR>ΔH+ΔΓG_{\mathrm{VR}}>\Delta H+\Delta\Gamma; the full bound additionally accounts for the change in ρS\rho_S. The key mechanism is therefore useful information recovery under controlled representation complexity, rather than indiscriminate feature injection. The information term concerns the true object, not simply larger activation magnitude. Its benefit must be balanced against nuisance variation, captured by the theorem's complexity and residual terms.

Why the implemented retracing step is targeted. As layer-wise uncertainty rises and visual attention weakens, VERGE retraces at the first eligible threshold crossing. It uses all original visual tokens, rather than only either FFN prototype subset, retaining both localized and global evidence.

The current hidden state queries scale-adjusted keys and values from the original visual memory:

\Delta=(hK^\top)V.$$ The retrieved representation is then normalized to the corrected FFN-output scale and combined through $$\hat f=(1-\lambda)\tilde f+\lambda\bar\Delta.$$ Here $Z$ contains the original visual tokens, $\tilde f$ is the FFN output after channel correction, and $\bar\Delta$ is the scale-matched visual retrieval. The coefficients $s_1,s_2$ account for the relevant projection and visual-feature magnitudes. A single trigger and a controlled blending weight $\lambda$ make retracing a selective supplement to the existing hidden state, consistent with the theorem's emphasis on balancing recovered evidence and representational cost. **Paired Retracing ON/OFF evaluation.** We test **1,000 images per backbone**, with one eligible triggered object prediction per image. ON and OFF share image, prefix, hidden state, trigger layer, FFN correction, and decoding; only visual-memory reinjection differs. This isolates retracing from the other modules. The ON/OFF comparison offers a direct test: with every other component fixed, changes in correct-object support reflect revisiting the original visual memory at the triggered step. **LL Gain+** counts events where the annotated correct object gains log-probability with retracing ON: $$\log p_{\mathrm{ON}}(y^\star)> \log p_{\mathrm{OFF}}(y^\star).$$ **Useful** counts an increased correct-minus-incorrect logit margin against $y^-_{\mathrm{OFF}}$, the strongest incorrect object under OFF: $$\ell_{\mathrm{ON}}(y^\star)-\ell_{\mathrm{ON}}(y^-_{\mathrm{OFF}}) > \ell_{\mathrm{OFF}}(y^\star)-\ell_{\mathrm{OFF}}(y^-_{\mathrm{OFF}}).$$ The fixed competitor prevents gains from changing the reference object. **Failed** is the complementary rate of nonpositive margin changes. LL Gain+ measures true-object support; Useful measures discrimination. |Model|LL Gain+↑|Useful↑|Failed↓| |-|-|-|-| |LLaVA-7B|87.9%|89.5%|10.5%| |LLaVA-13B|86.6%|87.9%|12.1%| |Qwen2.5-VL|85.2%|86.6%|13.4%| |InternVL3|83.7%|85.0%|15.0%| |GLM-4V|81.9%|83.2%|16.8%| Retracing raises true-object log-probability in **81.9-87.9%** of eligible events and improves object margins in **83.2-89.5%**. LLaVA-7B attains 87.9%/89.5% on these measures; GLM-4V attains 81.9%/83.2%. The pattern spans diverse multimodal backbones. **Connecting these diagnostics to the theorem.** Higher true-object likelihood indicates stronger target support; a larger margin against the fixed incorrect object indicates better discrimination. Useful exceeds LL Gain+ on every model, so retracing improves object separation as well as target confidence. These paired results assess the **predictive effect** of recovered evidence. Theorem 2 supplies the statistical perspective: target-information gain must outweigh complexity and residual costs. A one-shot uncertainty trigger, scale matching, and controlled blending are the implemented safeguards for this trade-off. The matched ON/OFF results connect these choices to stronger image-grounded predictions. **In summary,** retracing selectively revisits original visual evidence at uncertain steps. Across five backbones, matched ON/OFF comparisons show greater correct-object likelihood and stronger margins against fixed incorrect objects. These results support Theorem 2's information-recovery motivation and its emphasis on controlling representational complexity.

We thank the reviewer for requesting CHAIR/Recall ablations that isolate FFN channel correction. Magnitude-matched controls also test which channels are corrected.

W2&Q2: Controlled channel-correction ablation. VERGE combines channel correction (C), visual retracing (R), and prior-calibrated decoding (P). R+P retains the last two modules but disables C; Full adds C. Both use identical images, prompts, layers, retracing triggers, decoding branches, and evaluation. Their difference isolates C.

VERGE computes cosine affinity of FFN gate-weight rows to high-norm and evidence prototypes. Mask mm suppresses channels aligned with the former, and bias gg enhances channels aligned with the latter:

d=\tilde a-a=a\odot(m-\mathbf1)+g.$$ Bounded suppression ($c_{\min}$) retains global cues, while enhancement uses positive evidence affinity. Both act **inside the FFN**, keeping visual tokens unchanged. The two context-dependent prototypes guide different channel sets: high-norm affinity determines bounded suppression, while evidence affinity determines enhancement. This changes FFN responses, not source tokens. **What each control changes.** **Suppression only** applies $a\odot m$ without $g$; **Enhancement only** applies $a+g$. **Random channels** uses a Gaussian update matched to $d$ in dimension and L2 norm. **Shuffled channels** permutes the actual correction coordinates, preserving their values and norm while breaking the prototype-guided channel assignment. Comparing Shuffled with Full directly tests whether the channel mapping, rather than update magnitude alone, matters. Shuffled is stricter than Random: it keeps every correction value and its L2 norm but moves values to different channels. The missing cosine-guided pairing, not perturbation size, is the key difference from Full. **Object-level ablation on LLaVA-1.5-7B.** CHAIR-S measures affected descriptions, CHAIR-I hallucinated object mentions, Recall genuine-object coverage, and POPE object-existence accuracy. Lower CHAIR and higher Recall/POPE are better. |Configuration|CHAIR-S↓|CHAIR-I↓|Recall↑|POPE↑| |-|-|-|-|-| |R+P|48.4|14.1|80.0|87.0| |+Suppression|46.9|13.3|80.5|87.2| |+Enhancement|47.3|13.5|80.4|87.1| |+Random channels|48.2|14.0|80.1|87.0| |+Shuffled channels|48.1|13.9|80.2|87.0| |+Full correction|**45.8**|**12.8**|**81.1**|**87.4**| Full improves on R+P by **2.6 CHAIR-S points**, **1.3 CHAIR-I points**, **1.1 Recall points**, and **0.4 POPE points**. Lower hallucination coincides with stronger genuine-object coverage. CHAIR-S counts descriptions containing hallucinated objects; CHAIR-I counts hallucinated object mentions. Both improve with Full. Recall also rises, so gains do not simply reflect fewer object mentions. Suppression alone reduces CHAIR-I by 0.8 points, enhancement by 0.6, and their combination by 1.3. Random and Shuffled yield 14.0 and 13.9, near R+P. Shuffled retains update values and magnitude, so the advantage of Full highlights **visual-prototype-guided assignments**. **Why this is different from direct visual-token replacement.** Centroid replacement substitutes selected visual embeddings with the image-wide mean, altering direction and norm. Norm-only rescales embeddings without rotating them; Direction-only shifts them toward the centroid while restoring their norms. Unlike these token-space changes, VERGE preserves tokens and corrects downstream FFN channels. A matched comparison shows the trade-off: |Condition|L7 CI↓|L7 Recall↑|Qwen CI↓|Qwen Recall↑| |-|-|-|-|-| |R+P|14.1|80.0|9.5|69.0| |+Norm-only|13.7|79.8|9.3|68.8| |+Direction-only|13.8|78.6|9.4|68.2| |+Centroid|13.5|77.3|9.2|67.1| |+FFN (Full)|**12.8**|**81.1**|**8.9**|**69.4**| Centroid lowers CHAIR-I but drops Recall to 77.3 on L7 and 67.1 on Qwen. FFN correction achieves lower CHAIR-I **and** higher Recall on both models, retaining useful evidence. Norm-only preserves direction while changing token magnitude; Direction-only changes direction but restores magnitude; Centroid alters both. These controls distinguish embedding distortion from FFN modulation and explain Recall differences. **Cross-backbone isolation.** We repeat matched R+P versus Full comparisons across five backbones. |Model|R+P CI|Full CI↓|R+P Recall|Full Recall↑| |-|-|-|-|-| |LLaVA-7B|14.1|**12.8**|80.0|**81.1**| |LLaVA-13B|13.6|**12.9**|78.5|**79.2**| |Qwen2.5-VL|9.5|**8.9**|69.0|**69.4**| |InternVL3|8.3|**8.1**|69.9|**70.2**| |GLM-4V|8.9|**8.5**|73.1|**73.4**| Adding C reduces CHAIR-I by **0.2-1.3 points** and increases Recall by **0.3-1.1 points** across all backbones. The effect persists beyond LLaVA across Qwen2.5-VL, InternVL3, and GLM-4V. Within each architecture only C changes, so the comparison measures its added value. Together with component splits and norm-matched channel shuffling, these matched gains isolate an independent FFN benefit and validate prototype-guided choices of **where**, not only **how much**, to correct.

We kindly remind you that the rebuttal period is drawing to a close. We sincerely hope to clarify the misunderstandings in our paper and address your concerns, and we truly appreciate your taking the time to read our response.

Reviewer_h4SuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VERGE, a training-free framework for mitigating hallucinations in MLLMs. Motivated by case studies linking high-norm visual tokens to off-target attention and hallucinated object predictions, VERGE combines FFN channel correction, uncertainty-triggered visual retracing, and text-prior contrastive decoding. Evaluation across five backbones examines hallucination reduction and general capability preservation.

Critical strengths
  1. Interesting case studies. The analyses of visual-token norms, layer-wise alignment, and token-replacement interventions provide useful observations about visual grounding and hallucination.
  2. Broad empirical evaluation. Experiments across five backbones and complementary benchmarks suggest that VERGE can reduce hallucinations while largely preserving general multimodal capabilities.
Critical weaknesses
  1. The connection between high norms and abnormality is insufficiently justified. The empirical associations and replacement experiments are suggestive, but do not establish that high-norm tokens reliably represent harmful signals rather than useful global information. Treating the remaining tokens as an evidence-bearing subset therefore requires stronger justification. The theoretical analysis assumes alignment between the correction direction and faithful prediction, without establishing why the proposed norm-based partition and channel correction satisfy this assumption. Consequently, the theory offers limited guidance for the actual method design.
  2. Heavy reliance on heuristic hyperparameters. VERGE introduces numerous choices, including the partition ratio, correction strengths, intervention layers, uncertainty threshold, retracing strength, and contrastive-decoding parameters. Several are configured separately for each backbone. The parameter-selection protocol and transferability of these settings are insufficiently established, making it difficult to separate methodological gains from tuning benefits.
Questions for authors
  1. How reliably does token norm distinguish harmful visual signals from useful global representations? Can the proposed partition be compared with random partitions and alternative abnormality criteria?
  2. Can the theoretical alignment assumption be verified for the actual correction direction, and does the theory yield actionable guidance for choosing intervention strengths?
  3. How were the model-specific configurations selected? Were tuning and evaluation data separated, and how does a fixed configuration transfer across backbones?
4 comments

Response 1 — W1 & Q1: Why high-norm visual tokens?

We thank the reviewer for asking why high-norm tokens are selected. We examine spatial evidence, matched token-selection controls, and alternative visual modifications.

1. Large-sample motivation and functional interpretation

Fig. 1 finds high-norm tokens outside object regions that still draw attention. Fig. 2 identifies recurrent high-norm positions and correlates their attention mass with dispersion. Fig. 3 shows stronger late-layer alignment of hallucinated, rather than grounded, object tokens with high-norm features. Together, these observations link strong visual responses to off-target predictions.

Fig. 3 also compares replacement of high- and low-norm tokens by a global visual centroid. Replacing high-norm tokens reduces hallucination, while replacing low-norm tokens causes larger recall losses. High-norm positions are less likely to lie within labeled object regions, supporting selective rebalancing instead of feature deletion.

High norm marks potential over-influence, not intrinsically harmful content. These tokens can carry valuable global context. VERGE preserves them while reinforcing local object evidence; the remaining tokens form an evidence prototype, not a hard object/background classification.

2. Fixed-budget token-selection comparison

For LLaVA-1.5-7B, every method selects the same 10% of projected visual tokens, keeping retracing and decoding fixed. Random samples positions uniformly; Low-norm picks the smallest L2 norms; Attn.-score ranks mean instruction-to-visual attention; Attn.-dispersion uses entropy-derived spatial-attention attribution; and High-norm picks the largest L2 norms. All choices feed the identical prototype-based correction pipeline, isolating the effect of the selection criterion.

SelectionCHAIR-I↓CHAIR-S↓Recall↑POPE↑
Random14.425.678.786.2
Low-norm15.126.775.585.3
Attn.-score14.025.378.886.4
Attn.-dispersion13.724.979.186.6
High-norm12.823.181.187.4

High-norm ranks first on all metrics. Relative to Random, CHAIR-I/CHAIR-S drop by 1.6/2.5 points and Recall/POPE rise by 2.4/1.2. Relative to Attn.-dispersion, the strongest attention control, CHAIR-I falls by 0.9 and Recall increases by 2.0 points. All variants use equal selection budgets.

Attention criteria rank where decoding attends; norms rank the strength of incoming visual vectors. VERGE then uses these vectors for context-aware channel selection.

Magnitude, direction, and centroid controls. Here all conditions retain uncertainty-triggered retracing (R) and language-prior decoding (P), with the same images, prompts, layers, and decoding settings. R+P has these two modules but no FFN correction. Random rescale attenuates random positions using a matched scalar-strength budget. Norm-only rescales high-norm tokens by positive factors, preserving their directions. Direction-only moves selected vectors toward the within-image centroid and then restores their original norms. Centroid directly replaces each selected vector with the mean of all image tokens, changing both magnitude and direction. Full FFN instead corrects intermediate language-model channels without replacing original visual tokens.

ConditionL7 CHAIR-I↓L7 Recall↑Qwen CHAIR-I↓Qwen Recall↑
R+P14.180.09.569.0
Random rescale14.079.99.568.9
Norm-only13.779.89.368.8
Direction-only13.878.69.468.2
Centroid13.577.39.267.1
Full FFN12.881.18.969.4

Centroid replacement erases each selected token's individual spatial feature by substituting a single image-wide average, so object-specific details can be lost. Indeed, its Recall is 77.3 on L7 and 67.1 on Qwen. VERGE instead leaves every visual token intact, uses high-norm and evidence prototypes to identify FFN channels, attenuates the former through a bounded multiplicative mask, and boosts the latter through an additive bias. This yields lower CHAIR-I (12.8/8.9) and higher Recall (81.1/69.4) on L7/Qwen, demonstrating the benefit of channel rebalancing over direct token replacement.

3. Why the method protects useful global semantics

VERGE weights its two prototypes by similarity to the current hidden state and uses FFN gate-weight alignment to identify channels favoring high-norm context or complementary visual evidence.

The FFN suppression mask has a floor c_min to retain scene context. If uncertainty rises, retracing revisits all original tokens, including high-norm ones, and blends their evidence with the corrected FFN output. This retains global structure and local detail.

The two comparisons jointly support norm-guided selection and prototype-guided channel correction: the former identifies influential visual responses, and the latter improves grounding without discarding useful scene evidence.

Response 2 — W1 & Q2: Connecting the theory to VERGE's realized correction

We thank the reviewer for requesting a direct test of Theorem 1 on the implemented FFN correction and guidance for selecting its strength.

1. Regarding the implementation

VERGE builds high-norm (cRc^{R}) and evidence (cEc^{E}) prototypes. Their similarity to FFN gate weights yields channel scores bRb^{R} and bEb^{E}. It then applies bounded suppression and positive evidence enhancement:

m=\operatorname{clip}(1-\beta_{\rm eff}\sigma(b^R),c_{\min},1),\qquad g=\alpha_{\rm eff}\operatorname{ReLU}(b^E).$$ Uncertainty scales the strengths: $$\alpha_{\rm eff}=\alpha r(u),\quad \beta_{\rm eff}=\beta r(u),\quad r(u)=(1-\eta)+\eta u.$$ The realized FFN update $d=\tilde a-a$ is written as $$\tilde a=a+r(u)d_0.$$ Before clipping, the unscaled prototype-guided direction is $$d_0=-\beta\ a\odot\sigma(b^R)+\alpha\ \operatorname{ReLU}(b^E).$$ When the mask reaches its $c_{\min}$ floor, the realized correction $d$ incorporates clipping. The alignment tests use this **actual update**. ### 2. Direct alignment measurements on actual corrections For the true object token $y^\star$ and the highest-scoring incorrect object $y$, the logit-margin gradient along the implemented update is $$M_y(a)=\ell_{y^\star}(a)-\ell_y(a),\qquad A_y=\nabla_a M_y(a)^\top d,\qquad d=\tilde a-a.$$ A positive $A_y$ locally favors the correct object. **Random** draws an isotropic Gaussian vector matched to $d$ in dimension and L2 norm, testing arbitrary perturbations of equal strength. **Reversed** uses $-d$, keeping the same channels and magnitude but reversing their direction. Contexts, layers, and object candidates are identical. |Direction|L7|L13|Qwen|Intern|GLM| |-|-|-|-|-|-| |Random|52.3|51.6|52.0|50.6|50.9| |Reversed|10.7|11.9|13.0|14.4|15.6| |VERGE|**89.3**|**88.1**|**87.0**|**85.6**|**84.4**| Positive-alignment rates (%) are **84.4–89.3 for VERGE**, near 50 for norm-matched Random, and 10.7–15.6 for Reversed. This isolates **correction orientation**: the reversed vector keeps the same channels and magnitude but loses favorable alignment. **Additional diagnostics.** **Top-5 Align+** requires positive gradient alignment against *all five* highest-scoring incorrect objects; it is stricter than the single-competitor test. **Margin gain+** counts contexts where the actual applied correction increases the correct-versus-incorrect logit margin with other modules fixed. **Step+** tests whether first-order gain exceeds the curvature-dependent penalty, using estimated local curvature along that update. |Model|Top-5 Align+|Margin gain+|Step+| |-|-|-|-| |L7|86.7|88.6|78.1| |L13|85.0|87.3|77.0| |Qwen|84.2|86.1|75.4| |Intern|82.4|84.5|72.6| |GLM|80.8|83.7|70.9| **Top-5 Align+ reaches 80.8–86.7%**; **Margin gain+ reaches 83.7–88.6%**; and **Step+ reaches 70.9–78.1%**. These tests separate a positive local derivative, an observed finite margin change, and a curvature-aware lower bound. For Step+, the curvature-aware condition is $$\nabla_a M_y(a)^\top d>\frac{L_y}{2}\|d\|_2^2.$$ Here $L_y$ denotes local curvature. Thus, the step-size check uses the **actual, potentially clipped correction** rather than an abstract direction. ### 3. Intervention intensity and actionable guidance For correction $\tilde a=a+r(u)d_0$, Theorem 1 gives the lower bound $$\Delta(\tilde a)-\Delta(a)\ge r(u)A_\star-\frac{L_\star}{2}r(u)^2\|d_0\|_2^2.$$ Under positive alignment and local smoothness, the guaranteed gain is positive when $$0<r(u)<\frac{2A_\star}{L_\star\|d_0\|_2^2}.$$ The useful interval depends on both alignment $A_\star$ and update norm $\|d_0\|_2$, giving practical guidance beyond a fixed arbitrary multiplier. The bound favors **moderate intervention**: alignment benefits grow linearly while the curvature penalty grows quadratically. VERGE controls strength through $c_{\min}$ and uncertainty scaling. To investigate strength empirically, we vary only a shared multiplier $s$ applied to the two correction strengths: $$\alpha'=s\alpha,\qquad\beta'=s\beta.$$ Retracing, prior-calibrated decoding, uncertainty scaling, and intervention layers stay fixed. **$s=0$ disables only FFN correction**; positive values scale FFN strengths. The LLaVA-1.5-7B sweep therefore isolates channel strength from changes in retracing and prior calibration: |Scale s|Margin Gain↑|CHAIR-I↓|Recall↑| |-|-|-|-| |0|0.00|14.3|79.0| |0.5|+0.19|13.2|80.3| |1.0|**+0.28**|**12.8**|**81.1**| |1.5|+0.14|13.4|80.0| **$s=1.0$** maximizes margin gain and recall while minimizing CHAIR-I. The decrease in margin gain at $s=1.5$ matches the theorem's trade-off between directional improvement and quadratic curvature. **Together, matched directions, realized gains, and controlled strength changes show how VERGE's prototype-guided correction increases faithful-token preference while remaining within an effective intervention range.**

Response 3 — W2 & Q3: Hyperparameter selection and cross-backbone transfer

We thank the reviewer for asking about parameter sensitivity, selection, and cross-backbone transfer. The evidence combines controlled coefficient sweeps with evaluation under shared, source-transferred, and per-model parameter vectors.

1. Parameter roles and existing sensitivity study

VERGE's parameters have distinct roles. ρ\rho selects the high-magnitude token subset; α\alpha and β\beta control FFN enhancement and bounded suppression; and the layer interval specifies where correction occurs. η\eta and u0u_0 regulate uncertainty-aware intervention; λ\lambda blends retrieved visual evidence; and δ\delta together with the plausibility-filter settings calibrates the text-only prior.

These settings separately govern selection, rebalancing, retracing, and decoding, making one-factor-at-a-time sensitivity analysis meaningful.

The sensitivity analysis varies α\alpha, β\beta, ρ\rho, λ\lambda, and δ\delta through seven-point grids on full MME, changing one coefficient at a time while holding the remaining settings at each backbone's reference values. This isolates each parameter's effect under a fixed model configuration.

Quantified sensitivity. Each cell below is the maximum minus minimum full-MME score within that coefficient's seven-point sweep—not a selected parameter value or accuracy percentage. The same sweep definition is applied to all four backbones.

Backboneαβρλδ
Qwen2.514.6058.0322.1613.2520.30
GLM-4V45.3275.2516.3618.05126.12
InternVL315.6330.9213.2213.5322.81
LLaVA-1.510.2937.463.643.254.97

The dominant sensitivity differs across backbones: β\beta spans 58.03 points on Qwen2.5, 30.92 on InternVL3, and 37.46 on LLaVA-1.5, whereas δ\delta spans 126.12 on GLM-4V. These within-backbone ranges identify the controls most relevant to calibration and show why a single universal optimum need not be assumed.

2. Explicit calibration and fair comparisons

Selection protocol. Seven-point one-factor sweeps characterize sensitivity, whereas final parameter choice is a separate calibration decision. A clean comparison uses disjoint calibration and test images, predeclared grids for coefficients, layers, and uncertainty thresholds, and a hallucination–recall objective. Configurations are frozen for testing so evaluation outcomes do not determine the setting.

For comparable tuning opportunities, VCD and PAI can use the same calibration/test separation and candidate-search budget. This directly separates parameter-search exposure from the effects of the inference mechanism.

Sensitivity sweeps test individual parameters; source transfer tests portability of the complete configuration.

3. Transfer without target-specific retuning

Greedy is unmodified decoding (no VERGE modules). Shared applies one identical set of FFN, retracing, and decoding parameters to all models. From L7 and From Qwen reuse a source backbone's complete parameter vector on each destination without retuning. Per-model uses separately selected configurations. The following figures are POPE accuracy percentages:

SettingL7L13QwenInternGLM
Greedy84.187.087.790.889.3
Shared86.587.988.690.989.9
From L787.487.988.090.789.4
From Qwen85.887.589.190.889.8
Per-model87.488.789.191.090.1

Shared improves over Greedy on all five backbones: +2.4 points on L7, +0.9 on L13, +0.9 on Qwen, +0.1 on Intern, and +0.6 on GLM. A single setting thus strengthens object-existence evaluation across heterogeneous models.

Source transfer is also effective. From L7 obtains 87.4% on L7 and reaches 87.9% on L13 and 88.0% on Qwen with unchanged parameters. From Qwen reaches 89.1% on Qwen, alongside 85.8% on L7, 87.5% on L13, and 89.8% on GLM. These outcomes demonstrate strong portability of the rebalancing mechanism across distinct backbones.

Per-model accuracy reaches 87.4, 88.7, 89.1, 91.0, and 90.1, while Shared provides gains across all five models without destination-specific search. Thus, both high-performance calibration and low-tuning deployment are supported.

4. What the comparisons establish

Sensitivity sweeps and transfer tests answer different questions. The former vary one coefficient while fixing the rest; the latter compare complete configurations across backbones under the same evaluation. POPE targets object existence and MME broader visual skills, so optimal settings may differ by dataset.

In summary, The sensitivity analysis provides a clearer understanding of how different hyperparameters affect VERGE's performance. Besides, a single shared configuration can consistently improve POPE performance across all five MLLM backbones. These results also demonstrate strong robustness to hyperparameter choices.

We kindly remind you that the rebuttal period is drawing to a close. We sincerely hope to clarify the misunderstandings in our paper and address your concerns, and we truly appreciate your taking the time to read our response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.