- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VERGE is a training-free framework for mitigating hallucination in multimodal language models. It partitions visual tokens by magnitude and uses the resulting prototypes to suppress or enhance FFN channels. It then retrieves visual features under high uncertainty and calibrates decoding against a text-only branch. The paper provides conditional theoretical analyses and evaluates five backbone configurations on hallucination and general multimodal benchmarks.
Critical strengths
- The paper addresses an important limitation of multimodal models without requiring additional training.
- The empirical analysis identifies an association between high-magnitude visual tokens and diffuse attention, together with increased alignment of hallucinated object tokens to these representations in later layers. These observations provide a useful basis for investigating failures of visual grounding.
- Evaluation spans several model families and includes both hallucination and general capability benchmarks.
Critical weaknesses
-
The motivating intervention does not isolate token magnitude. Replacing a token with the image mean also changes its semantic content and direction. Moreover, the paper provides limited direct evidence that prototype similarity identifies channels responsible for hallucination. The detailed diagnosis is concentrated on LLaVA, so performance gains on other backbones do not by themselves establish the same mechanism.
-
The contribution beyond existing methods is insufficiently isolated. Visual retracing and language-prior calibration are closely related to MemVR and PAI. The component ablation is useful, but measures full-MME performance, where the incremental effect of channel correction is modest. Hallucination-specific ablations are needed to establish whether this component materially improves the combined method.
-
The theory provides conditional justification without establishing that the implemented method satisfies its central conditions. The margin result assumes favorable alignment of the correction direction, while the retracing analysis requires sufficient information gain. Neither is verified for VERGE. In particular, the analysis does not establish why gate-weight/prototype similarities produce the required direction or provide an operational choice of correction strength.
-
The contrastive term and candidate filtering are not independently evaluated. The contrastive coefficient also determines the plausibility threshold, so the reported sweep changes both mechanisms simultaneously. This prevents attribution of the decoding gains to language-prior subtraction. Separate controls under the same decoding rule are needed. POPE answer frequencies and class-specific errors would also help distinguish improved grounding from a shifted response tendency.
-
Parameter sensitivity is examined, but the selection protocol remains unclear. The method uses backbone-specific coefficients, thresholds, and intervention layers, and the reported sweeps show different responses across models. Without a clearly separated validation procedure, these results do not establish that the selected configuration transfers to unseen evaluation data.
Questions for authors
- Can norm-only interventions and measurements on another backbone establish the proposed link between token magnitude, selected FFN channels, and hallucination?
- What is the incremental benefit of channel correction on CHAIR or HallusionBench when retracing and prior calibration are already enabled?
- How often does the implemented correction satisfy the theorem’s alignment and step-size conditions? Does the theory predict successful and unsuccessful interventions?
- Can prior subtraction and plausibility filtering be varied independently under identical decoding settings?
4 comments
We test how VERGE's implemented interventions relate to Theorems 1 and 2 through the realized FFN update and paired Retracing ON/OFF diagnostics.
W3&Q3: Empirical verification of Theorem 1. Cosine similarity between FFN gate rows and visual prototypes guides bounded suppression and enhancement. The realized update to activation is
Here includes actual masks and uncertainty scaling. For true object and incorrect object , define . Its alignment is
Positive favors the true object locally. Given smoothness bound ,
The first term measures direction, the second penalizes strength. Since cosine similarity selects channels but the theorem concerns faithful margins, we test the link through actual downstream-logit gradients, not similarity alone.
To examine both alignment and its prediction effects, Align+ checks positive alignment against the leading wrong object; Top-5+ requires it against five incorrect candidates. Margin gain+ records actual positive logit-margin changes with other modules fixed. Step+ checks the estimated sufficient condition , comparing first-order gain with curvature cost. Together they test direction, finite-step success, and strength.
| Model | Align+ | Top-5+ | Margin gain+ | Step+ |
|---|---|---|---|---|
| LLaVA-7B | 89.3 | 86.7 | 88.6 | 78.1 |
| LLaVA-13B | 88.1 | 85.0 | 87.3 | 77.0 |
| Qwen2.5-VL | 87.0 | 84.2 | 86.1 | 75.4 |
| InternVL3 | 85.6 | 82.4 | 84.5 | 72.6 |
| GLM-4V | 84.4 | 80.8 | 83.7 | 70.9 |
Across backbones, Align+ reaches 84.4–89.3%, Top-5+ 80.8–86.7%, and realized Margin gain+ 83.7–88.6%. Step+ holds in 70.9–78.1%, selecting contexts where the estimated first-order benefit exceeds the curvature cost. Thus Step+ predicts sufficient-margin improvement, while Margin gain+ separately records actual success. A nonpositive Step+ indicates absence of this sufficient certificate, not necessarily a reduced margin.
To further isolate the role of channel direction, Random uses a Gaussian update matched to in L2 norm; Reversed applies , keeping norm and channel pattern but reversing direction. Image, prefix, layer, and competitor are fixed. Alignment rates (%) are:
| Direction | L7 | L13 | Qwen | Intern | GLM |
|---|---|---|---|---|---|
| Random | 52.3 | 51.6 | 52.0 | 50.6 | 50.9 |
| Reversed | 10.7 | 11.9 | 13.0 | 14.4 | 15.6 |
| VERGE | 89.3 | 88.1 | 87.0 | 85.6 | 84.4 |
Random reaches 50–52% and Reversed only 10.7–15.6%, isolating the prototype-guided direction rather than update norm.
We also examine how the theoretical trade-off guides intervention strength. Theorem 1 expresses the nominal update as , where . The lower bound on faithful-margin improvement is
The linear benefit competes with quadratic cost. With other modules fixed, yields margin gains and CHAIR-I . Nominal also maximizes Recall (81.1). This non-monotonicity offers practical strength guidance and distinguishes effective from over-strong updates.
Theorem 2: Visual-information recovery. At triggered contexts , Theorem 2 compares pre/post-retracing representations through target-information gain:
Theorem 2 predicts improved complexity when ; the full bound also involves residual . Direct estimation is difficult: and require conditional mutual-information/entropy estimates for high-dimensional hidden states, depends on nuisance-factor sensitivity, and on surrogate losses. We therefore indirectly assess recovered target evidence through paired prediction outcomes, while VERGE controls added complexity via one-shot retrieval of all original visual tokens, scale matching, and bounded blending.
To isolate retracing, we compare ON/OFF on 1,000 images per backbone (one triggered object event each), holding image, prefix, state, layer, FFN correction, and decoding fixed. LL Gain+ tracks true-object log-probability increases; Useful tracks larger margins against OFF's fixed strongest wrong object; Failed counts nonpositive margin changes.
| Model | LL Gain+↑ | Useful↑ | Failed↓ |
|---|---|---|---|
| LLaVA-7B | 87.9% | 89.5% | 10.5% |
| LLaVA-13B | 86.6% | 87.9% | 12.1% |
| Qwen2.5-VL | 85.2% | 86.6% | 13.4% |
| InternVL3 | 83.7% | 85.0% | 15.0% |
| GLM-4V | 81.9% | 83.2% | 16.8% |
Retracing improves true-object likelihood in 81.9–87.9% of cases and margins in 83.2–89.5%. LL Gain+ measures target support; Useful tests discrimination against OFF's fixed incorrect competitor. These paired, controlled prediction gains provide strong empirical evidence of recovered visual information, consistent with Theorem 2's mechanism.
We first isolate the decoding components and then test whether a fixed VERGE configuration transfers across model backbones.
W4&Q4: Isolating the decoding contributions. Unlike PAI's visual-attention reweighting and logit calibration, VERGE applies channel correction upstream and combines text-prior subtraction with an image-conditioned plausibility filter. The two decoding operations have different roles: prior subtraction adjusts scores of text-favored candidates, whereas filtering restricts selection to image-supported candidates. We therefore test them independently to identify the source of decoding gains under matched settings.
With image-conditioned and text-only log probabilities , the score is
The image-conditioned plausibility filter retains candidates satisfying:
To isolate their contributions, we decouple the cutoff from . Prior-only applies without filtering; Filter-only applies without subtraction; Combined activates both at fixed strengths, while Baseline disables both. All configurations share identical inputs and visual-side modules, while the filtering threshold is held fixed to assess the effect of prior subtraction.
Checking grounding beyond answer frequency. To distinguish better object decisions from a general tendency to answer 'no', we also inspect POPE's class-specific behavior. Yes% records positive answers, FPR absent objects answered yes, and FNR present objects answered no. CHAIR-I separately measures hallucinated mentions in free-form descriptions.
| Prior | Filter | CHAIR-I↓ | POPE↑ | Yes% | FPR↓ | FNR↓ |
|---|---|---|---|---|---|---|
| Off | Off | 14.1 | 86.1 | 58 | 21.9 | 5.9 |
| On | Off | 13.3 | 86.5 | 55 | 18.5 | 8.5 |
| Off | On | 13.7 | 86.4 | 54 | 17.6 | 9.6 |
| On | On | 12.8 | 87.4 | 52 | 14.6 | 10.6 |
Prior-only yields CHAIR-I 13.3 and POPE 86.5; Filter-only gives 13.7 and 86.4. Joint decoding achieves 12.8 and 87.4, improving both metrics by 1.3 points over Neither. Yes% shifts from 58% to 52%. FPR drops by 7.3 points (21.9 to 14.6), while FNR rises by 4.7 (5.9 to 10.6). Since POPE is balanced, the larger FPR reduction yields a net +1.3 accuracy points. The accompanying CHAIR-I decrease supports fewer hallucinated object mentions, while both error classes show how the yes/no tendency changes. The 2×2 study isolates each operator and their complementary effect. Prior subtraction counters text-driven scores, while filtering excludes visually implausible candidates, explaining their complementary benefit.
W5: Parameter sensitivity and transfer. We separate two complementary analyses. Seven-point sweeps on full MME vary one FFN coefficient , partition ratio , retracing weight , or decoding strength while other coefficients, triggers, and layers remain at their model-specific reference values. This characterizes the effect of each choice. To examine reuse rather than retuning, the separate POPE transfer experiment freezes a shared or source-selected parameter vector before evaluating a destination backbone. The benchmark and decoding protocol remain unchanged across conditions. This directly tests performance with fixed source coefficients instead of optimizing anew for each target. The reported procedure characterizes sensitivity and cross-backbone transfer, without specifying an image-disjoint calibration/test split.
To test this distinction on POPE accuracy (%), we compare Greedy (unmodified model), Shared (one common configuration), From L7/From Qwen (complete source-selected vectors transferred without target retuning), and Per-model (individual settings). Each backbone uses the same evaluation across conditions.
| Setting | L7 | L13 | Qwen | Intern | GLM |
|---|---|---|---|---|---|
| Greedy | 84.1 | 87.0 | 87.7 | 90.8 | 89.3 |
| Shared | 86.5 | 87.9 | 88.6 | 90.9 | 89.9 |
| From L7 | 87.4 | 87.9 | 88.0 | 90.7 | 89.4 |
| From Qwen | 85.8 | 87.5 | 89.1 | 90.8 | 89.8 |
| Per-model | 87.4 | 88.7 | 89.1 | 91.0 | 90.1 |
Shared settings consistently outperform Greedy across all five backbones, with POPE accuracy gains of +2.4 (L7), +0.9 (L13), +0.9 (Qwen), +0.1 (Intern), and +0.6 (GLM). Configurations transferred from L7 or Qwen also achieve competitive performance without target-specific retuning, while per-model settings provide further improvements on some backbones. Although sensitivity analyses reveal different responses to individual hyperparameters, these consistent gains demonstrate VERGE's robustness under shared settings, while allowing for different optimal configurations across evaluation tasks.
In summary, the independent decoding experiments confirm the complementary contributions of prior subtraction and plausibility filtering to hallucination mitigation and object-level prediction. Together with the cross-backbone results, these findings highlight VERGE's effectiveness and practical applicability across diverse MLLMs.
We kindly remind you that the rebuttal period is drawing to a close. We sincerely hope to clarify the misunderstandings in our paper and address your concerns, and we truly appreciate your taking the time to read our response.
We thank the reviewer for asking about magnitude, FFN attribution, and component novelty. Our controls examine prototype-guided channel selection.
W1&Q1: Separating magnitude, direction, and channel effects. Figs. 2–3 link high norms to diffuse attention and hallucinated-object alignment in deeper layers. Such tokens can also encode useful scene context. The motivating Centroid intervention replaces selected embeddings with the image-wide mean, changing both magnitude and semantic direction.
Random rescale attenuates a randomly selected, equally sized token subset using the same strength as high-norm attenuation. Norm-only rescales the selected high-norm embeddings but preserves their feature directions. Direction-only shifts these embeddings toward the centroid while restoring each original L2 norm. Centroid replaces the full embeddings with the global image mean, altering direction and norm together. In contrast, FFN (Full) keeps every original visual token and changes downstream FFN channel responses based on cosine similarity with the visual prototypes.
To isolate the interventions, we hold retracing (R), prior calibration (P), and evaluation fixed. CHAIR-I (CI) tracks hallucinated object mentions, while Recall tracks real-object coverage on LLaVA and Qwen2.5:
Norm-only lowers CI from 14.1 to 13.7 on LLaVA and from 9.5 to 9.3 on Qwen, outperforming Random rescale. Centroid lowers CI further but reduces Recall to 77.3 and 67.1. Full FFN correction attains the lowest CI and highest Recall on both models: 12.8/81.1 and 8.9/69.4. These cross-backbone results favor magnitude-aware selection and visual preservation in VERGE
Why channel selection differs from token replacement. VERGE forms relevance-weighted prototypes from the high-norm subset and the remaining evidence subset. For each FFN gate-weight row, cosine similarity to the high-norm prototype guides bounded suppression, while similarity to the evidence prototype guides additive enhancement. For intermediate activation a, the realized channel correction is
d=\tilde a-a=a\odot(m-\mathbf 1)+g.$$ Mask $m$ is lower-bounded by $c_{\min}$ to preserve global cues, while $g$ enhances evidence-aligned channels. Unlike centroid replacement, VERGE retains original tokens and spatial detail, adjusting only their downstream channel influence. **W2&Q2: Isolating the FFN correction benefit.** MemVR retrieves visual memory through the FFN pathway, whereas PAI reweights visual attention and calibrates logits. **VERGE instead uses two visual prototypes to select intermediate FFN channels** for bounded suppression and evidence enhancement. Rather than replacing visual tokens or adjusting attention weights, this directly rebalances how specific FFN features influence downstream predictions. **To test whether this distinction reduces hallucination while preserving object recall**, we keep retracing, prior calibration, intervention layers, and decoding settings fixed, removing only FFN correction to obtain **R+P**. We then compare **Suppression only** ($a\odot m$), **Enhancement only** ($a+g$), and their full combination. To test whether the benefit comes from visually informed channel assignment rather than a generic perturbation, **Random channels** uses a Gaussian vector matched to the realized update $d$ in dimension and L2 norm. **Shuffled channels** permutes the actual correction values across FFN channels, retaining every value and its total norm but breaking the prototype-guided correspondence. Comparing these controls with Full isolates the importance of **where the correction is applied** under matched intervention magnitude. |Configuration|CHAIR-S↓|CHAIR-I↓|Recall↑|POPE↑| |-|-|-|-|-| |R+P|48.4|14.1|80.0|87.0| |+Suppression|46.9|13.3|80.5|87.2| |+Enhancement|47.3|13.5|80.4|87.1| |+Random channels|48.2|14.0|80.1|87.0| |+Shuffled channels|48.1|13.9|80.2|87.0| |+Full correction|**45.8**|**12.8**|**81.1**|**87.4**| CHAIR-S counts descriptions with hallucinated objects, while CHAIR-I counts hallucinated mentions. Full lowers CHAIR-S/I by **2.6/1.3 points** versus R+P and raises Recall/POPE by **1.1/0.4**. Suppression and enhancement help individually, but Full is best; norm-matched Random and Shuffled remain near R+P. These hallucination-specific gains isolate FFN correction beyond full-MME ablations. **In summary,** the separate magnitude and direction controls clarify the high-norm observation, the Qwen comparison extends it beyond LLaVA, and the randomized and shuffled FFN controls isolate the value of visually guided channel correction. Overall, VERGE improves grounding while retaining real-object coverage.