acceptodds
Under review as a conference paper at ICLR 2027

Perception-Cognition Calibration for Mitigating MLLM Hallucinations

Abstract

Multimodal Large Language Models (MLLMs) frequently suffer from hallucinations, generating content that is ungrounded in visual reality. We diagnose this pathology as a compound failure: perceptual deficiency caused by the misallocation of visual attention, and cognitive inertia stemming from an over-reliance on linguistic priors. In this paper, we propose Perception-Cognition Calibration (PeCo), a unified, training-free framework that mitigates MLLM hallucinations by synergizing visual sharpening with prior suppression. Specifically, to counteract perceptual deficiency, we introduce Spatio-Temporal Attention Rectification. This mechanism progressively reclaims the attention budget wasted on uninformative background artifacts (i.e., sink tokens) and redirects it to semantically critical regions, reinforcing fine-grained visual grounding. Furthermore, to overcome cognitive inertia, we devise Sensory-Deprived Contrastive Decoding. By splitting the inference flow into a visually enhanced stream and a sensory-deprived stream, we dynamically penalize hallucination-prone tokens driven by statistical language biases. Extensive evaluations across multiple benchmarks demonstrate that PeCo significantly outperforms existing methods, effectively mitigating hallucinations while enhancing general multimodal reasoning capabilities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MLLM Hallucination, Contrastive Decoding, Perception-Cognition Calibration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UxEkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PeCo is a training-free inference-time framework that mitigates MLLM hallucinations by addressing two diagnosed failure modes: perceptual deficiency and cognitive inertia. It combines Spatio-Temporal Attention Rectification (STAR), which reclaims sink-token attention and redistributes it via spatially-smoothed maps with temporal compensation, and Sensory-Deprived Contrastive Decoding (SDCD), which contrasts a visually enhanced stream against a blind stream retaining only sink tokens. Experiments cover eight MLLM families across POPE, CHAIR, MME, and general VL benchmarks, with ablations on hyperparameters and components.

Critical strengths
  1. Compelling root-cause diagnosis. The zero-out ablation (masking sink tokens → 3.3% drop vs. non-sink → 26%) and sensory deprivation experiment provide direct mechanistic evidence, elevating the paper above generic contrastive-decoding variants.

  2. Coherent dual-intervention design. STAR and SDCD map directly onto the two diagnosed failures; Table 6 confirms their synergy.

  3. Broad validation. Consistent gains across eight architectures, supporting architecture-agnostic claims.

  4. Practical, plug-and-play. No fine-tuning, external detectors, or preference data required.

Critical weaknesses
  1. Novelty relative to VAR/VCD is asserted, not demonstrated. VAR also reallocates sink attention; VCD also contrasts distorted visual streams. Spatial smoothing and temporal compensation are incremental; the blind stream is a narrow contrastive variant. The paper needs a sharper—ideally empirical—argument for why PeCo succeeds where prior methods fail, beyond modest margin gains.

  2. Missing ablation: fully blind stream vs. sink-token-retaining stream. Appendix D argues retention is necessary for stability, but no empirical comparison is provided. This load-bearing design choice needs direct validation.

  3. Narrow hallucination taxonomy. POPE/CHAIR target object existence; attribute, relational, and counting hallucinations are not rigorously evaluated. Figure 8 is only anecdotal.

  4. SDCD doubles decoding cost via two forward passes; latency should be reported in the main text for a method positioned as deployable.

Questions for authors
  1. All evaluated models are standard Transformers. How would STAR/SDCD adapt to linear-attention or state-space VLMs where sink phenomena may differ?

  2. Does spatial smoothing conflate spatially adjacent but distinct objects (e.g., scattered small instances)?

  3. What δ is used in Eq. 10, and how sensitive are results to it?

  4. Is the optimal anchor layer absolute (layer 3) or relative (~1/3 depth)? Would layer 3 remain optimal for deeper/13B models?

3 comments
AuthorsOpenReview ↗

Response to Reviewer UxEk (1/3)

We sincerely thank the reviewer for the constructive and positive feedback. Below are our point-by-point responses to your concerns.

W1: Novelty beyond VAR/VCD.

Thank you for this constructive comment. We have clarified PeCo’s novelty and added direct comparisons with VAR+VCD and ASCD. PeCo introduces two complementary mechanisms:

  • STAR: spatially and temporally structured redistribution. VAR redistributes reclaimed attention proportionally to existing weights, while ASCD steers selected text-centric heads. STAR uses spatially smoothed importance maps and a decoding-dependent reclamation schedule to incorporate local visual context and compensate for declining visual attention during generation.
  • SDCD: a sink-aware contrastive reference. VAR+VCD uses a distorted image, while ASCD suppresses highly attended visual tokens. SDCD suppresses non-sink visual attention in selected image-centric heads while retaining sink tokens, defining the intervention through sink identity and head function.

The direct comparisons show substantial reductions in hallucination: PeCo lowers CHAIR_S by 12.2 and 8.9 points relative to VAR+VCD and ASCD, respectively, while also improving CHAIR_I and both POPE metrics (Table A).

Ablations further support these design choices. Within the perception-enhancement setting, removing spatial smoothing or temporal compensation raises CHAIR_I from 8.6 to 10.2 or 9.1, respectively (Table 7). Adding SDCD further lowers CHAIR_S/I from 35.1/8.6 to 33.3/8.0 (Table 6). Together, the direct comparisons establish the additional value of the complete framework, while the ablations support its individual design choices.

Table A. Performance comparison on LLaVA-1.5-7B.

MethodPOPE-F1 ↑POPE-Acc. ↑CHAIR_S ↓CHAIR_I ↓
VAR+VCD86.386.845.513.4
ASCD85.685.542.212.7
PeCo86.987.233.38.0

W2: Fully blind versus sink-retaining negative streams.

Thank you for highlighting this missing ablation. We conducted additional experiments on LLaVA-1.5-7B to compare three settings: no cognition calibration (CC), CC with a fully blind negative stream, and CC with a sink-retaining negative stream.

Fully blind CC performs worse than omitting CC on all four metrics in this setting. Retaining sink tokens instead improves POPE accuracy by 1.2%. The key insight is that stronger visual deprivation does not necessarily produce a more useful contrastive reference. Contrastive decoding requires negative logits that meaningfully expose language-driven biases. Complete visual suppression may disrupt pretrained attention dynamics and introduce artifacts that distort the contrastive penalty. Retaining sink tokens is intended to preserve computational anchors while suppressing informative visual access.

Table B. Ablation of negative-stream construction.

MethodPOPE-F1 ↑POPE-Acc. ↑CHAIR_S ↓CHAIR_I ↓
Without CC86.286.535.18.6
Fully blind CC85.886.035.88.7
Sink-retaining CC (PeCo)86.987.233.38.0

W3: Attribute, relation, and counting hallucinations.

Thank you for this constructive comment. Our evaluation already extends beyond object existence through the counting, spatial-position, and color subsets of MME (Table 1). As summarized in Table C, PeCo improves performance across all three subsets, providing quantitative evidence of its benefits for counting, spatial-position judgments, and color recognition. These results support PeCo’s broader applicability to multimodal grounding and are consistent with its design: it calibrates visual attention and output distributions at inference time without relying on decoding rules specific to object-existence judgments.

Table C. Results on the counting, spatial-position, and color subsets of MME.

MethodCounting ↑Spatial position ↑Color ↑
Beam Search124.67114.00151.00
PeCo155.00135.67160.00
Improvement+30.33+21.67+9.00
AuthorsOpenReview ↗

Response to Reviewer UxEk (2/3)

W4: Decoding cost should appear in the main text

Thank you for the constructive suggestion. Appendix B reports the inference-time and memory overhead of PeCo. We will move this comparison into the main text to make the performance–cost trade-off more explicit.

Q1: Adaptation to linear-attention and state-space VLMs.

Thank you for this constructive comment. We view PeCo's underlying principles as potentially transferable beyond standard Transformers.

  • STAR: When an architecture provides interpretable attention maps and identifiable sink behavior, attention reclamation and redistribution could be adapted to strengthen informative visual pathways.
  • SDCD: Its contrastive formulation operates on output logits and is therefore less tied to a particular attention mechanism. In principle, it could contrast an evidence-supported stream with a visually attenuated reference constructed through architecture-appropriate interventions, such as modifying visual-input contributions or state updates.

As noted in Appendix E, these extensions have not yet been validated. Future work will assess our method’s generalizability to linear-attention and state-space VLMs.

Q2: Can spatial smoothing conflate nearby distinct objects?

Thank you for this insightful comment. Yes, nearby objects may appear as a single salient region in the smoothed importance map. However, this does not necessarily undermine its purpose: the map estimates regional importance for allocating additional attention, rather than separating individual object instances. When adjacent objects are both relevant, highlighting their shared neighborhood can still direct the reclaimed budget toward useful visual evidence.

Importantly, smoothing is applied only to the redistribution map; it does not pool visual feature embeddings or replace the original non-sink attention. As defined in Eq. (6),

α~i,j=αi,j+Ω qi,j,\tilde{\alpha}_{i,j} = \alpha_{i,j} + \Omega\ q_{i,j},

where qi,jq_{i,j} determines the additional allocation. The original attention differences therefore remain part of the final distribution. For example, if neighboring tokens receive equal redistribution weights, their original attention difference is preserved exactly:

\Longrightarrow\ \tilde{\alpha}_{i,j}-\tilde{\alpha}_{i,k} = \bigl(\alpha_{i,j}-\alpha_{i,k}\bigr) + \Omega\bigl(q_{i,j}-q_{i,k}\bigr) = \alpha_{i,j}-\alpha_{i,k}.$$ Thus, grouping adjacent objects in this auxiliary map does not itself merge their representations or eliminate their original attention contrast. Although smoothing does not guarantee unchanged token rankings, instance-level separation is not required for its intended role of identifying useful regions for attention reinforcement.
AuthorsOpenReview ↗

Response to Reviewer UxEk (3/3)

Q3: Value and sensitivity of δ\delta in Eq. (10).

We set δ=0.1\delta=0.1, as stated in Section 5.1 (p. 7, line 348). This parameter controls the plausibility filter: a token is eligible for contrastive decoding only if its probability under the enhanced branch is at least δ\delta times the maximum probability:

\mathcal{V}_{\mathrm{cand}} = \left\\{ y \mid p_+(y) \geq \delta \max_{w} p_+(w ) \right\\}.

Although δ\delta is fixed, the probability cutoff and candidate set adapt at each decoding step. This filter limits the promotion of low-probability tokens by contrastive subtraction. A smaller δ\delta retains more candidates, while a larger δ\delta imposes stricter filtering and may exclude valid alternatives.

Additional experiments on LLaVA-1.5-7B show limited sensitivity across the tested values δ∈0.05,0.10,0.15\delta\in\\{0.05,0.10,0.15\\} (Table D). The default δ=0.1\delta=0.1 achieves the best result on every metric among these settings.

Table D. Sensitivity to the plausibility threshold on LLaVA-1.5-7B.

δ\deltaPOPE-F1 ↑POPE-Acc. ↑CHAIR_S ↓CHAIR_I ↓
0.0586.686.933.68.3
0.1086.987.233.38.0
0.1586.887.033.48.1

Q4: Anchor-layer selection.

Thank you for the constructive comment. For the evaluated 7B/8B backbones, which have comparable decoder depths, we use the third decoder layer as the default anchor, rather than one third of the total depth.

The anchor is intended to capture an early stage where sufficient vision–language interaction has occurred to construct a context-dependent reference, while visual influence remains amenable to attenuation. Intervening too early may remove useful multimodal context; intervening too late may weaken the contrast because substantial visual information has already entered the hidden states.

The location of this stage can depend partly on model depth. In our additional experiments on the deeper 13B backbone, layer 5 performed better among the evaluated anchor choices. This observation is consistent with a slightly later intervention being preferable in a deeper model. We therefore treat the anchor as a model-dependent early-layer setting.


Finally, thanks again for your time and helpful comments. We hope your concerns are well addressed.

Reviewer_gsJQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

PECO is a training-free framework for mitigating MLLM hallucinations. It combines Spatio-Temporal Attention Rectification, which redistributes sink-token attention using spatial smoothing and a decoding-dependent schedule, with Sensory-Deprived Contrastive Decoding, which contrasts the enhanced branch against an internally vision-suppressed branch. The paper reports improvements on hallucination and general vision-language benchmarks across multiple models.

Critical strengths
  1. Important problem and clear overall design. The paper addresses visual hallucinations in multimodal models and presents a clear framework combining attention interventions with contrastive decoding.

  2. Broad empirical evaluation. The paper reports improvements over the base model on five general vision-language benchmarks and extends the evaluation to seven additional models. This provides encouraging evidence of utility beyond a single task or backbone.

Critical weaknesses
  1. Insufficient comparison with closely related methods. The VAR paper[1] already reports VAR+VCD experiments. This combination uses VAR to enhance visual attention in the positive branch and a distorted image to construct the negative branch for contrastive decoding. Similarly, ASCD[2] uses internal attention interventions to strengthen visual grounding in the positive branch and suppress attention to critical visual tokens in the negative branch. Both therefore share PECO's overall strategy of contrasting visually enhanced and visually weakened branches, although their specific intervention rules differ. However, the paper compares PECO against VAR and VCD individually, without evaluating VAR+VCD or ASCD (Table 1). Without direct comparisons and a clearer discussion of these methodological differences, the additional value of PECO remains insufficiently established.

[1] Kang, Seil, et al. "See what you are told: Visual attention sink in large multimodal models." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Wang, Yujun, et al. "Ascd: Attention-steerable contrastive decoding for reducing hallucination in mllm." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 12. 2026.

  1. Unresolved implementation details raise concerns about soundness, reproducibility, and generalizability. Under Eq. (7) and the default parameters in Section 5.1, γ(t)=1.05−0.55e−t/2>1\gamma(t)=1.05-0.55e^{-t/2}>1 for integer decoding steps t≥5t\geq5. Consequently, the update \check{\alpha}{i,j}=(1-\gamma(t))\alpha{i,j} produces negative sink-token attention weights unless additional handling is applied. Additionally, Eq. (5) requires a 2D spatial neighborhood, whereas InstructBLIP supplies the LLM with projected Q-Former query outputs without a predefined 2D spatial arrangement. The corresponding mapping or architectural adaptation is unclear. Furthermore, the method relies on model-specific sink dimensions DsinkD_{\mathrm{sink}}, but Appendix A does not fully specify their identification procedure, and Table 8 omits the configuration for Qwen3-VL despite its inclusion in Table 5. These issues raise concerns about reproducibility and applicability across different model architectures.
Questions for authors
  1. Implementation clarification: Does the implementation γ(t)\gamma(t), or does it allow negative attention weights?

  2. Evidence needed to assess the current contribution: Could you provide direct comparisons against VAR+VCD and ASCD to clarify the additional value of PECO's specific design choices?

  3. Architectural and reproducibility clarification: How is the 2D neighborhood in Eq. (5) defined for InstructBLIP? Additionally, how are the model-specific sink dimensions DsinkD_{\mathrm{sink}} identified, and what values were used for Qwen3-VL, which is missing from Table 8?

4 comments
AuthorsOpenReview ↗

Response to Reviewer gsJQ (1/2)

We sincerely thank the reviewer for the constructive feedback. Below are our point-by-point responses to your concerns.

W1 & Q2: Relationship to VAR+VCD and ASCD.

Thank you for highlighting these closely related methods. We agree that VAR+VCD and ASCD combine visually enhanced and weakened branches. PeCo's contribution lies in the specific attention-redistribution and counterfactual-construction mechanisms:

  • Positive branch: VAR redistributes reclaimed attention in proportion to existing attention weights, while ASCD steers selected text-centric heads. PeCo's STAR instead uses spatially smoothed importance maps and a decoding-step-dependent reclamation schedule, aiming to consolidate local visual evidence and compensate for declining visual attention during generation.
  • Negative branch: VAR+VCD uses a distorted image, whereas ASCD suppresses highly attended visual tokens. PeCo's SDCD suppresses non-sink visual attention in selected image-centric heads while retaining sink tokens. The distinction is important because high attention magnitude does not necessarily indicate semantic informativeness; SDCD uses sink identification to distinguish the visual pathways being attenuated.

To assess the additional value of these choices, we conducted direct comparisons on LLaVA-1.5-7B. PeCo outperforms both composite baselines on all four reported metrics. The gains are particularly pronounced in open-ended generation. These comparisons establish the additional value of the complete PeCo framework.

Complementary component evidence comes from Table 7: removing spatial or temporal rectification reduces the perception-enhancement MME subtotal from 632.67 to 617.33 or 620.67, respectively. These ablations support the design choices, while the new direct comparisons address their combined value relative to the closest baselines.

Table A. Performance comparison on LLaVA-1.5-7B.

MethodPOPE-F1 ↑POPE-Acc. ↑CHAIR_S ↓CHAIR_I ↓
Base85.484.051.015.2
VCD85.385.051.014.9
VAR86.086.552.414.5
VAR+VCD86.386.845.513.4
ASCD85.685.542.212.7
PeCo86.987.233.38.0

W2 & Q1: Clipping of the reclamation coefficient and attention validity.

We apologize for omitting the clipping operation from the mathematical description. Our implementation caps γ(t)\gamma(t) at 1, so the update does not produce negative sink-token attention weights. Specifically, the effective coefficient is

γeff(t)=min⁡(γbase+η(1−e−t/τ), 1).\gamma_{\mathrm{eff}}(t) = \min\left(\gamma_{\mathrm{base}} + \eta\left(1 - e^{-t/\tau}\right),\ 1\right).

Under the default parameters, its values at t=0,1,2,…t=0,1,2,\ldots are approximately

0.50,0.72,0.85,0.93,0.98,1.00,…0.50,0.72,0.85,0.93,0.98,1.00,\ldots

Thus, γeff(t)∈[0,1]\gamma_{\mathrm{eff}}(t)\in[0,1] and saturates at 1 from t=5t=5. The same clipped coefficient is used for both sink attenuation and budget reclamation:

= \bigl(1-\gamma_{\mathrm{eff}}(t)\bigr)\alpha_{i,j} \ge\ 0, \qquad \Omega_i(t) = \gamma_{\mathrm{eff}}(t)\sum_{j\in\mathcal{S}}\alpha_{i,j}.$$ where $S$ denotes the sink-token set being modified. Eq. (6) redistributes exactly this reclaimed mass to eligible non-sink visual tokens using nonnegative, normalized weights. Consequently, the modified attention row remains nonnegative and sums to 1. For $t\geq5$, sink attention is fully reclaimed within the selected image-centric heads and redistributed, while other heads remain unchanged. We will revise Eq. (7) and the implementation description to explicitly include this upper bound and clarify that the clipped coefficient is used consistently throughout the update. > **W2 & Q3a: The 2D neighborhood for InstructBLIP.** We apologize for the confusion caused by omitting this architecture-specific detail. **1) Architecture-aware adaptation.** For InstructBLIP, we disabled Spatially-Aware Importance Smoothing because its projected Q-Former queries do not have the predefined 2D patch-grid structure required by Eq. (5). The evaluated variant retains temporal attention rectification and Sensory-Deprived Contrastive Decoding (SDCD). **2) Empirical evidence of modularity.** As reported in Table 5, this adapted variant improves InstructBLIP's POPE-F1 from 84.4% to 86.1% and reduces CHAIR_I from 23.7% to 12.5%. These results show that the remaining components can jointly improve performance even when spatial smoothing is geometrically inapplicable, supporting the modular design of PeCo. We will explicitly label this configuration in Table 5.
AuthorsOpenReview ↗

Response to Reviewer gsJQ (2/2)

W2 & Q3b: How Dsink\mathcal{D}_{\mathrm{sink}} is identified.

We apologize for the insufficient description in Appendix A. Following the massive-activation characterization of Sun et al. (2024) [1] and the backbone-based approach of VAR, we identify sink dimensions through the following calibration procedure:

  1. Collect hidden states. We randomly sample 100 images from COCO val2014 and use captioning and question-answering prompts. We record token hidden states from the layer where massive activations first emerge through the penultimate decoder layer.

  2. Measure dimension-wise activation magnitudes. For each hidden state x∈RDx\in\mathbb{R}^{D}, we compute the RMS-normalized magnitude of each feature dimension:

rd(x)=∣x[d]∣1D∑k=1Dx[k]2.r_d(x) = \frac{|x[d]|}{\sqrt{\frac{1}{D}\sum_{k=1}^{D} x[k]^2}}.

  1. Identify persistent outlier coordinates. We select dimensions exhibiting normalized activation magnitudes rd(x)≥20r_d(x)\geq20 in at least 90% of the calibration examples. We additionally cross-check their massive-activation characteristics against Sun et al. [1] : absolute magnitudes exceeding 100 and approximately three orders of magnitude above the corresponding layer's median activation magnitude.

The resulting coordinates form Dsink\mathcal{D}_{\mathrm{sink}} and are kept fixed for each backbone across inputs and layers. We will expand Appendix A to document this procedure explicitly alongside the model-specific indices in Table 8.

W2 & Q3c: Missing Qwen3-VL configuration.

We apologize for omitting the Qwen3-VL configuration from Table 8. As described in Appendix A, sink dimensions are specified according to the underlying language backbone. For Qwen3-VL, we used the Qwen-backbone configuration in our experiments:

Dsink=458, 2570.\mathcal{D}_{\mathrm{sink}} = \\{458,\ 2570\\}.

We will add this missing entry to Table 8 to complete the implementation details.


Finally, thanks again for your time and helpful comments. We hope your concerns are well addressed.

Reference:

[1] Massive activations in large language models, 2024 COLM

AuthorsOpenReview ↗

Follow-up on Our Responses

Dear Reviewer,

Thank you again for your time and constructive feedback. We have provided detailed responses to your concerns, including additional experiments and clarifications. We would be grateful if you could review our responses and let us know whether any concerns remain. We would be happy to provide further clarification.

Best regards,

The Authors

Thank you for the additional experiments and clarifications. These results are helpful and should be incorporated into the manuscript, which may require more than a minor revision. Nevertheless, I still consider the methodological contributions of the proposed approach to be largely incremental compared with existing attention-based and contrastive decoding methods. Therefore, I will maintain my original rating.

Reviewer_kmqAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PECO, a training-free framework for mitigating hallucinations in multimodal large language models. It combines Spatio-Temporal Attention Rectification (STAR) to improve visual grounding with Sensory-Deprived Contrastive Decoding (SDCD) to suppress linguistic priors. Experiments across multiple benchmarks demonstrate improvements in hallucination reduction and general vision-language performance.

Critical strengths
  1. The two components are well motivated and complement each other. Attention redistribution addresses poor visual grounding, while contrastive decoding reduces the model’s reliance on linguistic priors. This gives the framework a clear way to address both perceptual and cognitive sources of hallucinations.
  2. PECO consistently reduces hallucinations across the benchmarks tested. The ablation results suggest that both components contribute to these gains.
Critical weaknesses
  1. Although PECO requires no additional training, it adds substantial inference cost. In Table 9, PECO takes 776 s compared with 301 s for vanilla decoding, or about 2.58 times as long. PECO is also slower than VCD. A clearer discussion of the trade-off between performance and inference time would help readers assess its practical value.
  2. PECO relies on several manually chosen settings, including the attention reclamation strength, the anchor layer, and the contrastive decoding strength. The paper includes some sensitivity analyses, but it is unclear whether the same settings work well across different architectures and tasks.
  3. The paper relies mainly on empirical results to support the proposed mechanisms. A more detailed theoretical explanation would help clarify why attention rectification and suppression of linguistic priors work together to reduce hallucinations.
Questions for authors
  1. How well does the sensory-deprived stream isolate linguistic priors? The stream retains visual sink tokens, so it may still carry some information about the image. How do you verify that its predictions mainly reflect linguistic priors rather than residual visual information or artifacts introduced by attention masking?
  2. Table 5 shows improvements across several MLLM architectures, but most comparisons with other hallucination mitigation methods are on LLaVA-1.5-7B. Could you extend these comparisons to other backbones, particularly those with different visual token structures?
5 comments
AuthorsOpenReview ↗

Response to Reviewer kmqA (1/3)

We sincerely thank the reviewer for the constructive feedback. Below are our point-by-point responses to your concerns.

W1: Computational overhead and the performance–cost trade-off.

We acknowledge that PeCo introduces additional inference cost through its two-stream design. Table 9 reports the following measurements for LLaVA-1.5-7B in the H800/POPE setting. Relative to baseline decoding, PeCo requires 2.58× the runtime and 8.78% more memory, while improving accuracy by 3.2%. Compared with VCD, it incurs an additional 95 s (+13.95%) and 296 MB (+1.77%), accompanied by an accuracy improvement of 2.2%, corresponding to a 14.67% relative reduction in prediction errors. PeCo also achieves higher accuracy than OPERA while requiring 60.25% less time and 22.30% less memory.

Although this inference overhead is meaningful, PeCo offers an efficient deployment option by eliminating the fine-tuning and parameter-optimization costs required by training-based mitigation approaches. Among the evaluated training-free alternatives, its additional cost over VCD yields a measurable improvement in grounding accuracy, supporting a favorable performance–cost trade-off for applications that prioritize reliable visual understanding. In the future, we will further optimize the two-stream inference pipeline to reduce latency and improve the efficiency of the overall framework.

MethodTime (s)Memory (MB)POPE accuracy (%)
Baseline30115,67384.0
VCD68116,75385.0
OPERA1,95221,94385.2
PeCo77617,04987.2

W2: Hyperparameter dependence and robustness.

Thanks for the constructive comment. We address robustness along two axes: transfer across tasks and consistency across backbones. The original Tables 3 and 4 already evaluate each γbase\gamma_{base} and β\beta setting on POPE, CHAIR, and the four-category MME subtotal. For example, β\beta∈{0.2,0.3,0.4} gives POPE accuracy of 86.5–87.2, CHAIR_S/I of 33.3–34.2/8.0–8.6, and MME scores of 629.80–643.67. The corresponding reported baseline values are 84.0, 51.0/15.2, and 565.34. This demonstrates a useful neighboring parameter region across three task families.

Additional cross-backbone sensitivity analysis. We further evaluated γbase\gamma_{base} and β\beta on Qwen3-VL using CHAIR. Both sweeps recover the same selected values as the original LLaVA analysis: γbase\gamma_{base}=0.5 and β\beta=0.3. Moreover, the neighboring γbase\gamma_{base} settings 0.3–0.7 give CHAIR_S/I of 15.4–16.0/4.0–4.3, while β\beta=0.2–0.4 gives 15.4–15.8/4.0–4.2. These measurements provide additional evidence that useful settings are not confined to the original backbone.

Table A. Reclamation-strength sensitivity on Qwen3-VL.

γbase\gamma_{base}CHAIR_S ↓CHAIR_I ↓
0.116.64.7
0.316.04.3
0.515.44.0
0.715.84.2
0.916.44.5

Table B. Contrastive-strength sensitivity on Qwen3-VL.

β\betaCHAIR_S ↓CHAIR_I ↓
0.116.04.3
0.215.64.1
0.315.44.0
0.415.84.2
0.516.24.5

Additional cross-task anchor analysis. Figure 7 already compares absolute anchor layers 1–5 on LLaVA-1.5-7B and Qwen2-VL using POPE. We additionally evaluated the anchor on LLaVA-1.5-7B using MME. The absolute third layer again achieves the highest score, consistent with the POPE results.

Table C. Anchor-layer sensitivity on LLaVA-1.5-7B.

Anchor layerMME↑
1629.30
2636.54
3643.67
4638.80
5632.66

Taken together, the original multi-task analyses, the additional Qwen3-VL sensitivity studies, and the MME anchor-layer evaluation suggest that PeCo does not require extensive hyperparameter tuning, with its default settings generalizing well across the evaluated models and tasks.

AuthorsOpenReview ↗

Response to Reviewer kmqA (2/3)

W3: Theoretical analysis of the joint intervention.

We agree that the empirical results should be complemented by a clearer mathematical explanation. Our analysis builds on the likelihood-ratio formulation[1], the conditional pointwise mutual information interpretation[2], and the correct-versus-hallucinated logit-gap analysis[3]. Below, we make explicit the conditions under which attention rectification and contrastive decoding can complement one another.

1. Attention rectification: improving the allocation of visual evidence.

Consider a normalized attention row aa. Let SS denote the sink-token set, ss the original attention distribution normalized within SS, and qq a nonnegative redistribution distribution supported on eligible non-sink visual tokens. Both ss and qq are extended by zero outside their respective supports. Reclaiming a mass Ω\Omega, with 0≤Ω≤∑j∈Saj0 \leq \Omega \leq \sum_{j \in S} a_j, gives the valid, mass-preserving update

a′=a+Ω(q−s).a' = a + \Omega(q-s).

For a target-object alignment score gj∈[0,1]g_j \in [0,1], defined as zero outside the visual region, the change in attention-weighted alignment is exactly

\langle a', g\rangle - \langle a, g\rangle &= \Omega\\langle q-s\, g\rangle \\ &= \Omega\left(\mathbb{E}_{j\sim q}[g_j] - \mathbb{E}_{j\sim s}[g_j]\right). \end{aligned}$$ Thus, rectification improves this grounding measure whenever the recipient distribution has greater expected target alignment than the reclaimed sink distribution. This identifies the required condition without assuming that every non-sink token is informative. Related attention-alignment analysis is demonstrated in SECOND[4]. **2. Contrastive decoding: amplifying evidence-supported preferences.** For a grounded candidate $c$ and a hallucinated alternative $h$ that both survive the plausibility filter, define their log-odds margins in the enhanced and sensory-deprived branches as $$m_+ = \log\frac{p_+(c)}{p_+(h)}, \qquad m_- = \log\frac{p_-(c)}{p_-(h)}.$$ Under PeCo's contrastive rule, the resulting margin satisfies the exact identity $$\begin{aligned} m_{\mathrm{PeCo}} &= \log\frac{p_{\mathrm{PeCo}}(c)}{p_{\mathrm{PeCo}}(h)} \\ &= (1+\beta)m_+ - \beta m_- \\ &= m_+ + \beta(m_+-m_-). \end{aligned}$$ For $\beta>0$, PeCo therefore increases the grounded candidate's relative odds over the enhanced branch alone precisely when $$m_+ > m_-.$$ This condition means that visual evidence gives the grounded candidate a stronger relative advantage in the enhanced branch than in the sensory-deprived branch. **3. Why the two interventions can complement one another.** The identity above separates their potential contributions. With the negative branch fixed, if attention rectification increases the positive margin by $\Delta>0$, the final margin increases by $$\Delta m_{\mathrm{PeCo}}=(1+\beta)\Delta.$$ With the positive branch fixed, if the attenuated reference reduces its grounded-versus-hallucinated margin by $r>0$, the final margin increases by $$\Delta m_{\mathrm{PeCo}}=\beta r.$$ Thus, improved visual grounding and an informative contrastive reference can jointly strengthen evidence-supported token preferences. These are conditional statements: attention alignment alone does not guarantee a favorable output margin, and improved candidate-level odds do not constitute an unconditional sequence-level hallucination guarantee. Table 6 provides empirical evidence consistent with this complementary design. The combined method outperforms either component alone on both backbones, supporting their empirical complementarity. Together, the conditional analysis and component ablations provide a concrete explanation of the joint intervention. | Backbone | Perception enhancement only | Cognition calibration only | Full PeCo | | ------------ | --------------------------: | -------------------------: | -------------: | | LLaVA-1.5-7B | 35.1 / 8.6 | 36.3 / 9.8 | **33.3 / 8.0** | | Qwen2-VL | 23.3 / 6.2 | 25.2 / 6.8 | **21.9 / 5.3** |
AuthorsOpenReview ↗

Response to Reviewer kmqA (3/3)

Q1: Effectiveness of the sensory-deprived stream.

Thank you for raising this question. SDCD is designed to construct a visually attenuated reference that emphasizes context-conditioned linguistic tendencies. It does not require the negative branch to be completely independent of the image. We clarify below the motivation for sink retention, the conditions under which residual information is tolerable, and the evidence supporting the resulting reference.

1. Why retain sink tokens?

SDCD suppresses non-sink visual attention in selected image-centric heads from the anchor layer onward, while retaining sink tokens. This design is motivated by their different observed contributions to visual understanding. In Figure 1, masking the identified sink set reduces VQA(v2) accuracy by 3.3%, whereas masking the non-sink set reduces it by 26.0%. This supports preferentially suppressing the more task-informative visual pathway.

Low semantic relevance also does not imply computational dispensability. StreamingLLM[5] demonstrates the computational importance of retaining attention sinks in streaming inference, while Barbero et al. [6] theoretically analyze their role in controlling representational over-mixing. These findings motivate preserving structural anchors while attenuating informative visual access. They provide a rationale for our design.

2. Exact prior isolation is not necessary for a beneficial contrastive effect.

For a grounded candidate cc and a hallucinated alternative hh that both survive the plausibility filter, define

\qquad m_- = \log\frac{p_-(c)}{p_-(h)},$$ where $p_+$ and $p_-$ denote the enhanced and sensory-deprived distributions. Let $p_0$ be an ideal text-conditioned reference under the same textual history, with $$m_0 = \log\frac{p_0(c)}{p_0(h)}.$$ Suppose residual visual information and masking artifacts perturb the relevant reference margin by at most $\varepsilon$: $$|m_- - m_0| \leq \varepsilon.$$ For $\beta>0$, PeCo then satisfies $$\begin{aligned} m_{\mathrm{PeCo}}-m_+ &= \beta(m_+-m_-) \\ &= \beta(m_+-m_0)-\beta(m_--m_0) \\ &\geq \beta\left(m_+-m_0-\varepsilon\right). \end{aligned}$$ Therefore, the grounded candidate's relative odds improve whenever $$m_+-m_0>\varepsilon.$$ This gives a precise sufficient condition: residual information or masking error is tolerable when its distortion of the reference margin is smaller than the visual-evidence advantage. The negative branch need not be a perfectly isolated language prior. **3. What does the current empirical evidence establish?** Table 6 shows that adding SDCD to perception enhancement improves both hallucination metrics on two backbones: | Backbone | Perception enhancement only | Full PeCo | | ------------ | --------------------------: | -------------: | | LLaVA-1.5-7B | 35.1 / 8.6 | **33.3 / 8.0** | | Qwen2-VL | 23.3 / 6.2 | **21.9 / 5.3** | These results establish the practical usefulness of the selected reference within the evaluated settings. Accordingly, we will replace the overly strong description “pure language priors” with **“a visually attenuated, context-conditioned reference.”** > **Q2: Generalization relative to competing methods on additional backbones.** Thank you for the constructive suggestion. To assess generalization beyond LLaVA, we extended the comparison to Qwen2-VL with representative baselines. PeCo achieves the lowest scores on both metrics, outperforming VAR, the strongest listed competitor, by 2.4 on CHAIR_S and 1.4 on CHAIR_I. Together with the cross-model evaluations in Table 5, these results strengthen the evidence that PeCo's benefits extend beyond a single backbone. **Table D. Performance comparison on Qwen2-VL.** | Method | CHAIR_S ↓ | CHAIR_I ↓ | | ------ | --------: | --------: | | Base | 25.0 | 7.3 | | DoLa | 24.6 | 7.0 | | VCD | 25.0 | 7.1 | | VAR | 24.3 | 6.7 | | PeCo | **21.9** | **5.3** | --- Finally, thanks again for your time and helpful comments. We hope your concerns are well addressed. **References:** [1] Contrastive Decoding: Open-ended Text Generation as Optimization, 2023 ACL [2] Trusting Your Evidence: Hallucinate Less with Context-aware Decoding, 2024 NAACL [3] Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization, 2024 NeurIPS [4] SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding, 2025 ICML [5] Efficient Streaming Language Models with Attention Sinks, 2024 ICLR [6] Why do LLMs attend to the first token? 2025 arXiv

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.