- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PeCo is a training-free inference-time framework that mitigates MLLM hallucinations by addressing two diagnosed failure modes: perceptual deficiency and cognitive inertia. It combines Spatio-Temporal Attention Rectification (STAR), which reclaims sink-token attention and redistributes it via spatially-smoothed maps with temporal compensation, and Sensory-Deprived Contrastive Decoding (SDCD), which contrasts a visually enhanced stream against a blind stream retaining only sink tokens. Experiments cover eight MLLM families across POPE, CHAIR, MME, and general VL benchmarks, with ablations on hyperparameters and components.
Critical strengths
-
Compelling root-cause diagnosis. The zero-out ablation (masking sink tokens → 3.3% drop vs. non-sink → 26%) and sensory deprivation experiment provide direct mechanistic evidence, elevating the paper above generic contrastive-decoding variants.
-
Coherent dual-intervention design. STAR and SDCD map directly onto the two diagnosed failures; Table 6 confirms their synergy.
-
Broad validation. Consistent gains across eight architectures, supporting architecture-agnostic claims.
-
Practical, plug-and-play. No fine-tuning, external detectors, or preference data required.
Critical weaknesses
-
Novelty relative to VAR/VCD is asserted, not demonstrated. VAR also reallocates sink attention; VCD also contrasts distorted visual streams. Spatial smoothing and temporal compensation are incremental; the blind stream is a narrow contrastive variant. The paper needs a sharper—ideally empirical—argument for why PeCo succeeds where prior methods fail, beyond modest margin gains.
-
Missing ablation: fully blind stream vs. sink-token-retaining stream. Appendix D argues retention is necessary for stability, but no empirical comparison is provided. This load-bearing design choice needs direct validation.
-
Narrow hallucination taxonomy. POPE/CHAIR target object existence; attribute, relational, and counting hallucinations are not rigorously evaluated. Figure 8 is only anecdotal.
-
SDCD doubles decoding cost via two forward passes; latency should be reported in the main text for a method positioned as deployable.
Questions for authors
-
All evaluated models are standard Transformers. How would STAR/SDCD adapt to linear-attention or state-space VLMs where sink phenomena may differ?
-
Does spatial smoothing conflate spatially adjacent but distinct objects (e.g., scattered small instances)?
-
What δ is used in Eq. 10, and how sensitive are results to it?
-
Is the optimal anchor layer absolute (layer 3) or relative (~1/3 depth)? Would layer 3 remain optimal for deeper/13B models?
3 comments
Response to Reviewer UxEk (2/3)
W4: Decoding cost should appear in the main text
Thank you for the constructive suggestion. Appendix B reports the inference-time and memory overhead of PeCo. We will move this comparison into the main text to make the performance–cost trade-off more explicit.
Q1: Adaptation to linear-attention and state-space VLMs.
Thank you for this constructive comment. We view PeCo's underlying principles as potentially transferable beyond standard Transformers.
- STAR: When an architecture provides interpretable attention maps and identifiable sink behavior, attention reclamation and redistribution could be adapted to strengthen informative visual pathways.
- SDCD: Its contrastive formulation operates on output logits and is therefore less tied to a particular attention mechanism. In principle, it could contrast an evidence-supported stream with a visually attenuated reference constructed through architecture-appropriate interventions, such as modifying visual-input contributions or state updates.
As noted in Appendix E, these extensions have not yet been validated. Future work will assess our method’s generalizability to linear-attention and state-space VLMs.
Q2: Can spatial smoothing conflate nearby distinct objects?
Thank you for this insightful comment. Yes, nearby objects may appear as a single salient region in the smoothed importance map. However, this does not necessarily undermine its purpose: the map estimates regional importance for allocating additional attention, rather than separating individual object instances. When adjacent objects are both relevant, highlighting their shared neighborhood can still direct the reclaimed budget toward useful visual evidence.
Importantly, smoothing is applied only to the redistribution map; it does not pool visual feature embeddings or replace the original non-sink attention. As defined in Eq. (6),
where determines the additional allocation. The original attention differences therefore remain part of the final distribution. For example, if neighboring tokens receive equal redistribution weights, their original attention difference is preserved exactly:
\Longrightarrow\ \tilde{\alpha}_{i,j}-\tilde{\alpha}_{i,k} = \bigl(\alpha_{i,j}-\alpha_{i,k}\bigr) + \Omega\bigl(q_{i,j}-q_{i,k}\bigr) = \alpha_{i,j}-\alpha_{i,k}.$$ Thus, grouping adjacent objects in this auxiliary map does not itself merge their representations or eliminate their original attention contrast. Although smoothing does not guarantee unchanged token rankings, instance-level separation is not required for its intended role of identifying useful regions for attention reinforcement.Response to Reviewer UxEk (3/3)
Q3: Value and sensitivity of in Eq. (10).
We set , as stated in Section 5.1 (p. 7, line 348). This parameter controls the plausibility filter: a token is eligible for contrastive decoding only if its probability under the enhanced branch is at least times the maximum probability:
\mathcal{V}_{\mathrm{cand}} = \left\\{ y \mid p_+(y) \geq \delta \max_{w} p_+(w ) \right\\}.
Although is fixed, the probability cutoff and candidate set adapt at each decoding step. This filter limits the promotion of low-probability tokens by contrastive subtraction. A smaller retains more candidates, while a larger imposes stricter filtering and may exclude valid alternatives.
Additional experiments on LLaVA-1.5-7B show limited sensitivity across the tested values (Table D). The default achieves the best result on every metric among these settings.
Table D. Sensitivity to the plausibility threshold on LLaVA-1.5-7B.
| POPE-F1 ↑ | POPE-Acc. ↑ | CHAIR_S ↓ | CHAIR_I ↓ | |
|---|---|---|---|---|
| 0.05 | 86.6 | 86.9 | 33.6 | 8.3 |
| 0.10 | 86.9 | 87.2 | 33.3 | 8.0 |
| 0.15 | 86.8 | 87.0 | 33.4 | 8.1 |
Q4: Anchor-layer selection.
Thank you for the constructive comment. For the evaluated 7B/8B backbones, which have comparable decoder depths, we use the third decoder layer as the default anchor, rather than one third of the total depth.
The anchor is intended to capture an early stage where sufficient vision–language interaction has occurred to construct a context-dependent reference, while visual influence remains amenable to attenuation. Intervening too early may remove useful multimodal context; intervening too late may weaken the contrast because substantial visual information has already entered the hidden states.
The location of this stage can depend partly on model depth. In our additional experiments on the deeper 13B backbone, layer 5 performed better among the evaluated anchor choices. This observation is consistent with a slightly later intervention being preferable in a deeper model. We therefore treat the anchor as a model-dependent early-layer setting.
Finally, thanks again for your time and helpful comments. We hope your concerns are well addressed.
Response to Reviewer UxEk (1/3)
We sincerely thank the reviewer for the constructive and positive feedback. Below are our point-by-point responses to your concerns.
Thank you for this constructive comment. We have clarified PeCo’s novelty and added direct comparisons with VAR+VCD and ASCD. PeCo introduces two complementary mechanisms:
The direct comparisons show substantial reductions in hallucination: PeCo lowers CHAIR_S by 12.2 and 8.9 points relative to VAR+VCD and ASCD, respectively, while also improving CHAIR_I and both POPE metrics (Table A).
Ablations further support these design choices. Within the perception-enhancement setting, removing spatial smoothing or temporal compensation raises CHAIR_I from 8.6 to 10.2 or 9.1, respectively (Table 7). Adding SDCD further lowers CHAIR_S/I from 35.1/8.6 to 33.3/8.0 (Table 6). Together, the direct comparisons establish the additional value of the complete framework, while the ablations support its individual design choices.
Table A. Performance comparison on LLaVA-1.5-7B.
Thank you for highlighting this missing ablation. We conducted additional experiments on LLaVA-1.5-7B to compare three settings: no cognition calibration (CC), CC with a fully blind negative stream, and CC with a sink-retaining negative stream.
Fully blind CC performs worse than omitting CC on all four metrics in this setting. Retaining sink tokens instead improves POPE accuracy by 1.2%. The key insight is that stronger visual deprivation does not necessarily produce a more useful contrastive reference. Contrastive decoding requires negative logits that meaningfully expose language-driven biases. Complete visual suppression may disrupt pretrained attention dynamics and introduce artifacts that distort the contrastive penalty. Retaining sink tokens is intended to preserve computational anchors while suppressing informative visual access.
Table B. Ablation of negative-stream construction.
Thank you for this constructive comment. Our evaluation already extends beyond object existence through the counting, spatial-position, and color subsets of MME (Table 1). As summarized in Table C, PeCo improves performance across all three subsets, providing quantitative evidence of its benefits for counting, spatial-position judgments, and color recognition. These results support PeCo’s broader applicability to multimodal grounding and are consistent with its design: it calibrates visual attention and output distributions at inference time without relying on decoding rules specific to object-existence judgments.
Table C. Results on the counting, spatial-position, and color subsets of MME.