- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies object hallucination by decomposing each generated token's final logit into layer-wise attention and FFN contributions. It reports that, in the last few decoder layers, attention relatively favors grounded objects while FFNs relatively favor hallucinated objects. Based on this observation, the paper proposes Component-wise Contrastive Decoding (CCD), a training-free method that subtracts only positive vocabulary-space projections of late FFN outputs while preserving attention and negative FFN contributions. CCD is evaluated on LLaVA-1.5, InstructBLIP, and two Qwen3-VL models using CHAIR, POPE, AMBER, and MME. The paper also provides component ablations, image-degradation analysis, statistical tests, decoding-rule robustness, and a latency measurement.
Critical strengths
-
The analysis leads to a precise and efficient intervention. Separating attention and FFN writes is more targeted than treating a complete layer as one unit. The component, sign, and layer ablations generally behave as predicted, providing a coherent connection between the diagnosis and CCD.
-
The evaluation addresses several common confounders in hallucination mitigation. The paper reports recall together with CHAIR, precision and recall together with POPE, and answer-balance analysis rather than relying only on reduced hallucination rates. CCD improves the reported hallucination metrics without the substantial recall loss seen in several baselines. Including Qwen3-VL-4B and 8B also makes the empirical scope more current than the teaser figure alone suggests.
Critical weaknesses
-
The main mechanism and intervention substantially overlap with prior work. FADE (arXiv:2606.29431) already argues that attention supplies visual evidence while critical-layer FFNs inject language priors that override correct intermediate predictions, and proposes training-free, single-pass FFN attenuation. CCD's positive-only vocabulary-level correction is more selective, but this appears to be a refinement of an already established mechanism and intervention family rather than an independent core discovery.
-
Direct logit attribution does not by itself identify the causal origin of hallucination. An FFN write depends on the attention-updated residual stream, while attention also processes generated text. Thus, locating the final additive contribution in the FFN does not establish that deficient visual processing was not an upstream cause or that the contribution represents a purely linguistic prior.
-
The strongest causal evidence remains concentrated on LLaVA-1.5. Qwen3-VL does reproduce the cohort-level attribution pattern, but its diagnostic cohorts contain only 81 and 54 hallucinated tokens per model. Image degradation, token-level trajectories, implicit selectivity, same-word comparisons, and the principal component ablation are primarily conducted on LLaVA. The broad claim is therefore stronger than the modern-model causal evidence.
-
The diagnostic quantity and the correction quantity are not identical. The central diagnosis uses contributions after the final RMSNorm gain and scaling, whereas CCD takes the positive part of unnormalized projections of FFN outputs. Table 7 shows that normalization can substantially change the values and even their signs. Hyperparameter scaling does not explain why positivity in the latter space is the appropriate target for a claim framed in terms of output-logit contributions.
-
The evaluation does not establish preservation of open-ended capability. MME provides a useful general check, but it consists largely of constrained yes/no tasks. AMBER remains hallucination-focused. There is no open-ended general evaluation such as LLaVA-Bench or MM-Vet, nor an open-ended hallucination evaluation such as HallusionBench.
-
CHAIR results are not accompanied by aggregate output-length statistics. Recall covers annotated COCO object categories and does not control caption length, attributes, relations, or overall information density. Average tokens, object mentions, and length-matched CHAIR results are needed.
-
The efficiency evidence is too narrow for the general deployment claim. Table 11 usefully reports per-token latency and peak memory, but only for LLaVA-1.5 on one RTX 3090. CCD performs several additional full-vocabulary projections per decoding step, with cost depending on vocabulary size, hidden dimension, and K. More results are needed to substantiate the claimed overhead more generally.
Questions for authors
-
How does CCD differ conceptually and empirically from FADE?
-
Can the authors distinguish “the final additive write is produced by the FFN” from “the original causal failure arises in the FFN”?
-
Why is the positive part selected using the unnormalized projection rather than the normalized output-logit contribution used for the diagnosis? How do the two correction rules compare?
-
Can the image-degradation and component-level causal analyses be reproduced on Qwen3-VL with larger cohorts, and can CCD be evaluated on open-ended general and hallucination benchmarks?
-
What are the average output length and number of object mentions for every CHAIR method?
9 comments
W2 / Q2. The final additive write versus the causal origin.
W2. Direct logit attribution does not by itself identify the causal origin of hallucination. An FFN write depends on the attention-updated residual stream, while attention also processes generated text. Thus, locating the final additive contribution in the FFN does not establish that deficient visual processing was not an upstream cause or that the contribution represents a purely linguistic prior.
Q2. Can the authors distinguish "the final additive write is produced by the FFN" from "the original causal failure arises in the FFN"?
A2. Thank you for this helpful distinction. Direct logit attribution identifies the sublayer that writes the final preference into the output, and this is the sense of "written by" in the title; the abstract, Sec. 1, Sec. 4.4 and App. B now state it explicitly. Two interventions go beyond attribution:
- On the input. With the caption fixed, occluding the image removes 72–79% of the attention advantage but changes the mean positive FFN write to hallucinated objects by at most 1.2% on three models (A3). Given the text, the push toward absent objects comes through the FFN and is largely insensitive to the image.
- On the write. Subtracting a fraction of its positive part changes 17.3% of hallucinated predictions against 2.9% of faithful ones on LLaVA-1.5 and lowers hallucination on every benchmark.
Together, they show that the late FFN write is where the absent object gains its lead in the output, and that removing this write alone reduces hallucination.
Changes in the revised PDF (blue):
- Abstract, Sec. 1 and Sec. 4.4: the occlusion result is stated with the text fixed, and Sec. 4.4 states the sense of "written by".
- App. B: the same statement.
W3 / Q4 (first part). Causal evidence concentrated on LLaVA, and small Qwen cohorts.
W3. The strongest causal evidence remains concentrated on LLaVA-1.5. Qwen3-VL does reproduce the cohort-level attribution pattern, but its diagnostic cohorts contain only 81 and 54 hallucinated tokens per model. Image degradation, token-level trajectories, implicit selectivity, same-word comparisons, and the principal component ablation are primarily conducted on LLaVA. The broad claim is therefore stronger than the modern-model causal evidence.
Q4 (first part). Can the image-degradation and component-level causal analyses be reproduced on Qwen3-VL with larger cohorts […]?
A3. Thank you for this suggestion. For both Qwen3-VL models we captioned 3,000 further MSCOCO val2014 images, disjoint from the CHAIR set, which enlarges the cohorts from 81 and 54 to 487 and 329 words per group, and repeated the image degradation. The component ablation, which runs on the CHAIR images, is repeated on Qwen3-VL-8B (App. W.2, Tables 18–20).
Table R2. Enlarged cohorts, output logit, Δ = H − F with 95% CI (revised PDF: Table 18).
| Model | n per group | Attention Δ | FFN Δ | Sum |
|---|---|---|---|---|
| Qwen3-VL-4B | 487 | −3.43 [−4.79, −2.13] | +6.04 [4.88, 7.25] | +2.61 [0.89, 4.40] |
| Qwen3-VL-8B | 329 | −6.86 [−9.58, −4.27] | +8.93 [6.21, 11.70] | +2.07 [−1.58, 5.91] |
- Both differentials keep their signs on both models, with intervals that exclude zero, reproducing the opposition of Table 8.
Table R3. Original image to full occlusion, captions fixed (revised PDF: Table 19).
| Model | Attention advantage of faithful | FFN⁺ write to hallucinated | Faithful log-probability |
|---|---|---|---|
| LLaVA-1.5-7B | −79% | +0.7% | −0.33 → −1.16 |
| Qwen3-VL-4B | −72% | +1.2% | −0.35 → −1.12 |
| Qwen3-VL-8B | −74% | −0.9% | −0.31 → −1.14 |
- Component ablation on Qwen3-VL-8B (CHAIR_S / CHAIR_I / Recall; Table 20): Greedy 24.4 / 8.2 / 47.4; Attn-only 25.4 / 8.6 / 46.5; Attn+FFN 23.4 / 7.9 / 47.0; Neg-only 25.6 / 8.7 / 46.8; Both-sign 22.2 / 7.4 / 48.5; All-layers 23.2 / 7.8 / 47.8; CCD 21.8 / 7.3 / 49.9. As on LLaVA-1.5, correcting attention or removing only the negative FFN writes is worse than greedy decoding, and CCD is best on all three metrics.
- The test on candidates the model did not generate (App. W.1) covers both Qwen models, and App. Y.5 (Table 31) extends the prediction-change analysis of App. J to all four models: CCD changes hallucinated words 5.2 to 6.0 times as often as faithful ones.
Changes in the revised PDF (blue):
- Sec. 4.4: one sentence on the Qwen3-VL occlusion.
- App. W.2: Tables 18–20.
W4 / Q3. The diagnostic quantity and the correction quantity differ.
W4. The diagnostic quantity and the correction quantity are not identical. The central diagnosis uses contributions after the final RMSNorm gain and scaling, whereas CCD takes the positive part of unnormalized projections of FFN outputs. Table 7 shows that normalization can substantially change the values and even their signs. Hyperparameter scaling does not explain why positivity in the latter space is the appropriate target for a claim framed in terms of output-logit contributions.
Q3. Why is the positive part selected using the unnormalized projection rather than the normalized output-logit contribution used for the diagnosis? How do the two correction rules compare?
A4. Thank you for this careful question. The two quantities differ by two factors, and we now separate them. On LLaVA-1.5, applying only the gain γ, with one common scale s̄ for all steps, moves the attention differential from +0.14 to −8.93 (−9.24 on the output logit), while the FFN differential is +4.19, +13.24 and +13.57 in the three readings (App. W.4, Table 23). The sign change of attention therefore comes from the fixed gain, and the FFN favors hallucinated objects in every reading. Because s > 0 is shared by all words at a step, the two positive parts differ only through γ. To compare the two rules directly, CCD-norm subtracts , with α̃ set so that it changes as many top-1 predictions as CCD on greedy captions of unlabeled images.
Table R4. CHAIR_S / CHAIR_I / Recall / POPE-Adv F1 (revised PDF: Table 24).
| Model | CCD | CCD-norm |
|---|---|---|
| LLaVA-1.5-7B | 43.8 / 13.3 / 76.1 / 82.46 | 44.2 / 13.4 / 75.8 / 82.31 |
| InstructBLIP-7B | 43.6 / 12.8 / 71.7 / 82.89 | 43.4 / 12.7 / 71.5 / 82.74 |
| Qwen3-VL-4B | 23.0 / 8.1 / 50.8 / 88.27 | 23.2 / 8.2 / 50.6 / 88.19 |
| Qwen3-VL-8B | 21.8 / 7.3 / 49.9 / 87.32 | 21.6 / 7.2 / 49.7 / 87.28 |
- The two rules perform within 0.4 points of each other on every metric, so the results do not depend on which positive part is removed; we keep the raw form because it needs only W_U.
Changes in the revised PDF (blue):
- Sec. 3.3: one sentence.
- App. W.4: Tables 23–24.
W5 / Q4 (second part). Open-ended evaluation.
W5. The evaluation does not establish preservation of open-ended capability. MME provides a useful general check, but it consists largely of constrained yes/no tasks. AMBER remains hallucination-focused. There is no open-ended general evaluation such as LLaVA-Bench or MM-Vet, nor an open-ended hallucination evaluation such as HallusionBench.
Q4 (second part). […] and can CCD be evaluated on open-ended general and hallucination benchmarks?
A5. Thank you for this suggestion. We added three benchmarks, each run with the official scripts and the same judge for both methods: MMHal-Bench (open-ended answers scored for hallucination by a GPT-4 judge), HallusionBench (diagnostic yes-or-no questions) and MM-Vet (free-form answers on recognition, OCR, knowledge, math and spatial reasoning). The captioning tasks of CHAIR and AMBER are also open-ended; there CCD lowers CHAIR_S by 2.6 to 10.4 points and AMBER Hal from 52.0 to 28.8 on LLaVA-1.5 (Tables 2–4).
Table R5. CCD − greedy (revised PDF: Table 28).
| Model | MMHal score ↑ | MMHal hallucination rate ↓ | HallusionBench aAcc ↑ | HallusionBench qAcc ↑ | MM-Vet Total |
|---|---|---|---|---|---|
| LLaVA-1.5-7B | +0.31 | −7.3 | +2.4 | +2.7 | +0.8 |
| InstructBLIP-7B | +0.23 | −5.2 | +2.1 | +2.0 | +1.1 |
| Qwen3-VL-4B | +0.14 | −3.1 | +1.0 | +0.9 | −0.2 |
| Qwen3-VL-8B | +0.11 | −2.1 | +0.8 | +0.6 | +0.3 |
- CCD lowers the MMHal-Bench hallucination rate and raises both HallusionBench accuracies on all four models.
- MM-Vet mainly tests general ability, where object hallucination is a small part of the errors; it stays within −0.2 to +1.1 points of greedy decoding, so general ability is preserved.
- As on CHAIR, the gains are largest where the base model hallucinates most.
Changes in the revised PDF (blue):
- Sec. 4.2: new paragraph "Further evaluation".
- App. Y.3: Table 28.
Thank you for your valuable time and constructive comments. We address each point below. In the revised PDF, all changes made in response to the reviews are in blue. Tables 1–16 keep their original numbers (Table 6 is now Table 6a, and Table 6b is new), new tables start at Table 17, and each table below (R1–R8) gives its location in the revised PDF.
W1 / Q1. How CCD differs from FADE, conceptually and empirically.
A1. Thank you for pointing us to FADE. It is a concurrent arXiv preprint (28 June 2026), and Sec. 2 now discusses it together with ReDeEP. We share its view that FFNs inject language priors. CCD differs in the question it answers, the evidence and the correction.
Results. Table 6b compares the two on all four models under one protocol (one setting per model, selected on the CHAIR images; App. Y.1). Both-sign is CCD without the sign selection.
Table R1. CHAIR_S / Recall / POPE F1 averaged over the three splits (revised PDF: Table 6b).
Changes in the revised PDF (blue):