acceptodds
Under review as a conference paper at ICLR 2027

Object Hallucinations Are Written by Late Feed-Forward Layers

Abstract

Multimodal large language models often describe objects that are not in the image. Training-free decoding methods can reduce these hallucinations, but the most effective ones also make the model mention fewer of the objects that are present, and layer-level methods such as DoLa and DeCo help some models while hurting others. To understand why, we decompose the output logit of each generated object into the direct contributions of the attention and feed-forward network (FFN) sublayers at every layer. In four MLLMs, the two sublayers of the last few layers work against each other: attention favors objects present in the image, while the FFN favors hallucinated objects by a larger margin. We call this pattern FFN overcommitment. When the caption is held fixed and the image is progressively occluded, the attention advantage nearly vanishes but the FFN contribution to hallucinated objects stays the same, so the pull toward absent objects does not come from the visual input. Layer-level methods rescale both contributions together, so they weaken the support for present objects whenever they weaken the push toward absent ones. We therefore propose Component-wise Contrastive Decoding (CCD), which subtracts only the positive FFN contributions of the last few layers from the output logits and requires neither training nor an extra forward pass. Across four models, CCD reduces hallucination on CHAIR and AMBER and raises POPE accuracy on every split while preserving recall on CHAIR and POPE, whereas every compared method that reaches a lower caption hallucination rate does so by mentioning fewer of the objects that are present.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
hallucinations;Multimodal large language model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PjoaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies object hallucination by decomposing each generated token's final logit into layer-wise attention and FFN contributions. It reports that, in the last few decoder layers, attention relatively favors grounded objects while FFNs relatively favor hallucinated objects. Based on this observation, the paper proposes Component-wise Contrastive Decoding (CCD), a training-free method that subtracts only positive vocabulary-space projections of late FFN outputs while preserving attention and negative FFN contributions. CCD is evaluated on LLaVA-1.5, InstructBLIP, and two Qwen3-VL models using CHAIR, POPE, AMBER, and MME. The paper also provides component ablations, image-degradation analysis, statistical tests, decoding-rule robustness, and a latency measurement.

Critical strengths
  1. The analysis leads to a precise and efficient intervention. Separating attention and FFN writes is more targeted than treating a complete layer as one unit. The component, sign, and layer ablations generally behave as predicted, providing a coherent connection between the diagnosis and CCD.

  2. The evaluation addresses several common confounders in hallucination mitigation. The paper reports recall together with CHAIR, precision and recall together with POPE, and answer-balance analysis rather than relying only on reduced hallucination rates. CCD improves the reported hallucination metrics without the substantial recall loss seen in several baselines. Including Qwen3-VL-4B and 8B also makes the empirical scope more current than the teaser figure alone suggests.

Critical weaknesses
  1. The main mechanism and intervention substantially overlap with prior work. FADE (arXiv:2606.29431) already argues that attention supplies visual evidence while critical-layer FFNs inject language priors that override correct intermediate predictions, and proposes training-free, single-pass FFN attenuation. CCD's positive-only vocabulary-level correction is more selective, but this appears to be a refinement of an already established mechanism and intervention family rather than an independent core discovery.

  2. Direct logit attribution does not by itself identify the causal origin of hallucination. An FFN write depends on the attention-updated residual stream, while attention also processes generated text. Thus, locating the final additive contribution in the FFN does not establish that deficient visual processing was not an upstream cause or that the contribution represents a purely linguistic prior.

  3. The strongest causal evidence remains concentrated on LLaVA-1.5. Qwen3-VL does reproduce the cohort-level attribution pattern, but its diagnostic cohorts contain only 81 and 54 hallucinated tokens per model. Image degradation, token-level trajectories, implicit selectivity, same-word comparisons, and the principal component ablation are primarily conducted on LLaVA. The broad claim is therefore stronger than the modern-model causal evidence.

  4. The diagnostic quantity and the correction quantity are not identical. The central diagnosis uses contributions after the final RMSNorm gain and scaling, whereas CCD takes the positive part of unnormalized projections of FFN outputs. Table 7 shows that normalization can substantially change the values and even their signs. Hyperparameter scaling does not explain why positivity in the latter space is the appropriate target for a claim framed in terms of output-logit contributions.

  5. The evaluation does not establish preservation of open-ended capability. MME provides a useful general check, but it consists largely of constrained yes/no tasks. AMBER remains hallucination-focused. There is no open-ended general evaluation such as LLaVA-Bench or MM-Vet, nor an open-ended hallucination evaluation such as HallusionBench.

  6. CHAIR results are not accompanied by aggregate output-length statistics. Recall covers annotated COCO object categories and does not control caption length, attributes, relations, or overall information density. Average tokens, object mentions, and length-matched CHAIR results are needed.

  7. The efficiency evidence is too narrow for the general deployment claim. Table 11 usefully reports per-token latency and peak memory, but only for LLaVA-1.5 on one RTX 3090. CCD performs several additional full-vocabulary projections per decoding step, with cost depending on vocabulary size, hidden dimension, and K. More results are needed to substantiate the claimed overhead more generally.

Questions for authors
  1. How does CCD differ conceptually and empirically from FADE?

  2. Can the authors distinguish “the final additive write is produced by the FFN” from “the original causal failure arises in the FFN”?

  3. Why is the positive part selected using the unnormalized projection rather than the normalized output-logit contribution used for the diagnosis? How do the two correction rules compare?

  4. Can the image-degradation and component-level causal analyses be reproduced on Qwen3-VL with larger cohorts, and can CCD be evaluated on open-ended general and hallucination benchmarks?

  5. What are the average output length and number of object mentions for every CHAIR method?

9 comments

Thank you for your valuable time and constructive comments. We address each point below. In the revised PDF, all changes made in response to the reviews are in blue. Tables 1–16 keep their original numbers (Table 6 is now Table 6a, and Table 6b is new), new tables start at Table 17, and each table below (R1–R8) gives its location in the revised PDF.

W1 / Q1. How CCD differs from FADE, conceptually and empirically.

W1. The main mechanism and intervention substantially overlap with prior work. FADE (arXiv:2606.29431) already argues that attention supplies visual evidence while critical-layer FFNs inject language priors that override correct intermediate predictions, and proposes training-free, single-pass FFN attenuation. CCD's positive-only vocabulary-level correction is more selective, but this appears to be a refinement of an already established mechanism and intervention family rather than an independent core discovery.

Q1. How does CCD differ conceptually and empirically from FADE?

A1. Thank you for pointing us to FADE. It is a concurrent arXiv preprint (28 June 2026), and Sec. 2 now discusses it together with ReDeEP. We share its view that FFNs inject language priors. CCD differs in the question it answers, the evidence and the correction.

  • Question. We explain why decoders that remove the most hallucinated objects also remove present ones, and why layer-level methods help some models and hurt others: a layer-level contrast rescales the attention and FFN writes together (Sec. 3.4).
  • Evidence. We measure signed attention and FFN contributions on the output logit, with intervals on four models, and test them by intervention: the FFN write to absent objects changes by at most 1.2% when the image is occluded with the caption fixed, the same word gets a larger late FFN write when it is hallucinated, and the pattern holds for candidates the model did not generate.
  • Correction. FADE scales the whole FFN output of one layer inside the forward pass, with the layer and factor chosen per model and task. CCD changes no hidden state: it subtracts only the positive part of the late FFN projections from the logits, keeps the negative part, and uses one setting per model for every benchmark, which can also be set from unlabeled images (Sec. 3.3).

Results. Table 6b compares the two on all four models under one protocol (one setting per model, selected on the CHAIR images; App. Y.1). Both-sign is CCD without the sign selection.

Table R1. CHAIR_S / Recall / POPE F1 averaged over the three splits (revised PDF: Table 6b).

ModelGreedyFADEBoth-signCCD
LLaVA-1.5-7B54.2 / 72.5 / 79.3050.8 / 70.8 / 79.8444.6 / 72.8 / 84.8743.8 / 76.1 / 86.16
InstructBLIP-7B54.0 / 71.1 / 77.9545.2 / 70.4 / 77.7344.4 / 71.2 / 85.1243.6 / 71.7 / 86.38
Qwen3-VL-4B26.0 / 49.2 / 87.1024.6 / 49.0 / 87.5823.2 / 50.0 / 89.5823.0 / 50.8 / 90.60
Qwen3-VL-8B24.4 / 47.4 / 85.5123.0 / 47.2 / 85.9122.2 / 48.5 / 88.8321.8 / 49.9 / 90.14
  • CCD has lower CHAIR_S and higher recall and POPE F1 than FADE on every model.
  • Both-sign reaches nearly the same CHAIR as CCD, but the sign selection adds 0.5 to 3.3 recall points and 1.0 to 1.3 POPE F1 points.

Changes in the revised PDF (blue):

  • Sec. 2: new paragraph on FADE and ReDeEP.
  • Table 1: a row for FADE.
  • Table 6b and Sec. 4.3: FADE and Both-sign on four models.
  • App. Y.1: FADE configuration.

W2 / Q2. The final additive write versus the causal origin.

W2. Direct logit attribution does not by itself identify the causal origin of hallucination. An FFN write depends on the attention-updated residual stream, while attention also processes generated text. Thus, locating the final additive contribution in the FFN does not establish that deficient visual processing was not an upstream cause or that the contribution represents a purely linguistic prior.

Q2. Can the authors distinguish "the final additive write is produced by the FFN" from "the original causal failure arises in the FFN"?

A2. Thank you for this helpful distinction. Direct logit attribution identifies the sublayer that writes the final preference into the output, and this is the sense of "written by" in the title; the abstract, Sec. 1, Sec. 4.4 and App. B now state it explicitly. Two interventions go beyond attribution:

  • On the input. With the caption fixed, occluding the image removes 72–79% of the attention advantage but changes the mean positive FFN write to hallucinated objects by at most 1.2% on three models (A3). Given the text, the push toward absent objects comes through the FFN and is largely insensitive to the image.
  • On the write. Subtracting a fraction of its positive part changes 17.3% of hallucinated predictions against 2.9% of faithful ones on LLaVA-1.5 and lowers hallucination on every benchmark.

Together, they show that the late FFN write is where the absent object gains its lead in the output, and that removing this write alone reduces hallucination.

Changes in the revised PDF (blue):

  • Abstract, Sec. 1 and Sec. 4.4: the occlusion result is stated with the text fixed, and Sec. 4.4 states the sense of "written by".
  • App. B: the same statement.

W3 / Q4 (first part). Causal evidence concentrated on LLaVA, and small Qwen cohorts.

W3. The strongest causal evidence remains concentrated on LLaVA-1.5. Qwen3-VL does reproduce the cohort-level attribution pattern, but its diagnostic cohorts contain only 81 and 54 hallucinated tokens per model. Image degradation, token-level trajectories, implicit selectivity, same-word comparisons, and the principal component ablation are primarily conducted on LLaVA. The broad claim is therefore stronger than the modern-model causal evidence.

Q4 (first part). Can the image-degradation and component-level causal analyses be reproduced on Qwen3-VL with larger cohorts […]?

A3. Thank you for this suggestion. For both Qwen3-VL models we captioned 3,000 further MSCOCO val2014 images, disjoint from the CHAIR set, which enlarges the cohorts from 81 and 54 to 487 and 329 words per group, and repeated the image degradation. The component ablation, which runs on the CHAIR images, is repeated on Qwen3-VL-8B (App. W.2, Tables 18–20).

Table R2. Enlarged cohorts, output logit, Δ = H − F with 95% CI (revised PDF: Table 18).

Modeln per groupAttention ΔFFN ΔSum
Qwen3-VL-4B487−3.43 [−4.79, −2.13]+6.04 [4.88, 7.25]+2.61 [0.89, 4.40]
Qwen3-VL-8B329−6.86 [−9.58, −4.27]+8.93 [6.21, 11.70]+2.07 [−1.58, 5.91]
  • Both differentials keep their signs on both models, with intervals that exclude zero, reproducing the opposition of Table 8.

Table R3. Original image to full occlusion, captions fixed (revised PDF: Table 19).

ModelAttention advantage of faithfulFFN⁺ write to hallucinatedFaithful log-probability
LLaVA-1.5-7B−79%+0.7%−0.33 → −1.16
Qwen3-VL-4B−72%+1.2%−0.35 → −1.12
Qwen3-VL-8B−74%−0.9%−0.31 → −1.14
  • Component ablation on Qwen3-VL-8B (CHAIR_S / CHAIR_I / Recall; Table 20): Greedy 24.4 / 8.2 / 47.4; Attn-only 25.4 / 8.6 / 46.5; Attn+FFN 23.4 / 7.9 / 47.0; Neg-only 25.6 / 8.7 / 46.8; Both-sign 22.2 / 7.4 / 48.5; All-layers 23.2 / 7.8 / 47.8; CCD 21.8 / 7.3 / 49.9. As on LLaVA-1.5, correcting attention or removing only the negative FFN writes is worse than greedy decoding, and CCD is best on all three metrics.
  • The test on candidates the model did not generate (App. W.1) covers both Qwen models, and App. Y.5 (Table 31) extends the prediction-change analysis of App. J to all four models: CCD changes hallucinated words 5.2 to 6.0 times as often as faithful ones.

Changes in the revised PDF (blue):

  • Sec. 4.4: one sentence on the Qwen3-VL occlusion.
  • App. W.2: Tables 18–20.

W4 / Q3. The diagnostic quantity and the correction quantity differ.

W4. The diagnostic quantity and the correction quantity are not identical. The central diagnosis uses contributions after the final RMSNorm gain and scaling, whereas CCD takes the positive part of unnormalized projections of FFN outputs. Table 7 shows that normalization can substantially change the values and even their signs. Hyperparameter scaling does not explain why positivity in the latter space is the appropriate target for a claim framed in terms of output-logit contributions.

Q3. Why is the positive part selected using the unnormalized projection rather than the normalized output-logit contribution used for the diagnosis? How do the two correction rules compare?

A4. Thank you for this careful question. The two quantities differ by two factors, and we now separate them. On LLaVA-1.5, applying only the gain γ, with one common scale s̄ for all steps, moves the attention differential from +0.14 to −8.93 (−9.24 on the output logit), while the FFN differential is +4.19, +13.24 and +13.57 in the three readings (App. W.4, Table 23). The sign change of attention therefore comes from the fixed gain, and the FFN favors hallucinated objects in every reading. Because s > 0 is shared by all words at a step, the two positive parts differ only through γ. To compare the two rules directly, CCD-norm subtracts α~∑k[δ~kffn(y)]+\tilde\alpha\sum_k[\tilde\delta^{\mathrm{ffn}}_k(y)]_+, with α̃ set so that it changes as many top-1 predictions as CCD on greedy captions of unlabeled images.

Table R4. CHAIR_S / CHAIR_I / Recall / POPE-Adv F1 (revised PDF: Table 24).

ModelCCDCCD-norm
LLaVA-1.5-7B43.8 / 13.3 / 76.1 / 82.4644.2 / 13.4 / 75.8 / 82.31
InstructBLIP-7B43.6 / 12.8 / 71.7 / 82.8943.4 / 12.7 / 71.5 / 82.74
Qwen3-VL-4B23.0 / 8.1 / 50.8 / 88.2723.2 / 8.2 / 50.6 / 88.19
Qwen3-VL-8B21.8 / 7.3 / 49.9 / 87.3221.6 / 7.2 / 49.7 / 87.28
  • The two rules perform within 0.4 points of each other on every metric, so the results do not depend on which positive part is removed; we keep the raw form because it needs only W_U.

Changes in the revised PDF (blue):

  • Sec. 3.3: one sentence.
  • App. W.4: Tables 23–24.

W5 / Q4 (second part). Open-ended evaluation.

W5. The evaluation does not establish preservation of open-ended capability. MME provides a useful general check, but it consists largely of constrained yes/no tasks. AMBER remains hallucination-focused. There is no open-ended general evaluation such as LLaVA-Bench or MM-Vet, nor an open-ended hallucination evaluation such as HallusionBench.

Q4 (second part). […] and can CCD be evaluated on open-ended general and hallucination benchmarks?

A5. Thank you for this suggestion. We added three benchmarks, each run with the official scripts and the same judge for both methods: MMHal-Bench (open-ended answers scored for hallucination by a GPT-4 judge), HallusionBench (diagnostic yes-or-no questions) and MM-Vet (free-form answers on recognition, OCR, knowledge, math and spatial reasoning). The captioning tasks of CHAIR and AMBER are also open-ended; there CCD lowers CHAIR_S by 2.6 to 10.4 points and AMBER Hal from 52.0 to 28.8 on LLaVA-1.5 (Tables 2–4).

Table R5. CCD − greedy (revised PDF: Table 28).

ModelMMHal score ↑MMHal hallucination rate ↓HallusionBench aAcc ↑HallusionBench qAcc ↑MM-Vet Total
LLaVA-1.5-7B+0.31−7.3+2.4+2.7+0.8
InstructBLIP-7B+0.23−5.2+2.1+2.0+1.1
Qwen3-VL-4B+0.14−3.1+1.0+0.9−0.2
Qwen3-VL-8B+0.11−2.1+0.8+0.6+0.3
  • CCD lowers the MMHal-Bench hallucination rate and raises both HallusionBench accuracies on all four models.
  • MM-Vet mainly tests general ability, where object hallucination is a small part of the errors; it stays within −0.2 to +1.1 points of greedy decoding, so general ability is preserved.
  • As on CHAIR, the gains are largest where the base model hallucinates most.

Changes in the revised PDF (blue):

  • Sec. 4.2: new paragraph "Further evaluation".
  • App. Y.3: Table 28.
Reviewer_AJyaOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies where object hallucinations enter the output of MLLMs. Using direct logit attribution, it splits the logit of each generated object word into contributions from the attention and FFN sublayers of every layer, and compares balanced sets of faithful and hallucinated object words from captions of 500 MSCOCO images. In the last few layers of LLaVA-1.5-7B, InstructBLIP-7B, Qwen3-VL-4B, and Qwen3-VL-8B, attention favors faithful objects on the output logit, while the FFN favors hallucinated objects by a larger margin ("FFN overcommitment"). When the image is progressively occluded with the caption held fixed, the attention advantage nearly disappears but the positive FFN write to hallucinated objects stays flat. The paper then shows that layer-level methods such as DoLa and DeCo can only rescale the attention and FFN terms of the late update together, which explains why they lose recall or behave inconsistently across models. Based on this, it proposes Component-wise Contrastive Decoding (CCD), which subtracts a scaled sum of the positive FFN contributions of the last K layers from the logits at every step. CCD needs no training and no extra forward pass. Across four models it lowers CHAIR and AMBER hallucination, raises POPE F1 on every split, keeps or raises recall, and gets the best AMBER and MME totals, at about 1% latency overhead.

Critical strengths
  1. The analysis is careful and goes beyond a single correlation. The attention/FFN split holds on four models with confidence intervals (Table 8). The occlusion experiment with fixed captions (Fig. 3a) shows that only the attention side depends on the image, and Appendix M shows the late FFN surge depends on the context rather than on the word itself. Section 3.4 also gives an exact argument for why DoLa and DeCo cannot separate the two terms, which explains their inconsistent behavior across models in a way I found useful.
  2. CCD is simple and cheap, and it reduces hallucination without losing recall on all four models, under greedy, beam, and nucleus decoding (Tables 2-5, 15). This is the main practical difference from the other methods. Table 13 shows that every recent method that reaches a lower CHAIR_S than CCD gives up 5-8 recall points, while CCD gains 3.6.
Critical weaknesses
  1. The novelty is overstated, and closely related work is missing. ReDeEP (Sun et al., ICLR 2025) already argued that hallucination happens when FFNs push parametric knowledge into the residual stream while attention fails to carry the external evidence. Its mitigation, AARF ("Add Attention, Reduce FFN"), down-weights the FFN contributions. That work is on RAG in LLMs, but the mechanism and the fix are very similar, and it is not cited. FADE (arXiv, June 2026) makes almost the same claim for LVLMs: attention aggregates visual evidence, FFNs at critical layers inject language priors, and attenuating FFN outputs during decoding reduces hallucination. FADE is probably concurrent, so I don't hold it against the paper, but it should be discussed. Given these, I think the new parts are mainly the sign selection (subtracting only the positive FFN part) and doing the correction on the logits. The paper should present them that way and compare against plain FFN down-weighting on all models.
  2. Part of the diagnosis may follow from how the cohorts are built, and the hyperparameters seem to be tuned on the test benchmarks. Both cohorts are tokens the model actually generated, so both must have a high final logit. If attention gives hallucinated words less, the other terms have to make up the gap, so some FFN advantage is expected just from conditioning on the model's choice. Table 7 also shows that before the final normalization attention contributes almost equally to the two cohorts (-0.73 vs. -0.59). The attention advantage appears only after the gain and the 1/s scaling, while CCD itself uses the pre-normalization projections. On the method side, (alpha, K) differ per model (alpha ranges from 0.01 to 0.10), the sensitivity study uses the CHAIR test images, and I could not find a held-out set for choosing them.
Questions for authors

Questions that could change my assessment:

  1. How were (alpha, K) chosen for each model? Was any held-out data used? How does CCD do with a single shared setting across all four models?
  2. Can you test the diagnosis on candidates the model did not generate? For example, compare the late FFN writes to absent objects that rank high but are not chosen, or match the cohorts by position in the caption. This would separate FFN overcommitment from the selection effect.
  3. Table 6 has a "both-sign" FFN variant on LLaVA-1.5 that is close to plain FFN down-weighting (as in AARF or FADE). Can you report it on all four models and on POPE, so the value of sign selection is clear? Clarifications:
  4. On POPE the decision is made on the Yes/No token, not on an object word. Which of the two tokens gets the larger positive late FFN write, and how large is the correction relative to the Yes/No margin? With alpha = 0.01, InstructBLIP still gains about 10 accuracy points, so I would like to understand where this comes from.
  5. In Table 4, DoLa and DeCo have identical generative numbers on Qwen3-VL-8B (6.7 / 49.0 / 33.1 / 2.1). Is this a copy error? Suggestion (doesn't affect my score): Please cite and discuss ReDeEP and FADE.
9 comments

Thank you for your valuable time and constructive comments, including the careful checks of Table 7 and Table 4. We address each point below, starting with the three questions that you marked as able to change your assessment. In the revised PDF, all changes made in response to the reviews are in blue. Tables 1–16 keep their original numbers (Table 6 is now Table 6a, and Table 6b is new), new tables start at Table 17, and each table below (R1–R6) gives its location in the revised PDF.

Q1 / W2 (tuning). How were (α, K) chosen, and how does CCD do with one shared setting?

Q1. How were (alpha, K) chosen for each model? Was any held-out data used? How does CCD do with a single shared setting across all four models?

W2 (tuning). On the method side, (alpha, K) differ per model (alpha ranges from 0.01 to 0.10), the sensitivity study uses the CHAIR test images, and I could not find a held-out set for choosing them.

A1. Thank you for this question. Selecting (α, K) on 500 MSCOCO val2014 images that share no image with any evaluation set returns exactly the setting used in the paper for all four models, so the reported numbers are unchanged (App. X.1, Table 25). Each model uses this one setting for every benchmark; in the submission, the same selection (same candidates; lowest CHAIR_S among the settings whose recall is not below greedy) was run on the CHAIR images.

One shared setting. We apply (0.05, 3), the setting of both Qwen3-VL models, unchanged to LLaVA-1.5 and InstructBLIP; the last column is the label-free rule below.

Table R1. CHAIR_S / Recall / POPE-Adv F1 (revised PDF: Table 26).

ModelGreedyPer-model settingShared (0.05, 3)Shared rule
LLaVA-1.5-7B54.2 / 72.5 / 76.2843.8 / 76.1 / 82.4644.4 / 75.4 / 80.9344.4 / 75.7 / 82.18
InstructBLIP-7B54.0 / 71.1 / 75.2443.6 / 71.7 / 82.8945.6 / 70.6 / 81.1744.8 / 71.0 / 82.42
Qwen3-VL-4B26.0 / 49.2 / 85.2323.0 / 50.8 / 88.2723.0 / 50.8 / 88.2723.4 / 50.5 / 88.06
Qwen3-VL-8B24.4 / 47.4 / 83.6821.8 / 49.9 / 87.3221.8 / 49.9 / 87.3222.2 / 49.6 / 87.13
  • With one shared setting, CCD still lowers CHAIR_S by 2.6 to 9.8 points and raises POPE-Adv F1 by 3.04 to 5.93 points on every model (LLaVA-1.5 CHAIR entries at (0.05, 3) are from Table 10).

A shared rule. The same α has a different strength on different models: on greedy captions of 100 unlabeled images, α = 0.05 changes 1.1% of the predictions of LLaVA-1.5 but 4.4% of those of InstructBLIP, because the scale of Φ⁺ relative to the logit gaps differs. Since CCD changes a prediction exactly when α exceeds a threshold α* computed at that step (Eq. 8; formula in Sec. 3.3), we share this rate instead: α is the ρ-quantile of α*, with K = 3 and ρ = 2% fixed once from LLaVA-1.5. The rule stays within 1.2 CHAIR_S and 0.5 POPE-Adv F1 points of the per-model results, and it is what we recommend for a new model.

Changes in the revised PDF (blue):

  • App. O.2 and App. X.1 (Table 25): held-out selection.
  • Sec. 3.3: the label-free rule; App. X.2 (Table 26): the shared setting and the rule.

Q2 / W2 (selection). Does the diagnosis hold for candidates the model did not generate?

Q2. Can you test the diagnosis on candidates the model did not generate? For example, compare the late FFN writes to absent objects that rank high but are not chosen, or match the cohorts by position in the caption. This would separate FFN overcommitment from the selection effect.

W2 (selection). Part of the diagnosis may follow from how the cohorts are built, and the hyperparameters seem to be tuned on the test benchmarks. Both cohorts are tokens the model actually generated, so both must have a high final logit. If attention gives hallucinated words less, the other terms have to make up the gap, so some FFN advantage is expected just from conditioning on the model's choice.

A2. Thank you for this suggestion. Yes, the pattern holds. At every step at which the caption names an object, we take the candidates ranked 2 to 10 (rank 1 is the generated word under greedy decoding), keep those that map to MSCOCO categories, and pair each absent candidate with a present one from the same step, so the two groups share positions and prefixes and neither was generated.

Table R2. Absent minus present, on the output logit over the window W of Table 8, with 95% image-level bootstrap intervals (revised PDF: Table 17).

Modeln per groupAttention ΔFFN Δ
LLaVA-1.5-7B1968−4.13 [−4.97, −3.31]+6.37 [5.30, 7.46]
InstructBLIP-7B1513−4.48 [−5.08, −3.90]+7.34 [6.60, 8.07]
Qwen3-VL-4B887−1.54 [−2.47, −0.60]+3.41 [2.56, 4.27]
Qwen3-VL-8B694−2.47 [−4.33, −0.64]+4.83 [2.95, 6.71]
  • Absent candidates receive the larger FFN contribution and the smaller attention contribution on all four models.
  • The test you suggested shows that the pattern holds beyond the generated words, with position and prefix matched.

Changes in the revised PDF (blue):

  • Sec. 4.4: new paragraph.
  • App. W.1: Table 17.

W2 (normalization). The attention advantage appears only after normalization, while CCD uses the raw projections.

W2 (normalization). Table 7 also shows that before the final normalization attention contributes almost equally to the two cohorts (-0.73 vs. -0.59). The attention advantage appears only after the gain and the 1/s scaling, while CCD itself uses the pre-normalization projections.

A3. Thank you for this careful observation. The attention advantage comes from the gain γ, a fixed parameter, and not from the per-step scale 1/s. Applying γ with one common scale s̄ for all steps already moves the attention difference from +0.14 to −8.93, close to its output-logit value of −9.24.

Table R3. Three readings on LLaVA-1.5, last five layers, H − F (revised PDF: Table 23).

ReadingAttention ΔFFN Δ
Raw projection δ (Table 7)+0.14+4.19
Gain γ with a common scale s̄−8.93+13.24
Output logit δ̃ (Table 7)−9.24+13.57

CCD acts only on the FFN term, which favors hallucinated words in all three readings. To compare the two spaces directly, CCD-norm takes the positive part on the output logit,z^y=zy−α~∑k[δ~kffn(y)]+\hat z_y=z_y-\tilde\alpha\sum_k[\tilde\delta^{\mathrm{ffn}}_k(y)]_+. Since s is shared by all words at a step, it differs from CCD only through γ and a per-step factor 1/s on the strength. We set α̃ so that CCD-norm changes as many top-1 predictions as CCD on greedy captions of unlabeled images, so it is calibrated without labels.

Table R4. CHAIR_S / CHAIR_I / Recall / POPE-Adv F1 (revised PDF: Table 24).

ModelCCDCCD-norm
LLaVA-1.5-7B43.8 / 13.3 / 76.1 / 82.4644.2 / 13.4 / 75.8 / 82.31
InstructBLIP-7B43.6 / 12.8 / 71.7 / 82.8943.4 / 12.7 / 71.5 / 82.74
Qwen3-VL-4B23.0 / 8.1 / 50.8 / 88.2723.2 / 8.2 / 50.6 / 88.19
Qwen3-VL-8B21.8 / 7.3 / 49.9 / 87.3221.6 / 7.2 / 49.7 / 87.28
  • The two rules perform within 0.4 points of each other on every metric, so the results do not depend on which positive part is removed. We keep the raw form because it needs only W_U.

Changes in the revised PDF (blue):

  • Sec. 3.3: one sentence.
  • App. W.4: Tables 23–24.

W1 / Q3. ReDeEP and FADE, Both-sign on all models, and the contribution.

W1. The novelty is overstated, and closely related work is missing. ReDeEP (Sun et al., ICLR 2025) already argued that hallucination happens when FFNs push parametric knowledge into the residual stream while attention fails to carry the external evidence. Its mitigation, AARF ("Add Attention, Reduce FFN"), down-weights the FFN contributions. That work is on RAG in LLMs, but the mechanism and the fix are very similar, and it is not cited. FADE (arXiv, June 2026) makes almost the same claim for LVLMs: attention aggregates visual evidence, FFNs at critical layers inject language priors, and attenuating FFN outputs during decoding reduces hallucination. FADE is probably concurrent, so I don't hold it against the paper, but it should be discussed. Given these, I think the new parts are mainly the sign selection (subtracting only the positive FFN part) and doing the correction on the logits. The paper should present them that way and compare against plain FFN down-weighting on all models.

Q3. Table 6 has a "both-sign" FFN variant on LLaVA-1.5 that is close to plain FFN down-weighting (as in AARF or FADE). Can you report it on all four models and on POPE, so the value of sign selection is clear?

Suggestion. Please cite and discuss ReDeEP and FADE.

A4. Thank you for pointing us to ReDeEP and FADE. Sec. 2 now cites ReDeEP and discusses it together with the concurrent FADE; all three works connect FFNs to prior knowledge. Beyond this shared view, this paper contributes:

  1. An explanation of the recall trade-off. On the output logit, late attention favors present objects and the late FFN favors absent ones. Sec. 3.4 shows that layer-level methods can only rescale the two together, which explains why DoLa and DeCo lose recall where they reduce hallucination most and why they help some models while hurting others.
  2. Tests of this opposition by intervention. With the caption fixed, occluding the image removes 72–79% of the attention advantage but changes the FFN write to absent objects by at most 1.2% (now on three models); the same word receives a larger late FFN write when it is hallucinated (App. M); and the pattern holds for candidates the model did not generate (A2).
  3. A sign-selective correction at the logits. CCD subtracts only the positive part of the late FFN projections from the logits. It changes no hidden state, leaves the negative part untouched, and uses one setting per model for every benchmark. FADE scales the whole FFN output of one layer inside the forward pass, with the layer chosen per model and task.

Table 6b now reports FADE and Both-sign (close to plain FFN down-weighting) on all four models; FADE uses one setting per model selected on the CHAIR images, and Both-sign uses the setting of CCD, which is also recovered on held-out images (App. Y.1).

Reviewer_z3CHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates object hallucinations in multimodal LLMs using direct logit attribution. It identifies the phenomenon that in late transformer layers, attention promotes image‑grounded faithful objects while feed‑forward networks push harder for hallucinated objects. Existing layer‑level decoding methods rescale attention and FFN jointly, trading reduced hallucination for lower object recall. Built on this mechanistic finding, the authors propose CCD, a training‑free, single‑pass decoding intervention that only subtracts positive late‑layer FFN logit contributions. Evaluated across four MLLMs on CHAIR, POPE, AMBER and MME benchmarks, CCD cuts hallucination metrics while preserving or improving recall, with negligible computational overhead. Ablations and mechanistic analyses validate its design and the underlying sublayer‑level diagnosis.

Critical strengths

Using direct logit attribution, the paper uncovers the FFN‑overcommitment effect, which explains a known pain point of existing layer‑wise decoding methods. This interpretability provides a principled causal basis for the proposed intervention instead of heuristic engineering, and offers a new sublayer‑granular lens for studying MLLM hallucination beyond just performance numbers.

The proposed CCD is training‑free, single‑pass and introduces minimal inference overhead. Across four different multimodal models and multiple standard benchmarks (CHAIR, POPE, AMBER, MME), it reduces object hallucinations without the typical recall drop suffered by competing contrastive‑decoding baselines.

Critical weaknesses
  1. Mechanistic evidence is limited to decoder‑only pre‑normalization transformer architectures studied in the paper. The core FFN‑overcommitment diagnosis relies on direct‑logit‑attribution over additive residual‑stream outputs, and empirical evidence is only gathered for four pre‑normalization decoder‑based multimodal models. The paper does not verify whether this opposing attention‑FFN pattern generalizes to alternative architectures such as MoE‑LLMs, post‑norm transformers, or encoder‑decoder multimodal systems. If FFN‑overcommitment does not hold for other popular MLLM families, CCD’s underlying principle cannot be assumed to transfer. This limits how broadly we can trust the generalizability of both the interpretability conclusion and the decoding method, which weighs on the paper’s broader impact claim.

  2. CCD requires per‑model hyper‑parameter tuning (α, K) without automatic selection heuristic. CCD depends on two hyperparameters, i.e., correction strength α and number of late layers K. The authors tune these separately for each evaluated model. Although sensitivity sweeps are provided for LLaVA‑1.5, there is no proposed validation‑free rule to set α and K for unseen models. Without such guidance, end‑users must run costly grid‑search on held‑out data to deploy CCD on new checkpoints. This materially reduces its plug‑and‑play practical appeal for real‑world deployment, a key selling‑point for training‑free decoding interventions.

Questions for authors
  1. The core FFN‑overcommitment diagnosis and CCD build upon pre‑norm decoder‑only architectures. Can you provide diagnostic results checking whether the opposing attention‑FFN pattern also exists for at least one MoE‑based or post‑normalization multimodal model? Absence of this pattern would break CCD’s core operating assumption and narrow the generality of both your interpretability finding and decoding approach.

  2. The tune hyper‑parameters α (correction strength) and K (number of late layers) individually for each model in the experiments. Given a completely new unseen MLLM with no held‑out validation data available, what concrete heuristic can practitioners use to directly set α and K? If no usable heuristic exists, the practical zero‑shot deployability of CCD is substantially limited.

  3. The evaluation focuses heavily on object‑level hallucination. Does CCD introduce new failure modes for non‑object hallucination (e.g. incorrect attributes, spatial relations, counting errors)? Can you provide quantitative metrics measuring attribute / relation errors before and after CCD intervention? Large degradation on these dimensions would indicate CCD only solves a narrow subset of multimodal hallucination.

  4. In your analysis you observe that CCD mostly suppresses hallucinated tokens but can also modify faithful tokens. Can you quantify the rate and magnitude by which CCD corrupts originally correct top‑1 faithful predictions across your test benchmarks? A high corruption rate would represent an unreported safety risk for practical inference‑time usage.

6 comments

Thank you for your valuable time and constructive comments. We address each point below. In the revised PDF, all changes made in response to the reviews are in blue. Tables 1–16 keep their original numbers (Table 6 is now Table 6a), new tables start at Table 17, and each table below (R1–R5) gives its location in the revised PDF.

W1 / Q1. Does the attention/FFN opposition hold beyond dense pre-norm decoders?

W1. Mechanistic evidence is limited to decoder‑only pre‑normalization transformer architectures studied in the paper. The core FFN‑overcommitment diagnosis relies on direct‑logit‑attribution over additive residual‑stream outputs, and empirical evidence is only gathered for four pre‑normalization decoder‑based multimodal models. The paper does not verify whether this opposing attention‑FFN pattern generalizes to alternative architectures such as MoE‑LLMs, post‑norm transformers, or encoder‑decoder multimodal systems. If FFN‑overcommitment does not hold for other popular MLLM families, CCD’s underlying principle cannot be assumed to transfer. This limits how broadly we can trust the generalizability of both the interpretability conclusion and the decoding method, which weighs on the paper’s broader impact claim.

Q1. The core FFN‑overcommitment diagnosis and CCD build upon pre‑norm decoder‑only architectures. Can you provide diagnostic results checking whether the opposing attention‑FFN pattern also exists for at least one MoE‑based or post‑normalization multimodal model? Absence of this pattern would break CCD’s core operating assumption and narrow the generality of both your interpretability finding and decoding approach.

A1. Thank you for this question. Yes, the opposition holds in all three designs we tested: a mixture-of-experts decoder (Qwen3-VL-30B-A3B), a decoder that normalizes every sublayer output before adding it to the residual stream (Gemma-3-4B-it), and an encoder-decoder model (InstructBLIP-FlanT5-XL). The decomposition applies because each sublayer output is still added to the stream: the MoE FFN write is the router-weighted sum of the selected experts, the Gemma-3 write is the normalized output, and the FlanT5 attention write is decoder self-attention plus cross-attention to the encoder.

Table R1. Diagnosis over the last three decoder layers, with cohorts from 1,500 MSCOCO val2014 images; output logit, Δ = hallucinated − faithful, 95% CI (revised PDF: Table 21).

Modeln per groupAttention ΔFFN ΔSum
Qwen3-VL-30B-A3B (MoE)157−2.96 [−5.37, −0.58]+9.04 [5.02, 12.97]+6.08 [1.53, 10.66]
Gemma-3-4B-it213−4.07 [−6.85, −1.33]+12.94 [8.07, 17.93]+8.87 [3.31, 14.52]
InstructBLIP-FlanT5-XL (enc-dec)163−2.71 [−4.98, −0.49]+5.27 [2.80, 7.81]+2.56 [−0.62, 5.70]
  • On all three models, both the attention and the FFN differential exclude zero.
  • In the encoder-decoder model, the attention advantage comes from cross-attention to the encoder, which carries the image (−3.12 [−5.18, −1.09]); decoder self-attention, which reads only the generated text, stays near zero (+0.41 [−0.92, 1.70]).

Table R2. CCD − greedy on the same three models, with α set by the label-free rule of A2 and no tuning; POPE F1 averaged over the three splits (revised PDF: Table 22).

ModelΔCHAIR_SΔCHAIR_IΔRecallΔPOPE F1
Qwen3-VL-30B-A3B (MoE)−1.6−0.4+0.9+1.74
Gemma-3-4B-it−2.0−0.8+2.1+2.26
InstructBLIP-FlanT5-XL (enc-dec)−1.4−0.6+0.7+1.38

Changes in the revised PDF (blue):

  • Sec. 4.2: new paragraph "Further evaluation".
  • App. W.3: Tables 21–22.
  • App. B: the scope of the analysis is updated.

W2 / Q2. Setting α and K for a new model without validation data.

W2. CCD requires per‑model hyper‑parameter tuning (α, K) without automatic selection heuristic. CCD depends on two hyperparameters, i.e., correction strength α and number of late layers K. The authors tune these separately for each evaluated model. Although sensitivity sweeps are provided for LLaVA‑1.5, there is no proposed validation‑free rule to set α and K for unseen models. Without such guidance, end‑users must run costly grid‑search on held‑out data to deploy CCD on new checkpoints. This materially reduces its plug‑and‑play practical appeal for real‑world deployment, a key selling‑point for training‑free decoding interventions.

Q2. The tune hyper‑parameters α (correction strength) and K (number of late layers) individually for each model in the experiments. Given a completely new unseen MLLM with no held‑out validation data available, what concrete heuristic can practitioners use to directly set α and K? If no usable heuristic exists, the practical zero‑shot deployability of CCD is substantially limited.

A2. Thank you for this practical question. A new model can be set up from 100 unlabeled images: K = 3 for every model, and α set by a rule. By Eq. 8, at a step whose top-1 word is y∗y^\ast, CCD changes the prediction exactly when α exceeds

α∗=min⁡y: Φ+(y)<Φ+(y∗)zy∗−zyΦ+(y∗)−Φ+(y).\alpha^\ast=\min_{y:\,\Phi^+(y)<\Phi^+(y^\ast)}\frac{z_{y^\ast}-z_y}{\Phi^+(y^\ast)-\Phi^+(y)}.

The share of changed predictions therefore never decreases as α grows. The rule sets α to the ρ-quantile of α* over greedy captions of 100 unlabeled images, so that CCD changes a share ρ of the top-1 predictions along these captions. K = 3 and ρ = 2% were fixed once, from the held-out setting of LLaVA-1.5 (which changes 2.3%), and reused unchanged for every other model.

Table R3. The label-free rule against the per-model setting; POPE-Adv F1 is the Adversarial split, the hardest of the three (revised PDF: Table 26).

ModelSetting (α, K)CHAIR_S ↓CHAIR_I ↓Recall ↑POPE-Adv F1 ↑
LLaVA-1.5-7BGreedy54.218.572.576.28
Per-model (0.10, 3)43.813.376.182.46
Rule (0.0874, 3)44.413.375.782.18
InstructBLIP-7BGreedy54.018.171.175.24
Per-model (0.01, 5)43.612.871.782.89
Rule (0.0213, 3)44.813.271.082.42
Qwen3-VL-4BGreedy26.09.449.285.23
Per-model (0.05, 3)23.08.150.888.27
Rule (0.0471, 3)23.48.350.588.06
Qwen3-VL-8BGreedy24.48.247.483.68
Per-model (0.05, 3)21.87.349.987.32
Rule (0.0486, 3)22.27.549.687.13
  • The rule stays within 1.2 CHAIR_S points and 0.5 POPE-Adv F1 points of the per-model setting on every model.
  • On InstructBLIP it lowers CHAIR_S by 9.2 points while recall stays at the greedy level (71.0 vs. 71.1).
  • The three models in A1 use this rule unchanged.

Changes in the revised PDF (blue):

  • Sec. 3.3: new paragraph with the formula above.
  • App. X.2: derivation and Table 26.
  • App. B: the limitation on setting the hyperparameters is updated.

Q3. Does CCD introduce errors on attributes, relations or counting?

Q3. The evaluation focuses heavily on object‑level hallucination. Does CCD introduce new failure modes for non‑object hallucination (e.g. incorrect attributes, spatial relations, counting errors)? Can you provide quantitative metrics measuring attribute / relation errors before and after CCD intervention? Large degradation on these dimensions would indicate CCD only solves a narrow subset of multimodal hallucination.

A3. Thank you for this suggestion. CCD improves these categories on LLaVA-1.5 and InstructBLIP and keeps them stable on Qwen3-VL. Besides existence, AMBER asks discriminative questions about attributes, numbers and relations, and MME has Count and Position subtasks.

Table R4. Change from greedy decoding to CCD (AMBER: F1 points; MME: subtask score out of 200) (revised PDF: Table 27).

ModelAMBER AttributeAMBER NumberAMBER RelationMME CountMME Position
LLaVA-1.5-7B+6.2+4.1+3.5+6.67+3.33
InstructBLIP-7B+2.8+1.6+1.2+11.67+6.67
Qwen3-VL-4B+0.4−0.3+0.5+1.670.00
Qwen3-VL-8B+0.2+0.4−0.2−1.67+3.33
  • CCD improves every category on LLaVA-1.5 and InstructBLIP.
  • On the Qwen3-VL models, which already answer these questions well, every AMBER category stays within 0.5 F1 points of greedy decoding and every MME subtask within 3.33 points (out of 200).

Changes in the revised PDF (blue):

  • App. Y.2: Table 27.
  • Sec. 4.2: one sentence in the paragraph "Further evaluation".

Q4. How often does CCD corrupt a correct top-1 prediction?

Q4. In your analysis you observe that CCD mostly suppresses hallucinated tokens but can also modify faithful tokens. Can you quantify the rate and magnitude by which CCD corrupts originally correct top‑1 faithful predictions across your test benchmarks? A high corruption rate would represent an unreported safety risk for practical inference‑time usage.

A4. Thank you for raising this point. CCD rarely changes a correct prediction, and it fixes far more predictions than it breaks. A changed faithful word may become another present object or no object, so we also count faithful words that become an absent object. Word-level rates use teacher forcing on the diagnostic cohorts (enlarged cohorts for Qwen3-VL, App. W.2); POPE rates are over all 9,000 questions.

Table R5. How often CCD changes the top-1 prediction, in percent (revised PDF: Table 31).

ModelFaithful word changedFaithful → absent objectHallucinated word changedPOPE correct → wrongPOPE wrong → correct
LLaVA-1.5-7B2.90.417.30.826.92
InstructBLIP-7B3.40.618.51.1910.94
Qwen3-VL-4B2.30.412.30.773.73
Qwen3-VL-8B2.70.314.00.924.84
  • At most 0.6% of faithful words become an absent object.
  • On POPE, CCD corrects about 5 to 9 times as many answers as it breaks, and the difference between the last two columns equals the mean accuracy gain in Table 5, up to rounding.
  • At the caption level, CCD raises CHAIR recall on all four models (Tables 2–3), so it mentions more of the objects that are present.

Changes in the revised PDF (blue):

  • App. Y.5: Table 31.

Summary

  • W1 / Q1, architectures. The opposition and the gains of CCD hold on MoE, normalized-sublayer and encoder-decoder models (Tables 21–22).
  • W2 / Q2, setting α and K. A label-free rule sets α from 100 unlabeled images, within 1.2 CHAIR_S and 0.5 POPE-Adv F1 points of the per-model setting (Table 26).
  • Q3, non-object errors. Gains on LLaVA-1.5 and InstructBLIP; stable on Qwen3-VL (Table 27).
  • Q4, corrupted predictions. At most 0.6% of faithful words become an absent object; on POPE, CCD fixes about 5 to 9 answers for each one it breaks (Table 31).

The architecture check and the question on deployment without validation data have made the paper stronger. We are happy to discuss any remaining questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A note on the timeline relative to FADE

Dear Reviewers and Area Chair,

Thank you all again for the careful reviews and the thoughtful discussion.

Since the relation to FADE has been the main remaining point in the discussion, we would like to share one piece of information that was not part of our earlier responses. An earlier version of this work was submitted to a peer-reviewed venue on 5 May 2026, almost two months before FADE first appeared on arXiv on 28 June 2026. That first version already included FFN overcommitment, Component-wise Contrastive Decoding (subtracting only the positive late-layer FFN contributions), and the analysis of DoLa and DeCo. As this concerns a prior submission, we have provided the timestamped record to the Area Chair rather than here.

Our work was therefore developed independently of FADE and before it appeared. We hope this helps in assessing the relation between the two works, and we are grateful to all reviewers for their time and feedback.

With sincere thanks,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.