acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Object Hallucination in Large Vision-Language Models via Faith-guided Self-Correcting Decoding

Abstract

Large Vision-Language Models (LVLMs) have demonstrated strong capabilities in open-ended visual understanding, yet remain prone to object hallucinations, generating objects unsupported by the visual input. Existing training-free decoding methods typically correct next-token predictions using auxiliary visual signals, without explicitly assessing the visual support underlying the prediction. We observe that grounded and hallucination-prone predictions respond differently to targeted evidence weakening: grounded predictions tend to lose more predictive support when their attended visual cues are disturbed. Based on this observation, we introduce visual faithfulness, an online measure of visual support derived from intervention-induced prediction sensitivity. We further propose FaiD, a training-free decoding framework that uses visual faithfulness to continuously guide self-correction during generation. Predictions with stronger visual support are preferentially calibrated using positive evidence, while weakly supported ones receive stronger suppression. Extensive experiments across multiple LVLMs and benchmarks show that FaiD consistently reduces object hallucinations while largely preserving grounded content and open-ended response quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Object Hallucination, Large Vision-Language Models, Visual Support Probing, Visual Faithfulness, Faith-Guided Decoding.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RRSoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper is working on VLM object hallucination issue, and propose a training-free decoding method FaiD. Specifically, it proposes to perturb top-k visual tokens and measure logits change to get sensitivity score. Leveraging this score, FaiD the combines a AGLA-derived positive visual view with a correction based on original-versus-perturbed logits. Experiment over three backbones and four benchmarks demonstrate certain benefits.

Critical strengths
  1. The paper is well presented and crafted method is simple and easy to use.

  2. The reported improvements span several model families, with useful ablations. The paper examines alternative control signals, perturbation strategies, single-branch corrections, gating functions, and generation budgets.

Critical weaknesses
  1. The proposed method and analysis has little innovation. VGS-Decoding [1] already uses sensitivity to visual corruption to adapt candidate-token probabilities. CHASD [2] uses attention-selected regions for localized perturbation and conditional contrastive correction. Neither is discussed. FaiD’s continuous combination with a positive visual view may be distinct, but the evidence for its particular signal is limited.

  2. Propose method will induce substantial computational overhead. For example, on LLaVA1.5, FaiD takes18.6 seconds versus 9.4 seconds for AGLA, while improving CHAIR-S from 49.2% to 44.8%. This makes the additional inference cost consequential for deployment.

  3. There are missing of multiple recent baselines in comparison: VGS-Decoding[1], CHASD[2], SID[3]

[1] https://arxiv.org/abs/2603.20314 [2] https://arxiv.org/abs/2605.23344 [3] https://proceedings.iclr.cc/paper_files/paper/2025/file/3cc87f2bd3e3b4df8f9217326761c322-Paper-Conference.pdf

Questions for authors
  1. How are pseudo-grounded and pseudo-hallucinated tokens labeled? Which data are used for calibration and hyper-parameter selection, and are they separate from evaluation? Please also specify the Gaussian noise scale and sampling settings.
1 comment
AuthorsOpenReview ↗

Response to Reviewer RRSo

We sincerely thank you for recognizing the clarity and simplicity of FaiD and its improvements across multiple LVLMs. We appreciate the concerns about innovation, computational overhead, and the coverage of recent decoding baselines.

(Weakness.1) The proposed method and analysis has little innovation. VGS-Decoding [1] already uses sensitivity to visual corruption to adapt candidate-token probabilities. CHASD [2] uses attention-selected regions for localized perturbation and conditional contrastive correction. Neither is discussed. FaiD’s continuous combination with a positive visual view may be distinct, but the evidence for its particular signal is limited.

(Weakness.3) There are missing of multiple recent baselines in comparison: VGS-Decoding[1], CHASD[2], SID[3].

Response: Thank you for highlighting these closely related methods. FaiD differs from these methods in what the adaptive signal controls. It uses candidate-specific responses to visual intervention to continuously allocate weight between positive enhancement and contrastive correction, rather than adapting only negative-view construction, candidate probabilities, or branch activation.

MethodAdaptation target
SIDNegative-view construction
VGS-DecodingCandidate probabilities
CHASDContrastive-branch activation
FaiD (Ours)Candidate-wise enhancement–correction balance

We additionally compare CHAIR results on LLaVA-1.5-7B. SID is reproduced using its official implementation under matched settings. We omit a numerical comparison with VGS-Decoding because its reported evaluation concerns medical VQA rather than our general-domain setting.

Table R11. Comparison of multiple recent baselines.

MethodCHAIR-S ↓CHAIR-I ↓Recall ↑Length
SID48.213.5176.898.8
CHASD47.014.20——
FaiD (Ours)44.814.1177.098.9

In the comparison, FaiD lowers CHAIR-S by 3.4% relative to SID, while SID achieves lower CHAIR-I. Their recall and response lengths are similar, indicating different sentence-level and instance-level hallucination trade-offs. We have added the methodological discussion to the Introduction and Related Work, and the result comparison to Table 1.

(Weakness.2) Propose method will induce substantial computational overhead. For example, on LLaVA1.5, FaiD takes18.6 seconds versus 9.4 seconds for AGLA, while improving CHAIR-S from 49.2% to 44.8%. This makes the additional inference cost consequential for deployment.

Response: We acknowledge that FaiD introduces additional inference overhead due to visual support probing and positive evidence enhancement. However, the overhead varies across LVLMs. As shown in Table 10, on Qwen2.5-VL, FaiD reduces CHAIR-S from 27.4% to 19.8% compared with VCD, with latency increasing from 28.7s to 31.9s. Similarly, on InternVL3, FaiD reduces CHAIR-S from 31.4% to 23.6%, while latency increases only from 20.5s to 21.2s. Of course, we agree that reducing inference overhead remains an important direction for future improvement.

(Question.1) How are pseudo-grounded and pseudo-hallucinated tokens labeled? Which data are used for calibration and hyper-parameter selection, and are they separate from evaluation? Please also specify the Gaussian noise scale and sampling settings.

Response: We acknowledge the insufficient implementation details. We clarify the labeling procedure, parameter selection, noise scale, and decoding settings below.

Labels. We match generated object mentions to the CHAIR vocabulary and synonym mapping. A mention is pseudo-grounded if its normalized object category occurs in the corresponding MSCOCO-derived CHAIR annotations, and pseudo-hallucinated otherwise.

Parameter selection. Consistent with the hyperparameter selection protocol in prior training-free decoding studies, such as VCD[1], AGLA[2], and DeGF[3], we evaluate the faithfulness margin mm and perturbation ratio KK on CHAIR. We did not construct a separate held-out calibration split.

[1] Sicong Leng, et.al. Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding. In CVPR, pp. 13872–13882, 2024.

[2] Wenbin An, et.al. Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention. In CVPR, pp. 29915–29926, 2025.

[3] Ce Zhang, et.al. Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models. In ICLR, 2025.

Gaussian noise scale and sampling setting. We define the perturbation noise as standard Gaussian, ϵ∼N(0,I)\epsilon\sim\mathcal N(0,I), with zero mean and unit standard deviation (σ=1\sigma=1). We use greedy next-token selection, with a maximum of 1,024 new tokens for generative evaluation.

Reviewer_vEurOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Faith-guided Self-Correcting Decoding (FaiD), a training-free framework for mitigating object hallucinations in large vision-language models. The key idea is to estimate token-level visual faithfulness by measuring the change in prediction logits after perturbing highly attended visual features. The resulting signal is used to dynamically balance positive visual evidence enhancement and faith-based suppression during decoding. Experiments on three LVLMs across CHAIR, POPE, MME-Hallucination, and LLaVA-Bench-Wild demonstrate improvements in hallucination reduction and open-ended response quality. Additional ablations investigate the effectiveness of the faithfulness signal, intervention strategies, gating mechanisms, and computational efficiency.

Critical strengths
  1. An interesting perspective on visual support estimation. Instead of directly applying contrastive corrections, the paper estimates token-level visual faithfulness through targeted visual interventions and uses it to guide decoding. This provides a meaningful perspective on adaptive hallucination mitigation.

  2. Consistent empirical improvements with useful ablation studies. FaiD reduces object hallucinations across three LVLMs and multiple benchmarks. The ablations on control signals, intervention strategies, and gating mechanisms provide useful evidence for the effectiveness of the proposed framework.

Critical weaknesses
  1. The relationship between visual faithfulness and actual visual grounding is not sufficiently established. The proposed faithfulness score measures logit sensitivity to visual perturbations, which does not necessarily indicate whether a prediction is factually supported by the image. A visually dependent but incorrect prediction may also receive a high faithfulness score. Although Figure 1 shows different distributions for pseudo-grounded and pseudo-hallucinated tokens, their substantial overlap and the absence of more direct grounding evaluations limit the evidence supporting the central assumption.

  2. The contribution of faith-guided calibration is not sufficiently disentangled from existing decoding strategies. FaiD primarily relies on AGLA-based positive enhancement for high-faithfulness candidates and contrastive-style correction for low-faithfulness candidates. Although Table 6 demonstrates that continuous gating outperforms fixed fusion and hard gating, it remains unclear whether the improvements arise from the proposed visual faithfulness signal accurately identifying visually grounded predictions, or from adaptive reweighting of existing correction signals. More targeted analysis of the behaviors and contributions of the two branches would strengthen the methodological justification.

Questions for authors
  1. Regarding the validity of visual faithfulness. Could the authors provide a more direct evaluation of how well visual faithfulness distinguishes grounded from hallucinated object predictions, such as AUROC/AUPRC or token-level error analysis? In particular, how does the method handle visually dependent but incorrect predictions that may still receive high faithfulness scores?

  2. Regarding the mechanism of faith-guided calibration. Table 6 shows that continuous gating improves upon fixed fusion and hard gating. However, could the authors further analyze how the positive enhancement and faith-based correction branches affect grounded and hallucinated object predictions, respectively?

  3. Regarding the preservation of grounded content. Table 1 shows that object recall decreases on Qwen2.5-VL and InternVL3, despite improvements in hallucination metrics. Could the authors analyze whether the faith-based suppression mechanism disproportionately affects correctly grounded objects, particularly small, partially occluded, or low-attention objects?

  4. Regarding robustness and calibration. How sensitive is the faithfulness estimate to attention localization errors, Gaussian noise strength, and the calibration of the margin mm? Could the authors clarify the calibration procedure and evaluate whether the selected settings generalize across datasets without requiring dataset-specific tuning?

6 comments
AuthorsOpenReview ↗

Response to Reviewer vEur (Part 1/2)

We sincerely thank you for recognizing the perspective of visual support estimation and the empirical improvements of FaiD. We particularly appreciate the comments concerning the interpretation of faithfulness, the two correction components, and grounded content preservation.

(Weakness.1) The relationship between visual faithfulness and actual visual grounding is not sufficiently established. The proposed faithfulness score measures logit sensitivity to visual perturbations, which does not necessarily indicate whether a prediction is factually supported by the image. A visually dependent but incorrect prediction may also receive a high faithfulness score. Although Figure 1 shows different distributions for pseudo-grounded and pseudo-hallucinated tokens, their substantial overlap and the absence of more direct grounding evaluations limit the evidence supporting the central assumption.

(Question.1) Regarding the validity of visual faithfulness. Could the authors provide a more direct evaluation of how well visual faithfulness distinguishes grounded from hallucinated object predictions, such as AUROC/AUPRC or token-level error analysis? In particular, how does the method handle visually dependent but incorrect predictions that may still receive high faithfulness scores?

Response: Thank you for highlighting the distinction between visual dependence and factual correctness. Tables R1–R2 provide direct diagnostics and show overlapping faithfulness distributions: sensitivity to visual intervention does not guarantee factual correctness.

Table R1. Comparison of different control signals.

Control signalAUROC ↑CHAIR-S ↓CHAIR-I ↓Recall ↑Length
Confidence0.710121.47.255.2133.8
Negative entropy0.707620.67.655.4130.1
Top-K attention mass0.487820.46.754.9137.2
Faithfulness0.689419.86.654.4146.8

Table R2. Faithfulness distributions on the FaiD trajectory.

Proxy labelMeanMedianInterquartile rangeMean gate
Grounded1.6300.875−0.125 to 3.0000.470
Hallucinated1.0990.500−0.3125 to 1.8750.401

In particular, visually dependent but incorrect candidates follow the same gating rule. High $\phi $ favors the enhancement branch, although the final logit change depends on both branches’ signed corrections. If enhancement also supports an incorrect candidate, FaiD may retain or reinforce it. We will clarify the role of $\phi $ as a candidate-dependent intervention-response signal and explain how it guides the balance between evidence enhancement and contrastive correction.

(Weakness.2) The contribution of faith-guided calibration is not sufficiently disentangled from existing decoding strategies. FaiD primarily relies on AGLA-based positive enhancement for high-faithfulness candidates and contrastive-style correction for low-faithfulness candidates. Although Table 6 demonstrates that continuous gating outperforms fixed fusion and hard gating, it remains unclear whether the improvements arise from the proposed visual faithfulness signal accurately identifying visually grounded predictions, or from adaptive reweighting of existing correction signals. More targeted analysis of the behaviors and contributions of the two branches would strengthen the methodological justification.

(Question.2) Regarding the mechanism of faith-guided calibration. Table 6 shows that continuous gating improves upon fixed fusion and hard gating. However, could the authors further analyze how the positive enhancement and faith-based correction branches affect grounded and hallucinated object predictions, respectively?

Response: We conducted a analysis to clarify the actual contributions of both branches through signed-logit analysis. The enhancement and contrastive corrections are E=ρt(ztaug−ztraw)E=\rho _t (z_t^{aug}- z_t^{raw}) and $S=( 1-\rho _t ) (z_t^{raw}- z_t^{anti}) $, respectively.

Table R8. Analysis of positive enhancement and faith-based correction branches.

ObjectsAverage positive enhancementAverage faith-based correction
Grounded−0.0491−0.0525
Hallucinated−0.1156+0.0031

The results show that their signs depend on the branch outputs and gate weights alone do not determine whether a candidate is promoted or suppressed. In addition, the submitted Table 6 shows that FaiD achieves CHAIR-S/I of 19.8/6.6, compared with 22.4/7.7 for Pos. Only, and 22.2/6.9 for Direct Contrast (only faith-based correction). These results demonstrate lower hallucination rates of FaiD than the standalone and fixed-fusion configurations.

AuthorsOpenReview ↗

Response to Reviewer vEur (Part 2/2)

(Question.3) Regarding the preservation of grounded content. Table 1 shows that object recall decreases on Qwen2.5-VL and InternVL3, despite improvements in hallucination metrics. Could the authors analyze whether the faith-based suppression mechanism disproportionately affects correctly grounded objects, particularly small, partially occluded, or low-attention objects?

Response: We evaluated whether COCO-grounded categories mentioned by Regular remain in FaiD’s descriptions, using image–category pairs as the unit in Table R9.

Table R9. Ground-truth category preservation.

MethodRetainedDroppedNewTotal mentioned
Regular871——871
FaiD (Ours)744127112856

Object size. We use the original COCO instance-annotation area: small objects have A<322A<32^2 pixels², and large objects have A⩾962A\geqslant 96^2 pixels². To avoid assigning a category mention ambiguously to differently sized instances, this analysis includes only categories with exactly one non-crowd instance in the image.

Attention groups. At the Regular decoding steps, we measure attention mass within the COCO category-union mask, weighted by each visual token’s mask occupancy, and average over deduplicated steps per image–category pair. Among 826 eligible pairs, low and high attention denote the bottom and top quintiles, respectively. Groups are defined using Regular attention, not FaiD’s subsequent outputs.

Table R10. Conditional drop rates among Regular grounded mentions.

GroupRegular mentionDroppedDrop rate
Small object381231.58%
Large object303237.59%
Lowest attention quintile1663420.48%
Highest attention quintile165159.09%

As shown in Table R10, FaiD drops 12/38 small-object categories (31.58%), compared with 23/303 large-object categories (7.59%). Likewise, drop rates are 34/166 (20.48%) for low attention and 15/165 (9.09%) for high attention. These results indicate greater omission risk for small and low-attention categories. We have included these definitions and limitations in Section 4.3, under “Q4: Where Does Object Recall Decline?”

(Question.4) Regarding robustness and calibration. How sensitive is the faithfulness estimate to attention localization errors, Gaussian noise strength, and the calibration of the margin (m)? Could the authors clarify the calibration procedure and evaluate whether the selected settings generalize across datasets without requiring dataset-specific tuning?

Response: Thank you for requesting a clearer robustness and calibration analysis.

Attention localization. We compare targeted and random localization at identical decoding states and equal visual-token budgets. Targeted selection covers more of the referenced category-mask area (66.71% versus 49.65%).

Noise strength. We vary the noise standard deviation by factors of 0.5, 1.0, and 1.5, keeping other settings fixed. CHAIR-S/I are 20.2/6.88, 19.8/6.56, and 19.8/9.30, respectively. The increased CHAIR-I at scale 1.5 indicates sensitivity to stronger perturbations.

Margin and calibration. Appendix B.3 (Table 10) already evaluates m=0 and Q10~Q50 across three LVLMs. On Qwen2.5-VL, Q50 improves CHAIR-S/I from 22.6/7.2 to 19.8/6.6 relative to m=0. Results indicate model-dependent sensitivity. Hyperparameters were selected on CHAIR and reused on POPE and MME without additional dataset-specific tuning.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts. However, two concerns remain:

  1. Visual faithfulness: The AUROC results do not sufficiently establish faithfulness as a reliable measure of visual grounding.
  2. Correction mechanism: The signed-logit analysis does not fully clarify how the two branches contribute to hallucination suppression.

While the rebuttal addresses some concerns, these core issues remain unresolved. Therefore, I maintain my original rating.

Reviewer_Qd4MOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FaiD is a training-free decoding framework for object hallucination in LVLMs. At each step it (1) localizes the most-attended visual tokens, (2) constructs an "anti-evidence" view by perturbing them, (3) computes a faithfulness score as the raw-vs-intervened logit difference, (4) maps the score to a continuous sigmoid gate, and (5) interpolates between AGLA-style positive enhancement and faith-based suppression, under a plausibility constraint. Experimental evaluation is performed on LLaVA1.5-7B, Qwen2.5-VL-7B, and InternVL3-9B across CHAIR, POPE, MME-Hallucination, and LLaVA-Bench-Wild, with ablations on control signal, intervention design, gating, and cost.

Critical strengths
  1. Well-motivated and cleanly framed. The insight that high logits ≠ visual support, and that intervention sensitivity beats confidence/attention as a control signal, is intuitive and clearly argued.

  2. Training-free, model-agnostic, honest efficiency reporting. No parameter updates; transfers across architectures.

  3. Good reproducibility. Hyperparameters and sensitivity analyses are reported.

Critical weaknesses
  1. ϕ\phi validity is shown only downstream. The core claim is that ϕ\phi measures visual support, but validation is indirect (CHAIR/POPE gains).

  2. No compute-matched comparison to the closest baselines. FaiD's contribution over VCD is token-conditioned, step-wise intervention. Table 5's "Global" row is a start but is not framed as a compute-matched VCD comparison; nor is there a VCD variant using FaiD's extra forward passes. Without this, the gain is confounded with extra compute.

Questions for authors
  1. Can you report a token-level diagnostic—distribution of φ for object tokens in vs. not in COCO annotations, and AUC—to directly test the premise?

  2. How does FaiD compare to (a) VCD with equal forward passes, and (b) FaiD with a random or tuned-constant gate ρt\rho_t?

  3. What is run-to-run variance for CHAIR/POPE (≥3 seeds), and how many GPT-4V judge calls were averaged for Table 3?

  4. What fraction of decoding steps have Top-K attended tokens overlapping annotated objects, and how does ϕ\phi degrade under random localization?

  5. How sensitive are results to β\beta, and does β\beta interact with τ\tau?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Qd4M (Part 1/3)

We sincerely appreciate you for your careful review, insightful comments, and recognition of our work’s excellent motivation, novel method, and reproducibility. We tried our best to provide detailed clarifications for all your concerns as follows. Please do not hesitate to share any additional questions. We would be delighted to answer further.

(Weakness.1) ϕ\phi validity is shown only downstream. The core claim is that ϕ\phi measures visual support, but validation is indirect (CHAIR/POPE gains).

(Question.1) Can you report a token-level diagnostic—distribution of φ for object tokens in vs. not in COCO annotations, and AUC—to directly test the premise?

Response: Thank you for this suggestion. We conduct a token-level evaluation using object predictions and MSCOCO annotations. Specifically, we report the faithfulness distributions of grounded and hallucinated object predictions and AUROC.

Table R1. Comparison of different control signals.

Control signalAUROC ↑CHAIR-S ↓CHAIR-I ↓Recall ↑Length
Confidence0.710121.47.255.2133.8
Negative entropy0.707620.67.655.4130.1
Top-K attention mass0.487820.46.754.9137.2
Faithfulness0.689419.86.654.4146.8

These results indicate a distinction between standalone discrimination (AUROC in Table R1) and downstream decoding performance (CHAIR-S/CHAIR-I in Table R1). Although faithfulness has a lower AUROC than confidence and negative entropy, the faithfulness-based configuration achieves the lowest CHAIR-S and CHAIR-I.

Table R2. Faithfulness distributions on the FaiD trajectory.

Proxy labelMeanMedianInterquartile rangeMean gate
Grounded1.6300.875−0.125 to 3.0000.470
Hallucinated1.0990.500−0.3125 to 1.8750.401

The distribution (Table R2) summaries show higher mean and median faithfulness scores for grounded predictions, while the overlapping interquartile ranges indicate imperfect separation. Thus, visual faithfulness is intended to measure sensitivity to targeted visual evidence, rather than to serve as a standalone correctness detector. The corresponding results and analysis have been added to Appendix B.2.

AuthorsOpenReview ↗

Response to Reviewer Qd4M (Part 2/3)

(Weakness.2) No compute-matched comparison to the closest baselines. FaiD's contribution over VCD is token-conditioned, step-wise intervention. Table 5's "Global" row is a start but is not framed as a compute-matched VCD comparison; nor is there a VCD variant using FaiD's extra forward passes. Without this, the gain is confounded with extra compute.

(Question.2) How does FaiD compare to (a) VCD with equal forward passes, and (b) FaiD with a random or tuned-constant gate ρt\rho _t ?

Response: We address both factors using controlled comparisons.

Compute-matched VCD. We added a two-negative-view VCD-style control. At every step, it performs two independent forwards using the original replacement-noise operator and averages their contrastive corrections. This matches FaiD’s number of auxiliary branch forwards: z2neg−vcd=zraw+(zraw−z1anti)+(zraw−z2anti)2.z_{2neg-vcd}=z^{raw}+\scriptstyle{\frac{\left( z^{raw}-z_{1}^{anti} \right) +\left( z^{raw}-z_{2}^{anti} \right)}{2}}.

Table R3. Comparison with a compute-matched VCD-style variant.

MethodCHAIR-S ↓CHAIR-I ↓Recall ↑Length
Regular24.49.055.1134.5
Compute-matched VCD22.68.755.3140.3
FaiD (Ours)19.86.654.4146.8

The corresponding results and analysis have been added to Appendix B.3.

Alternative gates. The submitted Table 6 already includes constant gates with ρt\rho _t = 0, 0.5, and 1, as well as hard, linear, and sigmoid gates. We additionally compare these constant gates and random gates (ρt\rho _t~U[0, 1]). Random gate assigns an independent uniform gate to each plausible candidate at the current step.

Table R4. Comparison with random and constant gates.

Gate settingCHAIR-S ↓CHAIR-I ↓Recall ↑Length
ρt\rho _t = 0 (already in Table 7)22.26.954.5143.5
ρt\rho _t = 0.5 (already in Table 7)22.810.855.8135.8
ρt\rho _t = 1.0 (already in Table 7)22.47.757.5127.2
Random23.67.955.2137.6
FaiD (Ours)19.86.654.4146.8

The reported point estimates favor FaiD over the listed controls. These comparisons provide a more direct assessment of faithfulness-conditioned allocation under matched auxiliary forward counts.

(Question.3) What is run-to-run variance for CHAIR/POPE (≥3 seeds), and how many GPT-4V judge calls were averaged for Table 3?

Response: We additionally evaluate FaiD over 3 independent random seeds on CHAIR. The mean and standard deviation are shown in Table R5.

Table R5. CHAIR variation across three FaiD seeds.

SeedCHAIR-S ↓CHAIR-I ↓Recall ↑Length
119.86.5654.42146.78
221.66.5053.78140.48
321.07.5455.50148.77
Mean ± SD20.80 ± 0.926.87 ± 0.5854.57 ± 0.87145.34 ± 4.33

The GPT-4V evaluations in Table 3 use one judging calls per sample, without repeated-call averaging. We will clarify this protocol in Appendix C.1.

AuthorsOpenReview ↗

Response to Reviewer Qd4M (Part 3/3)

(Question.4) What fraction of decoding steps have Top-KK attended tokens overlapping annotated objects, and how does $\phi $ degrade under random localization? How sensitive are results to $\beta $, and does $\beta $ interact with $\tau $?

Response:

Category-conditioned Localization. We evaluate whether the selected visual patches cover the annotated regions of the currently mentioned object category. Specifically, for each annotation-grounded object mention, we obtain its corresponding COCO category through the CHAIR synonym mapping and merge the instance-segmentation masks of that category. We align this category-union mask with the model’s merged visual-token grid using area pooling. We compare attention-selected patches with the saved random selection at identical decoding states and the same patch budget.

Table R6. Category-conditioned localization.

Selection ratio KKSelected/TotalTargeted coverage (%)Random coverage (%)Difference (pp)
10%19/19624.539.63+14.90
20%39/19638.0419.82+18.22
30%58/19648.9029.45+19.44
40%78/19658.3039.46+18.84
50%98/19666.7149.65+17.06

From the results, we observe that targeted selection improves category-mask coverage over random selection by 14.90%-19.44% across KK =10%-50%. With the default selection ratio of KK=50%, targeted selection covers 66.71% of the category-mask area, compared with 49.65% for random selection. We have included the above analysis and localization visualizations in Category-conditioned Localization in 4.3 Section.

Sensitivity to $\beta $ and $\tau $. We conduct a controlled beta–tau sensitivity analysis on CHAIR benchmark under Qwen2.5-VL-7B.

Table R7. Sensitivity analysis of $\beta $ and $\tau $.

$\beta $$\tau $CHAIR-S ↓CHAIR-I ↓Recall ↑Length
0.10.526.49.5556.01136.78
0.11.025.010.0954.42135.08
0.12.022.09.2753.91129.31
0.30.525.48.6656.45146.14
0.31.022.68.4054.93137.78
0.32.024.68.5856.33143.34
0.50.522.26.8054.80147.19
0.5 (default)1.0 (default)19.86.5654.42146.78
0.52.020.06.1654.35151.46
0.70.521.68.6253.91144.26
0.71.021.06.4654.48146.14
0.72.021.67.1855.12141.71

The default configuration achieves the lowest CHAIR-S in the tested grid, while (β,τ)=(0.5,2)(\beta,\tau)=(0.5,2) achieves the lowest CHAIR-I. The effect of τ\tau varies with β\beta : increasing τ\tau consistently reduces CHAIR-I at β=0.5\beta=0.5, but not at β=0.1\beta=0.1 or 0.70.7. These results indicate configuration dependent trade-offs rather than uniform parameter insensitivity. The corresponding results and analysis have been added to Appendix B.6.

Comments on the paper

1 comment

Dear Reviewers,

The rebuttal deadline is approaching. I am leaving this friendly reminder to encourage active discussion. Thank you for your valuable time and engagement!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.