acceptodds
Under review as a conference paper at ICLR 2027

From Suppression to Repair: Mitigating Object Hallucination in Large Vision-Language Models via Localized Distribution Alignment

15 views

Abstract

Object hallucination remains a major obstacle for large vision-language models (LVLMs) to generate reliable content. An intuitive mitigation strategy is to suppress hallucination-related components in hidden representations. However, these components may also contain useful information, and suppressing them can weaken the model's multimodal capabilities. In this paper, we propose ResOT, a training-free method that repairs representations at inference time through localized distribution alignment. Specifically, ResOT projects dominant hallucinated directions away from the faithful subspace, forming a low-dimensional residual subspace for intervention. Within this subspace, ResOT uses Gaussian optimal transport (OT) to align the hallucinated distribution with the faithful one. The resulting map defines repair targets with minimal changes to the original representations. At inference, ResOT adaptively controls how far each token state moves toward its OT target. Experiments on three representative LVLMs show that ResOT substantially reduces object hallucination while improving image caption quality and multimodal performance across multiple benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Vision-Language Models, Multimodal Large Language Models, Hallucination Mitigation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YGUaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ResOT, which mitigates object hallucination by repairing hallucinationrelated representations rather than directly suppressing them. ResOT confines the intervention to the residual subspace and uses Gaussian optimal transport to move representations toward a faithful target with minimum expected squared displacement. A likelihood-based gate further adjusts how much of this correction is applied during decoding. Evaluation on LLaVA-1.5, mPLUG-Owl2, and InstructBLIP covers hallucination, caption quality, multimodal capabilities, calibration robustness, and inference cost.

Critical strengths
  1. The repair formulation is well motivated and supported by informative ablations. The method recognizes that hallucination-associated directions can also contain useful information, and replaces component removal with distribution-based correction. Table 4 helps separate the contributions of localization and repair: OT in the residual subspace outperforms OT in random or Nullu-style subspaces, while covariance-aware transport improves over mean-only alignment. These comparisons provide evidence that the gains depend on both where and how the intervention is applied.

  2. The empirical evaluation provides substantial evidence for the effectiveness of ResOT. ResOT is combined with LLaVA-1.5, mPLUG-Owl2, and InstructBLIP and conducted experiments on multiple benchmarks. The results demonstrate that ResOT reduces object hallucination while improving image caption quality and multimodal performance.

Critical weaknesses
  1. The fitted distribution and gate are validated mainly on complete descriptions, while intervention operates on generation prefixes. Section 3.1 fits the map using final-position states after complete faithful or hallucinated captions. Online repair instead acts at every next-token prediction position. The separability, alignment, and displacement analyses in Appendix D largely reuse complete-description representations, so they do not establish that the likelihood ratio identifies states needing repair during free-running generation. The paper acknowledges this shift in Appendix E, p. 21. This matters because OT optimality applies to its fitted source distribution; it does not establish that transporting an arbitrary prefix state improves grounding.

  2. The robustness of the advantage over the strongest competing methods is not fully established. Some leading results have small margins: LLaVA-1.5 POPE F1 improves from 86.46 with HulluEdit to 86.54, while mPLUG-Owl2 CHAIRs improves from 0.314 with VTI to 0.302. The main comparisons lack paired confidence intervals or significance analyses.

Questions for authors
  1. Can you evaluate how the actual likelihood gate relates to grounded versus hallucinated object predictions during free-running generation? A focused analysis of gate values at object-generating positions, including their behavior across generation positions, would help connect the complete-description calibration analyses to the online intervention.

  2. Could you report paired image-level uncertainty for the principal CHAIR and POPE comparisons against the closest baselines, keeping questions from the same image together when resampling? Could you also assess whether the rankings persist when the closest representation baselines receive comparable validation tuning on the same image split?

  3. Please report the final selected shared rank (r=k) and global repair strength (\alpha) for each backbone. Appendix B already specifies the search procedure and the per-layer 95th-percentile correction threshold. Could you additionally provide the exact norm-clipping formula and clarify whether clipping is applied before or after multiplication by the global strength and likelihood gate?

3 comments

We thank the reviewer for the positive assessment of our work and the constructive suggestions. We address your questions below.

W1 + Q1. Calibration-to-inference distribution shift

We acknowledge the reviewer’s concern about this potential distribution shift. We address this from two perspectives:

  • The successful use of similar calibration strategies in prior work [1–2] provides empirical support for this practice.
  • To assess its practical implications, we examine whether the calibrated repair signals remain selective on prefixes from freely generated captions. The following results suggest that the calibration-derived signals remain informative on generation prefixes despite potential representation shifts. (The intuitive example is illustrated in Appendix F, and the corresponding experimental results have been added to Appendix D.3.)

Setting. We randomly sample 100 COCO images and generate captions using Vanilla. We use GPT-6 Sol, followed by manual verification, to identify hallucinated spans. Keeping the generated prefixes unchanged, we evaluate the likelihood gate and repair magnitude immediately before each span.

Hallucinated SpanOther Span
token positiongaterepair magnitudegaterepair magnitude
1–320.86403.5530.65301.612
33–640.85313.3880.66471.641
65–960.79593.2770.63121.445
97–2560.73153.0450.61401.199

Across all evaluated token-position ranges, hallucinated spans receive higher average gate values and repair magnitudes than other spans.

W2 + Q2. Statistical significance and robustness of comparisons under comparable hyperparameter tuning.

We thank the reviewer for this suggestion. We repeat the comparisons using five random seeds. The experimental results are presented in the table below.

  • On POPE, ResOT consistently outperforms HulluEdit across all three subsets.
  • Although VTI achieves better mean CHAIR and BLEU-4 scores, its results exhibit substantially higher variance, while ResOT remains considerably more stable.
LLaVA-v1.5POPE randomPOPE popularPOPE adversarial
+ HulluEdit89.964 ± 0.0387.135 ± 0.0482.005 ± 0.08
+ ResOT90.011 ± 0.0287.426 ± 0.0182.402 ± 0.01
LLaVA-v1.5CHAIRsCHAIRiBLEU-4
+ VTI29.28 ± 4.508.23 ± 1.546.85 ± 0.71
+ ResOT30.64 ± 0.438.74 ± 0.096.07 ± 0.01

To assess the robustness of our comparisons with the strongest baselines, we conduct additional experiments below. We perform 10,000 paired image-level bootstrap resamples over 500 images. The experimental results are presented in the table below.

LLaVA-v1.5POPE F195% CI
ResOT − HulluEdit86.653[-0.203, +0.591]

To further examine the fairness of our comparisons, we conduct additional experiments on an independent validation set of 200 images, comprising 3,600 POPE-style questions. We fine-tune the hyperparameters of VTI, HulluEdit, and ResOT on the same validation set. The experimental results are presented in the table below.

LLaVA-v1.5Macro F1CHAIRsCHAIRiBLEU
+ VTI88.8735.509.660.2045
+ HulluEdit89.1661.0013.260.1684
+ ResOT89.5736.509.590.2202

Overall, while ResOT does not consistently outperform the strongest baselines on every individual metric, its main contribution lies in achieving a favorable balance among hallucination reduction, multimodal capability preservation, and generation quality. Our extensive experimental results across multiple benchmarks further support this claim.

Q3. Implementation Detail.

We apologize for the confusion. We have added the relevant clarifications in Lines 891–901.

Finally, we thank the reviewer again for their time and thoughtful feedback. We hope our responses have addressed your concerns, and we would be happy to clarify any remaining questions!

[1]. Liu S, Ye H, Zou J Y. Reducing hallucinations in large vision-language models via latent space steering. ICLR, 2025.

[2]. Li Z, Shi H, Gao Y, et al. The hidden life of tokens: Reducing hallucination of large vision-language models via visual information steering. ICML, 2025.

Thank you for the detailed experiments and explainations.

The prefix replay analysis substantially addresses my concern about whether the likelihood gate remains informative on freely generated prefixes.

The experiment of statistical significance and robustness of comparisons under comparable hyperparameter tuning is clear and informative.

The revised appendix also fully answers my questions about the selected hyperparameters and clipping order.

Overall, the rebuttal strengthens the paper, and I am maintaining my rating of 3 (weak acceptance).

We are glad that our responses have addressed your concerns!

Thank you again for your positive assessment and continued support.

Your careful review and constructive feedback have helped us strengthen the empirical analysis and clarify the method.

Reviewer_fATzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies object hallucination in large vision-language models from the perspective of representation repair. The authors argue that directly suppressing hallucination-related components may also remove useful information. Based on this motivation, they propose ResOT, a training-free inference-time intervention method. ResOT constructs a residual subspace from dominant hallucinated directions orthogonal to the dominant faithful subspace, fits Gaussian distributions to faithful and hallucinated representations, and uses optimal transport to move token states toward the faithful distribution. A likelihood-based gate adaptively controls the repair strength. Experiments on three LVLM backbones and multiple hallucination, captioning, and multimodal benchmarks show reduced object hallucination while generally preserving generation quality and broader capabilities.

Critical strengths
  1. The paper presents a coherent alternative to suppression-based intervention. The residual-subspace construction and the comparison among suppression, mean alignment, and Gaussian optimal transport provide evidence that repairing rather than deleting hallucination-related components can better preserve useful information.

  2. The empirical evaluation is broad and includes three LVLM backbones, several hallucination benchmarks, caption-quality metrics, general multimodal benchmarks, calibration-size and corruption analyses, and efficiency measurements. The results generally show that ResOT reduces hallucination without achieving this merely by producing less informative descriptions.

Critical weaknesses
  1. ResOT estimates its distributions from final-position representations of complete descriptions but applies the resulting maps to intermediate states during autoregressive generation. This calibration-to-inference distribution shift is acknowledged but not quantitatively analyzed, leaving the proposed repair interpretation only partially validated.

  2. The paper provides no qualitative success or failure cases. It is therefore difficult to determine whether ResOT genuinely corrects hallucinated objects, replaces them with visually grounded content, or simply avoids uncertain object mentions. Token-level analyses of the likelihood gate and repair magnitude are also missing.

Questions for authors
  1. Why is representation repair applied specifically to layers 16–31 for all three backbones? Was this range selected using validation data, prior empirical observations, or simply chosen as the latter half of the decoder? Please provide ablations comparing individual layers and different intervention ranges.

  2. Can the authors quantify the distribution shift between representations extracted from complete descriptions and intermediate states encountered during autoregressive generation, and verify that the learned Gaussian distributions and likelihood-based gate remain valid under this shift?

  3. Please provide representative success and failure cases showing the input image, Vanilla and ResOT outputs, and the corrected or omitted object mentions. Does ResOT correct hallucinated content, or does it partly reduce hallucination by generating more conservative descriptions?

  4. Can the authors visualize the token-level likelihood gate and repair magnitude during generation? In particular, does the gate become stronger before or during hallucinated object generation while remaining small for faithful object mentions?

4 comments

We thank the reviewer for the positive assessment of our work and the constructive suggestions. We address your questions below.

W1 + Q2 + Q4. Calibration-to-inference distribution shift.

We acknowledge the reviewer’s concern about this potential distribution shift. We address this from two perspectives:

  • The successful use of similar calibration strategies in prior work [1–3] provides empirical support for this practice.
  • To assess its practical implications, we examine whether the calibrated repair signals remain selective on prefixes from freely generated captions. The following results suggest that the calibration-derived signals remain informative on generation prefixes despite potential representation shifts.

We also visualize the token-level likelihood gate and repair magnitude during generation. The intuitive example is illustrated in Appendix F, and the corresponding experimental results have been added to Appendix D.3.

Setting. We randomly sample 100 COCO images and generate captions using Vanilla. We use GPT-6 Sol, followed by manual verification, to identify hallucinated spans. Keeping the generated prefixes unchanged, we evaluate the likelihood gate and repair magnitude immediately before each span.

Hallucinated SpanOther Span
token positiongaterepair magnitudegaterepair magnitude
1–320.86403.5530.65301.612
33–640.85313.3880.66471.641
65–960.79593.2770.63121.445
97–2560.73153.0450.61401.199

Across all evaluated token-position ranges, hallucinated spans receive higher average gate values and repair magnitudes than other spans.

W2. Qualitative cases and token-level analysis

We thank the reviewer for the helpful suggestion. We address this issue in two ways:

  • We have added qualitative case studies in Appendix F.
  • We examine how ResOT affects object predictions. The following results suggest that ResOT improves visual grounding rather than simply suppressing object predictions. The corresponding experimental results have been added to Appendix D.4.

Setting. We use COCO ground-truth annotations as reference captions and identify grounded object words based on the object labels. Using fixed prefixes, such as "This image contains a ___ ", we examine the token probabilities assigned by Nullu and ResOT to these grounded objects and the top-5 absent objects with the highest co-occurrence rates in COCO. Compared with Nullu, ResOT assigns higher probabilities to grounded objects while more strongly suppressing absent ones.

Δlog⁡ppresent−Δlog⁡pabsent\Delta\log p_{\mathrm{present}}-\Delta\log p_{\mathrm{absent}}Δlog⁡ppresent\Delta\log p_{\mathrm{present}}Δlog⁡pabsent\Delta\log p_{\mathrm{absent}}
Vanilla000
Nullu+0.3477-0.2774-0.6251
ResOT+0.7988+0.0335-0.7653

Q1. Selection of intervention layers.

We apologize for any confusion.

  • We initially selected layers 16–31 based on prior empirical findings from Nullu [4] and MPD [5], which also adopt this intervention range for hallucination mitigation.
  • To validate this choice for ResOT, we conduct ablations comparing different intervention layers and ranges. The results show that layers 16–31 provide the best overall performance among the tested settings, balancing hallucination reduction and generation quality.
Layer RangeCHAIRsCHAIRiBLEUMME
16-1847.0013.460.18611792.71
16-2044.2012.970.19231806.44
16-2241.8012.870.19951809.61
16-2438.6011.890.20601786.60
16-2636.2011.490.21321798.08
16-2832.4010.610.21781814.45
16-3028.409.350.22771805.81
16-3129.209.530.23341819.02

Q3. Case study and Visualization

We thank the reviewer for this helpful suggestion.

  • We have added case study in Appendix F.
  • To examine whether ResOT reduces hallucinations by generating less informative descriptions, we use GPT-assisted evaluation of detailedness in open-ended generation (Table 3). Compared with Vanilla, ResOT shows a slight decrease in detailedness on LLaVA-1.5, comparable performance on mPLUG-Owl2, and a slight improvement on InstructBLIP. Notably, ResOT achieves higher detailedness scores than Nullu across all three backbones. These results suggest that ResOT reduces hallucinations without consistently sacrificing descriptive detail.

Finally, We thank the reviewer again for their time and thoughtful feedback. We hope our responses have addressed your concerns, and we would be happy to clarify any remaining questions!

[1]. Liu S, Ye H, Zou J Y. Reducing hallucinations in large vision-language models via latent space steering. ICLR, 2025.

[2]. Li Z, Shi H, Gao Y, et al. The hidden life of tokens: Reducing hallucination of large vision-language models via visual information steering. ICML, 2025.

[3]. Zou X, Kang J, Kesidis G, et al. Understanding and rectifying safety perception distortion in vlms. NeurIPS, 2025.

[4]. Yang L, Zheng Z, Chen B, et al. Nullu: Mitigating object hallucinations in large vision-language models via halluspace projection. CVPR, 2025.

[5]. Zhu X, Fang J, Wang S, et al. Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation. ACL, 2026.

I thank the authors for the detailed response and the additional experiments.

The new results largely address my concerns. The token-level analysis on freely generated captions shows that gate values and repair magnitudes remain clearly higher at hallucinated spans across all position ranges. The grounded vs. absent object comparison supports the claim that ResOT improves visual grounding rather than simply suppressing object predictions, and the layer-range ablation justifies the choice of layers 16-31. The case studies in Appendix F clearly illustrate the difference between ResOT and Vanilla, and the resulting captions are reasonable.

I maintain my original rating of 3.

Thank you for your time!

We are glad that our responses have addressed your concerns.

We sincerely appreciate your constructive feedback, which will help us improve our manuscript.

Reviewer_Qp5sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ResOT, a training-free method for reducing object hallucination in LVLMs by repairing hidden representations instead of suppressing hallucination-related components. The method builds a residual subspace from faithful and hallucinated representations, then uses Gaussian optimal transport to move hallucinated states toward the faithful distribution. A likelihood-based gate controls the strength of the correction during generation. The method is evaluated on LLaVA-1.5, mPLUG-Owl2, and InstructBLIP across several hallucination, captioning, and multimodal benchmarks. The reported results show lower hallucination while generally preserving or improving caption quality and multimodal performance. The paper also includes ablations on the repair subspace, suppression versus OT repair, the gating mechanism, calibration size, and robustness to noisy calibration data.

Critical strengths
  1. The main idea is interesting and well motivated. Instead of removing hallucination-related information, the method tries to repair the representation and move it closer to the faithful distribution. I think this is a reasonable and practical direction compared with simple suppression-based methods.

  2. The experimental evaluation is strong and covers different hallucination and multimodal benchmarks on three LVLMs. The ablations are also useful because they show the effect of the residual subspace, the repair operation, and the gating separately, which gives more confidence in the method.

Critical weaknesses
  1. I am not fully convinced about the novelty of the main repair operation. Gaussian OT and affine representation steering are already studied in previous work, so the main new part seems to be the residual subspace and how it is applied for hallucination mitigation. I think this distinction should be made more clearly.

  2. I also have some concerns about the calibration setup. The method learns the repair distribution from complete faithful and hallucinated captions, but applies it to intermediate hidden states during generation. Also, the two caption sets come from different sources, so part of the learned difference may be related to caption style or length rather than hallucination itself. These points make it less clear what exactly the method is correcting.

Questions for authors

Can you clarify the main technical difference between ResOT and previous activation-transport or affine-steering methods? In particular, which part of ResOT should be considered the main new contribution beyond using Gaussian OT for representation alignment?

The calibration distributions are estimated from complete captions, but the repair is applied at every decoding step. Can you provide more evidence that these intermediate hidden states are still well represented by the same faithful and hallucinated distributions?

Since the faithful and hallucinated captions come from different sources, can you clarify how much of the learned residual subspace may capture differences in caption style or length rather than hallucination itself?

Can you report the average output length and number of mentioned objects for Vanilla, Nullu, and ResOT? This would help clarify whether part of the hallucination reduction is coming from shorter or more conservative outputs.

How sensitive are the results to the in-domain hyperparameter selection on COCO? Do the main conclusions still hold if the parameters are fixed or selected on a different validation set?

I am currently between 2 and 3, and I would raise my score to 3 (Weak acceptance) if the authors can satisfactorily address the novelty compared with the closest prior work and provide convincing evidence for the calibration/inference mismatch and possible caption-style confounding.

Suggestion: Evaluation on at least one recent LVLM backbone would further strengthen the generalization claim.

4 comments

Thank you for your positive feedback on our motivation and approach. We appreciate your constructive comments, which will help us improve our paper. Our responses are provided below.

W1 + Q1. Clarification of ResOT's contributions beyond Gaussian OT and affine steering methods.

We thank the reviewer for highlighting the connections to prior work on affine steering and activation transport [1, 2]. We agree that Gaussian OT and affine distribution alignment have been studied in prior work. Our main contribution is not to introduce a new OT formulation, but a repair-based approach to representation intervention for hallucination mitigation.

Our motivation stems from the observation that hallucination-related and faithful information can be entangled in representations. Directly suppressing these components in the original representation space may discard useful information for visual understanding.

Technically, ResOT is not simply applying an existing OT map to the original representations. It follows a locate-then-repair principle:

  • Where to repair: ResOT confines intervention to a targeted residual subspace to limit disruption to useful information elsewhere in the representation.
  • How to repair: We use the faithful distribution as the repair target and apply optimal transport to align the distributions with minimum displacement. This allows us to correct hallucination-related deviations while preserving useful information.
  • How to apply the repair: During autoregressive generation, a likelihood-based gate adaptively applies stronger corrections to more hallucination-prone states.

The broader evaluations also support our central claim: ResOT can reduce hallucination while better preserving the model's broader multimodal capabilities than suppression-based methods.

We have added a discussion of [1, 2] to clarify how ResOT differs from these methods (lines 149–154) and revised the Introduction to better explain our motivation and the rationale behind ResOT’s design (lines 38-42, 49-52).

W2 + Q2 + Q3. Calibration-to-inference distribution shift and potential confounding from caption style or length

We thank the reviewer for these insightful comments. We address the two concerns separately.

Part 1: Calibration-to-inference distribution shift

We acknowledge the reviewer’s concern about this potential distribution shift. We address this from two perspectives:

  • The successful use of similar calibration strategies in prior work [3–5] provides empirical support for this practice.
  • To assess its practical implications, we examine whether the calibrated repair signals remain selective on prefixes from freely generated captions. The following results suggest that the calibration-derived signals remain informative on generation prefixes despite potential representation shifts.

Setting. We randomly sample 100 COCO images and generate captions using Vanilla. We use GPT-6 Sol, followed by manual verification, to identify hallucinated spans. Keeping the generated prefixes unchanged, we evaluate the likelihood gate and repair magnitude immediately before each span. (An illustrative example is provided in Appendix F, and the corresponding experimental results have been added to Appendix D.3).

Hallucinated SpanOther Span
token positiongaterepair magnitudegaterepair magnitude
1–320.86403.5530.65301.612
33–640.85313.3880.66471.641
65–960.79593.2770.63121.445
97–2560.73153.0450.61401.199

Across all evaluated token-position ranges, hallucinated spans receive higher average gate values and repair magnitudes than other spans.

Part 2: Calibration data and caption-style confounding

We thank the reviewer for this constructive suggestion. We address this from two perspectives:

  • We initially adopted the same calibration dataset (LURE) as Nullu [6] to ensure a fair comparison.
  • To further assess whether the gains depend on differences in caption source, style, or length, we additionally calibrated both ResOT and Nullu using DetailVerifyBench [7]. The following results show that ResOT remains effective, suggesting that its improvements do not rely solely on differences in caption sources, styles, or lengths.

DetailVerifyBench: Its paired faithful and hallucinated descriptions are constructed from the same underlying captions through controlled hallucination injection, with nearly identical average lengths of 310.27 and 310.15 tokens, respectively. This setting reduces potential confounding from caption source, style, and length.

We re-selected ResOT’s hyperparameters on DetailVerifyBench using the same selection procedure as in our original experiments.

CHAIRsCHAIRiBLEU-1MMEPOPE F1
LLaVA-v1.50.5140.14510.17391785.5484.67
Nullu0.4940.14250.16941686.2382.63
ResOT0.4240.12330.20351806.3785.86

Q4. Average output length and object coverage.

We apologize for any confusion. Our results suggest that ResOT achieves a better balance between hallucination reduction and grounded object coverage than Nullu. To compare information retention at comparable hallucination levels, we adjust ResOT's repair strength α\alpha to match Nullu's CHAIRs as closely as possible. ResOT achieves higher object recall across all three backbones, while also obtaining lower CHAIRi. We also observe that increasing the repair strength can reduce object recall. This may be because the repair signals are estimated from finite calibration data, so stronger corrections may also affect grounded object predictions.

MethodCHAIRsCHAIRiRecallAvg length
LLaVA-v1.5Vanilla0.5140.145178.56%90.1
Nullu0.3940.115075.08%101.1
ResOT0.3940.113177.47%85.4
mplug-Owl2Vanilla0.5680.163977.51%92.2
Nullu0.4120.139574.48%86.3
ResOT0.4140.123377.24%87.3
InstructBLIPVanilla0.4620.131074.08%92.0
Nullu0.4400.142973.49%79.0
ResOT0.4400.126074.21%90.2

Q5. Sensitivity to hyperparameter selection on COCO.

We thank the reviewer for the helpful suggestion. We address this from two perspectives:

  • MMHal-Bench uses images not from COCO. With hyperparameters selected on COCO, ResOT improves the average score and reduces the hallucination rate compared to Vanilla across all three backbones (Table 7). This shows that the improvements extend beyond COCO.
  • We further use DetailVerifyBench for calibration and hyperparameter selection (see W2+Q2+Q3). ResOT still outperforms Vanilla and Nullu across all reported metrics. Together, these results support its effectiveness across different calibration and evaluation data sources.

Suggestion: More recent LVLM backbone

We thank the reviewer for the helpful suggestion. We conduct ResOT on InternVL2.5-8B and Qwen3-VL-8B. The results likewise demonstrate the effectiveness of ResOT on these two LVLMs.

POPE (F1 score)CHAIRMME
RandomPopularAdversarialCHAIRsCHAIRiRecallBLEUPerceptionCognition
InternVL2.5-8B90.9788.7488.1131.407.21969.210.30731681.83650.00
+ ResOT92.3190.0289.1226.806.21869.270.31581693.48660.36
Qwen3-VL-8B90.5588.1486.3642.809.58665.880.14761738.18637.86
+ ResOT91.4288.7486.7935.607.91365.770.14891743.74635.00

Finally, We thank the reviewer again for their time and thoughtful feedback. We hope our responses have addressed your concerns, and we would be happy to clarify any remaining questions!

[1]. Rodriguez P, Blaas A, Klein M, et al. Controlling language and diffusion models by transporting activations. ICLR, 2025.

[2]. Singh S, Ravfogel S, Herzig J, et al. Representation surgery: Theory and practice of affine steering. ICML, 2024.

[3]. Liu S, Ye H, Zou J Y. Reducing hallucinations in large vision-language models via latent space steering. ICLR, 2025.

[4]. Li Z, Shi H, Gao Y, et al. The hidden life of tokens: Reducing hallucination of large vision-language models via visual information steering. ICML, 2025.

[5]. Zou X, Kang J, Kesidis G, et al. Understanding and rectifying safety perception distortion in vlms. NeurIPS, 2025.

[6]. Yang L, Zheng Z, Chen B, et al. Nullu: Mitigating object hallucinations in large vision-language models via halluspace projection. CVPR, 2025.

[7]. Wang X, Zhang Y, Zhang X, et al. DetailVerifyBench: A Benchmark for Dense Hallucination Localization in Long Image Captions. arXiv:2604.05623, 2026.

We recognize that your time is valuable. As the discussion period draws to a close, we would be grateful to know whether our responses address your concerns, and whether anything would benefit from further clarification.

Thank you for your time and consideration!

Comments on the paper

1 comment
AuthorsOpenReview ↗

AC Summary

Dear reviewers and ACs,

We are sincerely grateful for the time and insightful feedback provided throughout this process.

Below, we have outlined the key points from the rebuttal process to help facilitate your decision-making.

Strengths Highlighted by the Reviewers

We are encouraged by the reviewers’ recognition of our work:

  1. Strong motivation: The idea is "interesting and well motivated" and represents "a reasonable and practical direction" (Qp5s), and the repair formulation is well motivated (YGUa).
  2. Well-Supported Method Design: The method provides “a coherent alternative to suppression-based intervention” (fATz), with benefits from both residual space localization and the repair operation (YGUa). The ablation study “gives more confidence in the method” (Qp5s).
  3. Strong Empirical Results: The evaluation is strong and broad (Qp5s, fATz), providing substantial evidence of hallucination reduction and capability preservation (YGUa).
  4. Clear Presentation: All reviewers found the paper “Clear and easy to follow” (Qp5s, fATz, YGUa).

Post-Rebuttal Discussion Status

  1. YGUa and fATz responded positively to our rebuttal and maintained 3 (Weak Acceptance).
  2. Qp5s initially stated that they were “between 2 and 3” and would raise their score to 3 if their concerns were satisfactorily addressed, but they have not yet responded to our rebuttal. We believe our clarifications and additional results substantially addressed their concerns.

Summary of Reviewer Concerns and Our Responses

ConcernReviewerResponse
Calibration-to-inference mismatchQp5s, fATz, YGUaPrefix replay analysis shows ResOT’s selectivity during free generation, as acknowledged by fATz and YGUa. (Appendix D.3)
Grounding and information preservationQp5s, fATzResOT achieves higher object recall at comparable hallucination levels. Object-level analysis and qualitative cases further support improved grounding, as acknowledged by fATz. (Appendix F, Appendix D.4)
Comparison robustness and generalizationQp5s, YGUaResOT remains effective across different calibration sets (DetailVerifyBench) and recent LVLMs (InternVL2.5-8B and Qwen3-VL-8B). Additional experiments support the robustness of our comparisons, as acknowledged by YGUa.
Intervention settings and implementationfATz, YGUaWe justified the intervention settings and clarified the implementation details, as acknowledged by fATz and YGUa (Implementation details: lines 891-901).
Novelty relative to prior transport methodsQp5sWe clarified how ResOT differs from prior methods and demonstrated its benefits in reducing hallucinations while preserving useful information. (lines 38-42, 49-52, 149-154)
Calibration data confoundingQp5sResOT remains effective on DetailVerifyBench, whose matched caption pairs reduce source, style, and length confounding.

Paper Summary

Our work addresses object hallucination in LVLMs. We show that suppression can disrupt faithful representations and degrade the model’s capabilities. We propose ResOT to expose and correct residual distributional mismatches while leaving components in the faithful subspace unchanged.

  • Analysis: Explains the limitations of suppression and provides theoretical support for our method design.
  • Effectiveness: Reduces hallucinations across models and benchmarks while preserving caption quality and broader multimodal capabilities (10 benchmarks and 3 representative LVLMs).
  • Efficiency: Requires no parameter updates and retains 81.24% of the original throughput, with only 8.15 MiB of additional memory.

During the remaining response period, please feel free to let us know if any further clarification or additional material would be helpful. We would be more than willing to respond promptly!

With best wishes,

Authors of Paper 11036

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.