acceptodds
Under review as a conference paper at ICLR 2027

Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations

Abstract

Hallucination remains a significant challenge in Large Vision-Language Models (LVLMs). Existing training-free methods generally mitigate hallucinations through contrastive decoding or visual enhancement, often increasing the relative influence of visual evidence during generation. This raises a fundamental question: Can LVLMs dynamically regulate the contributions of different context sources to suppress hallucinations? In this work, we investigate and quantify how LVLMs coordinate multiple context sources during decoding and examine how this intrinsic behavior can guide hallucination mitigation. We find that LVLMs exhibit an intrinsic vision-attending tendency that can guide adaptive visual steering, while textual contexts can also contribute to hallucination mitigation. Motivated by these findings, we propose AIMS (Adaptive Information Multi-source Steering), a lightweight training-free framework that adaptively coordinates visual, prefilled textual, and generated contexts during decoding. Specifically, AIMS constructs compact prototypes for the three context domains and estimates their affinities with the current query to determine head-wise steering weights. The resulting multi-source steering direction is applied to the query representation, enabling adaptive context integration without additional model training or auxiliary forward passes. Extensive experiments across multiple LVLMs and decoding strategies demonstrate that AIMS effectively mitigates object hallucination while maintaining competitive general-purpose multimodal capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LVLM, Hallucination

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3rrFOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how LVLMs utilize multiple context sources (image, prompt, and generation history) during generation to mitigate object hallucinations. The authors identify three key observations: (1) visual reliance peaks when generating object-related content in the early stage, (2) adaptive visual steering is preferable to fixed amplification, and (3) textual contexts provide complementary corrective signals. Based on these findings, they propose AIMS, a training-free method that adaptively steers attention queries using visual, prefilled-text, and generated-text contexts to mitigate hallucinations. Experiments across multiple models and benchmarks demonstrate its effectiveness in reducing hallucinations.

Critical strengths
  • The paper provides empirical evidence on how visual attention evolves during generation and how attention intervention can be designed more effectively for hallucination mitigation, yielding three useful observations. These findings naturally motivate a practical, training-free solution for mitigating object hallucinations in LVLMs.

  • The proposed AIMS is evaluated comprehensively across representative models, decoding strategies, and benchmarks, with results consistently supporting its effectiveness.

Critical weaknesses
  • While the paper presents useful findings that motivate the proposed method, these findings are validated only on Qwen2.5-VL-3B, raising concerns about their reliability.

  • The paper lacks ablations comparing adaptive single- and multi-context steering. Since prefilled texts (P) may also contain misleading information, such as POPE queries [R1] that explicitly mention absent objects, and generated text (G) may already contain hallucinations, it remains unclear whether incorporating P and G consistently improves over Adaptive-V.

[R1] Li Y, Du Y, Zhou K, et al. Evaluating object hallucination in large vision-language models. EMNLP, 2023.

Questions for authors

Please refer to the Critical Weaknesses section.

3 comments

We sincerely thank the reviewer for the thoughtful and constructive feedback, as well as for recognizing the value of our empirical findings and proposed method. We address each concern below.

W1: Generalizability of the empirical observations.

We thank the reviewer for this concern. To validate the generalizability of our observations, we conducted additional experiments on Qwen3.5-9B using the same 500 images. We divided generated sequences into five length-based bins with approximately equal sample counts and observed consistent visual attention trends across decoding steps, similar to Qwen2.5-VL-3B. These results further support the generalizability of our observations across LVLMs. We will include the corresponding curves in the revised manuscript.

W2: Potential risks of incorporating P/G.

We thank the reviewer for this concern. Although P/G may contain misleading information, they also provide useful context for instruction understanding and coherent generation. AIMS therefore adaptively weights V/P/G rather than uniformly enhancing textual information.

To investigate potential hallucination propagation through generated context, we analyzed 500 CHAIR responses generated by Qwen2.5-VL-3B under greedy decoding, including 222 containing hallucinations.

MetricDefinitionValue
First hallucination positionE[tfirst/L]\mathbb{E}[t_{\mathrm{first}}/L]36.80%
Post-hallucination mention rateE[Nhall/Nobj∣Nobj>0]\mathbb{E}[N_{\mathrm{hall}}/N_{\mathrm{obj}}\mid N_{\mathrm{obj}}>0]15.40%
Subsequent hallucinated mentionsE[Nhall]\mathbb{E}[N_{\mathrm{hall}}]1.20
New hallucinated mentionsE[Nnewmention]\mathbb{E}[N_{\mathrm{new_mention}}]0.41
New hallucinated categoriesE[Nnew]\mathbb{E}[N_{\mathrm{new}}]0.27

Here, NhallN_{\mathrm{hall}} and NobjN_{\mathrm{obj}} denote hallucinated and total object mentions after the first hallucination, respectively. NnewmentionN_{\mathrm{new_mention}} counts hallucinated mentions belonging to categories other than the first hallucinated category, while NnewN_{\mathrm{new}} counts distinct new hallucinated categories. The second metric is computed over 209 descriptions with subsequent object mentions, while the other metrics are computed over all 222 hallucinated descriptions.

Although the first hallucination occurs relatively early (36.8% of generation), the average hallucination rate among subsequent object mentions is only 15.4%. Moreover, of the 1.20 subsequent hallucinated mentions per description, only 0.41 involve new categories, corresponding to 0.27 distinct new categories. These results suggest that subsequent hallucinations mainly repeat existing errors rather than continuously introduce new object categories.

Moreover, Figure 5 directly compares adaptive single- and multi-context steering, showing that full AIMS achieves lower hallucination rates and higher F1 than V-only. Together, these results support the benefits of adaptively incorporating P/G despite their potential imperfections.

I appreciate the authors' detailed and timely response.

W1: Validation on a different VLM family (e.g., InternVL) would be ideal for demonstrating generality. That said, the additional Qwen3.5-9B results are acceptable.

W2: My concern has been addressed. I recommend incorporating the relevant discussion into the manuscript to better support the proposed method's effectiveness.

Overall, I will keep my positive score (3).

We sincerely thank the reviewer for the positive assessment and constructive feedback. We will incorporate the additional experimental results and relevant discussions into the revised manuscript. We also plan to further extend our implementation framework to support a broader range of LVLM architectures and facilitate cross-family evaluation in future work.

We appreciate the reviewer's thoughtful suggestions, which have helped us strengthen our work.

Reviewer_z15eOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AIMS, a training-free attention-steering method for reducing hallucinations in large vision-language models. AIMS combines visual context, prefilling text, and previously generated text, then adaptively weights these sources during decoding based on their query-dependent affinities. The authors evaluate the method on LLaVA and Qwen models across several hallucination benchmarks and report improvements over existing baselines, although the gains are not consistent across datasets and metrics.

Critical strengths
  • The method provides a training-free way to combine visual, prefilling-text, and previously generated context during decoding.
  • The online steering mechanism does not require recomputing the model for each context source, which supports practical inference.
  • The evaluation covers multiple model families and hallucination benchmarks.
  • Results generally improve on AMBER and FaithScore, with smaller gains on MME.
  • The paper reports latency and memory overhead, helping assess the practical cost of the method.
Critical weaknesses
  • W1: The paper does not clearly position AIMS relative to prior training-free steering methods. The novelty of combining visual and textual signals, and the specific benefit of the adaptive formulation, are not sufficiently established.
  • W2: The evaluation protocol does not provide enough detail on hyperparameter selection across decoding strategies and datasets, including whether separate development data was used. This makes the reported improvements difficult to interpret.
  • W3: Improvements are inconsistent or modest across benchmarks, while the method adds latency and nontrivial memory overhead. The paper does not show that these costs are justified, particularly on stronger models.
Questions for authors
  1. How does AIMS differ from prior training-free steering methods such as VTI (https://arxiv.org/abs/2410.15778) and GIFT (https://arxiv.org/abs/2510.22067)? Please clarify the specific contribution of the adaptive multi-context formulation and explain why GIFT is not included in the comparison.
  2. How were the steering and interpolation parameters selected for each model, decoding strategy, and benchmark? Were separate development splits used, or were any test results used for tuning?
  3. What is the additional inference and memory cost of AIMS compared with the baseline, and how does this overhead affect throughput at practical sequence lengths?
  4. Have you evaluated AIMS on a stronger LVLM with lower baseline hallucination rates? If so, does it still provide meaningful improvements?
3 comments

We thank the reviewer for the constructive feedback and address each concern below.

W1: Novelty and adaptive weighting.

1. Novelty. The novelty of AIMS lies not in simply combining V/P/G, but in leveraging the model's own evolving information-attending tendencies to mitigate hallucinations.

Observation 1 shows that visual attention is stronger during object description and decreases during elaboration and summarization. Our additional analysis reveals complementary trends: attention to P decreases, while attention to G increases. These findings suggest meaningful, though imperfect, preferences for different context domains throughout generation.

VTI derives steering directions from hallucinated and non-hallucinated examples in a separate calibration set, while VCD/SID construct negative counterparts for contrastive decoding. In contrast, AIMS derives its intervention signal from the LVLM's own evolving preferences for different context domains. Query affinities with V/P/G dynamically determine the steering direction and context contributions at each decoding step, without external calibration examples or constructed negative counterparts.

2. Adaptive weighting. We compare AIMS with Fixed-strength steering on Qwen2.5-VL-3B under greedy decoding. AIMS uses (τV,τP,τG)=(1.5,1.2,1.1)(\tau_V,\tau_P,\tau_G)=(1.5,1.2,1.1), while Fixed assigns unit weights to all branches.

α\alphaAIMS CSC_S ↓Recall ↑Fixed CSC_S ↓Recall ↑
0.0249.874.8052.874.75
0.0449.574.3051.073.35
0.0645.773.3045.270.81
0.0840.871.6925.460.15

AIMS achieves lower hallucination rates at comparable Recall and higher Recall at comparable hallucination rates. Together with the branch ablation in Figure 5, these results demonstrate the benefits of both adaptive weighting and multi-context steering over single-context steering. We will include the extended experimental results in the revised manuscript.

W2: Hyperparameter selection.

We primarily use shared configurations across benchmarks and decoding strategies, rather than extensive setting-specific tuning.

ModelExperimentsα\alpha(τV,τP,τG)(\tau_V,\tau_P,\tau_G)
LLaVA-1.5-7BAll0.05(1.5,1.2,1.1)
Qwen2.5-VL-3BMME0.08(1.0,1.0,1.0)
Qwen2.5-VL-3BOthers0.08(1.5,1.2,1.1)
Qwen3.5-9BMME0.30(1.0,1.0,1.0)
Qwen3.5-9BOthers0.40(1.5,1.2,1.1)

No separate development split or extensive hyperparameter tuning was used. We thank the reviewer again for pointing this out and will clarify the hyperparameter configurations and evaluation protocol in the revision.

W3: Performance–efficiency trade-off.

Table 9 shows negligible memory overhead (+0.24 MB, approximately 0.003%) on Qwen2.5-VL-3B. AIMS (0.0705 s/token) is comparable to MemVR (0.0698) and faster than VCD (0.0905), SID (0.1075), and VTI (0.2424), without additional forward passes.

We additionally evaluate efficiency on Qwen3.5-9B under greedy decoding.

MethodPeak Mem. (GB) ↓Time (s/token) ↓
Baseline17.79370.004493
AIMS17.79380.004709

AIMS incurs only a 4.81% increase in per-token latency with negligible memory overhead on Qwen3.5-9B, while reducing hallucinations across CHAIR and AMBER-G (e.g., CHAIR CSC_S: 47.4 to 40.0). This supports a favorable performance–efficiency trade-off on stronger LVLMs.

Q1: Comparison with VTI and GIFT.

GIFT identifies salient visual regions during prefill to guide decoding, while AIMS determines which context domains (V/P/G) to emphasize and by how much at each decoding step. The key novelty of AIMS is leveraging the LVLM's own evolving context preferences to dynamically coordinate multiple information sources, rather than relying on visual guidance derived during prefill.

We additionally evaluate GIFT on the same CHAIR data under greedy decoding, following its reported setting.

ModelMethodCSC_S ↓CIC_I ↓Recall ↑F1 ↑
Qwen2.5-VL-3BGIFT44.69.9071.2174.05
AIMS40.89.0771.6974.66
Qwen3.5-9BGIFT43.210.1069.6773.96
AIMS40.09.9167.0573.38

AIMS reduces hallucination rates on both models, with higher Recall/F1 on Qwen2.5-VL-3B, while GIFT retains higher Recall/F1 on Qwen3.5-9B. We will include these comparisons and the relevant citation in the revision.

Q2: Parameter selection.

We use largely shared configurations without a separate development split. Please refer to W2 for details.

Q3: Inference cost and throughput.

On Qwen2.5-VL-3B, AIMS achieves 14.17 tokens/s vs. 22.83 for Baseline, while outperforming most mitigation methods in inference speed. On Qwen3.5-9B, throughput decreases by only 4.59%. Please refer to W3 for detail.

Q4: Stronger LVLMs.

We have added Qwen3.8-27B results in our next comment. Please refer to it for details.

Q4: Stronger LVLMs.

We thank the reviewer for this insightful suggestion. We additionally evaluate AIMS on Qwen3.8-27B under greedy decoding, using α=0.3\alpha=0.3 and (τV,τP,τG)=(1.5,1.2,1.1)(\tau_V,\tau_P,\tau_G)=(1.5,1.2,1.1).

MethodCSC_S ↓CIC_I ↓Recall ↑F1 ↑Len
Baseline54.211.5674.9474.84258.91
AIMS51.611.2274.8175.17263.70

AIMS reduces both CSC_S and CIC_I while maintaining comparable Recall and improving F1 on Qwen3.8-27B. Notably, these improvements are achieved with slightly longer responses, demonstrating that AIMS remains effective on stronger LVLMs without sacrificing information coverage.

We sincerely thank the reviewer again for the constructive feedback, which has helped us strengthen our experimental evaluation and clarify the contributions of AIMS. We hope our responses and additional experiments address the reviewer's concerns.

Thank you again for your constructive feedback. As the discussion period is approaching its end, we would be grateful if you could let us know whether our responses and additional experiments have addressed your main concerns, or if any points still require clarification. We would be happy to provide further clarification promptly.

Reviewer_MkMCOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the object hallucination problem in large vision-language models (LVLMs) and proposes a training-free adaptive multi-context regulation framework, AIMS. Its core idea is to dynamically coordinate vision, prefilled text, and generation history according to the current generation state, rather than merely enhancing visual information. Specifically, AIMS leverages the existing Key Cache to construct three types of context prototypes, computes regulation weights that vary with attention heads and decoding steps according to the affinity between the current Query and each prototype, and fuses the weighted prototypes into a regulation target. After updating the Query, it performs the original attention computation. The entire process requires no additional training or auxiliary forward passes.

Critical strengths

1. The hallucination suppression results are consistent across models and decoding strategies and are supported by supplementary evaluations.

AIMS achieves the lowest CHAIRs\mathrm{CHAIR}_{s} in the table across all nine combinations of three models and three decoding methods in Table 2, indicating that its advantage is not limited to a single backbone or a specific decoding method. More importantly, on AMBER-G, AIMS achieves lower CHAIR and Hal than VTI in all six settings while maintaining comparable object coverage.

2. The method directly reuses the existing cache and requires no additional training or auxiliary forward passes, giving it practical value.

AIMS constructs context prototypes from the existing Key Cache, uses the affinity between the current Query and the prototypes to achieve adaptive regulation at the attention-head and decoding-step levels, and then continues with the original attention computation. This design requires no additional training or auxiliary model forward passes, and Table 9 shows that its additional peak memory overhead is small. Its value lies in providing a structurally simple multi-context intervention scheme usable at inference time, rather than relying on more complex training pipelines or additional model calls.

Critical weaknesses

1. The independent contribution of adaptive weighting still lacks sufficient controls on intervention strength.

The branch ablation in Figure 5 provides support for multi-context combination, but the necessity of adaptive weighting itself has not yet been sufficiently verified. Table 7 only compares fixed interventions with α = 0.01 and α = 0.1 against the adaptive method with α = 0.08: the former exhibit weak intervention and obvious quality degradation, respectively, and do not cover fixed strengths between them that might achieve a better trade-off. Therefore, the existing results are insufficient to distinguish whether the gains come from weight allocation that varies with the generation state or from a more appropriate effective intervention magnitude.

2. The decrease in hallucination is accompanied by changes in information coverage, and a better factuality–completeness trade-off has not yet been fully demonstrated.

In Table 2, under the Greedy setting of Qwen2.5-VL-3B, CHAIRₛ decreases from 56.0 to 40.8, but the average generation length also decreases from 404.2 to 284.2, and Recall decreases from 74.05 to 71.69. Under the Greedy setting of Qwen3.5-9B, CHAIRₛ decreases from 47.4 to 40.0, while Recall decreases from 74.11 to 67.05 and F1 decreases from 75.87 to 73.38. These results indicate that improvements in the main hallucination metrics are accompanied by the loss of some correct object information, and lower CHAIRₛ alone cannot be used to judge that overall generation quality is better.

The improvement over VTI on AMBER-G at comparable coverage is valuable positive evidence, but it does not yet fully explain the trade-offs in the above CHAIR settings. Therefore, it is still necessary to verify, under comparable information coverage or output length, whether the hallucination suppression advantage of the method is retained.

Questions for authors

1. Clarification of experimental configurations: How do the visual single-branch and the full method in Table 1, Table 6, and Figure 5 correspond to each other?

Adaptive-V in Table 1 and AIMS_NCS in Table 6 have entirely identical row results under Greedy at the shared displayed precision, but the former is described as an adaptive intervention using only the visual branch, whereas the latter is listed as an affinity-function variant of AIMS. Please explain this correspondence and clarify the context branches, affinity functions, and intervention strengths actually used by the two and by the V-only configuration in Figure 5. After controlling for other settings, can these results be used to determine the incremental benefit of adding P and G?

2. Validation of the adaptive mechanism: Is a fully tuned fixed multi-source intervention still weaker than AIMS?

For Table 7, could you add results for fixed multi-source intervention between α = 0.01 and α = 0.1, and clarify whether the fixed and adaptive methods use the same validation set and tuning budget? At comparable Recall or F1, does AIMS still achieve a lower hallucination rate?

3. Validation of the performance trade-off: Do the main hallucination gains still hold after controlling for information coverage?

Could you provide a CHAIRₛ–Recall trade-off curve as the intervention strength varies, or a comparison under a unified output-length constraint, and report image-level paired confidence intervals for the key differences?

1 comment

We thank the reviewer for the constructive feedback and for recognizing the effectiveness and practical value of AIMS.

Weakness 1: Insufficient controls for adaptive weighting.

We extend Table 7 with a finer-grained sweep of fixed steering strengths and an approximately strength-matched setting (α=0.046\alpha=0.046).

Methodα\alphaCSC_S ↓CIC_I ↓F1 ↑
Fixed0.0154.010.1174.45
Fixed0.0252.89.3075.44
Fixed0.0451.09.5474.89
Fixed0.04648.68.9075.16
Fixed0.0645.28.7074.45
Fixed0.0825.48.5870.72
Fixed0.108.04.6456.90
AIMS0.0840.89.0774.66

Fixed steering is highly sensitive to α\alpha: stronger intervention reduces hallucination but substantially degrades F1. At comparable F1, AIMS achieves lower CSC_S than Fixed (α=0.06\alpha=0.06): 40.8 vs. 45.2.

To control for intervention strength, the average affinity weight across V/P/G in AIMS is approximately 0.57 (vs. 1 for Fixed). Thus, its average affinity-scaled coefficient is 0.08×0.57≈0.0460.08\times0.57\approx0.046. Under this approximately matched coefficient, AIMS achieves lower CSC_S (40.8 vs. 48.6) with comparable F1 (74.66 vs. 75.16). This supports the benefit of adaptive weighting beyond selecting a suitable fixed strength.

Weakness 2: Factuality–completeness trade-off.

Table 2 shows that several mitigation methods generate substantially longer responses on Qwen3.5-9B. However, longer outputs may introduce unsupported details. Our additional case study during the rebuttal illustrates this issue: given an image of a person in a snowy landscape, Baseline and AIMS mainly describe visible elements, whereas SID and MemVR speculate about a polar expedition, emotions, and resilience.

We further evaluate PAI and SVAR, whose response lengths are closer to AIMS.

Qwen3.5-9B (Greedy)CSC_S ↓CIC_I ↓Recall ↑Len
PAI47.810.4969.21229.34
SVAR59.210.3170.44240.92
AIMS40.09.9167.05249.90

Despite slightly longer outputs, AIMS achieves lower CSC_S and CIC_I than both methods, with modestly lower Recall. Its hallucination reduction cannot be simply attributed to shorter outputs.

Q1: Clarification of single-branch and full AIMS configurations.

We thank the reviewer for identifying this error. Adaptive-V in Table 1 was mistakenly copied and should match V-only in Figure 5. AIMS_NCS in Table 6 is correct.

MethodCSC_S ↓CIC_I ↓Recall ↑F1 ↑Len
Adaptive-V48.69.6772.0274.28323.80

Full AIMS achieves lower hallucination rates and higher F1, supporting the benefit of incorporating P and G. We apologize and will correct Table 1.

Q2: Adaptive vs. fully tuned fixed steering.

Both methods are evaluated on the same 500 COCO images under identical experimental settings, without a separate validation split. We conduct a finer-grained sweep for Fixed, including an approximately strength-matched comparison, while keeping AIMS's sensitivity parameters fixed. The results support the benefit of adaptive weighting. Please refer to Weakness 1 for details.

Q3: Hallucination reduction at comparable information coverage.

1. CHAIR$_S$–Recall trade-off. We evaluate AIMS with varying α\alpha and fixed (τV,τP,τG)=(1.5,1.2,1.1)(\tau_V,\tau_P,\tau_G)=(1.5,1.2,1.1) on Qwen2.5-VL-3B under greedy decoding.

The trade-off curve consists of the following (CSC_S, Recall) points: (39.0, 70.30), (40.8, 71.69), (45.7, 73.30), (49.5, 74.30), and (49.8, 74.80).

We will include the corresponding trade-off curve in the revised manuscript.

2. Image-level paired confidence intervals. We evaluate Baseline and AIMS (α=0.08\alpha=0.08) on the same 500 COCO images with identical prompts, Qwen2.5VL-3B, greedy decoding, and a 512-token limit. We perform 10,000 paired bootstrap resamples, sampling 500 image IDs with replacement and using identical IDs for both methods. For each metric $M\in \lbrace C_S,\mathrm{Recall} \rbrace $:

ΔM(b)=MAIMS(b)−MBaseline(b).\Delta M^{(b)}=M_{\mathrm{AIMS}}^{(b)}-M_{\mathrm{Baseline}}^{(b)}.

The 95% CIs are the 2.5th and 97.5th percentiles of the paired differences.

MetricBaselineAIMSΔ\Delta (pp)95% CI
CSC_S56.040.8-15.20[-19.3,-9.0]
Recall74.0571.69-2.36[-3.95,-0.7]

These results confirm a statistically significant hallucination reduction with a moderate Recall decrease. The trade-off analysis further demonstrates hallucination mitigation at comparable information coverage.

Comments on the paper

1 comment

Dear Area Chair,

Thank you very much for your time and effort in handling our manuscript. As the rebuttal phase draws to a close, we would like to summarize the key points raised by the three reviewers and our corresponding responses to facilitate a quick overview of the review progress.

Strengths

  • Reviewer MkMC recognized the general applicability and generalizability of AIMS, highlighting its lightweight design and potential for practical deployment.
  • Reviewer z15e appreciated the extensive experimental evaluation and the practical relevance of the reported metrics.
  • Reviewer 3rrF acknowledged that our empirical observations naturally motivate a practical, training-free approach to mitigating object hallucinations in LVLMs, and appreciated the comprehensiveness of our experiments.

Notably, all three reviewers indicated that they learned something interesting or useful from our work.

Weaknesses & Questions

  • [MkMC] Effectiveness of adaptive steering: The reviewer questioned whether AIMS genuinely benefits from adaptive weighting rather than a well-tuned fixed steering strength. We conducted finer-grained fixed-strength sweeps and approximately strength-matched comparisons, demonstrating that AIMS achieves lower hallucination rates with comparable F1.

  • [MkMC] Factuality–completeness trade-off: The reviewer raised concerns that hallucination reduction might come at the cost of information coverage. We supplemented CHAIR–Recall trade-off analyses, paired bootstrap confidence intervals, and comparisons with methods producing similar response lengths, confirming substantial hallucination reduction while explicitly acknowledging the moderate Recall decrease.

  • [z15e] Motivation and distinction from VTI/GIFT: We clarified that AIMS is motivated by the LVLM's evolving attention preferences across different context domains, distinguishing it from VTI's calibration-based steering and GIFT's prefill-derived visual guidance. We further provided fixed-weight comparisons and additional GIFT experiments to demonstrate the effectiveness of adaptive multi-context steering.

  • [z15e] Hyperparameter selection: The reviewer raised concerns about parameter tuning and reproducibility. We clarified that AIMS largely uses shared configurations across benchmarks and decoding strategies, without extensive setting-specific tuning or a separate development split.

  • [z15e] Performance–efficiency trade-off: The reviewer questioned the additional inference cost. We provided latency, throughput, and memory analyses on Qwen2.5-VL-3B and Qwen3.5-9B, demonstrating competitive efficiency with negligible memory overhead.

  • [z15e] Generalization to stronger LVLMs: The reviewer requested evaluation on stronger models. We added experiments on Qwen3.8-27B, showing reduced hallucination rates with comparable Recall and improved F1.

  • [3rrF] Generalizability of empirical observations: The reviewer questioned whether the observed visual attention patterns generalize across LVLMs. We conducted additional experiments on Qwen3.5-9B, confirming consistent visual attention trends across decoding steps.

  • [3rrF] Potential risks of incorporating textual context: The reviewer raised concerns about misleading information in prefilled and generated contexts. We provided a statistical analysis of hallucination propagation across 500 CHAIR responses and additional evidence from branch ablations, supporting the effectiveness of adaptively incorporating P/G.

We sincerely thank you and all three reviewers again for your time and constructive feedback. We believe that the additional experiments and clarifications have strengthened our manuscript and addressed the major concerns raised during the review process. We will incorporate these results and clarifications into the revised manuscript.

Should any further questions or concerns arise before the discussion deadline, we would be happy to respond promptly.

Best regards,
The AIMS Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.