acceptodds
Withdrawn from ICLR 2027

Kestrel: Grounded Self-Refinement for LVLM Hallucination Mitigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large vision-language models (LVLMs) have become increasingly strong and perform better than earlier ones, yet their hallucination issues are not completely solved. Previous training-free hallucination reduction methods are mostly developed for earlier LVLMs and rely on correcting internal representation, which we show that they can bring no or negative gain when applied on stronger LVLMs. On the other hand, introducing complementary external visual evidence can provide additional grounding beyond the LVLM itself, but its potential unreliability motivates conservative evidence verification and refinement rather than direct correction. Therefore, we propose **Kestrel**, a training-free framework for stronger LVLM hallucination mitigation that operationalizes conservative external-evidence intervention through explicit visual grounding and evidence-verified iterative self-refinement. In detail, Kestrel first collects explicit visual evidence and converts tool outputs into structured textual evidence. Second, to take full advantage of this evidence, Kestrel verifies it via the same LVLM judge for evidence checking, then iteratively self-refines answers based on verified evidence to reduce the risk of over-correction. Extensive experiments on stronger LVLMs show that Kestrel consistently improves performance across hallucination benchmarks. In particular, Kestrel improves Qwen3-VL by an average of +3.82% on POPE and +3.87% on MME-Hallucination, and improves InternVL3.5 by an average of +3.49% and +2.69%, respectively. The grounding agent contributes a relative gain of +2.14%, while the integrated self-refinement module contributes +2.23% on POPE. To conclude, our Kestrel effectively alleviates the hallucination in stronger LVLMs with transparent verification traces for diagnosis and analysis.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hallucination Mitigation, Large Vision Language Model, Self-Refinement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CEkxOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Kestrel, a training-free framework for mitigating hallucinations in large vision-language models (LVLMs). It first converts an image question into verifiable claims, uses SAM3 to collect visual grounding evidence, and constructs structured textual observations from the grounding outputs. The target LVLM then verifies the claims and performs evidence-gated, history-aware refinement. Experiments with Qwen3-VL-8B and InternVL3.5-8B report improvements on POPE and MME-Hallucination. The paper also includes component ablations, prediction-transition statistics, efficiency measurements, and additional evaluations in the appendix.

Critical strengths

1. Relevant problem and interpretable design. The paper addresses residual hallucinations in stronger LVLMs and the risk of unnecessarily changing correct predictions. Using explicit visual evidence to guide verification and correction is well motivated, and the intermediate claims and evidence references make the decision process easier to inspect.

2. Useful empirical evaluation. The paper compares against a range of training-free mitigation methods on two recent LVLM backbones. It reports improvements across the main benchmarks and includes ablations, efficiency analysis, prediction transitions, and supplementary CHAIR and seed-robustness results.

Critical weaknesses

1. Limited methodological novelty. Kestrel largely integrates established ideas, including claim construction, external visual grounding, LVLM-based verification, and iterative correction. Woodpecker, for example, has already explored a multi-stage pipeline involving visual knowledge validation and hallucination correction. Kestrel differs in its SAM3-based evidence collection and conservative update strategy, but the paper does not make a sufficiently convincing case for the methodological significance of these additions beyond system integration.

2. Limited evidence for the contribution of the central refinement mechanisms. In Table 3, removing the evidence-gated update lowers POPE MS-COCO accuracy from 89.45% to 89.30%, while removing history-aware refinement yields 89.32%. These gains of 0.15 and 0.13 percentage points are small relative to the improvement of the complete system. Moreover, Appendix G reports up to 0.14 points of variation across random seeds on POPE MS-COCO. The ablations therefore do not yet convincingly establish the distinct benefits of the two mechanisms emphasized in the paper, although the overall evidence-based pipeline does improve performance.

3. Insufficient assessment of verification reliability. Existence and count evidence depend on SAM3 predictions, while some textual observations are produced by an LVLM; the target LVLM then performs verification and refinement. This design could propagate errors from grounding or evidence construction into subsequent decisions. The paper provides prediction-transition, qualitative, and seed-robustness analyses, but does not directly assess calibration of the verifier's confidence scores or how the gating mechanism responds to incorrect or incomplete external evidence.

4. Substantial inference cost relative to the incremental refinement gain. Appendix C (Table 6) reports average latency of 18.75 s for Kestrel versus 0.78 s for the Qwen3-VL baseline (24.03×). Accuracy after the first iteration is already 89.28%, compared with 89.45% for the complete pipeline. Although the paper reports per-round outcomes, an ablation without self-refinement, and early stopping, it does not establish a sufficiently clear latency–accuracy advantage over simpler tool-assisted alternatives under comparable inference budgets.

5. Incorrect supplementary code archive. The supplied supplementary ZIP contains a MedDiM medical multimodal modeling codebase rather than an identifiable Kestrel implementation. Its README and training configuration refer to MIMIC-CXR and medical multimodal training, and the proposed SAM3 grounding, claim verification, and evidence-gated refinement code could not be located in the archive. This conflicts with the paper's reproducibility statement (p. 10), which says that the full code is included in the supplementary material. Consequently, the provided archive does not allow the reported Kestrel implementation to be reproduced or checked.

Questions for authors
  1. What is the main methodological contribution beyond earlier multi-stage visual verification and correction methods such as Woodpecker? Which empirical results most directly isolate that contribution?
  2. Given the small effects in Table 3, can the authors provide uncertainty estimates or additional evidence for the benefits of evidence-gated updating and history-aware refinement?
  3. How are the claim-specific confidence thresholds chosen, and how does the method behave when SAM3 detections or LVLM-generated textual evidence are incorrect or incomplete?
  4. Can the authors provide a latency–accuracy comparison against simpler tool-assisted correction approaches under comparable inference budgets?
  5. The submitted supplementary ZIP appears to contain the MedDiM project rather than Kestrel. Could the authors confirm whether the wrong archive was submitted and provide the corresponding implementation?
  6. Figure 5's MS-COCO transition percentages imply approximately 84.3% initial accuracy and 88.1% final accuracy, whereas the corresponding Table 1 averages are approximately 87.37% and 89.45%. Were different subsets or aggregation protocols used?
Reviewer_XBkeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies hallucination mitigation in recent LVLMs. The authors observe that several methods developed for earlier models provide negative gains on stronger backbones. The paper proposes Kestrel, a training-free framework that uses SAM3 to collect explicit visual evidence. Kestrel verifies individual claims against this evidence and iteratively refines the answer when a correction is sufficiently supported. Experiments on Qwen3-VL and InternVL3.5 show consistent improvements on POPE and MME-Hallucination. The ablations suggest that structured evidence and verification-guided refinement are the main sources of improvement.

Critical strengths
  1. The paper identifies an important change in hallucination mitigation for stronger LVLMs. Several methods developed for earlier models provide limited benefits on these stronger backbones. The paper also highlights the need to correct remaining errors without changing answers that are already correct. The transition analysis is useful because it separates successful corrections from over-correction.
  2. Kestrel provides an interpretable approach and is evaluated carefully. The method uses explicit SAM3 evidence, claim-level verification, and conservative answer updates. The experiments cover two recent LVLMs, several POPE variants, and four MME-Hallucination categories. The component ablations, evidence ablations, seed experiments, and latency analysis help explain both the gains and the computational cost.
Critical weaknesses
  1. The novelty relative to existing training-free correction methods is not fully clear. Woodpecker and DeGF already use external feedback to verify or correct LVLM outputs. A closely related work, See It, Say It, Sorted (CVPR 2026), also uses iterative visual evidence collection and a textual evidence pool, but it is not discussed in the paper. The paper should explain more clearly how these differences lead to a new technical contribution.
  2. The paper does not fully establish the value of the later refinement rounds. On MS-COCO, the first iteration already reaches 89.28, while the full pipeline reaches 89.45. Meanwhile, the full pipeline increases latency from 0.78 seconds to 18.75 seconds. The paper reports this comparison only for MS-COCO. Results for the one-round and full versions across the other benchmarks would provide a clearer picture of the accuracy–cost trade-off.
Questions for authors
  1. How does Kestrel differ from See It, Say It, Sorted in its problem setting and correction mechanism? What do the authors consider the main technical contribution unique to Kestrel?
  2. Can the authors compare the one-round and full versions of Kestrel on all evaluated benchmarks? When do the later refinement rounds improve the answer, and when do they introduce errors?
  3. How would an existing correction method perform if it received the same SAM3 evidence and a similar inference budget?
Reviewer_C1xVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Kestrel, a training-free framework for mitigating hallucinations in stronger LVLMs. Kestrel converts visual question-answering decisions into verifiable claims, constructs visual and structured textual evidence using SAM3-based grounding, and employs the same LVLM for claim verification and confidence-gated, history-aware refinement. The authors report that the method outperforms the base models and several hallucination mitigation baselines across POPE and MME-Hallucination.

Critical strengths
  1. Systematic comparisons on strong backbones. Tables 1-2 evaluate hallucination mitigation methods on Qwen3-VL-8B and InternVL3.5-8B across POPE settings and MME categories, highlighting limitations of existing methods on recent backbones.
  2. Extensive supporting analyses. Tables 3-5 assess the contributions of pipeline components and visual and textual evidence. Results across three random seeds for Qwen, a human preference study, and an efficiency analysis further inform the assessment of effectiveness, stability, and cost.
Critical weaknesses
  1. The value of the key new mechanisms remains unclear. Given prior work on evidence-based correction and consistency checking, such as Woodpecker and LogicCheckGPT, gating and history are central to the methodological contribution. Yet removing them reduces accuracy only from 89.45 to 89.30 and 89.32, respectively (Table 3). These small conditional gains provide limited evidence of their added value, particularly whether gating prevents erroneous revisions of initially correct answers.
  2. The performance-cost trade-off requires further comparison. The full pipeline increases latency from 0.78 to 18.75 seconds (24.03×) and peak GPU memory usage to 1.23× the baseline, while improving POPE-COCO accuracy from 87.37 to 89.45 (Table 6). Although this overhead is acknowledged, comparisons with major mitigation baselines under matched settings, including latency and GPU memory, are needed to assess practical applicability.
Questions for authors
  1. To complement the component ablations, could the authors evaluate a Qwen-Agent baseline equipped with the same SAM3 tool under matched settings? This would help distinguish the contributions of tool capabilities from those of the verification and update mechanisms.
  2. The documentation and evaluation entry points in the supplied supplementary code package appear not to correspond to Kestrel. Could the authors provide a clarification about this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.