acceptodds
Under review as a conference paper at ICLR 2027

PAVER: Dual-Source Parsing and Premise-Aware Routing for Reliable LVLM Responses

Abstract

Large vision-language models (LVLMs) can produce unreliable responses for two distinct reasons: a user question may presuppose visual conditions that do not hold, while the model may introduce unsupported content in its answer. These failures require different treatment because a visual proposition may be a required question premise, queried content, or a model assertion. We introduce PAVER (Premise- and Answer-aware Visual Evidence Routing), a training-free framework that operationalizes this distinction asymmetrically. Required question premises determine whether the question remains valid, whereas entities exposed by an initial draft only guide targeted visual inspection. Grounding DINO and Florence-2 provide shared visual evidence for premise-aware routing and final response generation. The router selects refusal when a required premise is contradicted and otherwise selects evidence-grounded re-answering. Across six benchmarks and two LVLM backbones, PAVER reduces answer-side hallucination and improves premise-sensitive reliability. Factorized ablations further show complementary roles: the draft-guided pathway improves answer-side reliability, while explicit premise modeling improves premise-sensitive behavior. These results support interaction-role-aware evidence routing as a practical principle for reliable LVLM responses.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Vision-Language Models, Multimodal Hallucination, Training-Free Hallucination Mitigation, Multimodal Reliability, Premise-Aware Routing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kmDhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies two sources of unreliable LVLM responses: a question may contain required premises that conflict with the image, and the model's answer may introduce content unsupported by visual evidence. The authors propose PAVER, a training-free framework that separately parses required premises from the question and inspection targets exposed by an initial draft. The former determine whether the question remains answerable, whereas the latter only guide Grounding DINO and Florence-2 in acquiring visual evidence. A premise-aware router then chooses between refusal and evidence-grounded re-answering. Experiments with LLaVA-1.5-7B and Qwen3-VL-8B on six benchmarks report improvements in open-ended hallucination, object-existence judgments, false-premise recognition, and selective answering. Pathway ablations, response-length controls, and intermediate-representation audits are used to support the claimed division of labor: premise modeling handles question validity, while draft-guided inspection improves answer-side reliability.

Critical strengths
  1. The paper does not treat every visual conflict as grounds for refusal, which directly avoids the flawed policy of refusing whenever a mentioned entity is not detected.

  2. The evaluation covers both answer-side and premise-side reliability. The six benchmarks span open-ended generation, object existence, counting, position, color, false premises, and insufficient context.

  3. The ablations and representation audits are well targeted.

Critical weaknesses
  1. The main comparison changes both inference compute and available evidence, so it is not fair. PAVER includes draft generation, two parsing stages, Grounding DINO, Florence-2, routing, and final generation. Consequently, part of the improvement may come from additional visual models and multi-stage computation rather than premise/query role modeling itself.

  2. The selected operating point entails substantial over-refusal and information-loss trade-offs that are not summarized under a common utility.

  3. The premise-side evidence depends heavily on one automatic judge, while the human validation is small and lacks agreement statistics.

Questions for authors
  1. With the draft, Grounding DINO/Florence-2 outputs, number of LVLM calls, and token budget held fixed, how does a role-agnostic baseline perform if all evidence is given to a single answering prompt?

  2. Does the router expose a tunable threshold or confidence signal that would allow the authors to report curves over refusal rate, false-refusal rate, incorrect-answer rate, and answer coverage?

5 comments
AuthorsOpenReview ↗

Response to Reviewer kmDh (Part 1)

Response to Reviewer kmDh

We thank you for recognizing the importance of distinguishing required premises from queried content, the breadth of our evaluation, and the targeted ablations. To address the remaining concerns, we have added a routing ablation with matched evidence and inference budgets, a comparison with Woodpecker[1], a cost-sensitive JBA analysis, and a full-split audit of premise extraction.

1. Premise-aware routing improves reliability under matched evidence and inference budgets

With the draft, Grounding DINO/Florence-2 outputs, number of LVLM calls, and token budget held fixed, how does a role-agnostic baseline perform if all evidence is given to a single answering prompt?

PAVER uses separate LVLM calls for routing and generation, so merging them into a single answering prompt removes one call when the rest of the pipeline is unchanged. We therefore use two complementary controls: Full w/o Router tests the single-prompt design, and Generic Router tests the routing instruction with the call count and token budgets fixed.

A single answering prompt lowers MME-Hall from 658.3 to 415.3 (Table 8). Full w/o Router keeps the draft, extracted premises, grounding targets, and Grounding DINO and Florence-2 outputs fixed. It gives all visual evidence to one prompt that decides whether to answer and generates the final response.

With the same LVLM calls, token budgets, and visual evidence, Full PAVER outperforms Generic Router by 182.59 points on MME-Hall and 26.67 points on HaloQuest Overall (Table R1). Generic Router retains the explicit premises and the same final generation procedure. Only the routing instruction changes: it asks the model to refuse invalid questions and answer otherwise, without explicitly distinguishing required premises from queried content.

Table R1. Routing ablation with matched visual evidence and inference budgets (LLaVA-1.5-7B).

BenchmarkMetricGeneric RouterFull PAVER
MME-HallTotal Score ↑475.74658.33
Router REFUSE Rate ↓58/240 (24.17%)0/240 (0%)
HaloQuestIC ↑23.3360.00
FP ↑39.5375.08
VC ↑31.2836.31
Overall ↑33.8360.50

Generic Router refuses 42 of the 60 MME-Hall existence questions, while Full PAVER routes all 60 to ANSWER. These questions illustrate why the distinction matters: when a question asks whether an object is present, its absence supports the answer No.

Together, these complementary ablations demonstrate the benefits of premise-aware routing instructions and separating routing from response generation, beyond the effects of additional visual evidence and inference computation.

2. Comparison with multi-stage verification and correction methods

Consequently, part of the improvement may come from additional visual models and multi-stage computation rather than premise/query role modeling itself.

We reproduced Woodpecker using its released implementation. Woodpecker extracts concepts from a draft, formulates verification questions, checks visual evidence with external tools, and corrects the answer, making it a relevant comparison for the concern about additional tools and inference stages.

Table R2. Comparison with Woodpecker on LLaVA-1.5-7B.

BenchmarkMetricWoodpeckerPAVER
POPEAccuracy ↑76.6487.76
F1 ↑73.7286.80
HaloQuestIC ↑40.0060.00
FP ↑38.8775.08
VC ↑20.1136.31
Overall ↑33.5060.50
AMBERCHAIR ↓10.94.1
Cover ↑49.853.1
Hal ↓44.414.8
Cog ↓4.10.8

Table R3. Efficiency comparison on the first 50 AMBER Generation examples.

MethodLatency (s) ↓Peak GPU Memory (MB) ↓
Woodpecker20.35 ± 5.6736,516
PAVER14.92 ± 5.8819,750

PAVER outperforms Woodpecker across all reported benchmark metrics (Table R2), while achieving lower end-to-end latency and peak GPU memory usage (Table R3). These results demonstrate PAVER's advantages in both effectiveness and efficiency over this multi-stage verification-and-correction framework.

AuthorsOpenReview ↗

Response to Reviewer kmDh (Part 2)

3. Quantifying refusal trade-offs and information retention

The selected operating point entails substantial over-refusal and information-loss trade-offs that are not summarized under a common utility.

To quantify the trade-off between correctly rejecting false-premise questions and incorrectly rejecting valid questions, we introduce a cost-sensitive decision utility. We evaluate this utility on all 6,540 JBA examples using the benchmark judge's per-example classifications of the final responses. With false-premise questions as the positive class, TPTP and FNFN count correctly identified and missed false premises, respectively; TNTN and FPFP count valid-premise questions correctly accepted and incorrectly classified as false-premise, respectively. We define decision-level utility as

Uλ,μ=TP+TN−λFP−μFNN,N=TP+TN+FP+FN.U_{\lambda,\mu}=\frac{TP+TN-\lambda FP-\mu FN}{N},\qquad N=TP+TN+FP+FN.

Each correct premise decision receives +1+1. The penalties λ\lambda and μ\mu apply to false-positive and false-negative premise classifications, respectively. We fix μ=1\mu=1 and vary λ\lambda to evaluate the same final-response predictions under different error costs. The false-positive rate, FP/(FP+TN)FP/(FP+TN), is the proportion of valid-premise questions classified as false-premise.

Table R4. Cost-sensitive decision utility on JBA (LLaVA-1.5-7B). Utility values are multiplied by 100.

MethodFalse-premise recall ↑\uparrowFalse-positive rate ↓\downarrowU0.5,1U_{0.5,1} ↑\uparrowU1,1U_{1,1} ↑\uparrowU1.5,1U_{1.5,1} ↑\uparrowU2,1U_{2,1} ↑\uparrow
Vanilla22.29%1.59%21.1920.8020.4020.00
SHIELD30.19%4.92%26.5825.3524.1222.89
PAVER72.50%36.74%44.9535.7526.5517.35

PAVER achieves the highest decision utility under equal error costs, outperforming SHIELD by 10.40 points and Vanilla by 14.95 points. Beyond this equal-cost setting, PAVER maintains the highest utility for false-positive penalties below approximately λ≈1.65\lambda \approx 1.65 (with μ=1\mu=1), highlighting the trade-off between higher false-premise recall and additional false positives.

For information retention, at comparable response lengths, PAVER achieves lower hallucination rates and higher object coverage than length-matched Vanilla and MARINE on AMBER (Table 3), demonstrating a better hallucination–coverage trade-off.

Table R5. Hallucination and object coverage at comparable mean response lengths (AMBER Generation, Qwen3-VL-8B).

MethodMean length (words)CHAIR ↓\downarrowCover ↑\uparrowHal ↓\downarrowCog ↓\downarrow
Vanilla (regenerated)59.85.660.725.71.0
MARINE (regenerated)60.05.962.628.11.9
PAVER (native)61.03.863.420.11.0

4. Expanded validation of premise extraction

The premise-side evidence depends heavily on one automatic judge, while the human validation is small and lacks agreement statistics.

We added a full-split premise-extraction audit using GPT-6.1 Sol as an additional evaluator. It assessed all 6,540 LLaVA JBA premise sets, complementing the original condition-blind human audit of 100 stratified questions. The criterion is strict: a premise set is correct only if it includes every required premise and contains no unnecessary or incorrect premise. Omissions, unsupported additions, semantic distortions, and leakage of queried content into the premise set are all counted as errors.

As shown in Table R6, GPT-6.1 Sol judges 86.36% of the premise sets extracted by PAVER's frozen LVLM backbone to be correct across all 6,540 JBA examples. This result is similar to the 83% Fully Correct rate obtained in our previous human validation (Table 9), providing further evidence of the reliability of LVLM-based premise extraction in PAVER.

Table R6. Full-split premise-extraction accuracy on JBA (LLaVA-1.5-7B), evaluated by GPT-6.1 Sol.

TotalCorrectIncorrectAccuracy ↑
6,5405,64889286.36%
AuthorsOpenReview ↗

Response to Reviewer kmDh (Part 3)

5. Confidence thresholds

Does the router expose a tunable threshold or confidence signal that would allow the authors to report curves over refusal rate, false-refusal rate, incorrect-answer rate, and answer coverage?

The current router outputs a discrete ANSWER/REFUSE decision and does not expose a tunable confidence threshold. It selects REFUSE when visual evidence contradicts a required premise and ANSWER otherwise.

In our early experiments, we tried prompting the model to output confidence scores and using an explicit threshold to determine whether to refuse. However, we found that the model tended to be overconfident, leading to poor performance. This observation is consistent with prior work on confidence elicitation in LLMs [2] and LVLMs [3] [4]. These studies distinguish verbalized confidence from more informative probability- or consistency-based measures, which would require a separate calibration design and validation. We therefore retain a discrete premise-aware router and the present experiments evaluate the effectiveness of this routing policy.

Taken together, we hope these additional results and clarifications adequately address your concerns. If so, we would greatly appreciate it if you could reconsider your assessment and rating.


References

[1] Yin, Shukang, et al. "Woodpecker: Hallucination correction for multimodal large language models." Science China Information Sciences 67.12 (2024): 220105.

[2] Xiong, Miao, et al. "Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms." International Conference on Learning Representations. Vol. 2024.

[3] Ding, Zhikai, Shiyu Ni, and Keping Bi. "Do LVLMs Know What They Know? A Systematic Study of Knowledge Boundary Perception in LVLMs." EMNLP (Findings). 2025.

[4] Groot, Tobias, and Matias Valdenegro-Toro. "Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models." Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024). 2024.

Reviewer_qqtzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles two distinct sources of unreliable LVLM responses: false question premises and unsupported content introduced by the model's own answer. It proposes PAVER, a training-free framework that classifies each visual proposition by interaction role — required premise, queried content, or answer assertion — and routes visual evidence asymmetrically: only a contradicted required premise triggers refusal, queried content just determines the answer value, and draft assertions guide targeted inspection. Shared evidence from Grounding DINO and Florence-2 drives both premise-aware routing and evidence-grounded re-answering. The main contribution is the interaction-role asymmetric-routing problem framing, plus a comprehensive evaluation.

Critical strengths
  1. A clear, distinguishable design — interaction-role asymmetric evidence routing. Classifying visual propositions by interaction role (required premise / queried content / answer assertion) and applying an asymmetric decision rule is a genuine design that unifies the answer-centric and false-premise research lines, not engineering integration.
  2. Comprehensive and honest evaluation. Evaluation covers six benchmarks × two backbones, plus factorized ablations, efficiency, entity-level audits, and a human audit, with a matched ablation control and complete reproducibility docs. Furthermore, it is honest: it discloses the VC drop, a genuine over-refusal failure, the framework's inability to strengthen the backbone, and flags causal-attribution limits in the audit.
Critical weaknesses
  1. Core results rest on single-run point estimates with no statistical uncertainty. The key comparisons in the main tables (Tables 1–6) are largely single-run point estimates without multi-run variance or significance tests, and several improvements are modest.
  2. The core decision mechanism is carried entirely by ICL prompts, with robustness unmeasured.
Questions for authors
  1. Statistical uncertainty. Are the key comparisons in Tables 1–6 single-run point estimates? Can you provide mean ± std over ≥3 seeds, and state whether paired significance tests were conducted?
  2. Mechanism robustness. The role decision is carried entirely by ICL prompts. Please report: (a) the effect of demonstration count (LLaVA 4 vs Qwen 25) on premise extraction accuracy; (b) stability under different prompt templates for the same backbone; (c) measured router error rates (false refusal / false answer).
4 comments
AuthorsOpenReview ↗

Response to Reviewer qqtz (Part 1)

Response to Reviewer qqtz

We thank you for recognizing PAVER's interaction-role formulation as a distinct methodological contribution and for the positive assessment of our evaluation. We have added four analyses addressing the requested checks: three-seed comparisons with paired significance tests, a demonstration-count study, router prompt-template comparisons, and direct measurements of routing errors.

1. Statistical uncertainty: consistent gains across matched seeds

Can you provide mean ± std over ≥3 seeds, and state whether paired significance tests were conducted?

The original Tables 1–6 report single-run estimates. We have now added a supplementary three-seed evaluation on MME-Hall with LLaVA-1.5-7B, comparing PAVER with three representative baselines under matched seeds 42, 3407, and 2026.

Table R1. MME-Hall results across three matched seeds (LLaVA-1.5-7B).

MethodSeed 42Seed 3407Seed 2026Mean ± SD
MARINE508.33483.33476.67489.44 ± 16.69
SHIELD535.00563.33543.33547.22 ± 14.56
Vanilla561.67560.00575.00565.56 ± 8.22
PAVER658.33646.67646.67650.56 ± 6.74

PAVER leads in every matched run, with mean improvements of 85.00, 103.33, and 161.11 points over Vanilla, SHIELD, and MARINE, respectively. Its gains are large relative to the observed variation across seeds. We also performed two-sided paired t-tests on the seed-matched scores, applying Holm correction across the three comparisons. All three comparisons remain significant after correction (p<0.05p<0.05).

Table R2. Paired significance tests for the three-seed MME-Hall comparison.

ComparisonUnadjusted ppHolm-adjusted pp
PAVER vs. MARINE0.001330.00399
PAVER vs. SHIELD0.012260.01445
PAVER vs. Vanilla0.007230.01445

Given the limited rebuttal period, we prioritized LLaVA-1.5-7B on MME-Hall and will extend the multi-seed evaluation to additional methods and benchmarks as time permits.

2. Mechanism robustness: demonstration count, prompt templates, and routing errors

Mechanism robustness. The role decision is carried entirely by ICL prompts. Please report: (a) the effect of demonstration count (LLaVA 4 vs Qwen 25) on premise extraction accuracy; (b) stability under different prompt templates for the same backbone; (c) measured router error rates (false refusal / false answer).

(a) Demonstration-count sensitivity

We evaluated premise extraction on the same 100 stratified JBA questions (50 true-premise and 50 false-premise). Demonstrations were nested within each backbone: 2-shot ⊂ 4-shot ⊂ 8-shot for LLaVA and 8-shot ⊂ 16-shot ⊂ 25-shot for Qwen. The 0-shot condition retains the instructions and removes the demonstrations. All settings were assessed using the same premise-set correctness criteria and annotation procedure.

Table R3. Premise-extraction quality under different demonstration counts.

BackboneShotsFully Correct ↑Under-extraction ↓Over-extraction ↓Query Leakage ↓
LLaVA-1.5-7B063122519
LLaVA-1.5-7B26582717
LLaVA-1.5-7B4 (original)837109
LLaVA-1.5-7B87372016
Qwen3-VL-8B0846102
Qwen3-VL-8B891362
Qwen3-VL-8B1689471
Qwen3-VL-8B25 (original)91271

Both backbones benefit from demonstrations, but more shots do not consistently improve extraction quality. LLaVA performs best with the original 4-shot setting, while Qwen achieves comparable performance across different nonzero-shot settings. These results suggest that demonstration count should be selected based on the backbone rather than simply increased.

AuthorsOpenReview ↗

Response to Reviewer qqtz (Part 2)

(b) Router Prompt-Template Robustness

We compared three formulations of the same premise-aware routing policy on JBA, keeping the backbone, extracted premises, visual evidence, output schema, and decoding settings fixed:

  • Original: The detailed evidence-handling and routing policy used in the main experiments.
  • Step-by-step: The same policy organized into explicit premise verification, evidence assessment, and routing steps.
  • Simplified: A concise formulation preserving the premise/query distinction and refusal criterion.

Table R4. JBA performance across router prompt templates (LLaVA-1.5-7B).

Router promptFPC ↑F1 ↑TPIR ↑FPDP ↑
Original67.8769.2769.7566.31
Step-by-step66.3867.4067.5665.33
Simplified68.1267.2167.2569.09
Mean across templates67.4667.9668.1966.91
sample SD across templates0.941.141.371.95

PAVER maintains comparable performance across all three router prompt templates, consistently outperforming the strongest LLaVA baseline in Table 6 on both FPC (62.7) and F1 (44.7). These results suggest that PAVER's gains are robust to variations in router prompt wording and structure.

(c) Direct Router Error Rates

We evaluate two types of routing errors on JBA: incorrectly refusing valid questions and answering false-premise questions. Using JBA premise-validity labels as the reference, we assess the router's internal ANSWER/REFUSE decisions (Table R5).

Table R5. Internal routing errors on JBA (LLaVA-1.5-7B).

Router metricDefinitionRate
False refusal rateTrue-premise questions routed to REFUSE / all true-premise questions21.99%
False answer rateFalse-premise questions routed to ANSWER / all false-premise questions44.15%
Overall routing error rateIncorrect routing decisions / all questions33.06%

On POPE, MME-Hall, AMBER Generation, and OpenCHAIR, the full PAVER pipeline does not produce any erroneous refusals. POPE and MME-Hall primarily consist of straightforward visual questions (e.g., "Is there X in the image?") that generally do not contain premises requiring validation and thus do not trigger premise validation. Meanwhile, AMBER Generation and OpenCHAIR use open-ended prompts (e.g., "Describe the image in detail"), for which PAVER proceeds to answer rather than refusing.

Together, the additions provide statistical support for the MME-Hall gains, a direct assessment of demonstration sensitivity, evidence of robustness across router prompt formulations, and explicit measurements of routing errors. We hope these results address the concerns about uncertainty and prompt dependence.

I have read the authors' response, and I maintain my original score.

We sincerely appreciate your insightful comments and are pleased that our additional experiments and analyses have addressed your concerns.

Reviewer_ygZsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work identifies that LVLMS can give unreliable answers due to a) false presupposition in use query b) model hallucinations in its answers. They formalizes conceptual distinction between visual proposition. Required premises, where visual conditions that must hold for the question to be answerable. Queried content, visual propositions whose truth is precisely what the question asks the model to determine. Model assertions, visual claims introduced during answer generation.

It argues that these two mode of failures require different treatments as such they propose a PAVER training-free framework that address these two failures along two separate pathways to improve reliability in LVLM answers. Draft-guided visual evidence acquisition through Grounding DINO and Florence-2, and premise-aware routing between refusal and evidence-grounded re-answering.

Experiments were conducted across 2 LVLM backbones (llava-1.5-7b-hf and Qwen3-BL-8B-Instruct) on 6 benchmarks (OpenCHAIR, AMBER Generation, POPE-MSCOCO, MMME_Hall, Halo-Quest and JBA) , empirical results demonstrate that PAVER reduces answer-side hallucination and improves premise-sensitive reliability.

Critical strengths

The work introduces well formulated problem and draws distinction between required premises, queried content, and answer assertions provides a useful perspective on addressing the problem of reliability in LVLMs.

The proposed asymmetric treatment of question-derived premises and draft-derived grounding targets provides a reasonable mechanism for integrating question validity assessment with hallucination mitigation.

Evaluations across two different backbones and six benchmarks with PAVER demonstrates substantial improvements across several answer-side and premise-sensitive benchmarks support the contributions claims and generalization of method across models.

Critical weaknesses
  • A central idea of the paper is the LVLMs ability to Refuse given the a contradictory premise and generate a free form answer to explain the contradiction. However, it was not clear to me how these refusals are scored particularly on benchmarks like MME-Hall and POPE that requires binary answer.

  • While PAVER demonstrates substantial improvements over several training-free hallucination mitigation methods (VCD, DeCo, SHIELD, and MARINE), to my understanding these baselines primarily focus on decoding, representation-level interventions, or external visual guidance. They do not directly evaluate the proposed approach against closely related multi-stage verification and correction frameworks which I believe is the key novelty here.
    In particular, prior works such as Woodpecker, PELICAN, have explored training-free hallucination correction using intermediate representations and external visual evidence. Although PAVER's explicit distinction between question validity and answer-side hallucination is potentially meaningful, the experiments do not sufficiently establish the benefit of this formulation over existing verification-based approaches.

  • I found the presentation unnecessarily difficult to follow due to the introduction of numerous overlapping or closely related terms without clearly establishing their distinctions. For instance, the paper alternates between "draft assertions," "model assertions," and "draft-exposed entities," and between "candidate visual propositions," "query-role content," and "required premises." These terms refer to different levels of representation (propositions, entities, and grounding targets), but the manuscript does not consistently clarify how they relate to one another or if they all mean the same thing. More importantly, this ambiguity makes it difficult to understand what is actually represented and verified by the framework.
    Furthermore, notations and introduction of terms and abbreviations could be done more carefully for example introduction line 042 authors introduced "draft content" without defining for explaining what this is ,in Iine 82 ICL is introduced without the definition and in equation (1) π_draft is not defined. In fact I was unable to find π is defined clearly any where and can only be first inferred from line 225 "... dedicated routing prompt"
    While it is reasonable to use different terms or phrasing for the same concept, I encourage the authors to clearly distinguish conceptual propositions, implemented intermediate representations, and routing decisions and to clearly define the notations and terms utilized especially in the method section.

Questions for authors

What are the independent contributions of Grounding DINO, Florence-2, and draft-derived grounding targets? Author mentions the use of both grounding DINO and Florence-2 for evidence acquisition I would like to have seen the specifics on the models and how each contributes independently and also on decision or reason on why both grounding DINO and Florence-2 are both used.

Clarification of refusal handling in binary-answer benchmarks i.e MME-Hall. MME-Hall requires binary Yes/No responses, whereas PAVER can route examples to a REFUSE branch that produces a free-form explanation of a contradicted premise. It is unclear how such responses are handled by the official evaluation pipeline. Are refusal outputs counted as incorrect, excluded from evaluation, or normalized to binary predictions? Please clarify the exact scoring procedure for non-binary PAVER outputs and report the proportion of examples routed to REFUSE on MME-Hall and POPE.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ygZs (Part 1)

Response to Reviewer ygZs

We thank you for the constructive comments and for recognizing the value of our interaction-role distinction, asymmetric design, and comprehensive evaluation. To address the remaining concerns, we clarify the refusal handling and scoring procedures, add a direct comparison with Woodpecker, refine the representation terminology, and conduct new ablations to quantify the contributions of external visual tools.

1. Refusal handling and scoring on binary-answer benchmarks

Please clarify the exact scoring procedure for non-binary PAVER outputs and report the proportion of examples routed to REFUSE on MME-Hall and POPE.

Full PAVER's REFUSE rate is 0% on both MME-Hall and POPE across the two evaluated backbones (Tables 2 and 4). In these datasets, the questions are simple and straightforward visual questions (e.g., "Is there X in the image?") that do not contain premises and thus do not trigger premise validation.

For POPE and MME-Hall with non-binary outputs, our scoring protocol follows the same procedures as the other baselines and the official benchmark implementations. Specifically:

  • POPE. The parser takes the first case-insensitive, whole-word yes/no match. Exact responses y/true/1 and n/false/0 are also mapped to Yes and No, respectively. If no valid label is extracted, the example remains in the evaluation and is counted as FN for a Yes ground truth or FP for a No ground truth. An unparseable response therefore lowers accuracy and contributes to the corresponding error count for precision, recall, and F1.

  • MME-Hall. A valid binary response must begin with yes or no, ignoring case and the leading whitespace and punctuation/Markdown wrappers accepted by the parser. In the evaluation implementation used for these results, each category is scored as

    Acc=100×NcorrectNvalid\mathrm{Acc} = 100 \times \frac{N_{\mathrm{correct}}}{N_{\mathrm{valid}}}

    Acc+=100×Npair correctNimages\mathrm{Acc}^{+} = 100 \times \frac{N_{\mathrm{pair\ correct}}}{N_{\mathrm{images}}}

    Here, NvalidN_{\mathrm{valid}} counts parseable binary responses, and Npair correctN_{\mathrm{pair\ correct}} counts images for which both questions are answered correctly. The category score is Acc+Acc+\mathrm{Acc} + \mathrm{Acc}^{+}, and MME-Hall sums the four category scores. A non-binary response is excluded from the Acc\mathrm{Acc} denominator but prevents its image from contributing a correct pair to Acc+\mathrm{Acc}^{+}; that image remains in the Acc+\mathrm{Acc}^{+} denominator.

In the ablation studies, refusal cases occur in some variants. For example, although Full w/o Router retains the same premises and visual evidence, jointly performing routing and generation yields 76/240 non-binary refusals (31.67%) on MME-Hall and a score of 415.3, compared with 658.3 for Full PAVER (Table 8). These results highlight the importance of premise-aware routing.

2. Comparison with multi-stage verification and correction methods

They do not directly evaluate the proposed approach against closely related multi-stage verification and correction frameworks which I believe is the key novelty here.

We added a comparison with Woodpecker [1] using its released implementation. Woodpecker is a representative multi-stage framework for hallucination mitigation in vision-language models. It extracts concepts from a draft response, generates verification questions, checks external visual evidence, and corrects the response.

PELICAN [2] is also relevant, but we did not reproduce it because no official implementation is available. Reproducing its framework would require substantial engineering effort and reimplementation of external tool integrations, making faithful reproduction difficult and potentially compromising the fairness of the comparison.

PAVER improves on Woodpecker across all reported metrics, including +13.08 points in POPE F1, +27.00 points in HaloQuest overall accuracy, and a 29.6-point reduction in AMBER Hal, alongside higher object coverage.

Table R1. Comparison with Woodpecker on LLaVA-1.5-7B.

BenchmarkMetricWoodpeckerPAVER
POPEAccuracy ↑76.6487.76
F1 ↑73.7286.80
HaloQuestIC ↑40.0060.00
FP ↑38.8775.08
VC ↑20.1136.31
Overall ↑33.5060.50
AMBERCHAIR ↓10.94.1
Cover ↑49.853.1
Hal ↓44.414.8
Cog ↓4.10.8
AuthorsOpenReview ↗

Response to Reviewer ygZs (Part 2)

As shown in Table R2, PAVER achieves lower end-to-end latency and peak GPU memory usage than Woodpecker, while also delivering stronger benchmark performance.

Table R2. Efficiency comparison on the first 50 AMBER Generation examples.

MethodLatency (s) ↓Peak GPU Memory (MB) ↓
Woodpecker20.35 ± 5.6736,516
PAVER14.92 ± 5.8819,750

3. Representation terminology and notation

I encourage the authors to clearly distinguish conceptual propositions, implemented intermediate representations, and routing decisions and to clearly define the notations and terms utilized especially in the method section.

We appreciate this criticism and agree that the current presentation sometimes mixes conceptual propositions, implemented intermediate representations, and routing actions. The intended hierarchy is as follows:

  1. Question-level propositions. ZqZ_q denotes candidate visual propositions evoked by the question. The conceptual role mapping ρq\rho_q distinguishes PREMISE, a condition required for the question to remain answerable as written, from QUERY, content placed under inquiry. Here, ρq\rho_q describes the selection criterion induced by the premise-extraction prompt. In implementation, PAVER explicitly extracts only the required-premise set PqP_q, while queried content remains in the original question.
  2. Required premises. PqP_q is the explicit premise list produced by the question-side parser. It provides the conditions used to assess question validity. Queried content remains in the original question, where visual evidence determines the answer rather than requiring the queried proposition to be true.
  3. Draft assertions and grounding targets. a(0)a^{(0)} is the LVLM's initial response, generated before evidence acquisition and routing. “Draft assertions” and “model assertions” refer to claims expressed in this response. “Draft-exposed entities” are entity mentions extracted from those claims and combined with question-derived terms to form the grounding targets UaU_a. The implemented draft-augmented representation is therefore a set of visual inspection queries. These queries guide evidence acquisition for a newly generated final response.
  4. Visual evidence and routing decisions. Grounding DINO and Florence-2 provide the shared observations EE. Using the original image, question, PqP_q, and EE, the router selects r∈{ANSWER,REFUSE}r \in \{\mathrm{ANSWER}, \mathrm{REFUSE}\}. Its policy permits refusal only when a required premise is clearly contradicted; otherwise, it selects ANSWER.

We also confirm the specific editorial omissions identified by the reviewer. We will define “draft” at its first mention, expand ICL as in-context learning at first use, and explicitly introduce πdraft\pi_{\mathrm{draft}}, πroute\pi_{\mathrm{route}}, πref\pi_{\mathrm{ref}}, and πans\pi_{\mathrm{ans}} as the prompting instructions for draft generation, routing, refusal generation, and evidence-grounded answering, respectively. We will consistently use draft assertions for claims in the initial response, required premises for PqP_q, grounding targets for UaU_a, and routing decision for rr.

We will carefully review and correct these presentation issues in the final version.

4. Complementary contributions of Grounding DINO, Florence-2, and draft-guided targets

What are the independent contributions of Grounding DINO, Florence-2, and draft-derived grounding targets?

We added new ablations to quantify the contributions of external visual tools. We separately remove Grounding DINO, Florence-2, or both, while keeping all other components unchanged.

We first evaluate whether these visual-tool ablations affect premise-sensitive reliability on JBA.

Table R3. Visual-tool ablations on JBA (LLaVA-1.5-7B).

ConfigurationFPC ↑F1 ↑TPIR ↑FPDP ↑
w/o DINO67.069.270.064.8
w/o Florence-267.466.967.067.7
w/o All Tool Evidence66.966.967.066.7
Full PAVER67.969.369.866.3
SHIELD62.744.758.086.0
Vanilla60.436.056.093.3

Premise-sensitive performance remains stable across visual-tool ablations, with JBA FPC ranging from 66.9 to 67.9. Notably, even the lowest FPC (66.9) and F1 (66.9) across these ablations outperform the strongest LLaVA baseline, SHIELD, which achieves 62.7 FPC and 44.7 F1. This demonstrates that PAVER's premise-sensitive gains persist even without external visual-tool evidence.

AuthorsOpenReview ↗

Response to Reviewer ygZs (Part 3)

We further evaluate the contributions of external visual tools on AMBER Generation. Here, Cover (↑) measures how many ground-truth objects are mentioned in the response, while CHAIR, Hal, and Cog (↓) measure hallucination-related errors.

Table R4. Visual-tool ablations on AMBER Generation (LLaVA-1.5-7B).

ConfigurationCHAIR ↓Cover ↑Hal ↓Cog ↓
w/o DINO6.449.328.23.0
w/o Florence-24.139.99.50.6
w/o All Tool Evidence3.237.47.60.3
Full PAVER4.153.114.80.8
MARINE5.149.723.42.3
Vanilla9.550.342.94.2

The ablations reveal complementary contributions from the two visual tools:

  • Removing Grounding DINO increases Hal from 14.8 to 28.2 and CHAIR from 4.1 to 6.4, demonstrating the importance of targeted localization for hallucination mitigation.

  • Removing Florence-2 substantially reduces Cover from 53.1 to 39.9, despite lowering Hal to 9.5, highlighting the importance of global semantic context in enabling the model to describe more objects present in the image.

  • Removing both tools further reduces Hal to 7.6, but Cover drops to 37.4, the lowest among all configurations. Note that relatively high Cover scores (measuring the descriptive coverage of ground-truth objects in input images) are generally desirable in visual question-answering scenarios and are often important for accurately addressing users' questions, e.g., by providing concrete descriptions of relevant objects that make the answers easier to understand. Thus, reducing hallucinations at the expense of object coverage is not desirable.

  • Compared with Vanilla and MARINE, Full PAVER achieves the highest Cover (53.1) and lower CHAIR, Hal, and Cog. MARINE also uses external visual models (DETR and RAM++) to extract object-level information and guide generation. Compared with MARINE, PAVER improves Cover from 49.7 to 53.1 while reducing Hal from 23.4 to 14.8, demonstrating better hallucination control without sacrificing object coverage.

These results motivate combining draft-guided target selection, Grounding DINO's targeted localization, and Florence-2's global semantic context. Draft-derived grounding targets guide DINO toward entities introduced in the initial response, including potentially unsupported content. Together, these complementary components enable PAVER to preserve richer visual information while effectively mitigating hallucinations.

Together, the clarified refusal handling, new Woodpecker comparison, and visual-tool ablations provide stronger empirical support for PAVER's reliability and the complementary roles of its components. We have also clarified the representation hierarchy and implementation details. We hope these additional results and clarifications address your concerns and support a more favorable assessment of our work.


References

[1] Yin, Shukang, et al. "Woodpecker: Hallucination correction for multimodal large language models." Science China Information Sciences 67.12 (2024): 220105.

[2] Sahu, Pritish, Karan Sikka, and Ajay Divakaran. "Pelican: Correcting hallucination in vision-llms via claim decomposition and program of thought verification." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.