acceptodds
Under review as a conference paper at ICLR 2027

Question-Guided Self-Distillation for Reducing False-Positive Hallucination in Vision-Language Models

Abstract

Vision-language models can affirm nonexistent objects, incorrect attributes, and invalid relations, or answer questions whose visual premises are false. Reduc- ing these false positives requires correcting unsupported content while retaining valid answers. We investigate question-guided self-distillation, which uses fac- tual conditions implicit in the original question as training guidance. Verification subquestions are supplied to a frozen base model without subquestion answers. A student generates responses from the original image and question and learns from the conditioned teacher’s token distributions on those same response prefixes. A second distribution-matching term constrains departure from the base model un- der the original input. Both terms use token-level Jensen–Shannon divergence; deployment requires neither question decomposition nor additional verification calls. With Qwen3-VL-8B and a 60K training pool, wh paired accuracy improves over our FINER-DPO baseline from 46.62% to 57.99% on FINER-CompreCap and from 36.42% to 47.01% on FINER-DOCCI. On DOCCI wh queries, negative- query accuracy rises from 46.09% to 61.03%, while positive-query accuracy falls from 83.60% to 80.63%. Other query settings and general capability evaluations reveal further trade-offs. These results show improved direct false-premise han- dling with the complete training recipe, while identifying selective correction and capability retention as remaining challenges.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language Models, Multimodal Hallucination, False-Positive Hallucination, On-Policy Self-Distillation, Question Decomposition

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_F5qTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes question-guided self-distillation (a form of OPSD) for reducing false-positive hallucinations in VLMs. A frozen teacher receives verification subquestions as privilige knowledge, while the student learns to answer directly through on-policy distribution matching and a reference objective. Results show improvements on FINER wh questions and general benchmarks.

Critical strengths
  1. The paper addresses an important reliability issue and explicitly examines both false-premise correction and valid-answer retention.
  2. The method requires no additional verification at deployment. The teacher-context ablation supports the usefulness of question-specific guidance.
  3. Experiments on two model backbones show consistent wh improvements, while pair-level analyses help explain the corrections and regressions.
Critical weaknesses
  1. Antidote [1] is relevant prior work on counterfactual presuppositions and factual-prior-guided self-correction by preference optimization, but is not discussed. The paper should clarify how its answer-free, question-derived guidance differs from factual-prior guidance and what additional contribution it offers.

  2. The methodological novelty appears incremental: the approach combines OPD, question-derived teacher context, and reference regularization. The current ablation does not isolate the contributions of on-policy sampling and the reference term, leaving the benefit beyond teacher prompting unclear.

  3. Improvements are concentrated on wh questions. Table 1 shows regressions on several object and relation settings across both backbones. On DOCCI wh, positive-query accuracy also drops from 83.60% to 80.63%, indicating unresolved over-rejection.

  4. Table 3 uses different evaluation processing across methods. Appendix B reports answer normalization and additional V* visual verification for the proposed model, while Base and DPO retain direct results.

[1] Wu et al., “Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception,” CVPR 2025.

Questions for authors
  1. A discussion of Antidote and evaluation on CP-Bench would help clarify the contribution and generalization. Comparing question-derived guidance with factual-prior guidance would further strengthen this analysis.
  2. Ablations on on-policy sampling and reference regularization, together with stronger teacher-prompt controls, would help establish the contribution of each component.
  3. Further analysis of the non-wh regressions and increased rejection rate of valid questions would help clarify the method’s benefits and limitations.
7 comments

Weakness 1 & Question 1: A discussion of Antidote and evaluation on CP-Bench would help clarify the contribution and generalization. Comparing question-derived guidance with factual-prior guidance would further strengthen this analysis.

Thank you for pointing out Antidote. We trained OPD on LLaVA-1.5-7B, matching the baseline used in the original Antidote work, and conducted additional experiments using GPT-4o as the judge and following the CP-Bench V2 evaluation procedure publicly released by the Antidote authors.

1. Gains and Trade-offs on CP-Bench V2

ModelCPQ Reject RateTPQ Accept RateAccuracyFinal Score
Base3.00%98.00%50.50%31.50%
Ours82.80%75.00%78.90%80.46%
Antidote73.10%95.50%84.50%79.94%

The original Antidote paper reports results on CP-Bench V1, whereas the version currently released by the authors is V2. Our model use a publicly available mirror of the CP-Bench dataset and follow the V2 evaluation procedure. To facilitate interpretation, we mathematically convert V1 metrics reported in the original Antidote paper (the Base and Antidote results) into the same metric format used in V2. Antidote’s official public resources do not include the complete training dataset or the trained model weights; therefore, we did not independently reproduce its training.

  • OPD improves false-premise rejection. Our model achieves a substantially higher CPQ rejection rate than the other models, consistent with our goal of reducing false-positive hallucinations in VLMs.

  • OPD still exhibits reduced true-premise acceptance. Suppressing hallucinations induced by false premises leads OPD to adopt a more conservative response tendency, while Antidote performs better at accepting true-premise questions.

2. Differences from Factual-prior Guidance

  • Antidote: Uses facts such as object presence or absence in synthetic data to guide self-correction, then constructs preference pairs for DPO. It also incorporates true-premise questions to mitigate excessive caution.

  • Ours: Provides only conditions to verify, without verification answers. The student matches the teacher's distribution along its own response prefixes, without requiring preconstructed preferred and dispreferred response pairs.

Weakness 2 & Question 2: Ablations on on-policy sampling and reference regularization, together with stronger teacher-prompt controls, would help establish the contribution of each component.

Thank you for your suggestion. We separately examine the contributions of on-policy sampling and reference regularization and introduce additional teacher-prompt controls.

1. Contribution of On-policy Sampling. Using Qwen3-VL-8B-Instruct as the baseline, we compare the paper's on-policy sampling approach with a variant that uses fixed samples from the initial student.

ModelObjAttrRelWHMacro Average
OPD75.1777.1166.2157.9969.12
Fixed Initial Prefixes74.4173.7663.9752.3766.13

Metric: CompreCap paired accuracy (%, higher is better); the macro average weights the four question types equally.

The variant uses fixed prefixes generated in advance by the initial student, while student parameters continue to update. Both settings use the same question-specific verification context, loss function, and training budget.

On-policy sampling improves performance across all four question types, supporting its contribution in this setting: as the student policy evolves, distillation along its current response trajectories is more effective than reusing fixed prefixes from the initial policy.

2. Individual Contributions of the Guidance Loss and Reference Loss

We evaluate four models on the CompreCap dataset. The table below reports paired accuracy for Obj, Attr, Rel, and WH, along with accuracy on WH questions with correct and incorrect premises.

ModelObjAttrRelWHWH Correct-Premise Acc.WH Incorrect-Premise Acc.
Base74.3873.8264.2547.1385.9157.93
Guidance-only73.7177.1167.0158.9477.4579.60
Reference-only74.2273.8264.4948.0186.4858.31
Full OPD (Ours)75.1177.2366.1758.1283.1372.96

The guidance loss provides the main corrective gains: compared with Base, WH paired accuracy improves by +11.81%, while accuracy on WH questions with correct premises decreases by 8.46%. This indicates that strengthening correction alone can reduce the ability to answer valid questions.

The reference loss complements the guidance loss: it preserves gains over the base model on Attr, Rel, and WH while improving Obj accuracy over Guidance-only to 75.11%, exceeding Reference-only and the baseline. This suggests that the reference term constrains capability shifts during correction, helping balance corrective gains with capability preservation.

3. Teacher-prompt Controls

Differences across question types: To analyze the effects of different teacher prompts on performance, we evaluate the same frozen Qwen3-VL-8B baseline on CompreCap and DOCCI under identical evaluation conditions, comparing three input settings: the original direct input , a generic verification reminder, and question-specific verification subquestions . Paired accuracy for each question type is reported below.

DatasetTypeDirect(Qwen v3)GenericSpecificS−D (pp)S−G (pp)
CompreCapObj74.7376.1674.67−0.06−1.49
CompreCapAttr73.3475.1376.69+3.36+1.56
CompreCapRel64.1167.6665.47+1.36−2.20
CompreCapWH47.2558.5070.44+23.18+11.94
DOCCIObj59.2659.6859.78+0.52+0.10
DOCCIAttr62.5864.7766.47+3.88+1.70
DOCCIRel53.6355.5954.84+1.21−0.75
DOCCIWH37.7347.7359.09+21.36+11.36

The benefits of different teacher prompts vary across question types. Generic improves all question types on both datasets over Direct, while Specific further improves WH performance. These results indicate that generic reminders provide broader verification benefits, whereas question-specific subquestions primarily benefit WH premise verification, consistent with our goal of improving WH performance on FINER.


Weakness 3 & Question 3: Further analysis of the non-wh regressions and increased rejection rate of valid questions would help clarify the method’s benefits and limitations.

Thank you for your suggestion. This pattern is consistent with our design rationale. Motivated by FINER's limited ability to mitigate false-positive hallucinations in WH questions, we developed OPD building on FINER's study of false-positive hallucinations, aiming to improve its weaker false-premise handling in WH questions while preserving performance on other question types as much as possible.

To this end, we combine the guidance loss with on-policy sampling. Compared with FINER-DPO, this yields clear improvements in WH paired accuracy across both datasets and both backbones. For Qwen3-VL-8B, the gains on the two datasets reach 11.37% and 10.59%, respectively. The non-WH regressions are smaller than the WH gains, resulting in better overall performance.

To mitigate the increased rejection of valid questions induced by the guidance loss, we introduce a reference loss anchored to the frozen base model. Compared with using the guidance loss alone, this raises WH positive-query accuracy from 77.45% to 83.13%.


Other Weaknesses:

Regarding the concern that the benefit may be limited to teacher prompting, we clarify that the full method not only improves teacher prompting but also transfers the guidance along the student's actual response trajectories and removes the additional guidance context at deployment. The additional on-policy sampling ablation supports this contribution. We will include the non-WH regressions and over-rejection of valid questions in the Limitations section of the revised paper. Regarding evaluation processing, we clarify that no additional verification was used; only response formatting was standardized for a fairer comparison.

Dear Reviewer F5qT,

We sincerely appreciate your time and effort in providing such thoughtful reviews and insightful comments.

Could you please kindly let us know if our rebuttal has addressed your concerns?

If you have any further questions regarding our work, we would be delighted to address them.

If our responses have addressed your concerns, we would greatly appreciate your reconsideration of your overall assessment.

Best regards,

Authors

Reviewer_pYMxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the tendency of vision-language models to affirm nonexistent objects, incorrect attributes or relations, and questions containing false visual premises. The authors propose question-guided self-distillation. Verification subquestions derived from the original question are supplied as additional context to a frozen teacher, without providing answers to those subquestions. The student still answers from the original image and question and matches the teacher's token-level distributions along prefixes sampled from the student's own responses. A second Jensen-Shannon divergence term discourages excessive departure from the original model's distribution. At deployment, neither question decomposition nor additional verification calls are required. Experiments show substantial improvements in paired accuracy for wh-type positive-negative question pairs. However, accuracy on positive questions and several object, relation, and external factuality evaluations decreases, indicating that over-rejection and capability retention remain unresolved.

Critical strengths

The paper targets a specific and important form of hallucination: incorrectly accepting a false visual premise. Paired accuracy is an appropriate metric because it evaluates both correction of false premises and retention of valid answers. Using factual conditions implicit in the question as training guidance while preserving direct inference is practically appealing. The token-level self-distillation objective and original-model reference term are described in sufficient technical detail. A notable strength is the transparent presentation of negative results: the paper explicitly reports degradation on positive questions, regression in non-wh categories, mixed transfer to external factuality benchmarks, and the effects of evaluation-stage processing. The pair-level analysis of corrected and regressed examples also helps identify where the measured improvements actually originate.

Critical weaknesses

The current experiments do not provide a fully controlled comparison with FINER-DPO. The paper acknowledges differences in the training pools and some evaluation settings, that DOCCI was consulted during development, and that several general-capability scores use additional normalization or visual verification. Consequently, key comparisons may be confounded by data and processing differences. The main experiments appear to lack multi-seed uncertainty estimates, and the teacher-context ablation is based on a single run. The existing ablation compares question-specific context with only one generic reminder, but does not separately isolate the effects of question decomposition, current-student sampling, the reference objective, or the choice of Jensen-Shannon divergence. Improvements are concentrated in negative wh questions, while positive questions and several external benchmarks regress. The broad claim of reducing hallucination should therefore be stated more cautiously.

Questions for authors
  1. Could the authors compare FINER-DPO and the proposed method using exactly the same 60K training examples, training steps, hyperparameters, and evaluation processing?
  2. Could the authors report means and variances over multiple random seeds and provide paired significance tests?
  3. Can the effects of the verification subquestions, current-student sampling, original-model reference term, Jensen-Shannon divergence direction, and lambda weight be separately ablated?
  4. Which prompts or hyperparameters were developed using DOCCI, and could this have indirectly adapted the method to the test distribution?
  5. How can the increase in negative-query accuracy be retained without reducing positive-query accuracy?
6 comments

Weakness 1 & Question 1: Could the authors compare FINER-DPO and the proposed method using exactly the same 60K training examples, training steps, hyperparameters, and evaluation processing?

Thank you for your suggestion. We compared OPD and FINER-DPO again across the four question types on CompreCap and DOCCI using exactly the same 60K training examples, training steps, hyperparameters, and evaluation processing. The resulting trends and conclusions remain consistent with those in the original paper, demonstrating the stability of our method.

CategoryBaseOPDFINER-DPO
CompreCap Attr73.8277.2377.12
CompreCap Obj74.3875.1176.40
CompreCap Rel64.2566.1769.10
CompreCap WH47.1358.1246.79
DOCCI Attr62.6165.3966.12
DOCCI Obj59.1259.1062.78
DOCCI Rel53.8455.6158.92
DOCCI WH37.8946.9236.42

Weakness 2 & Question 2: Could the authors report means and variances over multiple random seeds and provide paired significance tests?

Thank you for your suggestion. Training in this field entails substantial computational costs. Most studies use a single fixed random seed and compare performance across multiple tasks and datasets to assess the effectiveness of their methods.

We follow FINER’s evaluation framework to ensure consistency, covering object, attribute, relation, and WH questions on FINER-CompreCap and FINER-DOCCI, alongside AMBER, CRPE_R, DASH-B, MMStar, ChartQA, MMVP, NaturalBench, and V*. Compared with FINER, our method achieves consistent improvements in WH paired accuracy, negative-query accuracy, and performance on AMBER and other external hallucination benchmarks. These results support the effectiveness of our method in handling false premises and addressing some external visual factuality tasks.

Using per-question prediction records, we conducted a paired statistical analysis comparing our method with FINER on the same evaluation samples:

Model / DatasetWH Paired Accuracy Gain (%)95% CI for the Difference
Qwen3 / DOCCI+10.59[9.82, 11.35]
InternVL / CompreCap+7.45[5.61, 9.29]
InternVL / DOCCI+5.64[4.97, 6.34]

These results provide statistical evidence for the improvements in WH paired accuracy in these settings, supporting our method's effectiveness in handling false premises.


Weakness 3 & Question 3: Can the effects of the verification subquestions, current-student sampling, original-model reference term, Jensen-Shannon divergence direction, and lambda weight be separately ablated?

Thank you for your suggestion. We address these components individually in the tables below.

1. Contributions of Question Types

To investigate factors that may account for these differences, we compare three settings using the same frozen Qwen3-VL-8B baseline under identical evaluation conditions on CompreCap and DOCCI: the original direct input, a generic verification reminder, and question-specific verification subquestions. Paired accuracy for each question type is reported below.

DatasetTypeDirect (Qwen3)GenericSpecificS−D (%)S−G (%)
CompreCapObj74.7376.1674.67−0.06−1.49
CompreCapAttr73.3475.1376.69+3.36+1.56
CompreCapRel64.1167.6665.47+1.36−2.20
CompreCapWH47.2558.5070.44+23.18+11.94
DOCCIObj59.2659.6859.78+0.52+0.10
DOCCIAttr62.5864.7766.47+3.88+1.70
DOCCIRel53.6355.5954.84+1.21−0.75
DOCCIWH37.7347.7359.09+21.36+11.36

Direct scores lower on WH than on the other question types on both datasets. Generic improves performance across all question types, while Specific further outperforms Generic on WH by 11.94% and 11.36%, respectively; its additional gains on other types are small or negative. Thus, question-specific subquestions provide stronger premise verification and corrective guidance for WH questions, concentrating student gains on WH and aligning with our goal of improving FINER’s low WH accuracy.

2. Contribution of Current-student Sampling. Using Qwen3-VL-8B-Instruct as the baseline, we compare the paper's on-policy sampling approach with a variant that uses fixed samples from the initial student.

ModelObjAttrRelWHMacro Average
OPD75.1777.1166.2157.9969.12
Fixed Initial Prefixes74.4173.7663.9752.3766.13

Metric: CompreCap paired accuracy (%, higher is better); the macro average weights the four question types equally.

The variant uses fixed prefixes generated in advance by the initial student, while student parameters continue to update. Both settings use the same question-specific verification context, loss function, and training budget.

On-policy sampling improves performance across all four question types, supporting its contribution in this setting: as the student policy evolves, distillation along its current response trajectories is more effective than reusing fixed prefixes from the initial policy.

3. Contribution of the Original-model Reference Term

We evaluate four models on the CompreCap dataset. The table below reports paired accuracy for Obj, Attr, Rel, and WH, along with accuracy on WH questions with correct and incorrect premises.

ModelObjAttrRelWHWH Correct-Premise Acc.WH Incorrect-Premise Acc.
Base74.3873.8264.2547.1385.9157.93
Guidance-only73.7177.1167.0158.9477.4579.60
Reference-only74.2273.8264.4948.0186.4858.31
Full OPD (Ours)75.1177.2366.1758.1283.1372.96

The guidance loss provides the main corrective gains: compared with Base, WH paired accuracy improves by +11.81%, while accuracy on WH questions with correct premises decreases by 8.46%. This indicates that strengthening correction alone can reduce the ability to answer valid questions.

The reference loss (original-model reference term) complements the guidance loss: it preserves gains over the baseline on Attr, Rel, and WH while improving Obj accuracy over Guidance-only to 75.11%, exceeding Reference-only and the baseline. This suggests that the reference term constrains capability shifts during correction, helping balance corrective gains with capability preservation.

4. Contributions of the JSD Mixing Coefficient β and Reference Loss Weight λ

We compare model performance under different β and λ settings on the CompreCap dataset. The table below reports paired accuracy for each of the four question types and its macro average, as well as macro-average accuracy on questions with correct and incorrect premises.

SettingObjAttrRelWHMacro Paired Acc.Correct-Premise Macro Acc.False-Premise Macro Acc.
β=0.5, λ=0.575.2777.0566.2158.4369.2490.3074.61
β=0.5, λ=071.9075.6164.5361.5368.4089.3774.54
β=0.5, λ=275.1774.6664.4449.2165.8792.1769.64
β=0.1, λ=0.575.7576.9365.6157.2368.8889.7873.80
β=0.9, λ=0.575.8477.2366.7857.0469.2290.0974.46
  • β\beta has a limited impact on overall performance. With λ=0.5\lambda=0.5 fixed, relative to the default β\beta=0.5, decreasing β\beta to 0.1 reduces macro-averaged paired accuracy, correct-premise accuracy, and false-premise accuracy by no more than 0.81% each; increasing β\betato 0.9 reduces these metrics by no more than 0.21% each. The method’s overall gains do not depend on fine-tuning β\beta.
  • λ\lambda primarily affects trade-offs across question types and helps preserve valid answers. With β\beta=0.5 fixed, varying λ\lambda yields a maximum increase of 1.87% and a maximum decrease of 0.93% in correct-premise accuracy relative to the default setting. False-premise accuracy and macro-averaged paired accuracy both decrease, with maximum declines of 4.97% and 3.37%, respectively. These results indicate that a moderate reference constraint helps preserve correct-answering ability while maintaining false-premise correction. An overly strong reference constraint, however, yields gains in capability retention at the cost of larger losses in false-premise correction, particularly for WH questions.

Weakness 4 & Question 4: Which prompts or hyperparameters were developed using DOCCI, and could this have indirectly adapted the method to the test distribution?

We clarify that neither the prompts nor the training hyperparameters were developed or selected using FINER-DOCCI. Therefore, this selection process did not indirectly adapt the method to the DOCCI test distribution.


Weakness 5 & Question 5: How can the increase in negative-query accuracy be retained without reducing positive-query accuracy?

Thank you for your suggestion. Adding the reference term helps mitigate the loss in positive-query accuracy. To verify its contribution, we conducted a reference-loss ablation: keeping the guidance-term coefficient fixed, we compared the model using only the guidance termwith the full model that also includes the reference term (Ours). Results on CompreCap are shown below:

TypeAccG+\mathrm{Acc}^{+}_{\mathrm{G}}AccOPD+\mathrm{Acc}^{+}_{\mathrm{OPD}}AccG−\mathrm{Acc}^{-}_{\mathrm{G}}AccOPD−\mathrm{Acc}^{-}_{\mathrm{OPD}}
Obj96.60%96.98%74.03%75.56%
Attr89.99%90.35%78.97%80.05%
Rel88.18%91.59%67.15%68.32%
WH81.74%83.13%82.82%73.28%

G denotes Guidance-only, and OPD denotes our full model. Acc+\mathrm{Acc}^{+} and Acc−\mathrm{Acc}^{-} denote positive- and negative-query accuracy, respectively.

Adding the reference term improves both positive- and negative-query accuracy on Obj, Attr, and Rel. For WH, positive-query accuracy increases by 1.39 percentage points. Although negative-query accuracy decreases, it remains at 73.28%, substantially above Base's 58.18% (by 15.10 percentage points). These results support that the reference term can partially restore valid-answer retention while preserving corrective gains on negative queries, but does not fully eliminate the loss in positive-query accuracy relative to Base.

Reviewer_ZMSiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Vision-language models (VLMs) may generate responses that affirm the presence of nonexistent objects, assign incorrect attributes or relationships, or accept incorrect premises embedded in a question. To address these issues, this paper proposes a method in which sub-questions derived from the original question are provided to a teacher model only during training. The teacher’s behavior is then distilled into a student model. The proposed method generates verification sub-questions using rules tailored to each question type. The authors compare their method and the existing method using FINER-CompreCap and FINER-DOCCI, and examine both its effectiveness and the associated trade-offs.

Critical strengths

The proposed method offers several noteworthy strengths:


  • Learning without ground-truth answers to verification sub-questions.

The authors present a framework that generates verification sub-questions from the original question and leverages the teacher’s token distributions conditioned on student-generated response prefixes, without requiring ground-truth answers to these sub-questions. To the best of my understanding, the framework nevertheless relies on predefined question-type annotations in the dataset, such as object and WH categories. It may therefore be helpful to distinguish the absence of sub-question answer supervision from a fully label-free setting.


  • Notable improvements on WH questions.

The proposed method demonstrates particularly encouraging improvements in paired accuracy for WH questions compared with FINER-DPO 60K, as illustrated in Table 1. These gains should be considered alongside the trade-offs observed for other question types (Obj, Attr, and Rel), where performance may be lower than that of the comparison method.


  • Inference without the additional components.

An appealing aspect of the framework is that verification sub-questions, the teacher model, and the reference model are used only during training. The trained student model is designed to answer questions directly from an image and the original question, without requiring these additional components at inference time.

Critical weaknesses

I appreciate the authors' efforts to address this challenging problem. While recognizing the contributions of the proposed method, I have several concerns regarding its robustness, the evidence supporting the individual components, and the scope of its advantages over existing approaches. While the proposed method offers valuable contributions, several aspects would benefit from further clarification and empirical investigation.


  • The relationship between sub-question quality and final performance warrants further investigation.

The proposed method generates verification sub-questions using predefined rules. To the best of my understanding, their quality depends on both the assigned question category (e.g., object or WH) and the rules associated with that category. The reported results do not appear to establish how sensitive the method is to incorrect category assignments or imperfectly generated sub-questions. Evaluating these cases would help clarify whether the benefits primarily depend on accurate question decomposition or whether the method remains robust when the decomposition is imperfect.


  • The individual contributions of the loss components could be more clearly established.

The framework incorporates reference and guidance components, but their respective contributions are not yet sufficiently clear from my reading of the reported experiments. In particular, a more systematic sensitivity analysis of β\beta and λ\lambda, together with ablations that isolate each component, would help distinguish their individual effects from their combined impact. Such analyses would provide stronger quantitative evidence regarding the role of each component and the robustness of the method to coefficient selection.


  • The scope of the advantages over DPO requires further clarification.

As discussed in the “Strengths” section, the proposed method achieves encouraging improvements on WH questions. However, these gains are not consistent across question categories, with lower performance than the DPO baseline reported for Obj, Attr, and Rel questions. I therefore interpret the current evidence as supporting benefits under particular false-premise question settings, rather than establishing a general advantage over DPO. Additional experiments and discussion would help identify the conditions under which the proposed method is preferable, explain the category-dependent trade-offs, and clarify the scope of its advantages over DPO.

Questions for authors

I thank the authors for their work on this important problem. To better understand the proposed method and the scope of its applicability, I would appreciate clarification on the following points.


  • Q1. Robustness to the quality of generated sub-questions,

The proposed method generates verification sub-questions using rules tailored to each question type. Could the authors clarify how errors in question-type assignments or imperfections in the generated sub-questions affect the final performance? In particular, it would be helpful to understand how robust the method is when the sub-questions do not accurately reflect the original question. If my understanding is correct, question-type labels are provided as metadata in the dataset. Could the authors confirm whether this is the case? If so, I would appreciate further discussion of how the method could be applied to data without question-type labels or to questions that combine elements of multiple types.


  • Q2. Contributions of individual loss components and sensitivity to λ\lambda and β\beta.

Could the authors clarify the respective contributions of the reference loss and the guidance loss to the observed performance improvements? Ablation results isolating these components would be helpful in understanding their individual roles. It would also be valuable to understand the method’s sensitivity to the JSD mixing coefficient β\beta and λ\lambda. How does varying these coefficients affect accuracy on questions with incorrect premises and those with correct premises, respectively? Such an analysis would help clarify the trade-offs associated with coefficient selection.


  • Q3. Variation in improvements across question types and trade-offs relative to DPO

In Table 1, improvements over FINER-DPO 60K appear particularly pronounced for WH questions, whereas performance on Obj, Attr, and Rel questions is comparable to or lower than that baseline in some settings. Could the authors offer insight into the factors that may account for these differences across question types? It would also be helpful to clarify the conditions under which the proposed method would be preferable to DPO, together with the associated trade-offs. A discussion of specific application scenarios, where possible, would further strengthen the practical interpretation of the results. Clarifying both the benefits and limitations would help readers better understand the intended scope of the proposed method.

5 comments

Weakness 1 & Question 1: The proposed method generates verification sub-questions using rules tailored to each question type. Could the authors clarify how errors in question-type assignments or imperfections in the generated sub-questions affect the final performance? In particular, it would be helpful to understand how robust the method is when the sub-questions do not accurately reflect the original question. If my understanding is correct, question-type labels are provided as metadata in the dataset. Could the authors confirm whether this is the case? If so, I would appreciate further discussion of how the method could be applied to data without question-type labels or to questions that combine elements of multiple types.

Thank you for your questions. We address the following four points.

1. Question types are read directly from dataset labels.

Each training example is already labeled as Obj, Attr, Rel, or WH. The model does not predict question types, so no assignment errors are introduced by type prediction.

2. Performance remains relatively stable with imperfectly generated sub-questions.

We use the same 60K training examples, initialization, and training hyperparameters, and the same evaluation procedure on the full CompreCap dataset, to compare the following settings:

  • Neutral-50%: Replace one sub-question in half of the training examples with a generic verification reminder.
  • Misleading-50%: Modify one known semantic condition in the same examples and at the same sub-question positions, leaving the remaining sub-questions unchanged.
ModelObjAttrRelWH
Original-60K75.6277.1164.5856.66
Neutral-50%75.1775.3764.4455.97
Misleading-50%77.2776.0964.9554.45

Compared with the original setting, changes across all question types are below 3%, with a maximum of 2.21%. The original setting still achieves the highest WH accuracy, at 56.66%, indicating that accurate question-specific verification guidance benefits WH questions. This aligns with our goal of improving WH question answering, which has lower scores and greater difficulty handling incorrect premises in FINER.

3. Question-type labels are provided as dataset metadata.

Question-type labels are provided as metadata in the dataset.

4. Applicability to data without question-type labels or questions that combine elements of multiple types.

Our method is developed and studied on the four question types with question-type labels defined in FINER. Applying our method to data without question-type labels or to questions that combine elements of multiple types is out of scope for this work.


Weakness 2 & Question 2: Could the authors clarify the respective contributions of the reference loss and the guidance loss to the observed performance improvements? Ablation results isolating these components would be helpful in understanding their individual roles. It would also be valuable to understand the method’s sensitivity to the JSD mixing coefficient and . How does varying these coefficients affect accuracy on questions with incorrect premises and those with correct premises, respectively? Such an analysis would help clarify the trade-offs associated with coefficient selection.

Thank you for your suggestion. We have added loss ablations and coefficient sensitivity experiments.

1. Individual Contributions of the Guidance Loss and Reference Loss

We evaluate four models on the CompreCap dataset. The table below reports paired accuracy for Obj, Attr, Rel, and WH, along with accuracy on WH questions with correct and incorrect premises.

ModelObjAttrRelWHWH Correct-Premise Acc.WH Incorrect-Premise Acc.
Base74.3873.8264.2547.1385.9157.93
Guidance-only73.7177.1167.0158.9477.4579.60
Reference-only74.2273.8264.4948.0186.4858.31
Full OPD75.1177.2366.1758.1283.1372.96

The guidance loss provides the main corrective gains: compared with Base, WH paired accuracy improves by +11.81%, while accuracy on WH questions with correct premises decreases by 8.46%. This indicates that strengthening correction alone can reduce the ability to answer valid questions.

The reference loss complements the guidance loss: it preserves gains over the base model on Attr, Rel, and WH while improving Obj accuracy over Guidance-only to 75.11%, exceeding Reference-only and the base model. This suggests that the reference term constrains capability shifts during correction, helping balance corrective gains with capability preservation.

2. JSD Mixing Coefficient β and Reference Loss Weight λ

We compare model performance under different β and λ settings on the CompreCap dataset. The table below reports paired accuracy for each of the four question types and its macro average, as well as macro-average accuracy on questions with correct and incorrect premises.

SettingObjAttrRelWHMacro Paired Acc.Correct-Premise Macro Acc.False-Premise Macro Acc.
β=0.5, λ=0.575.2777.0566.2158.4369.2490.3074.61
β=0.5, λ=071.9075.6164.5361.5368.4089.3774.54
β=0.5, λ=275.1774.6664.4449.2165.8792.1769.64
β=0.1, λ=0.575.7576.9365.6157.2368.8889.7873.80
β=0.9, λ=0.575.8477.2366.7857.0469.2290.0974.46
  • β\beta has a limited impact on overall performance. With λ=0.5\lambda=0.5 fixed, relative to the default β\beta=0.5, decreasing β\beta to 0.1 reduces macro-averaged paired accuracy, correct-premise accuracy, and false-premise accuracy by no more than 0.81% each; increasing β\betato 0.9 reduces these metrics by no more than 0.21% each. The method’s overall gains do not depend on fine-tuning β\beta.
  • λ\lambda primarily affects trade-offs across question types and helps preserve valid answers. With β\beta=0.5 fixed, varying λ\lambda yields a maximum increase of 1.87% and a maximum decrease of 0.93% in correct-premise accuracy relative to the default setting. False-premise accuracy and macro-averaged paired accuracy both decrease, with maximum declines of 4.97% and 3.37%, respectively. These results indicate that a moderate reference constraint helps preserve correct-answering ability while maintaining false-premise correction. An overly strong reference constraint, however, yields gains in capability retention at the cost of larger losses in false-premise correction, particularly for WH questions.

Weakness 3 & Question 3: In Table 1, improvements over FINER-DPO 60K appear particularly pronounced for WH questions, whereas performance on Obj, Attr, and Rel questions is comparable to or lower than that baseline in some settings. Could the authors offer insight into the factors that may account for these differences across question types? It would also be helpful to clarify the conditions under which the proposed method would be preferable to DPO, together with the associated trade-offs. A discussion of specific application scenarios, where possible, would further strengthen the practical interpretation of the results. Clarifying both the benefits and limitations would help readers better understand the intended scope of the proposed method.

Thank you for your suggestion. We use teacher diagnostics to explain the sources of the gains and the scope of applicability.

1. Differences across question types:

To investigate factors that may account for these differences, we compare three settings using the same frozen Qwen3-VL-8B baseline under identical evaluation conditions on CompreCap and DOCCI: the original direct input , a generic verification reminder , and question-specific sub-questions. Paired accuracy for each question type is reported below.

DatasetTypeDirect(Qwenv3)GenericSpecificS−D (%)S−G (%)
CompreCapObj74.7376.1674.67−0.06−1.49
CompreCapAttr73.3475.1376.69+3.36+1.56
CompreCapRel64.1167.6665.47+1.36−2.20
CompreCapWH47.2558.5070.44+23.18+11.94
DOCCIObj59.2659.6859.78+0.52+0.10
DOCCIAttr62.5864.7766.47+3.88+1.70
DOCCIRel53.6355.5954.84+1.21−0.75
DOCCIWH37.7347.7359.09+21.36+11.36

Direct scores lower on WH than on the other question types on both datasets. Generic improves performance across all question types, while Specific further outperforms Generic on WH by 11.94% and 11.36%, respectively; its additional gains on other types are small or negative. Thus, Question-specific subquestions provide stronger premise verification and corrective guidance for WH questions. This concentrates the student’s gains on WH, consistent with our design goal of improving the relatively low WH accuracy reported in FINER.

2. Conditions Under Which OPD Is Preferable to FINER-DPO and the Associated Trade-offs.

To clarify the scope of applicability, we compare OPD and FINER-DPO on DOCCI WH:

MetricFINER-DPOOPDChange (%)
Correct-premise accuracy83.60%80.63%−2.97
Incorrect-premise accuracy46.09%61.03%+14.94
Paired accuracy36.42%47.01%+10.59

Compared with FINER-DPO, OPD improves accuracy on questions with incorrect premises by 14.94% and paired accuracy by 10.59%, while accuracy on questions with correct premises decreases by 2.97%. Therefore, OPD is more suitable when tasks primarily involve WH questions requiring visual premise verification and incorrectly accepting false premises carries a higher cost; the trade-off is reduced accuracy on some valid questions.

3.Scope of Applicability and Limitations.

Our method aims to mitigate VLM hallucinations, particularly false-positive hallucinations that incorrectly accept false visual premises. We focus on improving WH question answering, a weaker setting in FINER hallucination evaluation, to strengthen the model’s ability to identify and correct incorrect premises while retaining performance on other question types as much as possible. The method also improves performance on some tasks in general fine-grained and multimodal non-fine-grained hallucination evaluations, indicating that its benefits extend beyond FINER. Its limitations are that gains are not entirely consistent across question types, and stricter premise verification may lead to erroneous rejection of valid questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.