- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces Model-Substitution Risk Control (MSRC), a training-free method for reducing visual hallucinations in VLMs. Rather than simply deleting unsupported object claims, MSRC first attempts to replace them with visually supported alternatives. The method checks each claim against the image and uses the frozen VLM to propose replacements for unsupported claims. A replacement is accepted only if it passes additional checks and visual verification; otherwise, the original claim is deleted. The authors use held-out data to calibrate the verification thresholds, aiming to reduce hallucinations while retaining useful information. Experiments across image captioning, visual question answering, and hallucination benchmarks show improvements in hallucination rates and answer quality over several baselines.
Critical strengths
The paper measures hallucination risk over the original set of claims rather than only the claims that remain after filtering. This is useful because simply deleting claims should not be rewarded without considering how much information is lost. The comparison with verification-only filtering is useful, as both methods use the same verification and calibration setup, making it easier to assess the contribution of substitution.
Critical weaknesses
The paper presents MSRC as a risk-control method, but the proposed certification does not provide a formal guarantee on population risk. It is therefore unclear what level of risk control the method actually provides. The method focuses on correcting object claims in generated text, but it is not clear how it handles yes/no, counting, and attribute questions, especially given the large improvements reported on HaloQuest, POPE, and MME. It is not clear whether all baselines were evaluated under the same settings, including the evaluation judges, image subsets, and model backbones.
Questions for authors
Could the authors explain how MSRC handles short-answer VQA, particularly yes/no, counting, and attribute questions? The method mainly describes object-claim replacement and deletion, so it is unclear how the results on POPE and MME are obtained. How often do substitutions repeat object categories already mentioned in the caption, and are they correct at the object-instance and location level? Could the authors clarify how much of the improvement over verification-only filtering comes from substitution rather than differences in the selected verification thresholds?
6 comments
Dear Reviewer,
Thank you for your time and constructive comments. We kindly invite you to revisit our rebuttal and let us know if any questions remain unanswered.
Dear Reviewer,
We would like to kindly follow up on our previous message and request you to revisit our rebuttal. We would greatly appreciate any further feedback or clarification on whether your concerns have been addressed.
Thank you for your time and consideration.
Reviewer GSTT
We thank the reviewer for their thoughtful questions and constructive feedback.
W1 — Level of risk control. We agree: the submitted criterion is not a high-probability bound, and Table 8's "Bound ≤ α" and Figure 4's "certified" overstate it. We now give a formal test, fixed before scoring, on 1,500 never-scored COCO images per backbone (disjoint from calibration and evaluation; image as the unit; α = δ = 0.05):
The level provided is P(certify ∧ R(λ̂) > α) ≤ δ: the chance of certifying a policy whose expected per-image rate of unsupported COCO object claims exceeds 5% is at most 5%. All six tests also pass jointly (Bonferroni). It covers COCO object existence only, not attributes, counts or relations. The paper's 250-image half does not pass (0.060, 0.069, 0.060), and we withdraw that reading.
W2/Q1 — Short-answer handling. Eq. (4) edits object claims in free text and has no short-answer rule:
MSRC thus has no mechanism for the MME gains; Table 13 was not reproduced and moves to the appendix, marked. POPE asks exactly the object-existence question our verifier scores, so MSRC answers it directly from V; we keep Table 12 and will document this setting. Table 12 (average over random, popular and adversarial splits, accuracy / F1):
HaloQuest uses a cascade: entity verification, a TF-IDF question-type classifier and a logistic-regression selector (C = 0.5) choosing negation, abstention or the original answer. Substitution is off (calibration selected τ_sub = 2.0).
Nesting changes accuracy by −1.5 to +0.2 points; we withdraw the visual-challenge property .
W3 — Judges, subsets and backbones. No. Cited rows of Table 1 use their authors' splits, scorers and decoders; our HaloQuest and MMHal scores use Gemini 3.7 Flash, not GPT; MME mixes Qwen-VL baselines with our Qwen2.5-VL-3B, and CEBC uses Qwen2-VL-7B. Protocol alone can flip a result:
Q2 — Repeats, instances and location.
Repeats are common, but a repeated category is rarely a duplicate object (1 of 50, 2.0%). At instance and location level, 65.7% are fully correct, 16.4% have the right object in the wrong place or with wrong details, and 16.4% name an absent object. The audit (67 of 150 random items, one author, backbone-blind) is preliminary.
Q3 — Substitution vs. thresholds. τ₀ is identical for MSRC and VOF (Table 8, and across the α sweep), so thresholds explain nothing; the lanes differ only in substitution:
Substitution adds 0.10 to 0.22 points of coverage but gives no meaningful CHAIR reduction, so the hallucination reduction comes from calibrated verification and deletion, not substitution. The submitted Table 1 gap does not match a stored run, and we replace those rows.