acceptodds
Under review as a conference paper at ICLR 2027

MARVEL: Multi-Agent Reinforced Visual Examination for Hallucination

Abstract

Hallucination remains a critical challenge for vision-language models (VLMs), especially in real-world scenarios where models generate image-inconsistent content. Existing mitigation methods typically rely on human annotations or external supervision, limiting their scalability to large-scale unlabeled image corpora. We propose MARVEL, a multi-agent framework that mitigates hallucination in VLMs via reinforced visual examination, without any ground-truth labels. MARVEL casts factual verification as a multi-agent examination process: model responses are decomposed into atomic claims, reformulated as visual questions, and examined against the image to produce consistency signals that serve as reinforcement learning rewards. To improve reward reliability, we further introduce a Visual Claim Regulation (VCR) mechanism that encourages informative, image-grounded descriptions while suppressing trivial or redundant claims. We evaluate MARVEL on 12 benchmarks spanning hallucination evaluation (+3.30), real-world VQA (+1.74), and general multimodal understanding benchmarks (+4.07), where it consistently outperforms existing methods without human-annotated factual supervision. Moreover, MARVEL generalizes effectively across model scales, demonstrating its potential as a scalable hallucination mitigation solution on large-scale real-world unlabeled image corpora.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Hallucination Mitigation, Multi-Agent, Self-Verification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UG1tOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MARVEL, an RL-based method for mitigating visual hallucinations without relying on external supervision. To construct reward signals for GRPO, the authors employ multiple agent roles to decompose the Solver's response into atomic claims, convert them into visual question-answer pairs, and verify their consistency with the input image. To prevent reward hacking in visual tasks, the authors further introduce a Visual Claim Regulation (VCR) mechanism that removes redundant claims, encourages informative descriptions, and penalizes claims that can be verified without relying on meaningful visual information.

Experiments on the Qwen2.5-VL model series across 12 hallucination and general multimodal understanding benchmarks demonstrate that MARVEL effectively reduces hallucinations while improving general multimodal performance. Ablation studies further validate the contributions of the individual VCR components.

Critical strengths
  • Simple and practical framework. The proposed method is straightforward and intuitive. It does not require human annotations or introduce additional inference-time overhead.

  • Visual-task-specific reward design. The authors explicitly consider the characteristics of visual tasks and design reward mechanisms to address potential reward hacking in visual self-verification.

Critical weaknesses
  • Limited coverage of object existence. Object existence is one of the most fundamental aspects of visual hallucination. While Section 3.3 explicitly mentions object existence as a type of atomic claim, the claim extraction prompt in Appendix A.1 restricts the supported claim types to color, position, and quantity, seemingly excluding object existence. Interestingly, Table 1 shows a slight decrease in existence recognition performance compared to the backbone model, which might be related to this limitation.

  • Heuristic reward design. Some components of the reward function appear somewhat heuristic and may introduce unintended biases. For example, the claim-count reward rcntr^{cnt} uses a fixed target ntarget=constantn_{\text{target}}=\text{constant}, although the appropriate number of claims naturally varies with the complexity of the query. This may encourage the model to generate unnecessary details for simple questions or insufficient details for more complex ones. Similarly, the quality penalty rqualr^{qual} assumes that claims remaining verifiable under image degradation are less informative. However, certain large-scale visual structures are inherently more robust to degradation, which does not necessarily make the corresponding claims less meaningful. These design choices may therefore introduce undesirable biases into the training process.

Questions for authors
  • Why does the claim extraction prompt in Appendix A.1 exclude object existence claims, despite their importance for visual hallucination mitigation and their inclusion in the description of atomic claims in Section 3.3?
7 comments
AuthorsOpenReview ↗

Response to Reviewer UG1t [1/2]

Thank you for the reviews.

To Q and W1: Section 3.3 lists object existence among the claim types (Lines 193–194) while the extractor admits colour, position and quantity. Appendix B.1 states the extractor's three types and how existence is adjudicated.

Existence is excluded as a claim type because a bare existence question is the degenerate case VCR suppresses, and the logs now let us show this rather than assert it. Resolving the quality penalty by claim type over the 2,835 rollouts behind Figure 7 (Table 18 of the revised version), the penalty rate rises monotonically with how degradation-invariant a claim is: colour 1.1%, lateral positions 9.9%, counts of 2–3 12.9%, counts of one 27.6%, flip-invariant positions (center/top/bottom) 43.4%. "Is there an X?" survives all three operators and therefore sits past the end of that scale — which is exactly what Figure 3 shows for claim (1), "These are some objects", labelled Trivial. Adding it as a type would feed rcntr_{cnt} with claims that rqualr_{qual} then removes.

Existence is nevertheless adjudicated, through the attribute claims, and it fires continuously. For an object the image does not support, the Checker returns "unknown" (Lines 864–874) and ANSWER CHECK scores 0 by rule (Lines 876–885); measured abstention is 20.9% of colour claims, 7.8% of quantity claims and 3.3% of position claims, each scoring zero. Because rconsr_{cons} is a mean while rcntr_{cnt} is capped (Equation (6)), elaborating on an unsupported object never pays: at n=4n = 4 verified claims, a fifth unsupported claim costs 0.13–0.17 in rconsr_{cons} and gains 0.10 in rcntr_{cnt}.

On the AMBER Existence change, the structural fact is headroom. Across the eight per-metric columns of Table 1, AMBER Existence has the smallest headroom of all of them — the base model is at 94.90, leaving 5.10 points — and headroom predicts the gain at Pearson r=+0.77r = +0.77, Spearman ρ=+0.71\rho = +0.71 (n=8n = 8, Table 21 of the revised), with MME Color capturing 50% of its headroom and AMBER Relation 36%. On the discriminative object-existence benchmark with room to move, POPE at 5,127 samples (Line 958), MARVEL improves 86.51 → 87.04. If the missing type degraded existence grounding, POPE is where it would appear.

AuthorsOpenReview ↗

Response to Reviewer UG1t [2/2]

To W2: Fixed ntarget=5n_{target} = 5. At convergence the cap is binding, so rcntr_{cnt} exerts no marginal pressure to add claims: median n=6n = 6 against ntarget=5n_{target} = 5, and 92.8% of responses have n≥5n \geq 5 at step 200 (25.5% at step 0), where Equation (6) is saturated at λcnt=0.5\lambda_{cnt} = 0.5. A saturated term cannot encourage unnecessary detail. The query-complexity mismatch also does not arise in training, because the claims rcntr_{cnt} counts come from a fixed description clause that is identical for all 32K examples; the varying question is verified separately as a single QA pair and does not enter rcntr_{cnt}.

The measured behaviour matches the cap: claims per response rise from 3.63 to about 6.2 and then plateau, response length settles at about 126 words, 86.2% of responses carry all three claim types, and no response yields an empty claim set (Lines 1112–1119). Halving or doubling λcnt\lambda_{cnt} or λqual\lambda_{qual} moves the AMBER average by at most 0.91 points, every setting staying +2.6 to +3.5 above the base model, and λcnt=1.0\lambda_{cnt} = 1.0 slightly lowers it (Lines 451–453, 1024–1025).

Degradation-robust structures. Your reading of the mechanism is right, and its consequence is bounded. Because the operators are composed and Equation (7) requires the degraded answer to still match the original-image reference, claims whose evidence the operators destroy are exempt rather than penalised: grayscale removes colour evidence, 96.6% of degraded colour answers are "unknown", and colour claims are penalised 1.1% of the time. The claims that do get selected are the blur-invariant ones, at 43.4% for flip-invariant positions and 27.6% for counts of one.

Three measurements bound how far this can bias training. ∣rqual∣|r_{qual}| is 5.9% of mean rSVr_{SV} at convergence (rconsr_{cons} 0.787, rcntr_{cnt} 0.489, ∣rqual∣|r_{qual}| 0.071), with mean ∣rqual∣=0.085|r_{qual}| = 0.085 against the rcntr_{cnt} cap of 0.5 and only 0.39% of rollouts reaching the cap. Its triviality labels agree with human judgment in 89.5% of cases (Table 16). And the claim mix does not collapse, staying at colour 40.5% / position 24.4% / quantity 35.0% (Figure 7(d)). Consistently, rqualr_{qual} is the smallest contributor in ablation (−0.71 / −0.87 / −1.28, Table 4), yet removing it costs performance in all three categories. The term also selects in the intended direction: 87.7% of penalised claims were judged correct by the Checker against 61.9% of unpenalised claims, so it withdraws reward from claims that are easy to verify, which is the failure mode of Figure 2.

We welcome further discussion if any concerns remain.

Dear Reviewer UG1t,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

Reviewer_g57tOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MARVEL, a multi-agent reinforcement learning framework for mitigating hallucinations in vision-language models without ground-truth annotations. It employs four agent roles, Solver, Proposer, Regulator, and Checker, to generate responses, extract atomic visual claims, regulate claim quality, and verify factual consistency against images. Its main contribution is Visual Claim Regulation (VCR), which combines semantic deduplication, claim-count rewards, and image-degradation-based quality penalties. The resulting rewards optimize the Solver through Group Relative Policy Optimization (GRPO). Experiments cover 12 benchmarks and three model scales, 3B, 7B, and 32B.

Critical strengths
  • Well-motivated problem formulation. MARVEL addresses the practical challenge of constructing visual factuality rewards without human annotations. Identifying reward exploitation through redundant and uninformative claims provides a reasonable motivation for the regulation mechanism.

  • Comprehensive framework and empirical evaluation. MARVEL integrates claim decomposition, visual verification, reward regulation, and reinforcement learning into a unified framework. Experiments across 12 benchmarks and three model scales report positive average improvements, while ablations provide evidence for the contributions of the proposed components.

Critical weaknesses
  • Restricted claim coverage relative to the method description. Section 3.3 broadly describes atomic visual claims, and Figure 3 includes examples involving object existence, relative spatial relations, and text on signs. However, the Claim Extract prompt in Appendix A.1 restricts extraction to color, position, and quantity. Positions are limited to five absolute image locations, and quantities require explicit numbers. This leaves no explicit extraction category for actions, complex relations, or textual semantics. Object existence may be checked indirectly through attribute claims, but its coverage without these attributes is unclear. The paper should reconcile these restrictions with its broader description of claim-level verification.

  • Limited methodological novelty. MARVEL builds on the Proposer–Checker paradigm by adding semantic deduplication, claim-count rewards, and image-degradation penalties. Only the Solver is updated, while the remaining roles use frozen models or fixed procedures. These are relatively straightforward extensions of self-verification and reward shaping. Although Table 4 supports their empirical usefulness, their advantages over simpler combinations of existing mechanisms are not sufficiently established.

  • Uneven gains relative to the training complexity. On POPE, MARVEL improves over the base model by 0.53 points, from 86.51 to 87.04, but slightly underperforms EvoLMM at 87.29. AMBER Existence decreases by 0.32 points relative to the base model. The VQA and mixed benchmark averages improve over EvoLMM by 0.86 and 1.52 points, respectively. These positive but uneven gains warrant a clearer assessment of the trade-off against additional training cost.

  • Insufficient justification of the image-degradation quality penalty. Answer consistency after degradation does not necessarily indicate that a claim is trivial or insufficiently image-grounded. Valid counts and coarse positions may remain recognizable after blurring or grayscale conversion. Conversely, grayscale conversion and horizontal flipping can remove or change the evidence required for color and positional claims. The penalty may therefore introduce category-dependent biases. Table 4 supports its aggregate contribution, but separate operator ablations and claim-category analyses are missing.

  • Incomplete analysis of verification reliability and efficiency. A frozen VLM-based Checker can share perceptual errors or language priors with the Solver, potentially rewarding incorrect claims. The reported human audit of 200 claims and Pearson correlation of 0.83 provide useful evidence, but do not characterize reward false positives in difficult cases. The number of majority-voting runs K in Equation (8) is also unclear. Table 6 reports 281.21 seconds per training step for MARVEL versus 180.02 seconds for SelfCheck, while EvoLMM’s runtime is estimated. Stage-wise timings and comparisons under matched total training compute would help assess efficiency and scalability.

Questions for authors
  1. Claim coverage. How should the restricted Claim Extract prompt in Appendix A.1 be reconciled with the broader description and examples in Section 3.3 and Figure 3? How are claims outside color, absolute position, and explicit quantity handled?

  2. Contribution of VCR. What distinguishes VCR from a self-verification framework augmented with semantic deduplication and claim-count regularization? Which comparison best isolates its additional benefit?

  3. Performance–complexity trade-off. How does MARVEL compare with EvoLMM and SelfCheck under matched total training compute? Do the reported advantages persist under this comparison?

  4. Validity of the quality penalty. Can the effects of Gaussian blur, grayscale conversion, and horizontal flipping be separated? How do these operators affect rewards for color, position, and quantity claims?

  5. Verification reliability. What Checker configuration and number of voting runs K are used? Can the authors characterize cases where agreement between the Solver and Checker incorrectly rewards a false claim?

4 comments
AuthorsOpenReview ↗

Response to Reviewer g57t [1/2]

Thank you for the reviews.

To Q4 and W4: Your mechanism analysis is testable, so we tested it. The operators are composed (Lines 229–231) and Equation (7) penalises a claim only when the degraded answer still matches the reference answer from the original image. Resolving the penalty by claim type on the 2,835 logged rollouts (16,627 scored items):

Claim typenPenalisedDegraded answer = "unknown"
Colour5,6531.1%96.6%
Position, left/right (flip-sensitive)7969.9%50.0% (all positions)
Position, center/top/bottom (flip-invariant)2,08343.4%—
Position, answer outside the five templates59423.1%—
Quantity, count 13,33327.6%59.5% (all quantities)
Quantity, count 2–31,09612.9%—
Quantity, count ≥ 44455.2%—
Appended answer item2,6164.4%93.4%

Grayscale and flip do not bias the penalty against colour and lateral-position claims. They remove the evidence those claims need, the degraded answer becomes "unknown", the match fails, and the claim is exempted — colour 1.1%, left/right 9.9%. Figure 3 shows this case. What the penalty does select is claims that survive heavy blur, which is what the term is for.

Three measurements bound how far that selection can bias training. The term is a small fraction of the signal: ∣rqual∣|r_{qual}| is 5.9% of mean rSVr_{SV} at convergence (step 200: rconsr_{cons} 0.787, rcntr_{cnt} 0.489, ∣rqual∣|r_{qual}| 0.071), with mean ∣rqual∣=0.085|r_{qual}| = 0.085 against the rcntr_{cnt} cap of 0.5 and only 0.39% of rollouts reaching that cap (Lines 1117–1119). Its labels agree with human triviality judgments in 89.5% of cases (Table 16). And it does not collapse the claim distribution, which stays at colour 40.5% / position 24.4% / quantity 35.0% (Figure 7(d)). Consistent with a small, correctly-aimed term, rqualr_{qual} is the smallest contributor in ablation (−0.71 / −0.87 / −1.28, Table 4).

The term also fires in the intended direction: 87.7% of penalised claims were judged correct by the Checker, against 61.9% of unpenalised claims. It removes reward from claims that are easy to verify, which is the "easy but weak response" mode of Figure 2, not from claims the Checker could not confirm. The breakdown above is Table 18 of the revised.

To Q5 and W5: K=5K = 5 in Equation (8); the three frozen roles are the same Qwen2.5-VL-7B-Instruct checkpoint as the Solver on a separate endpoint; ϕ(⋅)\phi(\cdot) is all-MiniLM-L6-v2; the degraded view uses blur radius 10, grayscale and horizontal flip, composed. Appendix B.1 of records all of it.

Three measurements characterise reliability. The Checker is stable: of the 16,938 scored items with all five votes, 93.3% are unanimous, 4.4% split 4–1 and 2.3% split 3–2. Its uncertainty costs the Solver reward rather than granting it, because abstention scores 0 — measured abstention is 20.9% of colour claims, 7.8% of quantity and 3.3% of position, so the asymmetry runs against the Solver throughout training. And the claim-level reward complies with its own scoring rule: of 15,350 claim items, 3 (0.02%) were credited while the reference answer was "unknown". The residual cases of that kind sit on the appended answer item and are abstention against abstention, where neither side asserts a fact. Table 20 of the revised manuscript reports all three measurements.

On shared priors between Solver and Checker, the audit is the evidence that they do not dominate the reward: rconsr_{cons} tracks human factuality at r=0.83r = 0.83 with 94.5% extraction faithfulness (Lines 413–414). A Checker confirming the Solver's own errors would not produce that correlation. The Checker also never sees yy (Lines 159–161, 244), so it cannot be anchored by the Solver's phrasing.

AuthorsOpenReview ↗

Response to Reviewer g57t [2/2]

To Q1 and W1: The extractor covers colour, five absolute positions and explicit numerals; Appendix B.1 states this and how existence is adjudicated. The restriction is what makes verification a binary comparison over a closed answer space, which is where the 94.5% faithfulness and r=0.83r = 0.83 come from.

Existence is adjudicated through the attribute claims: the Checker returns "unknown" for an unsupported object and ANSWER CHECK scores 0 (Lines 876–885), which happens on 20.9% of colour claims. Because rconsr_{cons} is a mean while rcntr_{cnt} is capped, an unsupported claim never pays — at n=4n = 4 verified claims a fifth costs 0.13–0.17 in rconsr_{cons} and gains 0.10 in rcntr_{cnt}. On actions and relations, the claim grammar does not bound what the reward improves: Relation is the largest single gain in Table 1 at +8.84, and MME Position gains +4.17, although neither is an extraction type.

To Q2 and W2: MARVEL w/o VCR is the Proposer–Checker paradigm transferred to vision with a consistency-only reward, which is the comparison that isolates VCR (Table 4):

HallucinationVQAMixed
Qwen2.5-VL-7B (base)85.2471.4354.22
MARVEL w/o VCR86.6271.5953.96
MARVEL88.5473.1658.29

The naive transfer does not gain less; on mixed benchmarks it falls below the base model (Lines 410–412), and on BLINK Multi-view reasoning it collapses to 9.02 against 33.83 for the base model and 41.35 for MARVEL. Figure 6 shows why: single-agent self-check, MARCH and w/o VCR all show rapid reward growth with entropy collapse, and only the full method holds entropy above 0.1 with monotone reward (Lines 387–389, 429–431). VCR is therefore not a regulariser added to a working method — without it the transfer does not work.

The exploit it blocks is one the policy actively pursues: raw claim production per response rises from 4.99 to 9.02 under training while the merged share rises from 27.2% to 31.5%, so repetition pressure grows rather than being hypothetical. Updating only the Solver is also deliberate: a jointly trained Checker co-adapts with the policy and makes the signal non-stationary, and a frozen Checker is what makes the audited reward the same reward used in training.

To Q3 and W3: The unevenness is explained by headroom. Across the eight per-metric columns of Table 1 (Table 21 of the revised PDF):

MetricBaseHeadroomΔShare of headroom
AMBER Existence94.905.10−0.32−6.3%
VSR90.649.36+1.3914.9%
MME Color90.0010.00+5.0050.0%
POPE86.5113.49+0.533.9%
AMBER Attribute84.4515.55+1.9212.3%
MME Position82.5017.50+4.1723.8%
MME Scene77.5022.50+4.7521.1%
AMBER Relation75.6624.34+8.8436.3%

Headroom and gain correlate at Pearson r=+0.77r = +0.77, Spearman ρ=+0.71\rho = +0.71 (n=8n = 8), and the method captures 19.5% of available headroom on average. The two cells you cite are the two with least room: AMBER Existence has the smallest headroom of all eight metrics at 5.10 points, and POPE is a saturated existence benchmark. EvoLMM's 87.29 on POPE is higher than ours and we do not claim that cell.

On cost, the measured comparison is 281.21 s/step against the 171.24 s single-agent anchor, 1.64×, with 1 of 4 agents trained (Lines 460–472). The EvoLMM entry is marked as an estimate in Table 6 itself and no claim of ours rests on it. The trade-off the reviewer asks about is one-off: the three verification agents are discarded after training, so the deployed model carries zero added inference cost while leading EvoLMM by +2.43 / +0.86 / +1.52 on the three category averages.

We welcome further discussion if any concerns remain.

Dear Reviewer g57t,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

AuthorsOpenReview ↗

Have we addressed your concerns?

Dear Reviewer g57t,

Thank you again for the detailed review. Appendix B of the revised PDF is new and marked in blue. Table 18 resolves rqualr_{qual} by claim type, with the Checker's abstention rate alongside it (Q4/W4), Table 20 reports the vote margins, abstention rates and rule compliance of the Checker (Q5/W5), and Table 21 relates each metric's gain to its headroom (Q3/W3). B.1 records the full agent and reward configuration, including the three frozen roles, K=5K = 5, and the composed degradation (Q1/W1, Q5/W5).

Does this address your concerns, or is there a specific analysis you would still like to see? We would be glad to clarify anything further before the discussion period closes.

Reviewer_5PJ5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MARVEL aims to reduce visual hallucination without using factual answer labels. It turns a model response into a set of claims and visual questions, checks those questions against the image, and uses the resulting consistency scores as reinforcement-learning rewards. Its main addition is Visual Claim Regulation, which removes repeated claims, encourages the response to contain enough visual information, and penalizes claims that can be answered even when the image is heavily degraded.

The method is trained on GQA questions in a describe-then-answer format, while the reward generates its own image-grounded reference instead of using the stored answer label. The authors evaluate three Qwen2.5-VL model sizes across hallucination-related, visual question answering, and general multimodal benchmarks.

Critical strengths

I like the main motivation of the paper. A self-checking model can obtain a high reward by making generic statements or repeating easy claims, and VCR is a sensible way to discourage these behaviors. Checking individual claims is also easier to interpret than assigning one score to an entire response.

The empirical study is broad. It includes component ablations, training dynamics, several model sizes, and a small human audit of the reward. The full method improves the reported aggregate results consistently, and removing each VCR component hurts performance.

Critical weaknesses

My main concern is that the description reward is narrower than the paper suggests. It extracts only color, absolute image position, and explicit quantity. The GQA answer is checked separately and can cover other answer types, but facts elsewhere in the free-form description may be ignored. This matters because the motivating examples emphasize relations, object existence, and text in the image, which the description extractor does not directly cover.

I would also like more direct evidence that MARVEL reduces hallucination rather than improving general visual task performance. Most of the main benchmarks are discriminative, and the captioning experiment reports text-overlap scores rather than factuality or hallucination rates. The paper does not report multi-seed uncertainty or show training directly on raw image collections. Finally, the anonymous code raises reproducibility questions about the deduplication rule and whether the checking agents are actually frozen.

Questions for authors
  1. How much factual content is missed by the three-type description extractor, and how much of the reward comes from the separate answer branch?

  2. Which checkpoints implement the checking agents, and how should the deduplication and frozen-agent behavior in the released code be reconciled with the paper?

  3. Can the authors provide multi-seed results and a direct free-form hallucination evaluation?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5PJ5 [1/2]

Thank you for the reviews.

To Q1: The reward has two sources and we measured the split on the 2,835 rollouts behind Figure 7 and Appendix A.5: claim verification carries 87.2% of rconsr_{cons} and the response's answer to the user question carries 12.8% (mean weight 1/(n+1)=0.1761/(n+1) = 0.176, mean contribution 0.081, mean rconsr_{cons} 0.634). Both references are produced by the Checker from the image (IMAGE QA, Lines 864–874). No stored GQA answer enters the reward at any point, which is the annotation-free property the paper claims (Lines 112, 142). Appendix B.1 states this term explicitly.

On missed content, the relevant question is whether a three-type sample is enough to rank responses by factuality, and the audit answers it: rconsr_{cons} correlates with human factuality judgments of the response at Pearson r=0.83r = 0.83, with 94.5% extraction faithfulness (Lines 413–414, Table 16). The reward needs to order responses, not enumerate every fact. For scale, each ~126-word response yields about 6.2 claims with 0.1% falling outside the three types (Lines 1112–1116).

To Q2: All three frozen roles — Proposer, Regulator, Checker — are the same Qwen2.5-VL-7B-Instruct checkpoint as the Solver, served on a separate inference endpoint, with K=5K = 5 in Equation (8) and ϕ(⋅)\phi(\cdot) = all-MiniLM-L6-v2. Nothing stronger than the base model appears anywhere in the reward, which is what makes the method annotation-free in the sense the paper claims. They are frozen by construction: the reward function calls them over that endpoint, and they are never in the optimizer's parameter group — this is what Section 3.2 (Line 151) and Table 6's #Agents trained/total = 1/4 record (Lines 467–470).

The deduplication rule is type-aware, as Line 216 states: τ=0.8\tau = 0.8 for cross-type pairs and τ/2=0.4\tau/2 = 0.4 for same-type pairs. Recomputed on the logged claim sets, 6,388 of 21,750 raw claims are merged (29.4%), 88.2% of them same-type. The reason the rule is aggressive on same-type pairs is visible in the training dynamics:

StepRaw claims / responseKeptMerged
04.993.6327.2%
1009.026.1831.5%
2008.556.1627.9%

Raw claim production nearly doubles under training while the merged share rises, so the repetition channel is one the policy actively pursues, and removing deduplication costs −1.92 / −1.13 / −0.43 (Table 4). Appendix B.1 gives the exact thresholds and these counts.

To Q3 and W2: Free-form factuality is measurable on the model's own descriptions, and the training logs already contain it. Scoring every claim in the free-form description against the image (Table 19 of the revised):

StepClaims / responseVerified rateVerified claims / responseTrivial / response
03.6345.0%1.640.29
1006.1870.1%4.331.33
2006.1679.5%4.900.70

Image-verified statements per description rise 1.64 → 4.90 while the verified share rises 45.0% → 79.5%. The rate does not improve by saying less or by retreating to safe claims: the claim count rises at the same time, and trivial claims per response fall from 1.33 to 0.70 while the rate keeps climbing. Reward hacking would show the opposite signature. The judge here is the pipeline the audit validates at r=0.83r = 0.83, and this decomposition of Figure 6(d) is a property of the free-form text, not of a multiple-choice benchmark.

The discriminative results agree with it where free-form hallucination should show: AMBER Relation +8.84, MME Color +5.00, Position +4.17 (Lines 319–323). On COCO captioning, MARVEL gains +6.81 BLEU-1 and +6.51 ROUGE-L over the base model while the no-VCR variant stays at base level (Table 17, Lines 1097–1109) — against a fixed reference set, so the gain is in overlapping content.

On variance, the paper reports three independently trained models: Figure 4's 3 × 3 scale × category grid is positive in all nine cells (Lines 362–373), across 12 benchmarks and 74,160 evaluation samples (Line 917). The independent units there are the three training runs, and they vary the backbone rather than the random seed, which is the harder perturbation: the effect survives a change of model capacity from 3B to 32B under an identical recipe. At that evaluation size the category averages are not sampling noise, and the aggregate movements are +3.30 / +1.74 / +4.07. Accordingly we rest the claim on those averages and not on any individual sub-point movement.

AuthorsOpenReview ↗

Response to Reviewer 5PJ5 [2/2]

To W1: The extractor covers colour, five absolute positions and explicit numerals; Appendix B.1 states this and how existence is adjudicated. The restriction is what makes the reward checkable at all: these are the three types with a closed, normalisable answer space, which is why verification reduces to a binary string comparison with 94.5% faithfulness and r=0.83r = 0.83 against human judgment.

Existence is adjudicated through the attribute claims rather than as a separate type, and it fires continuously rather than hypothetically. For an object the image does not support, the Checker returns "unknown" and ANSWER CHECK scores 0 by rule (Lines 876–885); measured abstention rates are 20.9% of colour claims, 7.8% of quantity claims and 3.3% of position claims, each scoring zero. Because rconsr_{cons} is a mean while rcntr_{cnt} is capped, elaborating on an unsupported object never pays: at n=4n = 4 verified claims, a fifth unsupported claim costs 0.13–0.17 in rconsr_{cons} and gains 0.10 in rcntr_{cnt}.

On the relational and text examples, the claim grammar does not bound what the reward improves: Relation is the largest single gain in Table 1 at +8.84 and MME Position gains +4.17, although neither is an extraction type. Training the Solver to make verifiable colour, position and count statements improves the grounding that relational judgments draw on.

We welcome further discussion if any concerns remain.

Dear Reviewer 5PJ5,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

AuthorsOpenReview ↗

Have we addressed your concerns?

Dear Reviewer 5PJ5,

Thank you again for the detailed review. Appendix B of the revised PDF is new and marked in blue. Table 19 decomposes free-form claim verification by training step: image-verified statements per description rise 1.64 → 4.90 and the verified share rises 45.0% → 79.5%, with the claim count rising at the same time (Q3/W2). B.1 states the second term of rconsr_{cons} and its 12.8% share, with no GQA label entering the reward (Q1), the type-aware deduplication thresholds and the full agent configuration (Q2), and the three claim types with how object existence is adjudicated (W1).

Does this address your concerns? We would be glad to clarify anything further before the discussion period closes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief introduction to the revised PDF

We thank all reviewers for their time and feedback.

All new content is appended as Appendix B. Appendix B analyses data already reported in the paper — the 2,835 training rollouts of Figure 7 and Appendix A.5, and the per-metric results of Table 1.

  • B.1 Reward and agent specification. The Solver's training instruction verbatim, so that nn is seen to count description claims only; the second term of rconsr_{cons} (the response's answer, 12.8% of the reward, with no GQA label entering it at any point); the type-aware deduplication thresholds (τ=0.8\tau = 0.8 cross-type, τ/2=0.4\tau/2 = 0.4 same-type, 29.4% of raw claims merged); the frozen roles (the same Qwen2.5-VL-7B-Instruct checkpoint as the Solver, K=5K = 5, ϕ(⋅)\phi(\cdot) = all-MiniLM-L6-v2, composed degradation); and the three claim types, with object existence adjudicated through them via the Checker's "unknown". (5PJ5 Q1/Q2/W1, g57t Q1/Q5/W1/W5, UG1t Q/W1)
  • B.2, Table 18. The quality penalty by claim type, with abstention alongside it: grayscale and flip exempt colour and lateral-position claims (1.1% and 9.9% penalised) rather than biasing against them, and ∣rqual∣|r_{qual}| is 5.9% of rSVr_{SV} at convergence. (g57t Q4/W4, UG1t W2)
  • B.3, Table 19. Free-form claim verification by training step: image-verified statements per description rise 1.64 → 4.90 and the verified share 45.0% → 79.5%, with the claim count rising at the same time. (5PJ5 Q3/W2)
  • B.4, Table 20. Checker reliability: 93.3% of the 16,938 five-vote items unanimous, abstention by claim type, and 3 of 15,350 claim items credited against the scoring rule (0.02%). (g57t Q5/W5)
  • B.5, Table 21. Headroom against gain over the eight per-metric columns of Table 1: Pearson r=+0.77r = +0.77, Spearman ρ=+0.71\rho = +0.71. (g57t Q3/W3, UG1t W1)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.