- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes MARVEL, an RL-based method for mitigating visual hallucinations without relying on external supervision. To construct reward signals for GRPO, the authors employ multiple agent roles to decompose the Solver's response into atomic claims, convert them into visual question-answer pairs, and verify their consistency with the input image. To prevent reward hacking in visual tasks, the authors further introduce a Visual Claim Regulation (VCR) mechanism that removes redundant claims, encourages informative descriptions, and penalizes claims that can be verified without relying on meaningful visual information.
Experiments on the Qwen2.5-VL model series across 12 hallucination and general multimodal understanding benchmarks demonstrate that MARVEL effectively reduces hallucinations while improving general multimodal performance. Ablation studies further validate the contributions of the individual VCR components.
Critical strengths
-
Simple and practical framework. The proposed method is straightforward and intuitive. It does not require human annotations or introduce additional inference-time overhead.
-
Visual-task-specific reward design. The authors explicitly consider the characteristics of visual tasks and design reward mechanisms to address potential reward hacking in visual self-verification.
Critical weaknesses
-
Limited coverage of object existence. Object existence is one of the most fundamental aspects of visual hallucination. While Section 3.3 explicitly mentions object existence as a type of atomic claim, the claim extraction prompt in Appendix A.1 restricts the supported claim types to color, position, and quantity, seemingly excluding object existence. Interestingly, Table 1 shows a slight decrease in existence recognition performance compared to the backbone model, which might be related to this limitation.
-
Heuristic reward design. Some components of the reward function appear somewhat heuristic and may introduce unintended biases. For example, the claim-count reward uses a fixed target , although the appropriate number of claims naturally varies with the complexity of the query. This may encourage the model to generate unnecessary details for simple questions or insufficient details for more complex ones. Similarly, the quality penalty assumes that claims remaining verifiable under image degradation are less informative. However, certain large-scale visual structures are inherently more robust to degradation, which does not necessarily make the corresponding claims less meaningful. These design choices may therefore introduce undesirable biases into the training process.
Questions for authors
- Why does the claim extraction prompt in Appendix A.1 exclude object existence claims, despite their importance for visual hallucination mitigation and their inclusion in the description of atomic claims in Section 3.3?
7 comments
Response to Reviewer UG1t [2/2]
To W2: Fixed . At convergence the cap is binding, so exerts no marginal pressure to add claims: median against , and 92.8% of responses have at step 200 (25.5% at step 0), where Equation (6) is saturated at . A saturated term cannot encourage unnecessary detail. The query-complexity mismatch also does not arise in training, because the claims counts come from a fixed description clause that is identical for all 32K examples; the varying question is verified separately as a single QA pair and does not enter .
The measured behaviour matches the cap: claims per response rise from 3.63 to about 6.2 and then plateau, response length settles at about 126 words, 86.2% of responses carry all three claim types, and no response yields an empty claim set (Lines 1112–1119). Halving or doubling or moves the AMBER average by at most 0.91 points, every setting staying +2.6 to +3.5 above the base model, and slightly lowers it (Lines 451–453, 1024–1025).
Degradation-robust structures. Your reading of the mechanism is right, and its consequence is bounded. Because the operators are composed and Equation (7) requires the degraded answer to still match the original-image reference, claims whose evidence the operators destroy are exempt rather than penalised: grayscale removes colour evidence, 96.6% of degraded colour answers are "unknown", and colour claims are penalised 1.1% of the time. The claims that do get selected are the blur-invariant ones, at 43.4% for flip-invariant positions and 27.6% for counts of one.
Three measurements bound how far this can bias training. is 5.9% of mean at convergence ( 0.787, 0.489, 0.071), with mean against the cap of 0.5 and only 0.39% of rollouts reaching the cap. Its triviality labels agree with human judgment in 89.5% of cases (Table 16). And the claim mix does not collapse, staying at colour 40.5% / position 24.4% / quantity 35.0% (Figure 7(d)). Consistently, is the smallest contributor in ablation (−0.71 / −0.87 / −1.28, Table 4), yet removing it costs performance in all three categories. The term also selects in the intended direction: 87.7% of penalised claims were judged correct by the Checker against 61.9% of unpenalised claims, so it withdraws reward from claims that are easy to verify, which is the failure mode of Figure 2.
We welcome further discussion if any concerns remain.
Dear Reviewer UG1t,
We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.
Response to Reviewer UG1t [1/2]
Thank you for the reviews.
To Q and W1: Section 3.3 lists object existence among the claim types (Lines 193–194) while the extractor admits colour, position and quantity. Appendix B.1 states the extractor's three types and how existence is adjudicated.
Existence is excluded as a claim type because a bare existence question is the degenerate case VCR suppresses, and the logs now let us show this rather than assert it. Resolving the quality penalty by claim type over the 2,835 rollouts behind Figure 7 (Table 18 of the revised version), the penalty rate rises monotonically with how degradation-invariant a claim is: colour 1.1%, lateral positions 9.9%, counts of 2–3 12.9%, counts of one 27.6%, flip-invariant positions (center/top/bottom) 43.4%. "Is there an X?" survives all three operators and therefore sits past the end of that scale — which is exactly what Figure 3 shows for claim (1), "These are some objects", labelled Trivial. Adding it as a type would feed rcnt with claims that rqual then removes.
Existence is nevertheless adjudicated, through the attribute claims, and it fires continuously. For an object the image does not support, the Checker returns "unknown" (Lines 864–874) and ANSWER CHECK scores 0 by rule (Lines 876–885); measured abstention is 20.9% of colour claims, 7.8% of quantity claims and 3.3% of position claims, each scoring zero. Because rcons is a mean while rcnt is capped (Equation (6)), elaborating on an unsupported object never pays: at n=4 verified claims, a fifth unsupported claim costs 0.13–0.17 in rcons and gains 0.10 in rcnt.
On the AMBER Existence change, the structural fact is headroom. Across the eight per-metric columns of Table 1, AMBER Existence has the smallest headroom of all of them — the base model is at 94.90, leaving 5.10 points — and headroom predicts the gain at Pearson r=+0.77, Spearman ρ=+0.71 (n=8, Table 21 of the revised), with MME Color capturing 50% of its headroom and AMBER Relation 36%. On the discriminative object-existence benchmark with room to move, POPE at 5,127 samples (Line 958), MARVEL improves 86.51 → 87.04. If the missing type degraded existence grounding, POPE is where it would appear.