acceptodds
Under review as a conference paper at ICLR 2027

VERVE: Verified Visual Search

Abstract

Recent visual search methods improve perception in high-resolution images, but these gains can come at a cost: reduced ability to recognize when no target satisfies the query (hallucination). We introduce Verified Visual Search (VERVE), a framework for measuring and mitigating this perception-hallucination trade-off. Our VERVE-MCQ benchmark pairs positive visual search queries with minimally edited negative counterparts for which no matching target exists, testing both target identification and query verification. We find that state-of-the-art visual search methods surprisingly underperform their own base models on VERVE-MCQ. To diagnose the root cause, VERVE-Binary simplifies the queries into a yes/no task. Yet the failure persists, indicating a broader loss of discriminativeness. To tackle this, we propose VERVE-Train, a GRPO-based method that trains models to verify rather than assume target existence, adding negative queries (plain mode) and further on-policy self-verification rewards (OPSV mode). Notably, our trained Qwen3.5-9B surpasses Gemini-3.7-Flash by 13.8% on VERVE-MCQ. Crucially, VERVE-Train effectively reduced hallucinations across six benchmarks and remain competitive on eight perception benchmarks. Furthermore, these improvements transfer across tasks, domains, and modalities on seven external benchmarks, demonstrating benefits beyond high-resolution visual search.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-modal Large Langugae Models, Hallucination, Visual Search

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3pJrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces VERVE, a framework for evaluating and mitigating the perception–hallucination trade-off in high-resolution visual search. It proposes VERVE-MCQ and VERVE-BINARY to assess models' ability to identify valid targets and reject false premises. The authors further introduce VERVE-TRAIN, a GRPO-based training framework that combines positive and negative queries with on-policy self-verification (OPSV) to improve visual discrimination.

Critical strengths
  • Insightful Empirical Findings. The paper provides empirical evidence that existing visual search methods can improve perception while degrading hallucination resistance compared to their base models.

  • Comprehensive Evaluation. The experiments cover multiple visual search methods and a broad range of perception, hallucination, and cross-task transfer benchmarks.

  • Strong Empirical Results. VERVE-TRAIN achieves substantial improvements on VERVE-MCQ while maintaining competitive perception performance and reducing hallucinations across multiple benchmarks.

Critical weaknesses
  • Limited Novelty of the Benchmark. The core task of distinguishing positive and negative visual queries has already been explored in benchmarks such as POPE, HallusionBench, FineCops-Ref, and GroundingME. Moreover, the benchmark construction pipeline appears relatively conventional. The authors should better clarify the fundamental differences and unique contributions compared to existing benchmarks.

  • Limited Contribution of OPSV. Most performance gains already come from Plain training. For instance, OPSV improves VERVE-MCQ accuracy by only 0.4 points over Plain on Qwen3-VL-8B. On perception and hallucination benchmarks, OPSV also shows marginal improvements or even lower performance compared to Plain.

  • Insufficient Baseline Comparisons. Although the paper compares against existing visual search and hallucination mitigation methods, differences in training data and computational budgets are not clearly controlled. In particular, a standard SFT baseline using the same training data would help isolate the contributions of GRPO and OPSV.

  • Limited Model Diversity. All three training backbones belong to the Qwen family. Experiments on other MLLM families would better support the claimed generalizability.

  • Limited Benchmark Characterization. Although Figure 6 reports the distribution across edit types and hops, further analysis of the benchmark's characteristics and coverage across different evaluation categories would help assess its representativeness and balance.

  • Computational Cost and Reproducibility. Although Table 6 provides training hyperparameters, GPU resources, actual training time, and the additional inference overhead of OPSV are not clearly reported. Statistical variability across multiple runs (e.g., standard deviations) is also missing. Additionally, more details about the two human annotators and their annotation protocol would improve reproducibility.

Questions for authors
  • Figure 4 shows that a negative query is accepted when the verifier answers "No" or the iteration limit is exceeded. However, it is unclear why a query would be accepted after repeated verification failures. This could introduce incorrectly labeled negative samples into the training data.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 3pJr (1/2)

We thank the reviewer for finding our results "insightful" and our evaluation "comprehensive". Please also see the General Response.

W1 (benchmark novelty). We added a comparison of benchmark settings with a discussion (new, App. A.1, Tab. 6); please see the paper for the full table. Compared with prior benchmarks, we believe VERVE-MCQ differs and contributes in five ways:

  1. To our knowledge, VERVE-MCQ is the only visual search benchmark that combines high-resolution images and small targets with paired positive and negative queries. It is also the hardest of the listed benchmarks: Qwen3-VL-8B fails 88.8% of paired MCQs.
  2. FineCops-Ref and GroundingME are grounding benchmarks: they ask for a bounding box, not for an answer about specific features of the target. The task is inherently different from VERVE-MCQ.
  3. HallusionBench and POPE are hallucination benchmarks with much lower image resolution than VERVE-MCQ. POPE is also nearly saturated: Qwen3-VL-8B fails only 11.5% of its questions.
  4. VERVE-MCQ enables unique diagnosis by edit type and hop count. Findings include relation and 2-hop edits being the hardest (Tab. 1).
  5. VERVE-MCQ reveals failures that existing benchmarks miss. Current SOTA visual search methods fall below their own base models under paired queries, and even frontier models struggle, with Gemini-3.7-Flash reaching only 41.5%.

W2 (OPSV). In point 2 of the General Response, we explain in detail what OPSV adds on top of plain mode, with the PEARL-style comparison. In short, OPSV trains the model to raise and answer its own verification questions before answering, so it targets faithful visual search on high-resolution images, which is exactly what VERVE-MCQ evaluates. There, OPSV beats plain on all three Qwen backbones, both over five inference seeds (Tab. 24) and in the open-ended setting (Tab. 20), and also by 3.8 points on the newly added MiMo-VL-7B (Tab. 1). The gain does not come from the prompt alone (Tab. 18). The core methodological contribution of OPSV is that the model raises verification questions by itself (on-policy): a PEARL-style baseline that receives off-policy verification questions during training stays below OPSV on all three backbones (App. D.10, Tab. 17). We therefore believe that OPSV is also interesting from a methodological perspective.

On the perception and hallucination benchmarks, which test answering and rejecting separately, OPSV performs on par with plain mode (averages within 1.7 points). We agree that the transfer gains come mainly from our constructed data rather than from the training mode, and we made this clear in Sec. 4.3.

W3 (comparison with SFT). We trained SFT on our 57K training set (Sec. 4.5, App. D.9, Tab. 16); the full results are in the paper. Each cell lists Base / SFT / VERVE (plain) / VERVE (OPSV):

VERVE-MCQPerception avg.Hallucination avg.
Qwen2.5-VL-7B11.4 / 29.3 / 33.5 / 40.857.5 / 53.9 / 60.8 / 60.272.3 / 72.1 / 75.9 / 75.1
Qwen3-VL-8B11.2 / 25.8 / 41.8 / 42.264.4 / 59.6 / 65.8 / 67.276.1 / 73.7 / 80.8 / 80.8
Qwen3.5-9B20.1 / 37.0 / 49.7 / 55.367.4 / 60.2 / 71.5 / 71.176.2 / 77.0 / 84.1 / 82.9

On VERVE-MCQ, SFT is below both VERVE modes on every backbone. SFT also lowers the perception average below the base model. This shows that SFT alone cannot reach the performance of our RL training, highlighting the effectiveness of VERVE-Train.

AuthorsOpenReview ↗

Response to Reviewer 3pJr (2/2)

W4 (Qwen-only backbones). We understand the concern. We added a discussion of our backbone choices to App. B. We chose backbones for which every baseline has a released checkpoint: each baseline post-trains only one or two Qwen models, and VERVE is the only method applied to all three generations. Despite sharing the Qwen name, these backbones differ in size, architecture (e.g., DeepStack in Qwen3-VL, Gated DeltaNet in Qwen3.5) and training data. To further address this concern, we additionally train VERVE on MiMo-VL-7B, a non-Qwen model, and report the results on VERVE-MCQ (Tab. 1) and 14 perception and hallucination benchmarks (Tab. 2) in the revised PDF. Selected results:

MiMo-VL-7BVERVE-MCQV*HR-4KRePOPEFINER
Base19.475.472.392.965.1
+ VERVE (plain)41.282.776.093.775.8
+ VERVE (OPSV)45.084.877.194.173.5

VERVE more than doubles its VERVE-MCQ paired accuracy (19.4 → 41.2 / 45.0, plain / OPSV) and improves all four benchmarks in the table above, so our method is not specific to the Qwen family.

W5 (benchmark coverage). Besides the breakdown by edit type and hop (Tab. 1), we further analyze the benchmark by question type and target size. VERVE-MCQ covers five question types (color 39%, identification 20%, OCR 16%, material/state 15%, structure 11%), and 70% of targets cover less than 5% of the image (Fig. 7). The table reports paired accuracy per group for the Qwen3.5-9B family (results for all backbones in App. D.11.8, Tab. 23):

ColorIdent.OCRStruct.M./S.<1%1–5%5–10%≥10%
Gemini-3.7-Flash42.943.140.534.641.735.141.847.146.7
Base18.123.318.620.722.320.316.819.330.2
+ VERVE (OPSV)57.254.659.150.550.852.055.456.460.4

VERVE improves every group on all three backbones, with similar gains on small and large targets, and the OPSV model beats Gemini on every group. The difficulty is also fairly uniform across question types and target sizes: no single category is especially low-scoring.

W6 (cost and reproducibility). Thank you for pointing this out. We added the following to the paper:

  • Training (Tab. 9): each model is trained for 150 steps on 28 GH200 GPUs, taking 16.7–19.0 hours (468–531 GPU-hours) per run; plain and OPSV cost about the same.

  • Inference (App. C.3, Tab. 12): on VERVE-MCQ, we report the mean output tokens and the total time for all questions on one GH200 (base / plain / OPSV):

    TokensTime (s)
    Qwen2.5-VL-7B13.0 / 3.5 / 287.7186 / 172 / 313
    Qwen3-VL-8B171.2 / 389.6 / 432.8535 / 560 / 517
    Qwen3.5-9B180.3 / 285.7 / 326.5768 / 655 / 662

    OPSV makes the model output more tokens, but on Qwen3-VL and Qwen3.5 its total time is not longer, because with batched generation in vLLM the total time is dominated by the longest sequences. Qwen2.5-VL is not a reasoning model and answers very briefly by default; OPSV makes it reason before answering, which costs more tokens, but the overhead stays acceptable.

  • Variance (Tab. 24): the std over five inference seeds is at most 1.6 points.

  • Human annotation (new, App. F.5): the full protocol, i.e., what annotators see, the three checks per negative, and the accept / revise / reject rules. The annotators are two PhD students with expertise in the field, each spending about 80 hours on annotation and validation. We will release the code, benchmark, training data and checkpoints for full reproducibility.

Q (accepting negatives after failed verification). App. B and Tab. 8 analyze this in detail. The per-round statistics of the propose-verify loop are:

RoundAcceptedRejected
155,514 (75.0%)18,480 (25.0%)
212,410 (16.8%)6,059 (8.2%)
32,747 (3.7%)3,312 (4.5%)

We keep the 3,312 samples still rejected in the final round for two reasons:

  1. They are only 4.5% of the samples, and they are used only for training, never in the benchmark.
  2. Samples that reach round 3 are hard for the verifier itself, often because the image is genuinely difficult. In a human check of 100 of them, 67 were valid negatives. Dropping them would mostly remove hard but correct samples and bias the training set toward easy cases.

We thank the reviewer again and are glad to clarify anything further.

I appreciate the authors' additional experiments and clarifications. However, my primary concern regarding the fundamental novelty of VERVE-MCQ remains unresolved.

Existing benchmarks have already explored closely related evaluation problems. In particular, FineCops-Ref evaluates positive and negative referring expressions involving fine-grained modifications to objects, attributes, and multi-hop relations, while GroundingME explicitly evaluates small-object grounding, spatial reasoning, and rejection of nonexistent targets.

While I acknowledge that VERVE-MCQ introduces a distinct evaluation format by combining high-resolution visual search with paired positive/negative multiple-choice questions, the authors primarily demonstrate differences in benchmark configurations rather than establishing a substantial conceptual advance over existing evaluations.

Furthermore, the paper's central observation—the perception-hallucination trade-off—is already supported by the authors' own experiments on existing benchmarks in Table 2. Although VERVE-MCQ enables a more controlled, paired evaluation, the additional scientific insight provided by this design, beyond what existing benchmarks can reveal, remains insufficiently justified. Greater benchmark difficulty alone does not establish conceptual novelty.

Overall, while the additional experiments strengthen the empirical evaluation, they do not sufficiently resolve my fundamental concern regarding the benchmark's novelty. Therefore, I maintain my original rating.

We thank the reviewer for the follow-up, and we appreciate that the additional experiments addressed the other concerns. The remaining concern is about one of our three main contributions, the VERVE benchmarks. Beyond the setting comparison in App. A.1 (Tab. 6), we show below, with new results, what VERVE-MCQ reveals that existing benchmarks cannot.

1. Adding negatives does not by itself expose the failure (new, App. A.1, Tab. 28). VERVE-MCQ shows that SOTA visual search methods fall below their own base models when the query carries a false premise about the searched target (Tab. 1). To test whether a benchmark with negatives already reveals this, we evaluate the same three SOTA methods (TreeVGR, ZwZ and Vision-OPD) and their base models on FineCops-Ref (Tab. 28):

ModelPositive (P@1)Negative (R@1)AUROC
Qwen2.5-VL-7B78.448.659.9
+TreeVGR71.643.755.0
Qwen3-VL-8B83.348.555.7
+ZwZ82.152.758.7
Qwen3.5-9B82.749.454.0
+Vision-OPD81.551.356.5

TreeVGR falls below its base model, but ZwZ and Vision-OPD score higher than their base models on negatives (R@1 +4.2 / +1.9) and on the joint metric (AUROC +3.0 / +2.5). On VERVE-MCQ, in contrast, all three methods fall below their base models (Tab. 1). We do not argue against FineCops-Ref, as it measures a different ability (grounding). But these results show that simply introducing negative queries does not guarantee a consistent failure mode for advanced models: the specific task and benchmark design still matter. We therefore believe that VERVE-MCQ is conceptually different.

2. VERVE offers a controlled setting that reveals a new failure mode. Tab. 2 shows the perception-hallucination trade-off, but its benchmarks differ in images and question types, so the drop may simply come from a distribution shift (e.g., smaller image resolution, different question types). In VERVE-MCQ, each negative is a minimal edit of a positive about the same target in the same image. On these matched items (Tab. 21), the three methods answer more positives correctly than their base models, yet their accuracy on negatives drops sharply (TreeVGR 36.6 → 11.4, ZwZ 31.0 → 12.2, Vision-OPD 40.5 → 27.3). Since the images and targets are identical, distribution shift cannot explain this. As the reviewer notes, VERVE-MCQ "enables a more controlled, paired evaluation"; this control is what lets us attribute the failure precisely to false premises about the searched target, which Tab. 2 cannot. To our knowledge, no prior visual search benchmark has revealed this failure or allowed such failure attribution.

3. Visual search and grounding are different tasks. Grounding benchmarks such as FineCops-Ref and GroundingME ask the model to output a box for a given description. Visual search benchmarks (V*, HR-Bench, ZoomBench, TreeBench) ask questions that require perceiving and reasoning about a tiny target inside a complex scene, and the answer concerns a specific feature of the target. Therefore, visual search is typically treated as a different line of work from grounding. VERVE-MCQ follows this visual search line, but unlike prior visual search benchmarks, it jointly tests whether a model can reason about a target that exists (positive query) and detect a false premise in the query (negative query). To our knowledge, no prior visual search benchmark has integrated false-premise detection into the visual search setting. The grounding benchmarks differ not only in task but also in design: FineCops-Ref does not enforce strict 1:1 positive-negative pairing, as 4,031 of its 9,605 test positives (42%) have no negative (new, App. A.1). FineCops-Ref also uses low-resolution images with large targets. GroundingME has a negative-only (Rejection) subset that is not paired with positives at all, so it cannot offer the controlled failure attribution that VERVE-MCQ does (see point 2).

4. Benchmark difficulty is a practical benefit. We agree that "greater benchmark difficulty alone does not establish conceptual novelty", and we do not claim it as such. Still, we believe difficulty adds practical value: V* already exceeds 90 and base models reach at least 86.9 on RePOPE and AMBER (Tab. 2), leaving little room to measure progress. On VERVE-MCQ, Qwen3-VL-8B fails 88.8% of the pairs (the most difficult benchmark in Tab. 6), and the failure extends even to frontier models, with Gemini-3.7-Flash reaching only 41.5% (Tab. 1). We hope VERVE-MCQ can therefore serve as a useful testbed for the community.

We hope this clarifies the novelty of VERVE-MCQ, and we are happy to discuss further.

Reviewer_NUXiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies verified visual search in high-resolution images: a model should answer questions about a small target when the query is valid, and reject the query when one of its premises is false. The contributions are threefold. (1) VERVE-MCQ: 1,759 human-verified question pairs built on ZoomBench and TreeBench images. In each pair, the negative twin edits a single object, attribute, or relation slot along an object-anchored referring chain, and the pair is scored with paired accuracy. VERVE-Binary (1,680 yes/no pairs) isolates the edited content. (2) An empirical finding: recent visual-search methods (TreeVGR, ZwZ, Vision-OPD, DeepEyes, Thyme) underperform their own base models on VERVE-MCQ, on VERVE-Binary, and on six hallucination benchmarks. The Binary results reveal a strong “target exists” bias: on GT=no questions these methods reach 10–55% accuracy, while their base models reach 75–93%. (3) VERVE-Train: GRPO on a 57K set derived from ZwZ-RL with a 2:1 positive-to-negative ratio. Negatives are synthesized with a caption → propose → verify loop. The method has two modes. Plain mode uses a correctness reward only. OPSV mode has the model emit its own premise checks and adds a coverage reward, computed against offline golden checks and gated by answer correctness. Validation covers three backbones (Qwen2.5-VL-7B, Qwen3-VL-8B, Qwen3.5-9B). On Qwen3.5-9B, VERVE-MCQ paired accuracy rises from 20.1 to 55.3 (Gemini-3.7-Flash: 41.5), the hallucination average rises from 76.2 to 84.1, and the perception average rises from 67.4 to 71.5. The paper also reports transfer to seven external benchmarks and ablations on the data ratio and reward components.

Critical strengths
  1. The finding that visual-search training hurts the model’s ability to reject absent targets is useful and well supported. Tab. 8 makes this very clear. DeepEyes-V2, for instance, gets 99.9% on GT=yes questions but only 10.3% on GT=no, and the same pattern shows up on the six hallucination benchmarks in Tab. 2 for all three backbone families. Current visual-search benchmarks don’t test this at all, so I think people working in this area will want to know about it.
  2. Plain mode is simple and works on all three backbones. Since a model only scores on a pair if it gets both questions right, the gains can’t come from just saying “no” more often. Positive accuracy actually goes up (56.0 to 63.9 on Qwen3.5-9B, Tab. 12), and V*/HR-Bench stay at about the level of the perception-focused baselines.
Critical weaknesses
  1. I’m not convinced that OPSV adds much over plain mode. On Qwen3-VL the two are nearly the same (41.8 vs. 42.2), and on the hallucination average in Tab. 2, plain mode is equal or better on all three backbones. The +13.8% over Gemini comes from the annealed Qwen3.5 run, where the coverage reward is removed after step 80 of 150, and only OPSV is tested with the self-verification prompt. The reward is also quite similar to PEARL (CVPR 2026). The paper separates the two by calling PEARL off-policy, but OPSV’s coverage reward also relies on golden checks generated offline.
  2. There is no positive-only GRPO baseline at the same 57K scale. Tab. 4 has a 1:0 setting, but only with 10K samples and one backbone. Without this baseline, I can’t tell how much of the improvement comes from the negatives and how much from running GRPO on ZwZ-RL data. I also worry about format overlap: training and test negatives are built the same way (one edited slot plus a “there is no such X” option). The external gains are mostly on false-premise benchmarks, while RePOPE and AMBER barely move and VidHalluc is below the base model for Qwen3-VL.
Questions for authors

Questions that could change my assessment:

  1. Could you add a positive-only GRPO run with the same budget (57K samples, 150 steps), on at least one backbone?
  2. How do the baselines do with the self-verification prompt? Could you also compare OPSV with a PEARL-style variant that rewards answers to the golden checks directly? And how did you choose the [40, 80] annealing window?
  3. How does the model do on negatives in a different format, e.g., open-ended questions with no denial option, or images where the target is simply not there rather than edited out?

Clarifications:

  1. The VERVE-Binary key was revised for the 201 cases where Gemini-3.7-Flash disagreed. What is Gemini's score on the original key? Its Binary score is used to explain its MCQ failures, so this matters.
  2. DeepEyes reports fewer hallucinations on POPE, but your results show the opposite. Do you know why?

Suggestion (doesn't affect my score):

It would help to discuss the differences from FINER, JBA, and PEARL. Confidence intervals would also be useful, since 584 positive questions are reused across 1,759 pairs.

6 comments
AuthorsOpenReview ↗

Response to Reviewer NUXi (1/2)

We thank the reviewer for the careful review and for finding our main finding "useful and well supported". Please also see the General Response.

W1 (OPSV vs. plain). In point 2 of the General Response, we explain in detail what OPSV adds on top of plain mode. In short, on VERVE-MCQ, the only benchmark that jointly tests answering and rejecting on the same visual search pair, OPSV beats plain on all three backbones: by +7.3 / +0.4 / +5.6 (Qwen2.5-VL / Qwen3-VL / Qwen3.5) in the main run (Tab. 1) and by +8.5 / +1.2 / +2.0 over five inference seeds (Tab. 24). On the newly added MiMo-VL-7B, OPSV also beats plain by 3.8 points (Tab. 1). On the other benchmarks, which test the two abilities separately, OPSV is on par with plain. We also clarify that annealing the coverage reward on Qwen3.5 does not mean that the reward has no effect. As shown in App. D.12 (Fig. 17), Qwen3.5 picks up the coverage reward very quickly (above 0.9 by step 60) and keeps it until the end of training, so the reward has already taught the model to self-verify. Once it saturates, it is nearly the same across rollouts, so we fade it out and let the remaining steps focus on correctness. Even without annealing, OPSV still beats plain on VERVE-MCQ (51.3 vs. 49.7, Tab. 25). Q2 below also discusses the annealing window.

W1 / Q2 (SV prompt for baselines; comparison with PEARL). We gave the self-verification (SV) prompt to all baselines (new, App. D.11.2, Tab. 18). VERVE-MCQ paired accuracy, default prompt → SV prompt (VERVE OPSV uses the SV prompt):

BasePrior methodVERVE (OPSV)
Qwen2.5-VL-7B11.4 → 14.0TreeVGR 6.0 → 6.740.8
Qwen3-VL-8B11.2 → 15.6ZwZ 7.6 → 5.642.2
Qwen3.5-9B20.1 → 22.5Vision-OPD 15.6 → 19.755.3

The SV prompt changes the baselines (base models and prior methods) by only −2.0 to +4.4 points. Even with it, the best baseline (22.5) stays far below VERVE OPSV.

We also clarify the key difference between OPSV and PEARL: who writes the checks (verification questions). PEARL gives the model sub-questions written in advance (off-policy) and rewards its answers to them. In OPSV, the model never sees the golden checks: it decides by itself what to check (on-policy), and the golden checks only serve as a hidden rubric passed to the reward model (not to the policy). We have updated the related work section (Sec. 5) to better distinguish VERVE from PEARL. We also trained the PEARL-style variant as suggested (new, App. D.10, Tab. 17), where the same golden checks are added to the prompt and the reward weights final-answer and check accuracy equally. Please see point 2 of the General Response for the full table. On VERVE-MCQ, PEARL-style reaches 24.8 / 30.0 / 41.3, below plain (33.5 / 41.8 / 49.7) and OPSV (40.8 / 42.2 / 55.3). This shows that PEARL-style training, where sub-questions are given to the model, is not as effective as teaching the model to ask and answer the right sub-questions (OPSV mode).

W2 / Q1 (positive-only GRPO). Thank you for suggesting this baseline. We trained positive-only GRPO on Qwen3-VL-8B with 57K samples and the same training budget as the full run (Sec. 4.5, App. D.12, Tab. 26). Please see point 1 of the General Response for the full table, where we also compare with SFT at the same scale. Positive-only training improves HR-8K (76.2 → 82.0) but lowers VERVE-MCQ (11.2 → 2.8) and RePOPE (94.0 → 90.0), the same trade-off we see in existing visual search methods such as ZwZ and Vision-OPD. Our training data with negatives and our training method mitigate this trade-off: VERVE reaches 41.8 / 42.2 on VERVE-MCQ (plain / OPSV) while also improving HR-8K (80.0 / 81.6) and RePOPE (94.7 / 95.1).

AuthorsOpenReview ↗

Response to Reviewer NUXi (2/2)

W2 / Q3 (format overlap). We understand the concern. To show that the performance gain does not come from format matching, we built an open-ended version of VERVE-MCQ with no options and no denial option, graded by an LLM judge (new, App. D.11.3, Tab. 20). Point 3 of the General Response, on train-test independence and format overlap, summarizes this result; the full numbers are:

Open-ended paired acc.Qwen2.5-VL-7BQwen3-VL-8BQwen3.5-9B
Base6.48.24.6
Prior method0.2 (TreeVGR)1.5 (ZwZ)3.8 (Vision-OPD)
VERVE (plain)16.022.022.3
VERVE (OPSV)19.223.225.7

In this open-ended setting, VERVE still improves its base models by 9.6–21.1 points, while prior methods fall below theirs. This confirms that our models learned faithful visual search rather than format matching.

W2 (gains mostly on false-premise benchmarks). We would like to point out that the gains are broad and not limited to false-premise benchmarks. VERVE beats the base model in 103 of 112 results in Tab. 2 (14 benchmarks × 4 backbones × 2 modes); RePOPE and AMBER move less because the base models already score ≥ 86.9. Importantly, prior visual search methods lose hallucination robustness, while VERVE beats them on the hallucination average of every backbone. VERVE also transfers to other tasks (Tab. 3), including benchmarks without false premises such as PerceptionRubrics and VL-RewardBench: it beats the base model in 25 of 28 results and ZwZ / Vision-OPD in 23 of 28.

Q2 (annealing window). On Qwen3.5-9B, the coverage reward reaches about 0.86 by step 40 and exceeds 0.9 by step 60 (Fig. 17). Once it saturates, it is nearly the same for all rollouts of a prompt, so it barely separates good from bad responses in the GRPO advantage, even though part of the reward is still tied to it; the correctness reward, in contrast, still has room to improve. We therefore choose the annealing window based on this observation and anneal the coverage reward over steps 40–80, so that the remaining training focuses on correctness. Since one run costs about 480 GPU-hours (Tab. 9), we did not grid-search this window. On Qwen3-VL (Fig. 17) and Qwen2.5-VL, the coverage reward rises much more slowly, so we do not anneal it there.

Q4 (Binary key). On the original key, Gemini-3.7-Flash reaches 82.44 paired accuracy, compared with 85.65 on the revised key. The human review did not simply follow Gemini: humans checked each of its 201 disagreements with the original key carefully, and only 70 led to a label change. With either key, Gemini stays far above its own VERVE-MCQ score (41.5) and above TreeVGR, ZwZ and Vision-OPD (55.5–59.5), so our conclusion does not change. All models are evaluated with the same final key.

Q5 (DeepEyes on POPE). We evaluate on RePOPE, not the original POPE, which explains the difference from the DeepEyes paper. RePOPE re-annotates POPE by hand, because POPE contains label errors and ambiguous cases: about 9.3% of its "yes" labels are wrong and 13.8% are ambiguous. RePOPE fixes the wrong labels and removes the ambiguous ones, so scores on the two benchmarks can differ.

Related work and variance. We added the discussion of FINER to Sec. 5 (PEARL is discussed under W1 / Q2). FINER studies fine-grained negatives on standard-resolution images. VERVE instead pairs each negative with a hard visual search positive on a high-resolution image, and it beats FINER-Tuning on all 6 hallucination benchmarks with Qwen3-VL-8B and on 4 of 6 with Qwen2.5-VL-7B (App. D.8, Tab. 15). We could not identify JBA with certainty; could the reviewer share its title or a link? For variance, please see point 4 of the General Response, where we report the five-seed inference results: the std is at most 1.6 points (Tab. 24), and accounting for it does not change any model ranking on VERVE-MCQ.

We hope these results address the reviewer's questions, and we are happy to discuss further.

Thank you for the detailed response. The positive-only baseline, the SV-prompt results, and the open-ended evaluation address Q1–Q3 well, and the clarifications on the Binary key and RePOPE are helpful.

However, my main concern from W1 remains. On Qwen3.5-9B, OPSV leads plain by only +2.0 across five seeds (+1.6 without annealing), within the reported std, and it is on par with plain on the other benchmarks. The new results thus make clear that the improvement comes mainly from the negative data, while OPSV, the main methodological component, adds little beyond it and remains close to PEARL. What is left is a useful benchmark and finding, plus a training recipe of synthesized negatives and standard GRPO. I find the methodological contribution incremental for ICLR, so I’ll keep my score.

Reviewer_u5AKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces VERVE to evaluate whether VLMs can both identify targets in positive examples and reject false premises in paired counterfactual examples that alter an object, attribute, or relation. The plain model trains on positive and negative examples with a correctness reward, while the OPSV variant additionally rewards the generation and verification of question-check items. The paper reports reduced hallucination while maintaining performance on several perception benchmarks, along with transfer results across tasks.

Critical strengths
  1. The task captures a practically important failure mode. Paired accuracy requires a model to answer both the positive and negative examples correctly, exposing failures that positive-example perception scores alone may miss.
  2. The data and training pipeline are described in useful detail. The paper discusses human filtering, sample ratios, GRPO settings, and OPSV prompts, and includes a small human evaluation.
  3. The main gains are substantial and accompanied by ablations. On VERVE-MCQ, Qwen3.5-9B improves from 20.1 to 49.7 with Plain and 55.3 with OPSV. The paper also ablates training strategies and reward components.
Critical weaknesses
  1. The independence of the training and benchmark data needs stronger evidence. The authors report image deduplication, distinct generation pipelines, and a TreeBench subset evaluation, but these do not fully rule out correlations in scene content, question templates, or source-data distributions.
  2. Generalization is not uniformly positive. Results on multiple benchmarks provide evidence of transfer, but some metrics decline relative to particular baselines. The paper should avoid implying that the method improves every transfer setting.
  3. The independent contribution of OPSV over Plain is not fully established. The gains from individual reward components are relatively small, and the overall improvement may reflect a combination of negative-example training, prompting, reward gating, and training strategy rather than self-verification alone.
Questions for authors
  1. Please report sample counts by edit type, hop count, and image source, together with confidence intervals or multi-seed results, to establish whether gains on subgroups are stable.
  2. Please provide more rigorous image-level, scene-level, and question-template isolation tests, together with details of the deduplication protocol.
  3. Please distinguish results by baseline and metric, rather than describing all transfer results as improvements.
4 comments
AuthorsOpenReview ↗

Response to Reviewer u5AK

We thank the reviewer for the careful review and the positive assessment. Please also see the General Response.

W1 / Q2 (train-test independence). We agree that this needs stronger evidence, and we now check independence at four levels (App. B.1):

  • Image level. We embed all training images and the benchmark images with SSCD, a copy-detection model, and treat pairs with similarity ≥ 0.90 as duplicates, following FineVision (Wiedmann et al., NeurIPS 2026). We find no duplicates, and for 99.7% of training images the highest similarity to any benchmark image is below 0.3 (Fig. 9).
  • Scene level. Scene similarity does not explain the gains. We split VERVE-MCQ into the half most similar and the half least similar to the training images, using SigLIP-2 embedding similarity. On Qwen3-VL-8B, VERVE gains about the same on both halves (plain: +33.0 vs. +28.4; OPSV: +32.7 vs. +29.4 over the base model), and the 95% CI of the gap includes zero. The same holds on all three backbones (Tab. 10).
Qwen3-VL-8BLeast similar halfMost similar halfGap [95% CI]
VERVE (plain)+33.0+28.4−4.6 [−12.1, +2.6]
VERVE (OPSV)+32.7+29.4−3.3 [−10.5, +4.3]
  • Question level. Training and benchmark negatives come from two separate pipelines that share no model (both prompts are in the new App. F.6). Training negatives are written by a text-only GPT-OSS-120B from Qwen3.5-27B image descriptions. Benchmark negatives are proposed by Gemini-3-Flash from the image, phrased by GPT-5, and then checked by humans, who discarded 977 of 2,736 and rewrote 617. To rule out a format shortcut, we also built an open-ended VERVE-MCQ (App. D.11.3, Tab. 20), where VERVE still improves its base models by 9.6–21.1 points.
  • Source level. The TreeBench subset was built independently of ZwZ-RL. On it, VERVE improves Qwen3-VL-8B from 8.1 to 39.0 and Qwen3.5-9B from 11.0 to 36.1 (Tab. 22).

Q1 (counts and variance). Fig. 6 already gives the number of pairs by hop and edit type. By image source, 524 positives come from ZoomBench and 60 from TreeBench. Breakdowns by question type and target size are in Fig. 7 and the new App. D.11.8 (Tab. 23). For variance, we added five-seed inference experiments with VERVE and one-pass baselines (App. D.11.7, Tab. 24): the std of the average is at most 1.6 points, while the gains of VERVE over the base models are 22.3–34.8 points, so variance does not affect the model rankings on VERVE-MCQ.

W2 / Q3 (generalization claim). Thank you for raising this concern. Counting all comparisons in our paper (four backbones, both modes), VERVE beats its base model in 128 of 140 cases. We have revised the abstract to say that the improvements "broadly extend to seven external benchmarks", so it no longer implies that VERVE improves every benchmark. VERVE beats every prior method on the hallucination average of every backbone. However, VERVE still trails ZwZ on the Qwen3-VL perception average by 0.5 points (67.2 vs. 67.7), as noted in our limitations (Sec. 6). We have also revised Sec. 4.3 to report the results separately by baseline and metric.

W3 (OPSV vs. plain). We thank the reviewer for the constructive feedback. Please see point 2 of the General Response, where we clarify the contribution of OPSV mode over plain mode. To separate the effect of each component:

  • Prompting: the SV prompt alone changes the baselines by only −2.0 to +4.4 (Tab. 18), and training with it but without the coverage reward gives 41.6, vs. 41.8 for plain (Tab. 5).
  • Gating: the ungated coverage reward gives 41.4, and the gated one gives 42.2 (Tab. 5).
  • On-policy checks (new): a PEARL-style baseline that is given the checks reaches only 24.8 / 30.0 / 41.3, below both plain and OPSV (Tab. 17; table in point 2 of the General Response).
  • Negative samples in training: a positive-only Qwen3-VL-8B baseline trained at the 57K scale lowers VERVE-MCQ by 8.4 points (11.2 → 2.8, Tab. 26), showing that negative samples are necessary. Point 1 of the General Response compares it with SFT and both VERVE modes.

Dear Reviewer u5AK,

Thank you again for your effort and feedback. We believe that our revision and response have addressed your concerns. As the discussion period ends in less than a day, we would appreciate your feedback if any concerns remain.

Best regards,

The Authors

Thank you for the response. The independence check of the proposed benchmark is presented at four levels, both visually and textually. However, my concern stems from the setting in which the negative examples are only marginally fabricated from the positive ones. Consequently, the difference distribution in the synthesized data may be strongly biased relative to that of real-world visual search settings. Given that data augmentation is central to the paper's main contributions—namely, the benchmark and the method—I will therefore keep my score.

Thank you for your feedback and for acknowledging our independence check of the benchmark (W1 / Q2 and point 3 of the General Response). Indeed, real-world users may make several errors in one query, or errors that are more obviously wrong, which in principle are easier to detect than the minimal edits in VERVE-MCQ. We keep minimally edited negatives mainly for two reasons:

  1. Accurate failure attribution. Since each negative edits a single slot of the referring chain in the question, we can attribute failures to a specific edit slot (Target, 1-Hop, 2-Hop) and edit type (obj, attr, rel) (Sec. 4.2, Tab. 1; the counts are in Fig. 6, see our response to Q1). For example, Qwen3.5-9B reaches 21.6% paired accuracy on attribute edits at the Target, but only 18.1% at 1-Hop and 9.5% at 2-Hop, and at 1-Hop it scores roughly half as much on relation edits (8.9%). Gemini-3.7-Flash follows the same trend.
  2. Benchmark difficulty. Even the strongest frontier model, Gemini-3.7-Flash, reaches only 41.5% paired accuracy (Tab. 1). Such difficulty reveals the failure mode clearly and can possibly help the community to advance their methods.

We understand that VERVE-MCQ does not cover all real-world use cases. This is why we evaluate VERVE-Train models on 21 external benchmarks: eight perception and six hallucination benchmarks (Sec. 4.3, Tab. 2), and seven cross-task benchmarks covering general understanding, captioning, reward modeling, grounding, medical and video hallucination (Sec. 4.4, Tab. 3). As noted in our response to W2 / Q3, VERVE beats its base model in 128 of 140 comparisons. Notably, on Qwen3.5-9B, VERVE-Train improves VL-RewardBench by 14.1%, GroundingME by 11.4% and captioning (PerceptionRubrics) by 5.4%, and it exceeds the reference models on all three (e.g., Qwen3-VL-235B on GroundingME, Tab. 3). In contrast, the prior method Vision-OPD, trained on the same backbone, falls below the base model on GroundingME (34.9 vs. 38.8) and captioning (50.4 vs. 50.8).

We hope this clarifies our design choices. As the discussion period comes to an end, we would like to thank you for your time and service.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response by Authors

We thank all reviewers for their time and constructive feedback. We are glad that reviewers found VERVE captures "a practically important failure mode" (u5AK), that our main finding is "useful and well supported" (NUXi), and that our empirical findings are "insightful" (3pJr).

The main shared requests were: (1) positive-only and SFT training at the full 57K scale (NUXi, 3pJr), (2) the added value of OPSV over plain mode (all reviewers), (3) stronger evidence that training and test data are independent (u5AK, NUXi), (4) variance estimates on VERVE-MCQ (all reviewers). Changes in the revised PDF are marked in blue.

1. Positive-only and SFT training at the full 57K scale

We trained two Qwen3-VL-8B baselines with the same 57K training budget: positive-only GRPO (Sec. 4.5, App. D.12, Tab. 26) and SFT (Sec. 4.5, App. D.9, Tab. 16).

Qwen3-VL-8BTraining data (57K)VERVE-MCQHR-8KRePOPE
Base–11.276.294.0
GRPO, positive-onlypositives only2.882.090.0
SFTVERVE-Train data25.875.194.7
VERVE (plain)VERVE-Train data41.880.094.7
VERVE (OPSV)VERVE-Train data42.281.695.1

"VERVE-Train data" mixes positive and negative questions at a 2:1 ratio; the positive-only run has no negatives.

Positive-only GRPO improves HR-8K but lowers VERVE-MCQ and RePOPE below the base model. This is the same trade-off we see in prior visual search methods, so GRPO on ZwZ-RL data alone cannot explain our gains. SFT on our 57K data helps much less than RL (25.8 vs. 41.8 on VERVE-MCQ), and it lowers the perception average below the base model on all three backbones (Tab. 16). So we need both our constructed data and RL training.

2. What does OPSV add on top of plain mode?

Plain mode is purely data-driven and already works well. OPSV additionally trains the model to raise and answer its own verification questions before answering. It therefore targets faithful visual search on high-resolution images: answer when the target exists, reject when it does not. VERVE-MCQ tests exactly this with paired questions, and there OPSV consistently outperforms plain mode:

  • VERVE-MCQ, mean of 5 seeds (Tab. 24): 41.1 vs. 32.6 (Qwen2.5-VL), 42.2 vs. 41.0 (Qwen3-VL), 53.6 vs. 51.6 (Qwen3.5), OPSV vs. plain.
  • Open-ended VERVE-MCQ without options (Tab. 20): OPSV is better on all three backbones (+1.2 to +3.4), so it does not overfit to the MCQ format.
  • With the same self-verification prompt, the baselines change by only −2.0 to +4.4 points (Tab. 18), while OPSV improves its base models by at least 25.6 points (Tab. 1), so the gain comes from training, not the prompt.
  • On the newly added MiMo-VL-7B backbone (Tab. 1), OPSV still outperforms plain by 3.8 points.
  • Inference cost is almost the same as plain mode on Qwen3-VL and Qwen3.5 (Tab. 12).

We further show that the "on-policy" part is necessary. We train a PEARL-style baseline on the same data (new, App. D.10, Tab. 17): GPT-OSS-120B writes the sub-questions in advance (off-policy), and the policy is rewarded for answering them correctly. VERVE-MCQ paired accuracy:

VERVE-MCQQwen2.5-VL-7BQwen3-VL-8BQwen3.5-9B
Base11.411.220.1
PEARL-style (off-policy checks)24.830.041.3
VERVE (plain)33.541.849.7
VERVE (OPSV, on-policy checks)40.842.255.3

The PEARL-style baseline improves over the base model but stays below both of our modes. So giving the model sub-questions is less effective than teaching it to ask and answer the right ones (OPSV mode).

On the perception and hallucination benchmarks, which test the two abilities separately, OPSV performs on par with plain mode (averages within 1.7 points). We agree that the transfer gains come mainly from our constructed data rather than from the training mode, and we made this clear in Sec. 4.3.

3. Train-test independence and format overlap

  • Images: an SSCD near-duplicate check found no duplicates, and for 99.7% of training images the highest similarity to any benchmark image is below 0.3 (App. B.1, Fig. 9).
  • Scenes: VERVE gains about equally on the VERVE-MCQ halves most and least similar to the training images, so scene similarity does not matter (Tab. 10).
  • Source: the TreeBench subset was built independently of ZwZ-RL, and all VERVE models still improve their base models by large margins on it (Tab. 22).
  • Format: on an open-ended VERVE-MCQ with no options (Tab. 20), VERVE still improves its base models by 9.6–21.1 points, while prior methods fall to 0.2–3.8.

4. Variance. Over five inference seeds, the std of VERVE-MCQ paired accuracy is at most 1.6 points, far below our 22.3–34.8 point gains over the base models (Tab. 24).

Other additions: benchmark comparison (Tab. 6), results by question type and target size (Tab. 23), training cost (Tab. 9), human verification protocol (App. F.5), negative-proposal prompts (App. F.6).

Dear Reviewers,

The discussion closes on November 18 (AoE), less than a day from now.

u5AK: we haven’t heard from you yet. Please respond to the authors’ rebuttal.
NUXi: please respond to the new results.
3pJr: please respond to the new comparison.

Even a short note on whether your concerns are resolved and whether your rating changes would help a lot.

Thank you,
Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.