- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces VERVE, a framework for evaluating and mitigating the perception–hallucination trade-off in high-resolution visual search. It proposes VERVE-MCQ and VERVE-BINARY to assess models' ability to identify valid targets and reject false premises. The authors further introduce VERVE-TRAIN, a GRPO-based training framework that combines positive and negative queries with on-policy self-verification (OPSV) to improve visual discrimination.
Critical strengths
-
Insightful Empirical Findings. The paper provides empirical evidence that existing visual search methods can improve perception while degrading hallucination resistance compared to their base models.
-
Comprehensive Evaluation. The experiments cover multiple visual search methods and a broad range of perception, hallucination, and cross-task transfer benchmarks.
-
Strong Empirical Results. VERVE-TRAIN achieves substantial improvements on VERVE-MCQ while maintaining competitive perception performance and reducing hallucinations across multiple benchmarks.
Critical weaknesses
-
Limited Novelty of the Benchmark. The core task of distinguishing positive and negative visual queries has already been explored in benchmarks such as POPE, HallusionBench, FineCops-Ref, and GroundingME. Moreover, the benchmark construction pipeline appears relatively conventional. The authors should better clarify the fundamental differences and unique contributions compared to existing benchmarks.
-
Limited Contribution of OPSV. Most performance gains already come from Plain training. For instance, OPSV improves VERVE-MCQ accuracy by only 0.4 points over Plain on Qwen3-VL-8B. On perception and hallucination benchmarks, OPSV also shows marginal improvements or even lower performance compared to Plain.
-
Insufficient Baseline Comparisons. Although the paper compares against existing visual search and hallucination mitigation methods, differences in training data and computational budgets are not clearly controlled. In particular, a standard SFT baseline using the same training data would help isolate the contributions of GRPO and OPSV.
-
Limited Model Diversity. All three training backbones belong to the Qwen family. Experiments on other MLLM families would better support the claimed generalizability.
-
Limited Benchmark Characterization. Although Figure 6 reports the distribution across edit types and hops, further analysis of the benchmark's characteristics and coverage across different evaluation categories would help assess its representativeness and balance.
-
Computational Cost and Reproducibility. Although Table 6 provides training hyperparameters, GPU resources, actual training time, and the additional inference overhead of OPSV are not clearly reported. Statistical variability across multiple runs (e.g., standard deviations) is also missing. Additionally, more details about the two human annotators and their annotation protocol would improve reproducibility.
Questions for authors
- Figure 4 shows that a negative query is accepted when the verifier answers "No" or the iteration limit is exceeded. However, it is unclear why a query would be accepted after repeated verification failures. This could introduce incorrectly labeled negative samples into the training data.
4 comments
Response to Reviewer 3pJr (2/2)
W4 (Qwen-only backbones). We understand the concern. We added a discussion of our backbone choices to App. B. We chose backbones for which every baseline has a released checkpoint: each baseline post-trains only one or two Qwen models, and VERVE is the only method applied to all three generations. Despite sharing the Qwen name, these backbones differ in size, architecture (e.g., DeepStack in Qwen3-VL, Gated DeltaNet in Qwen3.5) and training data. To further address this concern, we additionally train VERVE on MiMo-VL-7B, a non-Qwen model, and report the results on VERVE-MCQ (Tab. 1) and 14 perception and hallucination benchmarks (Tab. 2) in the revised PDF. Selected results:
| MiMo-VL-7B | VERVE-MCQ | V* | HR-4K | RePOPE | FINER |
|---|---|---|---|---|---|
| Base | 19.4 | 75.4 | 72.3 | 92.9 | 65.1 |
| + VERVE (plain) | 41.2 | 82.7 | 76.0 | 93.7 | 75.8 |
| + VERVE (OPSV) | 45.0 | 84.8 | 77.1 | 94.1 | 73.5 |
VERVE more than doubles its VERVE-MCQ paired accuracy (19.4 → 41.2 / 45.0, plain / OPSV) and improves all four benchmarks in the table above, so our method is not specific to the Qwen family.
W5 (benchmark coverage). Besides the breakdown by edit type and hop (Tab. 1), we further analyze the benchmark by question type and target size. VERVE-MCQ covers five question types (color 39%, identification 20%, OCR 16%, material/state 15%, structure 11%), and 70% of targets cover less than 5% of the image (Fig. 7). The table reports paired accuracy per group for the Qwen3.5-9B family (results for all backbones in App. D.11.8, Tab. 23):
| Color | Ident. | OCR | Struct. | M./S. | <1% | 1–5% | 5–10% | ≥10% | |
|---|---|---|---|---|---|---|---|---|---|
| Gemini-3.7-Flash | 42.9 | 43.1 | 40.5 | 34.6 | 41.7 | 35.1 | 41.8 | 47.1 | 46.7 |
| Base | 18.1 | 23.3 | 18.6 | 20.7 | 22.3 | 20.3 | 16.8 | 19.3 | 30.2 |
| + VERVE (OPSV) | 57.2 | 54.6 | 59.1 | 50.5 | 50.8 | 52.0 | 55.4 | 56.4 | 60.4 |
VERVE improves every group on all three backbones, with similar gains on small and large targets, and the OPSV model beats Gemini on every group. The difficulty is also fairly uniform across question types and target sizes: no single category is especially low-scoring.
W6 (cost and reproducibility). Thank you for pointing this out. We added the following to the paper:
-
Training (Tab. 9): each model is trained for 150 steps on 28 GH200 GPUs, taking 16.7–19.0 hours (468–531 GPU-hours) per run; plain and OPSV cost about the same.
-
Inference (App. C.3, Tab. 12): on VERVE-MCQ, we report the mean output tokens and the total time for all questions on one GH200 (base / plain / OPSV):
Tokens Time (s) Qwen2.5-VL-7B 13.0 / 3.5 / 287.7 186 / 172 / 313 Qwen3-VL-8B 171.2 / 389.6 / 432.8 535 / 560 / 517 Qwen3.5-9B 180.3 / 285.7 / 326.5 768 / 655 / 662 OPSV makes the model output more tokens, but on Qwen3-VL and Qwen3.5 its total time is not longer, because with batched generation in vLLM the total time is dominated by the longest sequences. Qwen2.5-VL is not a reasoning model and answers very briefly by default; OPSV makes it reason before answering, which costs more tokens, but the overhead stays acceptable.
-
Variance (Tab. 24): the std over five inference seeds is at most 1.6 points.
-
Human annotation (new, App. F.5): the full protocol, i.e., what annotators see, the three checks per negative, and the accept / revise / reject rules. The annotators are two PhD students with expertise in the field, each spending about 80 hours on annotation and validation. We will release the code, benchmark, training data and checkpoints for full reproducibility.
Q (accepting negatives after failed verification). App. B and Tab. 8 analyze this in detail. The per-round statistics of the propose-verify loop are:
| Round | Accepted | Rejected |
|---|---|---|
| 1 | 55,514 (75.0%) | 18,480 (25.0%) |
| 2 | 12,410 (16.8%) | 6,059 (8.2%) |
| 3 | 2,747 (3.7%) | 3,312 (4.5%) |
We keep the 3,312 samples still rejected in the final round for two reasons:
- They are only 4.5% of the samples, and they are used only for training, never in the benchmark.
- Samples that reach round 3 are hard for the verifier itself, often because the image is genuinely difficult. In a human check of 100 of them, 67 were valid negatives. Dropping them would mostly remove hard but correct samples and bias the training set toward easy cases.
We thank the reviewer again and are glad to clarify anything further.
I appreciate the authors' additional experiments and clarifications. However, my primary concern regarding the fundamental novelty of VERVE-MCQ remains unresolved.
Existing benchmarks have already explored closely related evaluation problems. In particular, FineCops-Ref evaluates positive and negative referring expressions involving fine-grained modifications to objects, attributes, and multi-hop relations, while GroundingME explicitly evaluates small-object grounding, spatial reasoning, and rejection of nonexistent targets.
While I acknowledge that VERVE-MCQ introduces a distinct evaluation format by combining high-resolution visual search with paired positive/negative multiple-choice questions, the authors primarily demonstrate differences in benchmark configurations rather than establishing a substantial conceptual advance over existing evaluations.
Furthermore, the paper's central observation—the perception-hallucination trade-off—is already supported by the authors' own experiments on existing benchmarks in Table 2. Although VERVE-MCQ enables a more controlled, paired evaluation, the additional scientific insight provided by this design, beyond what existing benchmarks can reveal, remains insufficiently justified. Greater benchmark difficulty alone does not establish conceptual novelty.
Overall, while the additional experiments strengthen the empirical evaluation, they do not sufficiently resolve my fundamental concern regarding the benchmark's novelty. Therefore, I maintain my original rating.
We thank the reviewer for the follow-up, and we appreciate that the additional experiments addressed the other concerns. The remaining concern is about one of our three main contributions, the VERVE benchmarks. Beyond the setting comparison in App. A.1 (Tab. 6), we show below, with new results, what VERVE-MCQ reveals that existing benchmarks cannot.
1. Adding negatives does not by itself expose the failure (new, App. A.1, Tab. 28). VERVE-MCQ shows that SOTA visual search methods fall below their own base models when the query carries a false premise about the searched target (Tab. 1). To test whether a benchmark with negatives already reveals this, we evaluate the same three SOTA methods (TreeVGR, ZwZ and Vision-OPD) and their base models on FineCops-Ref (Tab. 28):
| Model | Positive (P@1) | Negative (R@1) | AUROC |
|---|---|---|---|
| Qwen2.5-VL-7B | 78.4 | 48.6 | 59.9 |
| +TreeVGR | 71.6 | 43.7 | 55.0 |
| Qwen3-VL-8B | 83.3 | 48.5 | 55.7 |
| +ZwZ | 82.1 | 52.7 | 58.7 |
| Qwen3.5-9B | 82.7 | 49.4 | 54.0 |
| +Vision-OPD | 81.5 | 51.3 | 56.5 |
TreeVGR falls below its base model, but ZwZ and Vision-OPD score higher than their base models on negatives (R@1 +4.2 / +1.9) and on the joint metric (AUROC +3.0 / +2.5). On VERVE-MCQ, in contrast, all three methods fall below their base models (Tab. 1). We do not argue against FineCops-Ref, as it measures a different ability (grounding). But these results show that simply introducing negative queries does not guarantee a consistent failure mode for advanced models: the specific task and benchmark design still matter. We therefore believe that VERVE-MCQ is conceptually different.
2. VERVE offers a controlled setting that reveals a new failure mode. Tab. 2 shows the perception-hallucination trade-off, but its benchmarks differ in images and question types, so the drop may simply come from a distribution shift (e.g., smaller image resolution, different question types). In VERVE-MCQ, each negative is a minimal edit of a positive about the same target in the same image. On these matched items (Tab. 21), the three methods answer more positives correctly than their base models, yet their accuracy on negatives drops sharply (TreeVGR 36.6 → 11.4, ZwZ 31.0 → 12.2, Vision-OPD 40.5 → 27.3). Since the images and targets are identical, distribution shift cannot explain this. As the reviewer notes, VERVE-MCQ "enables a more controlled, paired evaluation"; this control is what lets us attribute the failure precisely to false premises about the searched target, which Tab. 2 cannot. To our knowledge, no prior visual search benchmark has revealed this failure or allowed such failure attribution.
3. Visual search and grounding are different tasks. Grounding benchmarks such as FineCops-Ref and GroundingME ask the model to output a box for a given description. Visual search benchmarks (V*, HR-Bench, ZoomBench, TreeBench) ask questions that require perceiving and reasoning about a tiny target inside a complex scene, and the answer concerns a specific feature of the target. Therefore, visual search is typically treated as a different line of work from grounding. VERVE-MCQ follows this visual search line, but unlike prior visual search benchmarks, it jointly tests whether a model can reason about a target that exists (positive query) and detect a false premise in the query (negative query). To our knowledge, no prior visual search benchmark has integrated false-premise detection into the visual search setting. The grounding benchmarks differ not only in task but also in design: FineCops-Ref does not enforce strict 1:1 positive-negative pairing, as 4,031 of its 9,605 test positives (42%) have no negative (new, App. A.1). FineCops-Ref also uses low-resolution images with large targets. GroundingME has a negative-only (Rejection) subset that is not paired with positives at all, so it cannot offer the controlled failure attribution that VERVE-MCQ does (see point 2).
4. Benchmark difficulty is a practical benefit. We agree that "greater benchmark difficulty alone does not establish conceptual novelty", and we do not claim it as such. Still, we believe difficulty adds practical value: V* already exceeds 90 and base models reach at least 86.9 on RePOPE and AMBER (Tab. 2), leaving little room to measure progress. On VERVE-MCQ, Qwen3-VL-8B fails 88.8% of the pairs (the most difficult benchmark in Tab. 6), and the failure extends even to frontier models, with Gemini-3.7-Flash reaching only 41.5% (Tab. 1). We hope VERVE-MCQ can therefore serve as a useful testbed for the community.
We hope this clarifies the novelty of VERVE-MCQ, and we are happy to discuss further.
Response to Reviewer 3pJr (1/2)
We thank the reviewer for finding our results "insightful" and our evaluation "comprehensive". Please also see the General Response.
W1 (benchmark novelty). We added a comparison of benchmark settings with a discussion (new, App. A.1, Tab. 6); please see the paper for the full table. Compared with prior benchmarks, we believe VERVE-MCQ differs and contributes in five ways:
W2 (OPSV). In point 2 of the General Response, we explain in detail what OPSV adds on top of plain mode, with the PEARL-style comparison. In short, OPSV trains the model to raise and answer its own verification questions before answering, so it targets faithful visual search on high-resolution images, which is exactly what VERVE-MCQ evaluates. There, OPSV beats plain on all three Qwen backbones, both over five inference seeds (Tab. 24) and in the open-ended setting (Tab. 20), and also by 3.8 points on the newly added MiMo-VL-7B (Tab. 1). The gain does not come from the prompt alone (Tab. 18). The core methodological contribution of OPSV is that the model raises verification questions by itself (on-policy): a PEARL-style baseline that receives off-policy verification questions during training stays below OPSV on all three backbones (App. D.10, Tab. 17). We therefore believe that OPSV is also interesting from a methodological perspective.
On the perception and hallucination benchmarks, which test answering and rejecting separately, OPSV performs on par with plain mode (averages within 1.7 points). We agree that the transfer gains come mainly from our constructed data rather than from the training mode, and we made this clear in Sec. 4.3.
W3 (comparison with SFT). We trained SFT on our 57K training set (Sec. 4.5, App. D.9, Tab. 16); the full results are in the paper. Each cell lists Base / SFT / VERVE (plain) / VERVE (OPSV):
On VERVE-MCQ, SFT is below both VERVE modes on every backbone. SFT also lowers the perception average below the base model. This shows that SFT alone cannot reach the performance of our RL training, highlighting the effectiveness of VERVE-Train.