- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes new benchmark called Necessity-Bench to evaluate visual grounding of VLM. Necessity-Bench measures three property including sufficiency, necessity, and specificity. This new benchmark is built upon CLEVR (geometric image) and GQA (natural image). Qwen3-VL evaluations on this benchmark demonstrates that larger models are more visually grounded. The authors further post-train Qwen3-VL on model’s own prediction of target bounding box and shows their metrics are not necessarily increasing despite the increase in the target accuracy.
Critical strengths
- The paper proposes simple formulation of three metrics.
- Necessity-Bench is evaluated on 3 scales of Qwen3-VL series.
Critical weaknesses
-
Generally, it reads as a technical report than conference paper. Writing seems like highly AI-generated (enumeration of detailed numbers instead of emphasizing few important numbers with key message), which severely reduces the readability. Lots of unnecessary phrases (e.g., L239-241) and redundancies in writing overall (e.g., L92-107 I don't think this paper has 6 contribution points, Related work can be more compressed).
-
Although the paper's main contribution is the benchmark. it does not share detailed construction pipeline and only showing single example for CLEVR and GQA. For example, how the target object is extracted from the query? If there are multiple target regions, which one is removed?
-
The benchmark images in Figure 1 seem like including some artifacts (e.g., rightmost image in GQA sample with blur-like artifacts, simple grey marking in CLEVR sample), which is not a practical settings for VLMs. Is there any chances that VLM misbehaves because of these artifacts?
-
The paper should include human evaluation and frontier model evaluation on this benchmark to provide comprehensive view. One model series evaluation is not sufficient.
-
The paper’s main finding in L260-263 is not surprising. Already exists bunch of works [1-3] showing that larger VLM is more visually grounded.
-
I am bit skeptical about Sec 6 experiment setting and results. It is well known that the quality of data matters for RL post-training as it is trained with few samples (around 10k) at academic level compared to SFT. I am not sure whether training on self-reported region is proper way to apply RL post-training as their labels will be inevitably noisy. Also the coordinate of the target object in image only represents one format of self-reported grounding. VLM can also report the grounding in long caption format (e.g., A ball is behind the cube on the top-left side of the image.), which is not tested.
[1] Lee, K. I., Kim, M., Yoon, S., Kim, M., Lee, D., Koh, H., & Jung, K. (2025, April). Vlind-bench: Measuring language priors in large vision-language models. In Findings of the Association for Computational Linguistics: NAACL 2025 (pp. 4129-4144).
[2] Jing, L., & Du, X. (2024). Fgaif: Aligning large vision-language models with fine-grained ai feedback. arXiv preprint arXiv:2404.05046.
[3] Wang, X., Cui, Y., Yao, X., Wang, S., Hu, G., & Qin, X. (2025). CHARTHAL: A Fine-grained Framework Evaluating Hallucination of Large Vision Language Models in Chart Understanding. arXiv preprint arXiv:2509.17481.
Questions for authors
- How well the proposed benchmark scores aligned with previous hallucination or visual grounding benchmarks?
- Does Necessity-Bench can be extended to long text generation (hallucination) benchmarks in addition to CLEVR and GQA?
2 comments
Response Part 2: W5, W6, Q1, Q2, and a new real-photo result
A strict counterfactual on real photographs (new; §4, pp. 5–6; Table 3, p. 6). For GQA "Are there any X?" questions with exactly one X, the answer after removal must change to no. On 425 detector-checked items:
- Models change to "no" only 53–62% of the time.
- The two inpainters agree.
- 8B is no more grounded than 2B (joint 0.553 vs 0.567).
W5. "Larger VLMs are more grounded" is not surprising. We accept the point. The submission did present the CLEVR scaling effect as its central positive finding, and the revision shifts the emphasis to where scale and grounding diverge. [1] measures language priors, [2] aligns models with fine-grained AI feedback, and [3] evaluates chart hallucination. None tests whether an answer causally depends on the oracle evidence; we now cite and discuss all three (§4, p. 5). On screened interventions:
- Domain-dependent. Scale helps on CLEVR in both families tested (Qwen3-VL 2B vs 8B, p = 7e-8; InternVL3.5, p = 0.003) but not on real photographs: no joint difference from 2B to 32B on GQA (paired McNemar, all p ≥ 0.12), and none on the strict real-photo tier.
- Not monotonic. Qwen3-VL-4B is at least as grounded as 8B (0.774 vs 0.745, p = 0.25).
- Decoupled from accuracy item by item. In every model, 20–31% of correct answers are not causally grounded (§4, p. 5).
- Separable by training. For the 2B model, self-report RL improved accuracy and every self-reported metric without a statistically detectable improvement in grounding (§6.1, pp. 7–8; Tables 7–8).
W6. RL setting: data, noisy labels, one self-report format.
-
The labels are exact. The reward compares the model's box with the oracle evidence box from the CLEVR program. "Self-reported" refers to what is optimised, the model's own claim of where it looked (§6.1, p. 7).
-
Data budget. We agree the two recipes are not a controlled ablation: they differ in training data, output format and steps, and training and evaluation share scenes (App. F, p. 18). What we can show:
- With similarly small budgets, the box reward improved 2B's accuracy and self-reported metrics without a statistically detectable improvement in grounding. On held-out screened removals, necessity went 0.565 → 0.529 (p = 0.18). Under the submitted edit it significantly lowers it (0.538 → 0.475, p = 7e-5; Table 8).
- The intervention reward raises it (screened removals: 0.679 → 0.751, p = 1e-4; §6.2, p. 9).
So the two recipes produce different outcomes on grounding. We do not attribute this to the reward signal alone.
-
Other self-report formats (new; §6.3, p. 10; Table 10, p. 10). The model states in one sentence which object it used:
- It names the oracle object in 75% (2B) / 87% (8B) of items.
- Naming predicts grounding (p ≤ 0.002), but 28–31% of items are named but not grounded.
- The irrelevant control object is mentioned in 27–31% of evidence sentences.
This is one additional format, not a long-form caption evaluation.
Q1. Alignment with hallucination and grounding benchmarks.
- POPE (App. E, p. 17; Table 15). The eight models fall in a narrow POPE band (F1 0.84–0.90). We found no significant association between POPE and grounding across the eight tested models (Spearman ρ = −0.19, p = 0.65). Qwen3-VL-2B and 4B tie on POPE (0.890 vs 0.896) yet differ by 0.19 in grounding (p = 6e-11).
- FPVG (App. B, p. 14). Adding our checks to FPVG's conditions rejects 14–22% of its positives.
Q2. Extension to long-text generation. Yes, in principle: the three conditions apply to a claim. The claim should be made from the evidence-only image, retracted or corrected when the evidence is removed, and kept when an irrelevant object is removed, with entailment replacing string equality. Our one-sentence self-report test is a first step, not a long-form hallucination benchmark. The Conclusion (§7, p. 10) states this as future work.
We hope the human annotation, the stricter metric and the new controls address the soundness concerns.
Response Part 1
We thank the reviewer for a detailed and constructive review. Every point led to a concrete change, most to a new experiment, including a human annotation study. (Part 1 of 2: W1–W4)
W1. Writing, redundancy, six contributions, long related work. We agree and rewrote the paper.
W2. Construction pipeline; target extraction; multiple targets. The full pipeline is in App. A (p. 13), with pseudo-code and per-family statistics (Table 11). §3 (pp. 3–4) summarises it, and Figs. 2–4 (pp. 15–16) show further examples.
count), the tested set (exist), the referent (query_*), and both operands for comparisons.W3. Could artifacts make the VLM misbehave? We answer with human annotation and three controls. Figure 1 (p. 8) now shows each example under both edit mechanisms.
GQA edits remain visible (5% judged natural). Together, the controls reduce concern about editing artifacts: visible editing alone changes only a small fraction of answers.
W4. Human evaluation; frontier models; one model series.