- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the important question: "Should the vision-language model re-examine a part of the input images or directly commit to the answer at a certain decision state"? The paper first introduces the ReSee benchmark containing visual search trajectories with re-examination or commitment decisions, and baseline results show that VLMs can be bad at making such decisions even with fair answer accuracy. To alleviate this issue, the paper proposes the CARE method, which employs reinforcement learning to strengthen the model's capability to determine if a further localized visual inspection should be carried out to collect more visual information while deriving the answer. The method assigns specialized rewards for the "verdict" token spans for the decision, which empowers the model to make better evaluations of whether current evidence is sufficient and valid. Extensive experiments show consistent performance gains over visual perception/search and OOD benchmarks, and ablation studies validate the main components in the method.
Critical strengths
- The question studied is well chosen. Modern visual search agents suffer from the problem of making the final answer wrong even if at some intermediate state the model has the correct evidence, as well as deriving premature answer without retrieving sufficient visual evidence. The paper's solution targets directly at this problem.
- The method is natural and logical. Expanding both re-examination and commitment trajectories provide contrasts between two possible outcomes, which supplies a natural direction for the model to improve the decision over whether to collect more visual evidence.
- The experiments are extensive and generally convincing. The paper evaluates a large number of VLMs and visual search agents over the ReSee benchmark to emphasize the current models' inabilities, and compare CARE with many other methods (SFT, GRPO, etc.) to demonstrate the advantage of the proposed method. The paper also performs ablations on paired verdict contrast, hyperparameter sensitivity, counterfactual utility and decision quality at equal visual cost. Studies on OOD generalization and scaling behaviors further consolidate the paper's claims.
Critical weaknesses
- The benchmarking strategy on ReSee is unclear. From section 5, appendix D and the anonymous repository, I infer that the authors let VLMs generate trajectories on their own instead of performing teacher forcing on ReSee trajectories. However, it remains unexplained why transferring the counterfactual necessity label from Qwen3-VL-Plus (for generating trajectories in ReSee) to the VLM being tested is reasonable.
- The authors seem to have mixed "whether the visual evidence is sufficient at the current decision state" and "whether the re-examination can lead to a better answer". It is possible that the visual evidence is genuinely insufficient at the current decision state, and the model decides to re-examine, but the re-examination fails due to perception failure. In this case, the decision is considered over-examination by the paper's criterion, but in fact it has chance to improve the results. In other words, the authors do not disentangle decision error and perception error, which undermines the validity of some experimental results, such as Table 1.
- ReSee trajectories may be post-hoc rationalization given the reference answer instead of honest perception-based reasoning. Appendix A.1 mentions "Qwen3-VL-Plus (Bai et al., 2025a) writes each trajectory from the image, question and reference answer, which anchors the search to the target evidence." This means that the model has access to the target answer in the beginning. However, in the uploaded samples, the model responds "All visual evidence confirms the woman with backpack is to the right of the beanie-wearer, yet the answer is B (left) ... Given the ground truth is B, I must have missed the correct pair", which demonstrates that the model may initiate re-examination based on the discrepancy between current evidence and the ground truth, instead of the insufficiency of visual evidence itself. Although the authors describe filtering against explicit mention of reference answer in Appendix A.2 ("the 459 trajectories whose reasoning quotes the reference answer leave supervised training"), it is still not guaranteed that the reference answer does not interfere with re-examination decisions.
- The authors do not compare the training efficiency of CARE with GRPO. CARE shows higher accuracy than GRPO at the same visual cost, which implies supreme efficiency at inference time; but how much more computational resource CARE costs than GRPO at training time is still unknown.
Questions for authors
- What is the composition of the improvement of CARE models compared to GRPO models, i.e. what are the proportions of corrected over-examination and premature commitment?
- What is the precise definition of the reward , and how is it computed?
4 comments
Authors' Rebuttal (2/2)
Continued: Authors’ Rebuttal (2/2)
(2) Labels Independent of the Reasoning. As stated in the Counterfactual Adjudication paragraph of Sec. 3, labels compare the outcomes of commitment and re-examination from the same state rather than following the generator's verdict or reasoning. To test whether answer conditioning still shapes the labels, we regenerate trajectories without the reference answer and adjudicate them with reference-blind continuations. The resulting labels agree with the ReSee labels of the same questions at Cohen's κ = 0.80, and CARE still leads GRPO. Please refer to Table R1 in our response to W1.
(3) Gains of CARE Independent of the Demonstrations. No trained or evaluated policy receives the reference answer as input: RL training uses only the questions, CARE's verdict credit comes from the policy's own paired continuations, and evaluated models search on their own. Tab. 10 in our paper shows that CARE leads GRPO when the cold start contains no ReSee re-examination (Stripped, 45.1 vs. 43.9) or no ReSee trajectory (External, 44.0 vs. 42.8). The cold start alone reaches only 22.1% precision, compared with 18.1% for always re-examining, so CARE's selective verdicts come from RL rather than from the demonstrations (App. C.1).
W4 [Training efficiency of CARE compared with GRPO]
Thank you for this suggestion. We measure the training cost of GRPO and CARE on 8 NVIDIA H100 GPUs and train two compute-matched GRPO baselines. Table R3 below reports the comparison.
Table R3. Training cost and single-seed compute-matched comparisons on Qwen3-VL-8B.
| Method | Output tokens | GPU hours | Macro | Bal. acc. | Looks ↓ |
|---|---|---|---|---|---|
| GRPO | 1.00× | 62 | 46.0 | 54.8 | 2.09 |
| GRPO, group size 10 | 1.25× | 76 | 46.4 | 55.5 | 2.08 |
| GRPO, 1.25 epochs | 1.25× | 78 | 46.2 | 55.1 | 2.06 |
| CARE | 1.24× | 75 | 48.5 | 65.9 | 1.97 |
CARE requires only 21% more GPU hours than GRPO. Compute-matched GRPO gains at most 0.4 macro and 0.7 balanced-accuracy points, and CARE still leads the stronger variant by 2.1 and 10.4 points. We have added Table R3 to App. F of the revised manuscript.
Q1 [Composition of the improvement over GRPO]
Fewer premature commitments account for 59% of the reduction in decision errors (premature commitments plus over-examinations) on Qwen3-VL-8B and 53% on InternVL3-8B, and fewer over-examinations account for the rest. The rates of Tab. 2 on the 52 warranted and 236 unwarranted decisions give the following counts. On Qwen3-VL-8B, decision errors fall from 72 to 55, with premature commitments falling from 40 to 30 and over-examinations from 32 to 25. On InternVL3-8B, they fall from 76 to 59, with premature commitments from 38 to 29 and over-examinations from 38 to 30. In answer accuracy, CARE ends 7 more warranted re-examinations with a correct answer and 4 fewer over-examinations with an incorrect answer. Please refer to Table R2 in our response to W2.
Q2 [Precise definition and computation of the reward R(ζ)]
R(ζ) is the answer reward plus a grounding reward, and all RL methods, rollouts, and replays share it (App. C.3):
Here is the final answer of , is the reference answer, and is the rule-based match of App. C.1. is the set of boxes inspected in , and is the set of target boxes. enters only for training questions whose target boxes are independently annotated, with . R(ζ) contains no format, length, or observation-cost term. The observation cost of CARE enters only the paired contrast of Eq. (3), and each replay is scored by the same R. We have added this definition to Sec. 4 of the revised manuscript.
We hope this fully addresses your questions and comments, and remain happy to engage in further discussion for the rest of the rebuttal period.
Rebuttal Acknowledgement
Many thanks to the authors for the detailed response and additional experiments. The reference-blind and policy-specific adjudication results, together with the compute-matched comparison and the additional clarifications, have adequately addressed my main concerns. Although the faithfulness of individual reference-conditioned trajectories cannot be fully established, the authors have shown that this issue does not materially affect the main conclusions. I therefore continue to support acceptance and will maintain my current score.
Dear Reviewer ZBZe,
Thank you for your thoughtful assessment. We are pleased that our response and additional experiments have addressed your main concerns, and we sincerely appreciate your continued support for acceptance.
Best regards,
The Authors
Authors' Rebuttal (1/2)
Thank you for providing us with your valuable feedback. We have carefully considered your comments and would like to address them as below.
W1 [Benchmarking strategy on ReSee and transfer of counterfactual labels to tested models]
Every evaluated model searches on its own without teacher forcing, and whether it re-examines on each evaluation question is compared with the adjudicated label of that question. ReSee applies the same labels to all models, which keeps decision metrics comparable. The policy-dependent utility τ^π enters only training, where CARE estimates it with its own continuations.
To check that these labels also hold for the evaluated policies, we conduct two additional adjudications of the 288 evaluation decisions. In the first, GRPO and CARE are adjudicated at their own decision states with eight self-generated continuations per verdict. In the second, Qwen3-VL-Plus regenerates each trajectory without the reference answer, and each decision state is adjudicated with four reference-blind continuations per verdict. Table R1 below shows the resulting labels and decision quality.
Table R1. Adjudication of the 288 evaluation decisions with Qwen3-VL-8B. Paired entries are GRPO/CARE.
Labels from each policy's own search agree with the ReSee labels of the same questions at Cohen's κ = 0.76 and 0.82, and labels from trajectories regenerated without the reference answer at 0.80. CARE leads GRPO in balanced accuracy under every set of labels, so evaluating these policies against the ReSee labels yields the same conclusion as evaluating each against its own labels. We have stated this evaluation protocol in App. C.1 of the revised manuscript and added Table R1 to App. A.3.
W2 [Disentangling decision error and perception error]
Thank you for this comment. Decision metrics and answer accuracy separate the two errors. In adjudication, the re-examining branch is the demonstrated continuation, which reaches the reference answer in 99.3% of trajectories, so a warranted label marks a state where committing with the gathered evidence yields an incorrect answer. Decision metrics compare the executed verdict with this fixed label (App. C.1). A re-examination at a warranted state therefore counts toward recall rather than over-examination, whether or not the model then perceives the evidence correctly, and a perception failure lowers answer accuracy instead. Tab. 1 in our paper shows that decision quality does not track answer accuracy: Qwen3.8-27B has the highest macro accuracy (70.1) but 49.3 balanced accuracy, below the 55.7 of the less accurate GLM-4.6V.
We further decompose the outcomes of GRPO and CARE on the 288 adjudicated decisions by label and executed verdict, and record the final answer after each re-examination. Table R2 below reports the results.
Table R2. Outcomes of GRPO and CARE with Qwen3-VL-8B on the 52 warranted and 236 unwarranted evaluation decisions.
CARE re-examines at 10 more warranted decisions and 7 fewer unwarranted ones, and 63.6% of its warranted re-examinations end with a correct answer, compared with 58.3% for GRPO.
W3 [Possible post-hoc rationalization of ReSee trajectories given the reference answer]
Reasoning that invokes the reference answer, as in the cited trajectory, is excluded from supervised training by the filtering step of App. A.2, does not determine the labels, and does not explain the gains of CARE. Specifically, we address your concerns from three aspects:
(1) Construction and Filtering of Answer-Aware Reasoning. The generator receives the reference answer so that its intermediate search steps reach the target evidence instead of reflecting its own perception errors. Its reasoning can therefore state a conflict with the reference answer, as in the cited trajectory. ReSee trajectories are used only for benchmarking and the SFT cold start (App. B.2), and the 459 trajectories that quote the reference answer, including this one, leave supervised training. The labels and evaluation of the benchmark do not use the reasoning text.
Please see the subsequent Authors' Rebuttal (2/2) for remaining responses.