- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper shows that blind RL training can recover substantial benchmark gains when images are restored at test. It then constructs counterfactual coordinate tasks that require visual evidence and target identification, showing that standard GRPO can improve discovery and grounding. Matched gray-image training and caption-shortcut controls support the role of training-time visual evidence. The authors describe this task-design principle as visual resolvability: the intended visual operation should be necessary for success, while the task remains learnable.
Critical strengths
- The paper studies an important problem: the distinction between test-time image dependence and learning from training images, and it is investigated through useful controlled comparisons.
- The study includes meaningful controls such as REAL/GRAY training, caption shortcuts, held-out scene densities with independently constructed evaluations, and additional training seeds.
Critical weaknesses
- The presentation makes the experimental argument difficult to follow.
- The operational meaning of visual resolvability needs a sharper definition. The principle combines visual necessity with learnability, but the audit measures positive image-induced success-rate differences and mixed correctness outcomes.
- Transfer is convincing within coordinate tasks but limited beyond that domain. Held-out densities and independently generated grounding tasks provide useful evidence, but the strongest gains remain in closely related coordinate-reading and point-localization settings. Training interventions are also confined to Qwen2.5-VL, and other families are audited rather than trained.
Questions for authors
- Does the practical mixture conclusion hold when preservation is measured against ordinary-only RL rather than the untrained model?
- How sensitive are the step-100 results to optimizer resets and KL-reference re-anchoring? Does uninterrupted training produce similar acquisition trajectories?
3 comments
Thank you for the detailed response. The response addresses most of my concerns, and I maintain my weak acceptance rating.
Thank you
Thank you for reading our response and for your support of the paper. Your questions led to the comparison against RL on ordinary data alone and to a sharper definition of visual resolvability, and the paper is better for both.
A clearer argument, the right baseline, a sharper definition
Thank you for your positive assessment of the paper, and for a review that put its question more clearly than we had: "the distinction between test-time image dependence and learning from training images". We were glad that you found the problem important and the controls meaningful.
You were also right that the argument was hard to follow. Because of that we rewrote much of the main text, redesigned Figure 1 so that it shows the argument in one picture, and added a Figure 2 that shows the tasks. Your two questions and your request for a sharper definition each changed the paper too. The revised PDF is now uploaded, and pages 1 to 4 show the difference most.
The revised paper makes the argument in four steps.
Figure 1 (p. 2) shows steps 1, 3 and 4, and Figure 2 (p. 3) shows the tasks. Every benchmark, statistic and term is now explained where it first appears.
Your question 1: does the mixture conclusion hold against ordinary-only RL? Yes, and you were right that RL on ordinary data alone is the reference to use. The paper now reports the mixture against it, at the same budget and on the same test items.
So one third of constructed data costs 0.029 on held-out ViRL39K and nothing measurable on MathVista or BLINK. It adds 0.285 in discovery and 0.112 on the grounding task, which asks for the coordinate of a named point (Section 7.1, pp. 9 and 10; Appendix M.2, p. 36).
Your question 2: how sensitive are the step-100 results to optimizer resets and KL re-anchoring? Very little. The paper's 100-step runs were trained in four segments, with a restart between segments. We trained one more run of the same configuration with no restart. On the confirmatory set (200 questions on scenes denser than any in training) it ends at 0.890, against 0.875 and 0.900 for the two segmented runs, and it is within 0.02 of them on the development scenes and the grounding task. The runs differ no more at step 100 than at step 30, before the first restart. This run keeps one KL reference throughout, so it tests the re-anchoring directly. In the segmented runs themselves, reward, entropy and gradient norm change less across a restart than within a segment (Section 5.4, p. 7; Appendices M.1 and M.5, pp. 35 to 38).
The sharper definition you asked for. Section 4 now states it formally (p. 6). A training item is visually resolvable for an operation when two things hold. First, the image raises the probability of a correct answer: Δq=b−qblind>0, where b and qblind are the untrained model's success rates with and without the image. Second, success is neither certain nor impossible, 0<b<1. GRPO learns from the differences among the responses it samples for one prompt, and if they are all right or all wrong there is nothing to learn from. As you observed, these are the two things the audit measures, and the definition is now stated in the same terms. The operation is set by how the question points to its target: marked, named, or given by a relation.
Finally, we are very grateful for your help and your recognition! Without your valuable feedback, we would have continued to view the way our article is presented solely from the author’s perspective—erroneously assuming that readers already understood concepts that we hadn’t actually explained clearly. In short, our article could not have been improved without your valueble assistance.