- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes CoSEC, a framework that combines a counterfactual grounding reward with a self-evolving synthetic curriculum for VLM reinforcement learning. The paper argues that outcome-only rewards cannot distinguish responses grounded in visual evidence from responses relying on prompt templates or other shortcuts, while repeatedly optimizing a counterfactual grounding reward on fixed data may introduce response-length dependence. For each correct response, a frozen scorer compares its likelihood under the original image, an answer-changing counterfactual image and a label-preserving nuisance image; the resulting diagnostics shape the reward and replay priority, while a pass-rate controller adjusts task difficulty and task-family allocation and a programmatic generator renews training examples. Experiments over five model scales from two Qwen-VL families report consistent improvements over outcome-only static GRPO on MME, MME-RealWorld and VSR, and additional analyses examine visual-evidence use, reward normalization, training stability and component ablations.
Critical strengths
- The problem studied is important and well motivated. Direct SFT or reinforcement learning with outcome-only rewards may improve answer accuracy without ensuring that the model actually uses task-relevant visual evidence rather than language priors, prompt templates or option biases. The paper directly targets this limitation.
- The core method is intuitive. Evaluating the same correct response under the original image, an answer-changing counterfactual image and a label-preserving nuisance image provides a natural way to reward sensitivity to task-relevant visual changes while penalizing sensitivity to irrelevant changes.
- The empirical evaluation is extensive within the Qwen-VL series. The primary comparison covers five model scales and multiple training runs, and the paper reports training trajectories rather than only selected checkpoints. The analyses of response-length growth, reward normalization and fixed-data collapse also provide useful observations about the behavior of counterfactual grounding rewards during training.
Critical weaknesses
- There are major inconsistencies in the paper and the supplementary material. (1) The main text mainly uses Macro-3, while the appendices use Macro-4 (including BLINK) as the primary metric. (2) The introduction claims "COSEC carries the same visual diagnostics from reward shaping into replay selection", but Appendix C mentions "Removing the diagnostic replay term changes the parent-sampling distribution by total variation 0.002–0.011". The small change of the distribution undermines the effectiveness of the incorporation of visual diagnostics in replay selection. (3) Figure 1 mentions "3D scene engine" for data generation, whereas in the submitted code there are only generators for 2D images containing simple geometric objects such as squares, circles and triangles. Also, the paper does not show any rendered examples of training data.
- The paper does not demonstrate that the benchmark improvements result from better visual grounding. The main results measure answer accuracy on MME, MME-RealWorld and VSR, while the visual probes in Table 3 only report the within-run change of COSEC without an outcome-only baseline. Moreover, Table 17 shows that the fixed-data baseline exhibits the same qualitative probe pattern, with improvements on intact and nuisance views but little improvement on blank or shuffled views. Therefore, these probes show that training on the synthetic tasks uses visual information, but do not establish that COSEC is more visually grounded than standard outcome-only training.
- The overall formatting and presentation may be inappropriate. In ICLR formatting instructions, it is explicitly required: "Place one line space before the table title, one line space after the table title, and one line space after the table." However, in the main text, the one line spaces after the table title and after the table are absent. Also, the subfigures shown in Figure 2 and Figure 3 are spliced together. For presentation, the paper contains a large amount of raw experiment results without necessary high-level interpretation, which makes the main argument difficult to follow.
- The experiments are limited to the Qwen-VL series, without extensions to other architectures, such as Gemma and LLaVA.
Questions for authors
- Why did the authors exclude BLINK from the main-text Macro-3 metric?
- What's the role of "3D scene engine" in the main figure? Is it genuinely implemented but not uploaded in the supplementary material, or is it for future work?