acceptodds
Withdrawn from ICLR 2027

Fix the Reward or Fix the Data? CoSEC, a Counterfactual Self-Evolving Curriculum for Grounded Vision–Language Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning with verifiable rewards is widely used to post-train vision–language models (VLMs). Because it scores only whether the final answer is correct, it does not separate answers that rely on the image from answers that rely on prompt templates, option priors, or rendering artifacts. Counterfactual grounding rewards address this by perturbing the visual evidence a question depends on and measuring whether the model's confidence in its answer moves. In current practice the signal is used once, to shape a reward on a fixed training set; the per-sample diagnostic does not feed back into the next round's data, and a shaped reward pinned to one batch leaves the policy room to exploit it. We introduce CoSEC, a Counterfactual Self-Evolving Curriculum in which a single set of counterfactual interventions drives both loops of grounded VLM RL: a self-consistent grounding reward, and a closed-loop curriculum that regenerates constructively-labeled data every round with no human annotation and no real training images. Our central finding concerns the interaction between the two. A grounding reward's effect reverses sign with the data regime, acting as a hazard on fixed data and as a net gain once the data is renewed, so the curriculum is what makes a shaped reward safe to optimize. Containing this failure also improves accuracy: across five VLM scales in two families, CoSEC raises out-of-distribution macro accuracy over outcome-only RL, and the advantage grows when output length is held fixed, indicating that the gain reflects grounding rather than verbosity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, counterfactual intervention, visual grounding reward, self-evolving curriculum, reward hacking

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JGxbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes CoSEC, a framework that combines a counterfactual grounding reward with a self-evolving synthetic curriculum for VLM reinforcement learning. The paper argues that outcome-only rewards cannot distinguish responses grounded in visual evidence from responses relying on prompt templates or other shortcuts, while repeatedly optimizing a counterfactual grounding reward on fixed data may introduce response-length dependence. For each correct response, a frozen scorer compares its likelihood under the original image, an answer-changing counterfactual image and a label-preserving nuisance image; the resulting diagnostics shape the reward and replay priority, while a pass-rate controller adjusts task difficulty and task-family allocation and a programmatic generator renews training examples. Experiments over five model scales from two Qwen-VL families report consistent improvements over outcome-only static GRPO on MME, MME-RealWorld and VSR, and additional analyses examine visual-evidence use, reward normalization, training stability and component ablations.

Critical strengths
  1. The problem studied is important and well motivated. Direct SFT or reinforcement learning with outcome-only rewards may improve answer accuracy without ensuring that the model actually uses task-relevant visual evidence rather than language priors, prompt templates or option biases. The paper directly targets this limitation.
  2. The core method is intuitive. Evaluating the same correct response under the original image, an answer-changing counterfactual image and a label-preserving nuisance image provides a natural way to reward sensitivity to task-relevant visual changes while penalizing sensitivity to irrelevant changes.
  3. The empirical evaluation is extensive within the Qwen-VL series. The primary comparison covers five model scales and multiple training runs, and the paper reports training trajectories rather than only selected checkpoints. The analyses of response-length growth, reward normalization and fixed-data collapse also provide useful observations about the behavior of counterfactual grounding rewards during training.
Critical weaknesses
  1. There are major inconsistencies in the paper and the supplementary material. (1) The main text mainly uses Macro-3, while the appendices use Macro-4 (including BLINK) as the primary metric. (2) The introduction claims "COSEC carries the same visual diagnostics from reward shaping into replay selection", but Appendix C mentions "Removing the diagnostic replay term changes the parent-sampling distribution by total variation 0.002–0.011". The small change of the distribution undermines the effectiveness of the incorporation of visual diagnostics in replay selection. (3) Figure 1 mentions "3D scene engine" for data generation, whereas in the submitted code there are only generators for 2D images containing simple geometric objects such as squares, circles and triangles. Also, the paper does not show any rendered examples of training data.
  2. The paper does not demonstrate that the benchmark improvements result from better visual grounding. The main results measure answer accuracy on MME, MME-RealWorld and VSR, while the visual probes in Table 3 only report the within-run change of COSEC without an outcome-only baseline. Moreover, Table 17 shows that the fixed-data baseline exhibits the same qualitative probe pattern, with improvements on intact and nuisance views but little improvement on blank or shuffled views. Therefore, these probes show that training on the synthetic tasks uses visual information, but do not establish that COSEC is more visually grounded than standard outcome-only training.
  3. The overall formatting and presentation may be inappropriate. In ICLR formatting instructions, it is explicitly required: "Place one line space before the table title, one line space after the table title, and one line space after the table." However, in the main text, the one line spaces after the table title and after the table are absent. Also, the subfigures shown in Figure 2 and Figure 3 are spliced together. For presentation, the paper contains a large amount of raw experiment results without necessary high-level interpretation, which makes the main argument difficult to follow.
  4. The experiments are limited to the Qwen-VL series, without extensions to other architectures, such as Gemma and LLaVA.
Questions for authors
  1. Why did the authors exclude BLINK from the main-text Macro-3 metric?
  2. What's the role of "3D scene engine" in the main figure? Is it genuinely implemented but not uploaded in the supplementary material, or is it for future work?
Reviewer_YKHXOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes COSEC. It addresses the concern that rewarding correct answers alone may encourage models to exploit textual shortcuts rather than rely on visual evidence. COSEC uses a frozen scorer to evaluate the same response under an original image, an answer-changing edit, and an answer-preserving perturbation. These comparisons shape the reward and prioritize examples for replay, while task-level success rates control difficulty and sampling proportions. Training data consist of programmatically generated geometric scenes covering counting, color, and spatial relations, with labels derived from scene geometry. Across five models, COSEC improves average endpoint accuracy over GRPO by 2.37 points on three benchmarks. Additional experiments suggest that data renewal can complement reward normalization and reduce training instability.

Critical strengths
  1. Joint design of rewards and training data. COSEC uses shared counterfactual diagnostics for both reward shaping and replay selection, providing a coherent approach to encouraging visual grounding while adapting the training distribution.

  2. Consistent overall gains across model scales. The primary comparison includes multiple independent runs across five model scales in two Qwen-VL families. COSEC improves average endpoint accuracy at every scale, with positive gains retained when all four evaluated benchmarks are included.

Critical weaknesses
  1. The details of training data are missing. COSEC’s main contribution is to connect reward shaping with training-data generation and renewal, yet the main text provides little concrete information about the resulting data. Appendix I.1 specifies three task families, 1200 examples per round, seven rounds, and a target replay fraction of 30%. However, it does not adequately explain the types of scenes generated and how the difficulty variable dd maps to specfic changes in visual complexity. Consequently, the paper describes how the controller updates difficulty more clearly than what those updates actually change in the data.
  2. The evaluation uses modified benchmark subsets and scoring settings. Appendix M.2 clarifies that the reported benchmark names refer to frozen subsets: 793 single-image questions from six BLINK tasks, 2374 MME questions, 1919 MME-RealWorld questions from five domains, and 1222 VSR questions. It also explicitly states that MME is evaluated using question-level accuracy rather than its official composite score. The paper does not adequately explain this choice.
  3. Poor presentation. The paper reports extensive, sometimes overly granular experiments, but does not clearly connect them to a coherent set of research questions or provide sufficient explanation of how individual results support its claims. The figures are similarly overloaded with details that obscure the main ideas. More concerningly, Figure 1 labels the generator as a ''3D scene engine'', whereas Appendix I.1 describes the implementation as rendering geometric primitives with PIL. Although the figure is presented as schematic, this discrepancy creates ambiguity about what was actually implemented. Consequently, I have serious doubts about the authenticity of the paper.
  4. The paper lacks an AI usage statement, and the citation formatting also appears to have issues.
Questions for authors
  1. What is the actual data-generation pipeline and how the difficulty variable dd changes the generated scenes?
  2. Why were modified benchmark subsets and nonstandard MME scoring used, and why was BLINK excluded from the primary aggregate during manuscript revision?
Reviewer_GKqROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This submission studies RL with verifiable rewards for VLM post-training and proposes Counterfactual Self-Evolving Curriculum (CoSEC). CoSEC builds on three components: 1) a dual intervention grounding reward scoring responses on the original image, a label-changing semantic edit, and a nuisance edit preserving the label. 2) a synthetic data generator with "leakage gates" to prevent shortcuts, and 3) a feedback controller and replay archive to adjust task difficulty. Experiments include multiple small (<=7B) models from the Qwen2.5/3 family on MME, MME-RealWorld, VSR, and Macro-3. CoSEC is compared to GPRO and fairly consistently shows smaller gains compared to GPRO.

Critical strengths
  • The idea to ground answers in the proposed counterfactual way is exciting (original, semantic change, nuisance change).
  • CoSEC usually outperforms GPRO by a small but consistent margin
  • Main experiments are conducted over multiple runs, therefore capturing statistical significance.
Critical weaknesses
  • All experiments are limited to smaller Qwen models, which raises the question of whether the method works on other model families and how it scales.
  • The controls in Sec. 6.3 are sometimes not very conclusive, as they are obtained from a single run and show rather minor accuracy changes, therefore it is unclear what the method actually benefits from.

Code: I ran parts of the submitted code and found at least two bigger issues

  • The fixed data control lacks nuisance images, and simply skips that reward.
  • Some of the "label-changing" edits do not change the answer. This is particularly evident for diagonal relations: 92/300 cases retained the original label. This is a lot of noise ...
Questions for authors

see above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.