- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies when RLVR can generalize under weak supervision, considering scarce training data, noisy rewards, and self-supervised proxy rewards across Qwen and Llama models and three domains: math, science, and graph reasoning.
Critical strengths
-
Motivation is clear.
-
Experiment setups are clear.
-
The intervention study is useful and reasonably controlled.
Critical weaknesses
For an empirical study, the experiment scale is way too scarce:
-
The paper positions itself as a systematic empirical study of RLVR generalization, but the domain coverage is similarly limited comparing to prior work [1].
-
Two model family coverage is similarly limited comparing to prior work [2].
-
The correlations with existing work [3] should be diven deeper instead of a brief mention in related work.
-
More RLVR methods should be evaluated.
Also, with these prior works, the novelty of the evaluation suite designed in this paper is questionable too.
[1] Cheng, Jorge Zhoujun, et al. "Revisiting reinforcement learning for llm reasoning from a cross-domain perspective." Advances in Neural Information Processing Systems 38 (2025).
[2] Hu, Chuxuan, et al. "Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?." International Conference on Learning Representations. Vol. 2026. 2026.
[3] Shao, Rulin, et al. "Spurious rewards: Rethinking training signals in rlvr." arXiv preprint arXiv:2506.10947 (2025).
Questions for authors
I think this is more a dataset and benchmark paper rather than a method paper.
3 comments
W2: Model coverage compared to Hu et al. [2]
Hu et al. [2] ask whether RL gains transfer to unseen domains. Their large model count comes from an observational study that evaluates 18 existing RL-trained models, which needs inference only. Where they run RL themselves, they use two base models (DeepSeek-R1-Distill-Qwen-1.5B and Llama-3.2-3B-Instruct).
Our paper studies RL training under weak supervision, and every experiment requires a RL training run. Our evaluation spans three types of weak supervision, three domains, and ten models from two families: six existing models (Qwen2.5-1.5B/3B, Qwen2.5-Math-1.5B/7B, Llama3.2-3B-Instruct, and Llama3.1-8B-Instruct), plus four Llama3.2-3B variants that we trained before RL (Base or CPT, each followed by Thinking or Non-Thinking SFT). Our interventional studies additionally involve substantial continual pretraining and SFT. So although we cover fewer models than [2], our study is not smaller in compute, because each of our models is trained and not only evaluated. We believe our experiments provide evidence for our claims by examining dimensions directly relevant to our research questions, including model family, size, specialization, and pre-RL training.
In general, we would like to clarify that model coverage should be guided by each work’s research questions, scope, and experimental setting, and by whether additional experiments are necessary to support its conclusions, rather than simply matching the scale and number of models evaluated in other studies.
However, we appreciate the reviewer’s suggestion on adding model families. To address the reviewer's concern, we add a third family: OLMo-2-1B, which we pretrain from scratch on a math-heavy corpus. We compare the 50B-token and 200B-token checkpoints under scarce data (MATH, N = 8):
| Model | Saturation step | MATH-500 gain | AMC gain | Avg. gain |
|---|---|---|---|---|
| OLMo-2-1B @ 50B tokens | 120 | +1.9 | +1.1 | +1.5 |
| OLMo-2-1B @ 200B tokens | 301 | +6.2 | +3.9 | +5.1 |
The checkpoint with more domain-specific (math-heavy) pretraining saturates later and generalizes better. This matches what we see in the paper for Qwen2.5-Math vs. Qwen2.5 (Table 1) and for our CPT model (Fig. 3).
We will add this result to the revised paper.
[2] Hu et al., "Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?"
W3: Relation to Shao et al. [3]
The main difference between our work and Shao et al. [3] is that Shao et al. show that spurious rewards help Qwen2.5-Math on math, and attribute it to code reasoning, a behavior that helps only Qwen2.5-Math models. In our work, we identify a property that can be measured for any model and domain, reasoning faithfulness, and find that it is associated with slower saturation and better generalization across three weak-supervision settings and three domains. It also applies to models that do not use code reasoning: with our pre-RL intervention, which increases reasoning faithfulness, a failing Llama model generalizes under weak supervision.
We will add the detailed comparison to the related work.
[3] Shao et al., "Spurious Rewards: Rethinking Training Signals in RLVR."
W4: Additional RLVR methods
We use GRPO as our main algorithm because most of the related work we compare with uses it. Beyond it, the paper already evaluates four variants:
- GRPO-POS and GRPO-NEG (Appendix F): GRPO with positive-only updates (equivalent to REINFORCE) and with negative-only updates. On SCIENCE, with 8 and 1024 samples, both show the same saturation and generalization behavior as standard GRPO (Figs. 24 and 25).
- Majority vote [6] and self-certainty [5] (Section 3.3): two RL methods that use proxy rewards without a verifier.
New: ProRL-style training [8]. We tested KL regularization (β = 0.01) with reference-policy resetting every 150 steps, for Llama3.2-3B-Instruct on MATH with N = 8, trained for 750 steps. The results are similar: the training reward still saturates early and the gains stay limited.
We will add this result to the revised paper.
[5] Zhao et al., "Learning to Reason without External Rewards." [6] Zuo et al., "TTRL: Test-Time Reinforcement Learning." [8] Liu et al., "ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models."
Thanks for your responses. As an empirical study I still feel the coverage is limited. I will maintain my score.
We appreciate the reviewer's feedback and are encouraged that the reviewer found the motivation and experiment setups clear and the intervention study useful and reasonably controlled. Below we respond to each weakness and to the question, and we will include these changes in the revised paper.
Scope of the paper
Our paper is not a dataset or benchmark paper, and we do not propose an evaluation suite. All training data and benchmarks are existing ones (Table 2). Our goal is to understand when and why RL works under weak supervision and how to make it work when it fails, through three questions:
Our findings are new relative to prior work on weak supervision [3, 4, 5, 6]. Prior work shows that weak supervision can work, but it does not show what separates the models that generalize from the ones that do not across model families and domains, or how to make a failing model generalize.
We appreciate the reviewer's suggestions and have added a new domain (W1), a new model family (W2), and new RL methods (W4).
W1: Domain coverage compared to Cheng et al. [1]
Cheng et al. [1] and our paper ask different questions. [1] covers six domains and studies cross-domain transfer in the standard RL setting, where multiple domains are necessary to establish their domain transfer claims . We study weak supervision: what happens when the data is scarce, the rewards are noisy, or there is no verifier.
Prior work on weak supervision is narrower than ours. Each paper studies one setting, mostly in the math domain: one training example [4], spurious rewards [3], self-certainty [5], or majority voting [6]. They show that weak supervision can work. Shao et al. [3] explain their result with code reasoning, a behavior that helps only Qwen2.5-Math models. We study three types of weak supervision across three domains with different pretraining exposure (MATH: high, SCIENCE: moderate, GRAPH: low) and two model families, and we evaluate on 7 to 11 in-domain and out-of-domain benchmarks per training domain (Table 2). We also identify a general property, reasoning faithfulness, that is associated with generalization across these models and domains.
However, we appreciate the suggestion to evaluate the approach on additional domains. Therefore, we use the setting in recent work [7] and train the same settings in the paper on chess.
New domain: chess. Following the setting of recent work [7], we ran RL on chess puzzle solving with a self-supervised test-time reward, using a 20M model. With short pretraining, the model saturates quickly and does not generalize: its puzzle-solving accuracy drops by 10.7 points. With longer domain-specific pretraining, the same model has a longer pre-saturation phase and generalizes: its puzzle-solving accuracy improves by 18.1 points. This is the same pattern as in our other three domains.
We will extend the chess experiments and add them, with the comparison to [1], to the revised paper.
[1] Cheng et al., "Revisiting Reinforcement Learning for LLM Reasoning from a Cross-Domain Perspective."
[3] Shao et al., "Spurious Rewards: Rethinking Training Signals in RLVR."
[4] Wang et al., "Reinforcement Learning for Reasoning in Large Language Models with One Training Example."
[5] Zhao et al., "Learning to Reason without External Rewards."
[6] Zuo et al., "TTRL: Test-Time Reinforcement Learning."
[7] Shen et al. "Understanding Reasoning from Pretraining to Post-Training."