acceptodds
Under review as a conference paper at ICLR 2027

When Can LLMs Learn to Reason with Weak Supervision?

Abstract

Large language models have achieved significant reasoning improvements through reinforcement learning with verifiable rewards (RLVR). Yet as model capabilities grow, constructing high-quality reward signals become increasingly difficult, making it essential to understand when RLVR can succeed under weaker forms of supervision. We conduct a systematic empirical study across diverse model families and reasoning domains under three weak supervision settings: scarce data, noisy rewards, and self-supervised proxy rewards. We find that generalization is governed by training reward saturation dynamics: models that generalize exhibit a prolonged pre-saturation phase during which training reward and downstream performance climb together, while models that saturate rapidly memorize rather than learn. We identify reasoning faithfulness, defined as the extent to which a model’s intermediate steps logically support its final answer, as the pre-RL property that predicts which regime a model falls into, while output diversity alone is uninformative. Motivated by these findings, we disentangle the contributions of continual pre-training and supervised fine-tuning, finding that SFT on explicit reasoning traces is necessary for generalization under weak supervision, while continual pre-training on domain data amplifies the effect. Applied together to Llama3.2-3B-Base, these interventions enable generalization across all three settings where the base model previously failed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Weak supervision, RLVR, Reasoning faithfulness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sMsJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when RLVR can generalize under weak supervision, considering scarce training data, noisy rewards, and self-supervised proxy rewards across Qwen and Llama models and three domains: math, science, and graph reasoning.

Critical strengths
  1. Motivation is clear.

  2. Experiment setups are clear.

  3. The intervention study is useful and reasonably controlled.

Critical weaknesses

For an empirical study, the experiment scale is way too scarce:

  1. The paper positions itself as a systematic empirical study of RLVR generalization, but the domain coverage is similarly limited comparing to prior work [1].

  2. Two model family coverage is similarly limited comparing to prior work [2].

  3. The correlations with existing work [3] should be diven deeper instead of a brief mention in related work.

  4. More RLVR methods should be evaluated.

Also, with these prior works, the novelty of the evaluation suite designed in this paper is questionable too.

[1] Cheng, Jorge Zhoujun, et al. "Revisiting reinforcement learning for llm reasoning from a cross-domain perspective." Advances in Neural Information Processing Systems 38 (2025).

[2] Hu, Chuxuan, et al. "Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Shao, Rulin, et al. "Spurious rewards: Rethinking training signals in rlvr." arXiv preprint arXiv:2506.10947 (2025).

Questions for authors

I think this is more a dataset and benchmark paper rather than a method paper.

3 comments

We appreciate the reviewer's feedback and are encouraged that the reviewer found the motivation and experiment setups clear and the intervention study useful and reasonably controlled. Below we respond to each weakness and to the question, and we will include these changes in the revised paper.

Scope of the paper

Our paper is not a dataset or benchmark paper, and we do not propose an evaluation suite. All training data and benchmarks are existing ones (Table 2). Our goal is to understand when and why RL works under weak supervision and how to make it work when it fails, through three questions:

  1. RQ1 (when). When does RL under weak supervision (scarce data, noisy rewards, proxy rewards) generalize? We find that the models that generalize are the ones whose training reward saturates slowly. Models that saturate quickly memorize the training samples and do not generalize.
  2. RQ2 (which property). We find that a longer pre-saturation phase and better generalization are correlated with higher reasoning faithfulness, not with higher output diversity.
  3. RQ3 (intervention). We take a Llama model that did not generalize under weak supervision and raise its reasoning faithfulness with domain-specific mid-training (CPT and Thinking SFT). The model then generalizes under all three weak-supervision settings.

Our findings are new relative to prior work on weak supervision [3, 4, 5, 6]. Prior work shows that weak supervision can work, but it does not show what separates the models that generalize from the ones that do not across model families and domains, or how to make a failing model generalize.

We appreciate the reviewer's suggestions and have added a new domain (W1), a new model family (W2), and new RL methods (W4).

W1: Domain coverage compared to Cheng et al. [1]

Cheng et al. [1] and our paper ask different questions. [1] covers six domains and studies cross-domain transfer in the standard RL setting, where multiple domains are necessary to establish their domain transfer claims . We study weak supervision: what happens when the data is scarce, the rewards are noisy, or there is no verifier.

Prior work on weak supervision is narrower than ours. Each paper studies one setting, mostly in the math domain: one training example [4], spurious rewards [3], self-certainty [5], or majority voting [6]. They show that weak supervision can work. Shao et al. [3] explain their result with code reasoning, a behavior that helps only Qwen2.5-Math models. We study three types of weak supervision across three domains with different pretraining exposure (MATH: high, SCIENCE: moderate, GRAPH: low) and two model families, and we evaluate on 7 to 11 in-domain and out-of-domain benchmarks per training domain (Table 2). We also identify a general property, reasoning faithfulness, that is associated with generalization across these models and domains.

However, we appreciate the suggestion to evaluate the approach on additional domains. Therefore, we use the setting in recent work [7] and train the same settings in the paper on chess.

New domain: chess. Following the setting of recent work [7], we ran RL on chess puzzle solving with a self-supervised test-time reward, using a 20M model. With short pretraining, the model saturates quickly and does not generalize: its puzzle-solving accuracy drops by 10.7 points. With longer domain-specific pretraining, the same model has a longer pre-saturation phase and generalizes: its puzzle-solving accuracy improves by 18.1 points. This is the same pattern as in our other three domains.

We will extend the chess experiments and add them, with the comparison to [1], to the revised paper.

[1] Cheng et al., "Revisiting Reinforcement Learning for LLM Reasoning from a Cross-Domain Perspective."

[3] Shao et al., "Spurious Rewards: Rethinking Training Signals in RLVR."

[4] Wang et al., "Reinforcement Learning for Reasoning in Large Language Models with One Training Example."

[5] Zhao et al., "Learning to Reason without External Rewards."

[6] Zuo et al., "TTRL: Test-Time Reinforcement Learning."

[7] Shen et al. "Understanding Reasoning from Pretraining to Post-Training."

W2: Model coverage compared to Hu et al. [2]

Hu et al. [2] ask whether RL gains transfer to unseen domains. Their large model count comes from an observational study that evaluates 18 existing RL-trained models, which needs inference only. Where they run RL themselves, they use two base models (DeepSeek-R1-Distill-Qwen-1.5B and Llama-3.2-3B-Instruct).

Our paper studies RL training under weak supervision, and every experiment requires a RL training run. Our evaluation spans three types of weak supervision, three domains, and ten models from two families: six existing models (Qwen2.5-1.5B/3B, Qwen2.5-Math-1.5B/7B, Llama3.2-3B-Instruct, and Llama3.1-8B-Instruct), plus four Llama3.2-3B variants that we trained before RL (Base or CPT, each followed by Thinking or Non-Thinking SFT). Our interventional studies additionally involve substantial continual pretraining and SFT. So although we cover fewer models than [2], our study is not smaller in compute, because each of our models is trained and not only evaluated. We believe our experiments provide evidence for our claims by examining dimensions directly relevant to our research questions, including model family, size, specialization, and pre-RL training.

In general, we would like to clarify that model coverage should be guided by each work’s research questions, scope, and experimental setting, and by whether additional experiments are necessary to support its conclusions, rather than simply matching the scale and number of models evaluated in other studies.

However, we appreciate the reviewer’s suggestion on adding model families. To address the reviewer's concern, we add a third family: OLMo-2-1B, which we pretrain from scratch on a math-heavy corpus. We compare the 50B-token and 200B-token checkpoints under scarce data (MATH, N = 8):

ModelSaturation stepMATH-500 gainAMC gainAvg. gain
OLMo-2-1B @ 50B tokens120+1.9+1.1+1.5
OLMo-2-1B @ 200B tokens301+6.2+3.9+5.1

The checkpoint with more domain-specific (math-heavy) pretraining saturates later and generalizes better. This matches what we see in the paper for Qwen2.5-Math vs. Qwen2.5 (Table 1) and for our CPT model (Fig. 3).

We will add this result to the revised paper.

[2] Hu et al., "Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?"

W3: Relation to Shao et al. [3]

The main difference between our work and Shao et al. [3] is that Shao et al. show that spurious rewards help Qwen2.5-Math on math, and attribute it to code reasoning, a behavior that helps only Qwen2.5-Math models. In our work, we identify a property that can be measured for any model and domain, reasoning faithfulness, and find that it is associated with slower saturation and better generalization across three weak-supervision settings and three domains. It also applies to models that do not use code reasoning: with our pre-RL intervention, which increases reasoning faithfulness, a failing Llama model generalizes under weak supervision.

We will add the detailed comparison to the related work.

[3] Shao et al., "Spurious Rewards: Rethinking Training Signals in RLVR."

W4: Additional RLVR methods

We use GRPO as our main algorithm because most of the related work we compare with uses it. Beyond it, the paper already evaluates four variants:

  • GRPO-POS and GRPO-NEG (Appendix F): GRPO with positive-only updates (equivalent to REINFORCE) and with negative-only updates. On SCIENCE, with 8 and 1024 samples, both show the same saturation and generalization behavior as standard GRPO (Figs. 24 and 25).
  • Majority vote [6] and self-certainty [5] (Section 3.3): two RL methods that use proxy rewards without a verifier.

New: ProRL-style training [8]. We tested KL regularization (β = 0.01) with reference-policy resetting every 150 steps, for Llama3.2-3B-Instruct on MATH with N = 8, trained for 750 steps. The results are similar: the training reward still saturates early and the gains stay limited.

We will add this result to the revised paper.

[5] Zhao et al., "Learning to Reason without External Rewards." [6] Zuo et al., "TTRL: Test-Time Reinforcement Learning." [8] Liu et al., "ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models."

Thanks for your responses. As an empirical study I still feel the coverage is limited. I will maintain my score.

Reviewer_3x7GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the effect of continual pre-training and midtraining on the followup RL training with weak supervision, e.g., with little training data, or noisy rewards or self-supervised rewards. Specifically, it shows that (1) with scarce training data, the success of the trained model depends on the base model and the training domain. (2) With noisy rewards, the so-called faster-saturating model-domain pairs are less robust to the noise. (3) Self-supervised proxy rewards generally fail. Furthermore, the authors find that the rapid saturation is not caused by the semantic diversity via LLM-as-a-judge, and that the reasoning faithfulness is aligned with the success under weak supervision. Lastly, the authors conduct control study of {continue pre-trianing (CPT), without CPT}×\times{thinking SFT, non-thinking SFT}, and find that CPT+thinking-SFT can yield the best trained model under weak supervision during RL.

Critical strengths
  • This paper considers how CPT and SFT can benefit RL training under weak supervision. The scenario where weak supervision exists is quite common as there are lots of tasks where precise reward cannot be given beyond math, though this paper studies the math task as a clean environment.
  • It implies that the faithfulness during RL training is the key element that enables generalization, instead of the semantic diversity of the rollouts.
Critical weaknesses
  • In takeaway in lines 259, 300 and 350, the authors conclude that the rapid saturation reflects memorization rather than learning. What do the authors precisely mean by "memorization" here? Is it high training reward, low training loss, or the model can output the same training token sequence during inference?
  • It seems the paper conveys the message that, only the process reward is more useful and accurate compared to outcome reward, in the sense that it yields models that consistently improve the performance and generalizes better than the counterparts.
  • The paper shows that when the reward rapidly saturates during training, the reasoning faithfulness decreases or reaches a plateau. This correlation does not provide a causal relationship between them.
Questions for authors
  • If using process-reward, without any midtraining steps, can the models also generalize after RL training? Can the authors comment on this: is midtraining with faithful reasoning rollouts an alternative of process reward verifier?

Minors:

  • In Line 247, based on the results in Figure 1, 9, 10, how do the authors conclude domain-specific pretraining is necessary to sustain the pre-saturation phase?
  • Line 292 is not a complete sentence.
5 comments

Thank you for the thoughtful review. We are glad the reviewer sees that weak supervision is a common practical setting and that reasoning faithfulness, not semantic diversity, separates the models that generalize. We address the weaknesses and questions below and will include all changes in the revised paper.

W1: Clarifying what we mean by "memorization"

By "memorization" we mean high training reward without generalization: the training reward on the training prompts rises very quickly to its maximum (Llama3.2-3B-Instruct, Fig. 1), but held-out performance, in-domain and out-of-domain, improves little and then stops or declines. Qualitatively, we find that Llama3.2-3B-Instruct shows higher diversity but lower faithfulness in reasoning traces (Fig. 26). These suggest that the model learns to get the training answers right without learning reasoning that transfers. In contrast, for Qwen2.5-Math-1.5B (Fig. 1) and our CPT + Thinking SFT model (Fig. 3), the training reward rises slowly and held-out performance rises with it.

We will clarify this definition in Section 3.1.

W2 & Q1: Our experiments use outcome rewards only

Our paper studies weak supervision with outcome-based rewards only, as explicitly stated in line 146-151 in the manuscript. Every RL run gives one reward per response: a binary check on the final answer, or majority vote / self-certainty in the proxy-reward setting. We do not use process-based rewards, and the paper never discusses them.

So our message is not that process reward is better than outcome reward. It is that, with the same outcome reward, models with higher reasoning faithfulness saturate more slowly and generalize better, both in-domain and out-of-domain.

On Q1: We agree that studying process reward based RL will be an interesting direction. However, we believe process-reward experiments are beyond the scope of this paper., We will note this as future work.

W3: The link between saturation and reasoning faithfulness is a correlation

We agree, and we do not claim a causal relationship. Our goal is to find whether any property of the base model, measured before RL, is associated with generalization under RL with weak supervision. We checked output diversity (Section 3.4), and token budget, initial accuracy, and response length (see our response to Reviewer Mjo8, W2). The property we find correlated with the outcome is reasoning faithfulness:

ModelReasoning faithfulnessSaturation stepAvg. gain on math evals
Instruct0.26≈60−3.8
CPT + Thinking SFT0.43≈300+9.2

We will revise the wording to say that reasoning faithfulness is associated with a longer pre-saturation phase and better generalization.

Minor points

Line 247. In Figs. 1, 9, and 10 we compare a model with domain-specific pretraining (Qwen2.5-Math-1.5B, which has 1T extra math tokens) against Qwen2.5-1.5B, and find that Qwen2.5-Math sustains a longer pre-saturation phase (302 vs. 170 steps on MATH, 268 vs. 161 on SCIENCE; Table 1). We see the same with our own domain-specific model: CPT + Thinking SFT sustains a longer pre-saturation phase than Llama3.2-3B-Instruct (≈300 vs. ≈60 steps; Fig. 3). So domain-specific pretraining can sustain the pre-saturation phase, but there may be other ways to do so. We will revise "is necessary to sustain" to "helps sustain".

Line 292. This is the continuation of a sentence that starts on the previous page: "On MATH and SCIENCE, Qwen models maintain gains under substantial corruption (up to γ = 0.7), while Qwen on GRAPH and Llama on MATH and SCIENCE degrade at γ ≥ 0.5." Figure 2 was placed between the two halves, which made it look incomplete. We will move the figure so the sentence is not split.

Thank the authors for their response.

For W1, I think “overfitting” better describes the observed phenomenon than “memorization”.

For W2, I fully acknowledge that the manuscript only adopts the outcome reward only. To clarify my question, I wish the authors to distinguish two statements: (1) the performance consistently improves and generalization get better because of the better faithfulness in the rollout; (2) the performance consistently improves and generalization get better because the rollout contains more correct steps. Furthermore, for your LLM-as-a-judge annotations, how is the faithfulness correlated with the fraction of correct steps in the rollout? If these two metrics share high correlation, how do we distinguish which is the true workhorse of the improved performance?

For W3, thanks for the clarification. Following the review of Reviewer Mjo8, I have a similar question. The faithfulness metric used in the paper does not capture whether the model actually relies on the reasoning process to derive its answer. Specifically, when using the thinking training data, the generalization performance improves, and the authors attribute this to the improved faithfulness of the reasoning data. However, as many factors have been changed when switching from non-thinking data to thinking data, it remains unclear how faithfulness alone influence the final performance, as the given new results at Reviewer Mjo8, W2 only exclude limited hypothesis. Furthermore, prior work [1] has already shown that SFT on explicit reasoning traces can improve the Causal Importance Reasoning (CIR) and Sufficiency of Reasoning (SR) under outcome-based RL. Therefore, the observed association between faithfulness and generalization seems to be insufficient to establish faithfulness is the underlying causal mechanism.

For the minors, please incorporate them in the revised version.

Based on the current discussions, I incline to maintain my evaluation.

[1] Yu, et al. Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning. ICML 2026.

We thank the reviewer for the follow-up.

(1) "Memorization" vs. "overfitting". By "memorization" we meant that the model gets the final answer right on the training prompts while its reasoning does not support that answer. We agree that "overfitting" is the better term, and we will use it throughout the paper.

(2) Faithfulness vs. correctness. We checked whether faithfulness is simply correctness under another name. It is not: in our rollouts the two separate in both directions. (Correctness here means correctness of the final answer.)

  • Correct but unfaithful. For the model that generalizes poorly (Llama3.2-3B-Instruct, Fig. 1), 43% of rollouts with a correct answer have unfaithful reasoning under weak supervision. The model reaches the right answer without reasoning that supports it.
  • Wrong but faithful. For the model that generalizes well (Qwen2.5-Math-1.5B, Fig. 1), 72% of rollouts with a wrong answer have faithful reasoning, against 28% unfaithful. The model reasons genuinely even when it does not reach the right answer.

So a rollout can be correct and unfaithful, or wrong and faithful: faithfulness is not a proxy for correctness.

(3) We do not claim faithfulness is a causal mechanism. We understand the reviewer's concern. As we showed in our response to Reviewer Mjo8 (W2), when we controlled for other factors (token budget, initial accuracy and response length), reasoning faithfulness was the only metric that differed between the models that generalize and those that do not. We agree that this does not prove reasoning faithfulness is the only cause. We will revise the wording in the paper to say that reasoning faithfulness is associated with slower saturation and better generalization, and we will add both tables to the revision.

Reviewer_Mjo8OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when GRPO improves reasoning under scarce examples, corrupted answer labels, and model-derived proxy rewards. Across mathematical, scientific, and graph reasoning tasks, it relates generalization to training-reward saturation and agreement between written reasoning and final answers. A crossed intervention involving continued pretraining and Thinking versus Non-Thinking SFT examines how pre-RL preparation changes subsequent learning. The contribution is an empirical account of both successful learning and reward exploitation under weak supervision, rather than a new optimization algorithm.

Critical strengths
  1. The paper compares multiple weak-supervision regimes within one framework and tracks downstream performance alongside training reward. This documents cases in which increased training reward is not accompanied by improved generalization, providing evidence about the limits of weak rewards.

  2. The intervention study tests changes to pre-RL training in addition to comparing model families. The behavioral analysis also includes judge-agreement checks and a manual audit, lending support to the reported textual patterns. Together, these analyses support the descriptive finding that pre-RL preparation affects subsequent weak-supervision outcomes.

Critical weaknesses
  1. The definition of the post-saturation gain is inconsistent with the reported negative values. A maximum improvement over an interval that includes the saturation checkpoint cannot be negative when the same measurement at that checkpoint is subtracted, yet values of −1.9 and −2.1 are reported. A difference between the defined and implemented statistics is one possible explanation; clarifying this distinction is needed to interpret the saturation analysis.

  2. The intervention does not isolate faithfulness as the cause of improved generalization. Thinking SFT uses roughly 1B tokens compared with 0.27B for Non-Thinking SFT, while the interventions can also change domain knowledge, initial accuracy, and response length; shared continued pretraining does not eliminate these confounds. The comparisons support an effect of the overall pre-RL preparation, while the separate causal contribution of faithfulness remains unresolved. Together with the metric-definition issue, this limits the strength of the proposed causal account.

Questions for authors

See Weaknesses.

1 comment

We thank the reviewer for their careful review and for recognizing the value of studying several weak-supervision settings and of the pre-RL intervention study. We address the weaknesses below and will include all changes in the revised paper.

W1: Clarifying the definition of the post-saturation gain

Thank you for pointing this out. Our implementation excludes the saturation checkpoint from the maximization. We will correct the typo in the definition to match the implementation: Δpost∗(n)(M):=max⁡t∈(tsat(n),T]M(n)(t)−M(n)(tsat(n))\Delta_{\mathrm{post}}^{*(n)}(M) := \max_{t \in (t_{\mathrm{sat}}^{(n)}, T]} M^{(n)}(t) - M^{(n)}(t_{\mathrm{sat}}^{(n)}). A negative value therefore means that performance at every checkpoint after saturation is below performance at the saturation checkpoint.

W2: Clarifying the link between CoT faithfulness and generalization, accounting for token budget, initial accuracy, and response length

We understand the concern that other factors could explain the result. We checked each factor the reviewer lists.

(1) Token budget and initial accuracy. Both SFT versions use the same prompts and solutions; Non-Thinking SFT is smaller (0.27B vs. 1B) only because the reasoning traces are removed. To test whether more tokens help, we compare with Llama3.2-3B-Instruct, which received more than 10B SFT tokens [1], about ten times our Thinking SFT, and starts from a similar accuracy.

ModelSFT tokensInitial accuracyCoT faithfulnessSaturation stepAvg. gain on math evals
Instruct>10B≈16%0.26≈60−3.8
CPT + Thinking SFT (Our)1B≈16%0.43≈300+9.2

With ten times more SFT tokens and the same initial accuracy, Instruct saturates early and gets worse. So neither token budget nor initial accuracy explains the gap, while CoT faithfulness clearly differs.

(2) Response length. Base + Thinking SFT and CPT + Thinking SFT produce responses of the same mean length, yet their CoT faithfulness and their gains differ.

ModelMean response lengthCoT faithfulnessAvg. gain on math evals
Base + Thinking SFT≈6K tokens0.10+3.4
CPT + Thinking SFT≈6K tokens0.43+9.2

(3) Domain knowledge. CPT + Non-Thinking SFT has the same 52B math tokens as CPT + Thinking SFT but gains only +1.7 vs. +9.2.

We agree that these comparisons do not prove CoT faithfulness is the only cause. In the paper, we will revise the wording to say that CoT faithfulness is associated with slower saturation and better generalization. We will add both tables to the revision.

[1] Grattafiori et al., "The Llama 3 Herd of Models."

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.