acceptodds
Under review as a conference paper at ICLR 2027

CARVE: Coordinating Imitation and Exploration through Reward Variance for LLM Reasoning Post-Training

Abstract

Reinforcement learning with verifiable rewards (RLVR) commonly follows supervised fine-tuning (SFT) in the post-training of reasoning models. However, the two stages often reuse the same problems even though group-relative RL depends on reward variation among rollouts. SFT can raise the solve rates of these problems and leave RL with many all-correct groups that provide no effective gradient, a failure mode we term variance starvation. Binary outcome rewards also cannot distinguish among correct derivations, while process supervision becomes less informative when trained on similarly saturated problems. To address these challenges, we propose CARVE, a variance-aware post-training framework with two coordinated designs. First, it separates SFT and RL problems and uses the fine-tuned parent to retain RL prompts with informative outcome variation. Second, it trains a process reward model on a subset of the screened problems, validates its selection ability before use, and restricts it to lowering the reward of weaker correct rollouts. The screened problems therefore provide both useful reward variation for policy optimization and informative supervision for process learning. Experiments across ten benchmarks and two backbone show that CARVE consistently outperforms shared-pool post-training baselines on mathematical reasoning while maintaining comparable performance on code and general reasoning. It achieves these gains with half the SFT data and substantially fewer RL updates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning with verifiable rewards, LLM reasoning, reward variance

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kXiFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed CARVE, which coordinates SFT and group-relative RL through data allocation, reward group variance filtering, and process-based reward adjustment. It uses a source-layered and problem agnostic SFT/RL split, then screens reserved RL problems using the post-SFT parent to retain those with both correct and incorrect sampled answers. A process reward model is trained on screened problems and admitted after held-out solution-selection validation. Its step scores contribute to rollout-level reward deductions for weaker correct solutions, with better granularity in all-correct groups. Experiments report mathematical reasoning improvements over shared-pool SFT-based baselines with fewer RL updates, alongside allocation, screening, reward, and sampling-budget controls.

Critical strengths

The paper provides useful ablations for analyzing the empirical effects of its choices. In particular, the fixed-parent overlap sweep holds RL pool size and update budget constant, and the pool-selection comparison matches parent, pool size, source composition, and updates (Figure 7 and Table 6). These results make the practical effect of allocation and selection more assessable. I acknowledge this empirical value, but it does not resolve my concern about the significance of the contribution.

Critical weaknesses
  1. Equal rewards yielding zero group-relative advantages is a direct property of GRPO's normalization (Eq. 1). The initial stage allocation uses a source-layered problem-text hash, followed by the straightforward rule of retaining problems with both successful and unsuccessful samples. The paper itself acknowledges established pass-rate filtering (section 3). In my own SFT/RL practice, I already separate the training data and have tried variance-based filtering as a heuristic step. My assessment is that the paper does not sufficiently establish new knowledge beyond these familiar design choices. The difficulty-split control and process-reward component should be acknowledged, but the contribution of coordinating them around the parent policy needs a stronger justification than demonstrating the familiar need for informative rollout rewards.

  2. In Table 2, the largest improvement is from shared-pool training to the disjoint split, 38.7% to 46.4%. Screening then raises accuracy to 47.0% while reducing updates from about 3.2k to 1.4k. These rows change both the SFT parent and training budget, so their differences do not isolate separation's mechanism. The fixed-parent overlap sweep supports an overlap effect in the reported setting, and Table 6 shows a 1.4-point benefit of informative-band selection over random selection under matched conditions. I therefore do not claim that screening has no effect. My concern is whether disjointness adds meaningful value once the pools are matched for post-SFT solve-rate distribution and receive the same screening. Group reward variation is the immediate learning mechanism, and the current evidence does not establish how much separation contributes beyond changing that distribution.

Questions for authors
  1. The PRM evaluates individual steps, but their scores are aggregated into a rollout reward and a single advantage is used across the response (Eq. 6 and 7). Why choose this integration instead of using process rewards as step-level learning signals?
  2. Do existing results distinguish shared and disjoint RL pools when the SFT parent, post-SFT solve-rate distribution, pool size, source composition, update budget, and screening rule are matched? The overlap sweep matches pre-SFT difficulty; evidence addressing the post-SFT distribution would strengthen the claim.
  3. Please clarify CARVE's main new contribution. Is it the SFT/RL data allocation, the post-SFT screening, the process-docking reward, or their coordination? Which reported comparisons demonstrate added value beyond the existing components?
Reviewer_4JQJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces CRAVE - a three-step recipe for zero-variance filtering (ZVF). They do this by first splitting SFT and RL dataset into disjoint prompts, then using SFT-ed checkpoint to do ZVF and after screening they train a PRM on these problems for the SFT-ed checkpoint and using a gate to decide whether to use the PRM during RL. When running GRPO, they "dock" the rewards to prevent PRM reward hacking in the following way:

  • incorrect rollouts get zero PRM credit
  • correct rollouts starts a R=2, based on a messy formula in Eq 6 (which i didn't fully understand)

The paper claim training with CRAVE is better than doing simply SFT and SFT + GRPO/DAPO/GRESO.

Critical strengths

I had a very hard time finding strengths in this paper despite carefully searching through the paper.

Critical weaknesses
  • One of the main problems that RL& SFT share overlapping prompts and this is the main cause of many zero-variance prompts during RL training is fabricated. Even the baseline cited in this paper DAPO and GRESO already had separate SFT and RL pools. While ZVF is a real problem in RL pipeline, but claiming the cause is the overlapping problems in RL and SFT is simply not true.
  • Unfair and arbitrary baselines and mis-represented results
    • I found it strange that Table 1 had a rows for Qwen2.5-3B-Instruct and -7B-Instruct but did not report their performance. And as the original Qwen2.5 technical reported, 3B-instruct achieved 86.7 and 65.9 on GSM8K and MATH500, which are significantly higher than all models trained from this checkpoint. This suggests 1) the proposed pipeline degraded the performance, 2) the selected dataset is too narrow that it made the model worse after training, 3) something might be wrong in the training pipeline.
    • DAPO and GRESO are arbitrarily handicapped because they are trained on the shared pools, which is not the case in the original papers.
  • The paper is written in an obscure way which seems to be intentionally difficult to understand: many unnecessary jargons were introduced and a lot of maths in the method sections are not necessary and over complicates otherwise straightforward concepts.

[1] https://arxiv.org/pdf/2412.15115

Questions for authors

N/A

Reviewer_sZiTOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies a problem with standard SFT→GRPO post-training: reusing the same problems across stages lets SFT saturate them, so most RL groups come out all-correct and contribute no gradient. CARVE fixes this by splitting the corpus into disjoint SFT and RL pools, using the SFT parent to screen the RL pool down to problems it solves only sometimes, and training a self-labeled PRM on those problems that is used solely to deduct reward from weaker correct rollouts. This restores reward variance inside all-correct groups without ever rewarding incorrect answers. On Qwen2.5-3B/7B it outperforms shared-pool baselines on math with half the SFT data and far fewer RL updates.

Critical strengths
  1. The paper identifies a real and under-examined coupling between SFT and group-relative RL: reusing problems across stages saturates them during SFT and starves GRPO of reward variance.
  2. The method and its motivation are presented clearly; the pipeline is easy to follow and the ablations (Table 2, Table 3, Fig. 5) isolate the individual components reasonably well.
Critical weaknesses
  1. Gains are reported against the weakest baseline. The delta row compares to SFT→GRPO, which is worse than SFT alone. Against SFT→DAPO the math gain is +2.2 (3B) / +1.6 (7B). The pipeline ends below its starting point, and Table 2 shows most of the gain comes from the different SFT initialization (38.7 → 46.4 from the split alone), not from screening or the PRM. Please report deltas against the base model and SFT→DAPO, and discuss explicitly in the paper.
  2. Single model family with contamination concerns. All results are on Qwen2.5-Instruct, trained on MATH/GSM8K. Prior work shows Qwen2.5 gains on these benchmarks even under spurious rewards [1] and does not transfer to other families; the strict/flexible flip in Table 4 is consistent with this. Results on a non-Qwen2.5 backbone (OLMo, Qwen3) are needed.
  3. Missing baselines. The paper targets SFT→RL coupling but only compares against RL-stage methods. Not compared: SFT-stage objectives that improve RL initialization without data splitting or a PRM ([2], [3]), mixed-policy methods ([4], [5]), and the stage-wise split methods it claims to improve on ([6]). A Disjoint Split + DAPO ablation is also needed to isolate screening/docking from the initialization. At least one method from each category should be compared and discussed with the proposed pipeline to strengthen the author's claim.

[1] Shao, Rulin, et al. "Spurious rewards: Rethinking training signals in rlvr." arXiv preprint arXiv:2506.10947 (2025).

[2] Li, Gaotang, et al. "Beyond log likelihood: Probability-based objectives for supervised fine-tuning across the model capability continuum." arXiv preprint arXiv:2510.00526 (2025).

[3] Wu, Yongliang, et al. "On the generalization of sft: A reinforcement learning perspective with reward rectification." International Conference on Learning Representations. Vol. 2026. 2026.

[4] Yan, Jianhao, et al. "Learning to reason under off-policy guidance." Advances in Neural Information Processing Systems 38 (2026): 117157-117186.

[5] Zhang, Wenhao, et al. "On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting." International Conference on Learning Representations. Vol. 2026. 2026.

[6] Hu, Hanxu, et al. "DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning." arXiv preprint arXiv:2603.11193 (2026).

Questions for authors
  1. Step segmentation (Section 3) is described only informally. Can you release the rules/code and report average steps per solution by source? Does the 20-step cap truncate many MATH/AIME solutions?
  2. PRM reuse for scalability. Must the PRM be retrained for every backbone and corpus, or can one PRM be reused across scales? Reusable PRMs could strengthen the claim made by the authors, and making this pipeline a scalable approach for general post-training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.