- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposed CARVE, which coordinates SFT and group-relative RL through data allocation, reward group variance filtering, and process-based reward adjustment. It uses a source-layered and problem agnostic SFT/RL split, then screens reserved RL problems using the post-SFT parent to retain those with both correct and incorrect sampled answers. A process reward model is trained on screened problems and admitted after held-out solution-selection validation. Its step scores contribute to rollout-level reward deductions for weaker correct solutions, with better granularity in all-correct groups. Experiments report mathematical reasoning improvements over shared-pool SFT-based baselines with fewer RL updates, alongside allocation, screening, reward, and sampling-budget controls.
Critical strengths
The paper provides useful ablations for analyzing the empirical effects of its choices. In particular, the fixed-parent overlap sweep holds RL pool size and update budget constant, and the pool-selection comparison matches parent, pool size, source composition, and updates (Figure 7 and Table 6). These results make the practical effect of allocation and selection more assessable. I acknowledge this empirical value, but it does not resolve my concern about the significance of the contribution.
Critical weaknesses
-
Equal rewards yielding zero group-relative advantages is a direct property of GRPO's normalization (Eq. 1). The initial stage allocation uses a source-layered problem-text hash, followed by the straightforward rule of retaining problems with both successful and unsuccessful samples. The paper itself acknowledges established pass-rate filtering (section 3). In my own SFT/RL practice, I already separate the training data and have tried variance-based filtering as a heuristic step. My assessment is that the paper does not sufficiently establish new knowledge beyond these familiar design choices. The difficulty-split control and process-reward component should be acknowledged, but the contribution of coordinating them around the parent policy needs a stronger justification than demonstrating the familiar need for informative rollout rewards.
-
In Table 2, the largest improvement is from shared-pool training to the disjoint split, 38.7% to 46.4%. Screening then raises accuracy to 47.0% while reducing updates from about 3.2k to 1.4k. These rows change both the SFT parent and training budget, so their differences do not isolate separation's mechanism. The fixed-parent overlap sweep supports an overlap effect in the reported setting, and Table 6 shows a 1.4-point benefit of informative-band selection over random selection under matched conditions. I therefore do not claim that screening has no effect. My concern is whether disjointness adds meaningful value once the pools are matched for post-SFT solve-rate distribution and receive the same screening. Group reward variation is the immediate learning mechanism, and the current evidence does not establish how much separation contributes beyond changing that distribution.
Questions for authors
- The PRM evaluates individual steps, but their scores are aggregated into a rollout reward and a single advantage is used across the response (Eq. 6 and 7). Why choose this integration instead of using process rewards as step-level learning signals?
- Do existing results distinguish shared and disjoint RL pools when the SFT parent, post-SFT solve-rate distribution, pool size, source composition, update budget, and screening rule are matched? The overlap sweep matches pre-SFT difficulty; evidence addressing the post-SFT distribution would strengthen the claim.
- Please clarify CARVE's main new contribution. Is it the SFT/RL data allocation, the post-SFT screening, the process-docking reward, or their coordination? Which reported comparisons demonstrate added value beyond the existing components?