- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ROSS, which reuses rollouts generated during RL or on-policy distillation instead of discarding them after one update. It keeps only verifier-positive rollouts. An LLM reviewer (GLM-5.2) then selects which tokens to supervise and drops rollouts that cannot be cleanly annotated. The model is trained with masked SFT on the result. On Qwen3.6-35B-A3B, ROSS improves over the upstream checkpoint and over SFT on all positive rollouts across math, code, multi-domain distillation and SWE-bench Verified. Ablations attribute most of the gain to token masking rather than trajectory filtering.
Critical strengths
- The idea is practical and cheap to adopt. The pass@1 vs. pass@32 analysis convincingly shows that historical rollouts contain behaviors the policy has not consolidated.
- The ablation that separates filtering from masking is clean and informative.
Critical weaknesses
- A strong external model chooses the supervised tokens, so part of the gain may be implicit distillation from GLM-5.2. I find this the most serious concern. Comparisons with self-review, a weaker reviewer, or random masking at the same exclusion rate are needed.
- Every result comes from a single training run without error bars, and several gains are small. The efficiency claim is also untested: the paper reports neither the annotation cost nor a compute-matched comparison with continued RL.
- The main results use one model and one upstream run per domain.
- Efficiency could be the method's biggest advantage: ROSS reuses existing rollouts and needs no new generation or reward computation. However, the paper reports neither the SFT cost nor the GLM-5.2 annotation cost. There is also no compute-matched comparison with continued RL. Without these numbers, the main practical selling point remains unsupported.
Questions for authors
- How much of the gain depends on GLM-5.2? Have you tried using the policy itself or a weaker model as the reviewer, or random masking at the same exclusion rate?
9 comments
Author Response to Reviewer eFUK (2/6): Annotation Quality and Semantic Masking
W1 and Q1 (continued). Reviewer Rationale, Annotation Quality, and Semantic Masking
2. Why Smaller Reviewers Can Be Effective
The results are consistent with the focused nature of ROSS annotation. The reviewer examines existing verifier-positive trajectories, identifies undesirable supervision targets such as corrected mistakes and redundant exploration, and selects useful content already present in the trajectory. Our analysis in Section 4.5 and Appendix E identifies corrected mistakes and redundant exploration as the dominant excluded-content categories:
- Math: together accounting for 92.0%–95.0% of sampled masked trajectories.
- Code: together accounting for 89.3%–92.2% of corresponding samples.
Reviewers still need to assess local correctness, reasoning completeness, and precise span boundaries, but the task primarily involves evaluating and selecting existing content. Together with the reviewer-sensitivity experiments, these observations support the effectiveness of appropriately capable smaller models for ROSS annotation.
3. Annotation Agreement and Quality
We further evaluated annotation reliability through large-scale inter-reviewer agreement and an independent quality assessment. Across 225,597 trajectories with resolved keep/drop decisions from both GLM-5.2 and Qwen3.8-Flash-Next, the two reviewers achieve 93.59% trajectory-level decision agreement and Cohen's κ = 0.780.
We additionally sampled 200 distinct-prompt trajectories (70 Math RL, 70 Code RL, and 60 MOPD). All three reviewers produced definitive decisions for these trajectories, yielding 600 candidate annotations. Kimi-K3.1-Pro conducted a blinded annotation-quality assessment with reviewer identities and original rationales hidden; manual spot checks informed iterative refinement of the evaluation procedure.
| Reviewer | Keep/drop agreement with assessor | Acceptable supervision among retained trajectories |
|---|---|---|
| GLM-5.2 | 93.0% | 80.5% |
| Qwen3.6-35B-A3B | 89.0% | 66.0% |
| Qwen3.8-Flash-Next | 97.0% | 91.4% |
We further evaluated the 153 trajectories retained by all three reviewers.
| Reviewer | Common-set supervision acceptability |
|---|---|
| GLM-5.2 | 83.7% |
| Qwen3.6-35B-A3B | 62.7% |
| Qwen3.8-Flash-Next | 91.5% |
Qwen3.8-Flash-Next achieves high agreement with the original reviewer and high supervision acceptability in the blinded assessment. On commonly retained trajectories, its acceptability exceeds GLM-5.2 by 7.8 percentage points. These results also show that reviewer size alone does not determine annotation quality.
4. Semantic Masking versus Matched Random Masking
We also conducted the requested random-masking control. The baseline uses exactly the same reviewer-retained trajectories, full contexts, initialization checkpoints, and SFT configurations as ROSS. For each trajectory, excluded contiguous spans are randomly relocated while preserving their number, length multiset, and exact supervised-token count per example. This is a stronger control than matching only the global exclusion rate, because each example's supervision budget and excluded-span geometry are preserved.
| Setting | ROSS w/o mask | Random span mask | ROSS | ROSS − Random |
|---|---|---|---|---|
| Math RL Avg. | 76.32 | 75.17 | 76.98 | +1.81 |
| Code RL Avg. | 43.04 | 42.84 | 44.21 | +1.37 |
| MOPD Avg. | 58.49 | 59.58 | 62.20 | +2.62 |
ROSS consistently outperforms random masking across all three settings and on all six MOPD benchmarks. These gains extend beyond reducing the amount of supervised content: reviewer-selected supervision positions provide additional learning value. Together, the smaller-reviewer experiments and matched random-masking controls address the concern that improvements may depend primarily on GLM-5.2.
Revised manuscript: Section 4.6 and Appendix J report reviewer sensitivity and annotation quality; Section 4.4, Table 3, and Appendix I provide the matched random-masking experiments.
Author Response to Reviewer eFUK (3/6): Uncertainty and 4B Model Results
W2. Reliability Across Training Runs and Uncertainty
We appreciate the concern regarding single-run evaluations and statistical uncertainty. The primary experiments involve independently trained Math RL and Code RL policies alongside a separate MOPD pipeline. We further evaluated ROSS with different reviewer configurations, initialization checkpoints, and model scales. These experiments provide complementary evidence across training conditions, although they do not constitute a systematic multi-seed analysis under identical configurations.
In particular, we extended evaluation beyond the primary 35B model to 4B Math RL and 4B Code RL, as well as a production-stage MOPD consolidation pipeline covering 11 domains (detailed under W3). These experiments show that ROSS's effectiveness extends beyond a single model scale and task-specific configuration. Given the cost of repeating full upstream RL/MOPD pipelines, we prioritized broader independent settings and controlled ablations during the rebuttal period. This evidence of applicability is distinct from formal multi-seed uncertainty estimates.
W3. Generalization Across Model Scales and Training Settings
We agree that evaluating ROSS beyond the primary 35B model is important, so we provide additional results from 4B Math RL, 4B Code RL, and a broader production-stage MOPD training pipeline.
1. Results on 4B Math RL and Code RL
The revised manuscript includes an independent Qwen3.5-4B Math RL experiment in Appendix G. We have also completed a 4B Code RL experiment, applying ROSS to its corresponding post-RL checkpoint.
4B Math RL
| Benchmark | RL Upstream | RL + ROSS | Δ |
|---|---|---|---|
| AIME 2025 | 45.42 | 60.94 | +15.52 |
| AIME 2026 | 48.54 | 68.33 | +19.79 |
| HMMT-Nov. | 62.19 | 61.67 | −0.52 |
| Math Avg. | 52.05 | 63.65 | +11.60 |
In this experiment, the 4B upstream model develops repetitive-answer behavior after RL. ROSS improves mathematical performance while suppressing this degeneration: the proportion of responses containing at least five boxed-answer markers decreases from 84.5% to 21.5%.
New 4B Code RL Results
| Benchmark | Code RL Upstream | Code RL + ROSS | Δ |
|---|---|---|---|
| LCB Generation | 47.43 | 50.67 | +3.24 |
| OJ overall | 17.46 | 20.26 | +2.80 |
| Code Avg. | 32.45 | 35.47 | +3.02 |
Code Avg. is the unweighted mean of the two reported code benchmarks.
The newly completed Code RL experiment provides complementary evidence: ROSS improves LCB Generation by 3.24 percentage points and OJ overall by 2.80 points, with a 3.02-point gain in their average. The 4B Math RL experiment also corrects a pronounced repetitive-output failure mode, while Code RL improves on both benchmarks. Together, these results show that ROSS can improve post-RL checkpoints at the 4B scale across mathematical reasoning and code generation, complementing the primary 35B experiments. The new 4B Code RL result is reported here and will be consolidated with the existing 4B Math RL study in the supplementary material.
Author Response to Reviewer eFUK (1/6): External Supervision and Reviewer Sensitivity
Response to Reviewer eFUK
We thank the reviewer for the constructive feedback on external reviewer supervision, reliability of the experimental improvements, generalization across model scales, and the cost of historical experience reuse. To address these concerns, we conducted reviewer-sensitivity experiments, annotation-quality assessments, token-budget-matched random masking controls, and evaluations across model scales and training settings. We also measured the complete annotation workload and estimated monetary costs using publicly available API pricing. We address each concern below.
W1 and Q1. Dependence on GLM-5.2 and the Role of External Supervision
We acknowledge that the LLM reviewer provides additional supervision through its selection decisions. An important characteristic of ROSS is that the reviewer selects supervision targets exclusively from existing self-generated trajectories, without generating replacement solutions. All SFT target tokens originate from the model's historical policies: the reviewer examines verifier-positive trajectories, identifies undesirable behaviors, and determines which original tokens should receive imitation loss. To investigate dependence on reviewer capability, we evaluated substantially smaller reviewers and introduced a token-budget-matched random masking baseline.
1. Reviewer Model Sensitivity
The original ROSS experiments use GLM-5.2, a mixture-of-experts model with approximately 753B total parameters, operating in high-thinking mode. We additionally evaluated two substantially smaller reviewers:
Qwen3.6-35B-A3B also shares the backbone of the upstream policies and uses their starting model, providing an approximation to same-backbone self-review. We repeated the complete annotation and downstream SFT procedures with these alternative reviewers.
Math RL
Code RL
MOPD
Both smaller reviewers improve the corresponding upstream policies across all three aggregate evaluations. Qwen3.8-Flash-Next exceeds the original GLM-5.2-reviewed ROSS in Math RL (77.20 vs. 76.98), Code RL (45.53 vs. 44.21), and MOPD (63.14 vs. 62.20). Qwen3.6-35B-A3B, which is substantially smaller than GLM-5.2, also consistently improves upon Upstream in the aggregate results. These experiments demonstrate that the effectiveness of ROSS is not specific to GLM-5.2: smaller reviewers can also select useful supervision and improve downstream performance.