acceptodds
Under review as a conference paper at ICLR 2027

ROSS: Relearning from Self-Generated Rollouts through Selective Supervision

Abstract

Large language model post-training generates self-generated rollouts through reinforcement learning and on-policy distillation, yet this experience is often treated as stale once the policy advances. Historical rollouts can remain compatible with a later policy while preserving behaviors that the policy no longer expresses reliably. However, they may also contain mistakes, abandoned attempts, and redundant actions that should not be imitated, motivating finer-grained selective supervision. We introduce **ROSS** (Relearning from Self-Generated **Ro**llouts through **S**elective **S**upervision), which preserves the full historical trajectory as context while applying loss only to selected model-generated continuations. Across domain-specific reinforcement learning, multi-teacher on-policy distillation, and agentic reinforcement learning, ROSS consistently improves upstream checkpoints and outperforms baselines across mathematics, code generation, instruction following, and software engineering. On Qwen3.6-35B-A3B, ROSS improves the six-benchmark MOPD average from 58.40% to 62.20% and SWE-bench Verified from 64.20% to 68.40%. These results show that self-rollout training leaves behind reusable behavioral experience that can yield further gains through offline supervised fine-tuning (SFT), without additional policy rollouts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning, Self-Generated Rollouts, Selective Supervision, Post-Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_eFUKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ROSS, which reuses rollouts generated during RL or on-policy distillation instead of discarding them after one update. It keeps only verifier-positive rollouts. An LLM reviewer (GLM-5.2) then selects which tokens to supervise and drops rollouts that cannot be cleanly annotated. The model is trained with masked SFT on the result. On Qwen3.6-35B-A3B, ROSS improves over the upstream checkpoint and over SFT on all positive rollouts across math, code, multi-domain distillation and SWE-bench Verified. Ablations attribute most of the gain to token masking rather than trajectory filtering.

Critical strengths
  1. The idea is practical and cheap to adopt. The pass@1 vs. pass@32 analysis convincingly shows that historical rollouts contain behaviors the policy has not consolidated.
  2. The ablation that separates filtering from masking is clean and informative.
Critical weaknesses
  1. A strong external model chooses the supervised tokens, so part of the gain may be implicit distillation from GLM-5.2. I find this the most serious concern. Comparisons with self-review, a weaker reviewer, or random masking at the same exclusion rate are needed.
  2. Every result comes from a single training run without error bars, and several gains are small. The efficiency claim is also untested: the paper reports neither the annotation cost nor a compute-matched comparison with continued RL.
  3. The main results use one model and one upstream run per domain.
  4. Efficiency could be the method's biggest advantage: ROSS reuses existing rollouts and needs no new generation or reward computation. However, the paper reports neither the SFT cost nor the GLM-5.2 annotation cost. There is also no compute-matched comparison with continued RL. Without these numbers, the main practical selling point remains unsupported.
Questions for authors
  1. How much of the gain depends on GLM-5.2? Have you tried using the policy itself or a weaker model as the reviewer, or random masking at the same exclusion rate?
9 comments
AuthorsOpenReview ↗

Author Response to Reviewer eFUK (1/6): External Supervision and Reviewer Sensitivity

Response to Reviewer eFUK

We thank the reviewer for the constructive feedback on external reviewer supervision, reliability of the experimental improvements, generalization across model scales, and the cost of historical experience reuse. To address these concerns, we conducted reviewer-sensitivity experiments, annotation-quality assessments, token-budget-matched random masking controls, and evaluations across model scales and training settings. We also measured the complete annotation workload and estimated monetary costs using publicly available API pricing. We address each concern below.


W1 and Q1. Dependence on GLM-5.2 and the Role of External Supervision

We acknowledge that the LLM reviewer provides additional supervision through its selection decisions. An important characteristic of ROSS is that the reviewer selects supervision targets exclusively from existing self-generated trajectories, without generating replacement solutions. All SFT target tokens originate from the model's historical policies: the reviewer examines verifier-positive trajectories, identifies undesirable behaviors, and determines which original tokens should receive imitation loss. To investigate dependence on reviewer capability, we evaluated substantially smaller reviewers and introduced a token-budget-matched random masking baseline.

1. Reviewer Model Sensitivity

The original ROSS experiments use GLM-5.2, a mixture-of-experts model with approximately 753B total parameters, operating in high-thinking mode. We additionally evaluated two substantially smaller reviewers:

  • Qwen3.6-35B-A3B: approximately 35B total parameters and 3B active parameters, using thinking mode.
  • Qwen3.8-Flash-Next: approximately 176B total parameters and 6B active parameters, using xhigh mode.

Qwen3.6-35B-A3B also shares the backbone of the upstream policies and uses their starting model, providing an approximation to same-backbone self-review. We repeated the complete annotation and downstream SFT procedures with these alternative reviewers.

Math RL

ReviewerAIME 2025AIME 2026HMMT-Nov.Math Avg.
Upstream74.8477.5073.2375.19
GLM-5.2 (original)76.4679.9074.5876.98
Qwen3.6-35B-A3B75.5279.9574.3876.61
Qwen3.8-Flash-Next77.7180.0573.8577.20

Code RL

ReviewerLCB GenerationOJBenchCode Avg.
Upstream58.9525.2242.09
GLM-5.2 (original)61.0527.3744.21
Qwen3.6-35B-A3B62.2925.6543.97
Qwen3.8-Flash-Next63.9027.1645.53

MOPD

ReviewerAIME 25AIME 26HMMT-Nov.LCB GenOJBenchIFBenchAvg.
Upstream72.8177.6071.2557.4326.7244.5858.40
GLM-5.2 (original)77.7680.4276.7760.7629.7447.7662.20
Qwen3.6-35B-A3B78.3380.9475.6359.5229.3146.6861.74
Qwen3.8-Flash-Next78.4480.9976.2562.8628.0252.3063.14

Both smaller reviewers improve the corresponding upstream policies across all three aggregate evaluations. Qwen3.8-Flash-Next exceeds the original GLM-5.2-reviewed ROSS in Math RL (77.20 vs. 76.98), Code RL (45.53 vs. 44.21), and MOPD (63.14 vs. 62.20). Qwen3.6-35B-A3B, which is substantially smaller than GLM-5.2, also consistently improves upon Upstream in the aggregate results. These experiments demonstrate that the effectiveness of ROSS is not specific to GLM-5.2: smaller reviewers can also select useful supervision and improve downstream performance.

AuthorsOpenReview ↗

Author Response to Reviewer eFUK (2/6): Annotation Quality and Semantic Masking

W1 and Q1 (continued). Reviewer Rationale, Annotation Quality, and Semantic Masking

2. Why Smaller Reviewers Can Be Effective

The results are consistent with the focused nature of ROSS annotation. The reviewer examines existing verifier-positive trajectories, identifies undesirable supervision targets such as corrected mistakes and redundant exploration, and selects useful content already present in the trajectory. Our analysis in Section 4.5 and Appendix E identifies corrected mistakes and redundant exploration as the dominant excluded-content categories:

  • Math: together accounting for 92.0%–95.0% of sampled masked trajectories.
  • Code: together accounting for 89.3%–92.2% of corresponding samples.

Reviewers still need to assess local correctness, reasoning completeness, and precise span boundaries, but the task primarily involves evaluating and selecting existing content. Together with the reviewer-sensitivity experiments, these observations support the effectiveness of appropriately capable smaller models for ROSS annotation.

3. Annotation Agreement and Quality

We further evaluated annotation reliability through large-scale inter-reviewer agreement and an independent quality assessment. Across 225,597 trajectories with resolved keep/drop decisions from both GLM-5.2 and Qwen3.8-Flash-Next, the two reviewers achieve 93.59% trajectory-level decision agreement and Cohen's κ = 0.780.

We additionally sampled 200 distinct-prompt trajectories (70 Math RL, 70 Code RL, and 60 MOPD). All three reviewers produced definitive decisions for these trajectories, yielding 600 candidate annotations. Kimi-K3.1-Pro conducted a blinded annotation-quality assessment with reviewer identities and original rationales hidden; manual spot checks informed iterative refinement of the evaluation procedure.

ReviewerKeep/drop agreement with assessorAcceptable supervision among retained trajectories
GLM-5.293.0%80.5%
Qwen3.6-35B-A3B89.0%66.0%
Qwen3.8-Flash-Next97.0%91.4%

We further evaluated the 153 trajectories retained by all three reviewers.

ReviewerCommon-set supervision acceptability
GLM-5.283.7%
Qwen3.6-35B-A3B62.7%
Qwen3.8-Flash-Next91.5%

Qwen3.8-Flash-Next achieves high agreement with the original reviewer and high supervision acceptability in the blinded assessment. On commonly retained trajectories, its acceptability exceeds GLM-5.2 by 7.8 percentage points. These results also show that reviewer size alone does not determine annotation quality.

4. Semantic Masking versus Matched Random Masking

We also conducted the requested random-masking control. The baseline uses exactly the same reviewer-retained trajectories, full contexts, initialization checkpoints, and SFT configurations as ROSS. For each trajectory, excluded contiguous spans are randomly relocated while preserving their number, length multiset, and exact supervised-token count per example. This is a stronger control than matching only the global exclusion rate, because each example's supervision budget and excluded-span geometry are preserved.

SettingROSS w/o maskRandom span maskROSSROSS − Random
Math RL Avg.76.3275.1776.98+1.81
Code RL Avg.43.0442.8444.21+1.37
MOPD Avg.58.4959.5862.20+2.62

ROSS consistently outperforms random masking across all three settings and on all six MOPD benchmarks. These gains extend beyond reducing the amount of supervised content: reviewer-selected supervision positions provide additional learning value. Together, the smaller-reviewer experiments and matched random-masking controls address the concern that improvements may depend primarily on GLM-5.2.

Revised manuscript: Section 4.6 and Appendix J report reviewer sensitivity and annotation quality; Section 4.4, Table 3, and Appendix I provide the matched random-masking experiments.

AuthorsOpenReview ↗

Author Response to Reviewer eFUK (3/6): Uncertainty and 4B Model Results

W2. Reliability Across Training Runs and Uncertainty

We appreciate the concern regarding single-run evaluations and statistical uncertainty. The primary experiments involve independently trained Math RL and Code RL policies alongside a separate MOPD pipeline. We further evaluated ROSS with different reviewer configurations, initialization checkpoints, and model scales. These experiments provide complementary evidence across training conditions, although they do not constitute a systematic multi-seed analysis under identical configurations.

In particular, we extended evaluation beyond the primary 35B model to 4B Math RL and 4B Code RL, as well as a production-stage MOPD consolidation pipeline covering 11 domains (detailed under W3). These experiments show that ROSS's effectiveness extends beyond a single model scale and task-specific configuration. Given the cost of repeating full upstream RL/MOPD pipelines, we prioritized broader independent settings and controlled ablations during the rebuttal period. This evidence of applicability is distinct from formal multi-seed uncertainty estimates.


W3. Generalization Across Model Scales and Training Settings

We agree that evaluating ROSS beyond the primary 35B model is important, so we provide additional results from 4B Math RL, 4B Code RL, and a broader production-stage MOPD training pipeline.

1. Results on 4B Math RL and Code RL

The revised manuscript includes an independent Qwen3.5-4B Math RL experiment in Appendix G. We have also completed a 4B Code RL experiment, applying ROSS to its corresponding post-RL checkpoint.

4B Math RL

BenchmarkRL UpstreamRL + ROSSΔ
AIME 202545.4260.94+15.52
AIME 202648.5468.33+19.79
HMMT-Nov.62.1961.67−0.52
Math Avg.52.0563.65+11.60

In this experiment, the 4B upstream model develops repetitive-answer behavior after RL. ROSS improves mathematical performance while suppressing this degeneration: the proportion of responses containing at least five boxed-answer markers decreases from 84.5% to 21.5%.

New 4B Code RL Results

BenchmarkCode RL UpstreamCode RL + ROSSΔ
LCB Generation47.4350.67+3.24
OJ overall17.4620.26+2.80
Code Avg.32.4535.47+3.02

Code Avg. is the unweighted mean of the two reported code benchmarks.

The newly completed Code RL experiment provides complementary evidence: ROSS improves LCB Generation by 3.24 percentage points and OJ overall by 2.80 points, with a 3.02-point gain in their average. The 4B Math RL experiment also corrects a pronounced repetitive-output failure mode, while Code RL improves on both benchmarks. Together, these results show that ROSS can improve post-RL checkpoints at the 4B scale across mathematical reasoning and code generation, complementing the primary 35B experiments. The new 4B Code RL result is reported here and will be consolidated with the existing 4B Math RL study in the supplementary material.

Reviewer_zahMOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether historical self-generated rollouts from earlier stages of language model post-training can provide useful supervision to later checkpoints. In this paper, the authors argue that successful behaviors discovered during reinforcement learning (RL) or on-policy distillation (OPD) may become less frequently expressed as training progresses, while successful trajectories can also contain intermediate mistakes, redundant reasoning, and unnecessary actions. To address this, the paper proposes ROSS (Relearning from Self-Generated Rollouts through Selective Supervision), an offline fine-tuning approach that first filters historical trajectories using outcome verification and then uses an LLM reviewer to identify informative segments. ROSS preserves the complete original trajectory as context but applies supervised fine-tuning loss only to selected model-generated tokens. Experiments on Qwen3.6-35B-A3B cover domain-specific mathematical and coding RL, multi-teacher on-policy distillation (MOPD), and agentic software engineering.

Critical strengths
  1. The paper addresses an interesting aspect of LLM post-training: historical self-generated trajectories may retain useful behaviors that are no longer reliably expressed by the latest policy. Reusing this experience is particularly relevant for RL and agentic training, where generating additional trajectories can be computationally expensive. The paper presents a practical approach for extracting additional value from already collected training data.
  2. The empirical results across multiple post-training settings are quite strong, alongside promising results on long-horizon agentic tasks.
  3. The ablations are quite useful for separating filtering from token-level masking.
  4. The proposed annotation pipeline explicitly distinguishes between generating new supervision and selecting existing model-generated content. The use of independent audit stages, source-grounded span selection, and deterministic token-mask compilation is a useful design choice. Preserving the complete original context while excluding undesirable targets also allows training on correction behaviors that depend on earlier trajectory states.
Critical weaknesses
  1. Although the historical-rollout setting is interesting, the main method combines successful-trajectory filtering, LLM-based segment selection, and masked SFT, all of which have substantial precedents.
  2. The unique benefit of historical rollouts is not adequately isolated. A central motivation is that historical trajectories preserve useful behaviors no longer reliably expressed by the current policy. However, the experiments do not directly compare historical data against equally curated rollouts generated by the current checkpoint.
  3. ROSS relies on GLM-5.2 in high-thinking mode to identify useful supervision targets and audit selected segments. This external model introduces additional supervision information into the training pipeline, even though it does not generate replacement responses, so there is a strong dependence on an external LLM reviewer without sufficient quality or cost analysis.
  4. The ROSS versus ROSS-without-masking ablation is useful, but the methods supervise different numbers of tokens. Consequently, part of the improvement could result from reducing the effective supervision volume or avoiding excessive imitation, rather than from selecting semantically informative segments. The experiments do not include random token masking with a matched exclusion rate, length-matched span selection, or simpler heuristics based on model confidence, token loss, or reasoning boundaries.
  5. Despite the relatively extensive evaluation, the paper does not provide sufficient uncertainty estimates or results across independent training seeds. The main experiments appear to rely on individual upstream runs followed by individual relearning configurations.
Questions for authors
  1. Can the authors compare ROSS trained on historical trajectories against ROSS trained on newly generated trajectories from the current checkpoint, keeping the number of examples, outcome filtering, and annotation procedure fixed?
  2. Can the authors compare the proposed masks against random masking with matched token exclusion rates, length-matched spans, and simpler confidence- or loss-based selection methods?
  3. How much computation is required to annotate the historical trajectory pools, including proposal, audit, and revision calls? How sensitive are the results to reviewer strength and annotation errors? Can the authors provide a small human-validated annotation sample or agreement analysis?
  4. Can the authors report SWE-bench Verified performance for ROSS without token masking, using exactly the same reviewer-accepted trajectories as the full agentic ROSS configuration?
9 comments
AuthorsOpenReview ↗

Author Response to Reviewer zahM (1/6): Contribution and Related Work

We thank the reviewer for recognizing the practical importance of reusing historical self-generated experience, the breadth of our empirical results, and the design of our source-grounded annotation pipeline. The review raises valuable questions about what historical experience contributes beyond existing selective-supervision techniques, the importance of reviewer decisions and semantic masks, and whether the observed improvements extend beyond individual training configurations. We address each point with controlled experiments, annotation analyses, and additional results from a production-stage training pipeline.

W1. The Contribution of ROSS Beyond Existing Selective-Supervision Techniques

We acknowledge that outcome filtering, LLM-based segment selection, and masked SFT have precedents. ROSS contributes a different post-training objective: recovering and consolidating useful self-generated experience that has accumulated during policy evolution. RL and on-policy distillation continuously create rollouts as part of upstream optimization. Once training advances, these trajectories are often discarded, although earlier checkpoints may have discovered behaviors that the final policy does not reliably reproduce.

Our preliminary analysis in Section 2 motivates this distinction. On 98 difficult problems with a successful historical solution, the final policy reaches only 37.7% mean pass@1, despite 99.0% pass@32. Thus, previously discovered solutions can remain available under repeated sampling without being consistently expressed. The historical rollout pool provides a way to reinforce these under-consolidated behaviors. ROSS makes this reuse viable by retaining the original context and selectively supervising suitable model-generated tokens, with independent audits and deterministic mask compilation.

This perspective also explains our relationship to related selective-supervision work. SWE-Prime primarily improves supervision extracted from successful agent trajectories, particularly for software engineering. ROSS focuses on the accumulated provenance and reuse of trajectories generated by earlier policies, with a unified offline stage evaluated after Math RL, Code RL, MOPD, and agentic RL. SWE-Prime is closely related concurrent work: ROSS began independently in May 2026, ahead of SWE-Prime's August 27 public release, and our ICLR submission followed in September.

The benefits are not restricted to further optimizing the checkpoint that generated the experience. Table 4 also compares relearning from identical historical trajectories and masks when initialized from Base and Upstream. The resulting scores differ by at most 0.58 points across the five reported measures, showing that the saved experience can benefit different initialization checkpoints. Together, these findings motivate ROSS as a complementary retrospective learning stage that reuses existing training by-products without requiring new policy-rollout generation or teacher-written replacement responses.

Revised manuscript: Sections 1–2 establish the motivation; Section 4.4 and Table 4 examine cross-initialization relearning; Section 5 discusses related work.

AuthorsOpenReview ↗

Author Response to Reviewer zahM (2/6): Historical vs. Final-Policy Rollouts

W2 and Q1. Historical Experience versus Newly Sampled Final-Policy Experience

We conducted a direct comparison between ROSS trained on accumulated historical rollouts and ROSS trained on newly generated rollouts from the final upstream checkpoint. For each of Math RL, Code RL, and MOPD, both sources use the same prompts, per-prompt sampling quotas, and total number of initial rollouts. We then apply the same outcome verifier, annotation procedure, and SFT recipe, starting from the same upstream checkpoint. The controlled quantity is the initial rollout-generation budget. Differences in verifier-positive yields arise naturally from the two rollout distributions and are retained in the end-to-end comparison.

SettingFinal checkpointHistorical windowInitial rollouts per groupHistorical verifier-positiveFinal-policy verifier-positive
Math RLiter 47r1–4849,15235,70035,484
Code RLiter 187r1–187191,488152,454115,000
MOPDiter 199First 200 steps51,20042,55836,877

Math RL

MethodAIME 2025AIME 2026HMMT-Nov.Math Avg.
Upstream74.8477.5073.2375.19
Final-policy rollouts + ROSS76.3578.3373.5476.07
Historical ROSS76.4679.9074.5876.98
Historical advantage (pp)+0.11+1.57+1.04+0.91

Code RL

MethodLCB GenerationOJBenchCode Avg.
Upstream58.9525.2242.09
Final-policy rollouts + ROSS58.5726.0842.32
Historical ROSS61.0527.3744.21
Historical advantage (pp)+2.48+1.29+1.89

MOPD

MethodAIME 25AIME 26HMMT-Nov.LCB GenOJBenchIFBenchAvg.
Upstream72.8177.6071.2557.4326.7244.5858.40
Final-policy rollouts + ROSS76.3578.8073.6558.0029.3145.8360.32
Historical ROSS77.7680.4276.7760.7629.7447.7662.20
Historical advantage (pp)+1.41+1.62+3.12+2.76+0.43+1.93+1.88

Historical ROSS performs better on every benchmark in these three comparisons, improving the averages by 0.91 points in Math RL, 1.89 in Code RL, and 1.88 in MOPD. The largest individual gains include +3.12 on MOPD HMMT-November and +2.76 on MOPD LCB Generation. These results directly substantiate the practical value of accumulated historical experience under a fixed initial generation budget. The historical data have already been generated during upstream training; the final-policy variant must produce new rollouts before running the same curation and SFT stages.

Revised manuscript: Section 4.2 and Table 1 present the main comparison; Appendix H and Table 12 document the rollout sources, generation budgets, and verification protocol.

AuthorsOpenReview ↗

Author Response to Reviewer zahM (3/6): Reviewer Sensitivity and Quality

W3 and Q3. External Reviewer Dependence, Annotation Reliability, and Computation

We agree that selection decisions made by the reviewer contribute supervision, even though all imitated text remains grounded in the original self-generated trajectory. We therefore evaluated reviewer strength, annotation agreement, supervision quality, and the cumulative cost of proposal, audit, revision, and retry calls.

1. Sensitivity to Reviewer Strength

The original reviewer is GLM-5.2 (approximately 753B total parameters) in high-thinking mode. We repeated the annotation and SFT pipeline with Qwen3.6-35B-A3B (approximately 35B total / 3B active parameters, thinking mode) and Qwen3.8-Flash-Next (approximately 176B total / 6B active parameters, xhigh mode).

Math RL

ReviewerAIME 2025AIME 2026HMMT-Nov.Math Avg.
Upstream74.8477.5073.2375.19
GLM-5.2 (original)76.4679.9074.5876.98
Qwen3.6-35B-A3B75.5279.9574.3876.61
Qwen3.8-Flash-Next77.7180.0573.8577.20

Code RL

ReviewerLCB GenerationOJBenchCode Avg.
Upstream58.9525.2242.09
GLM-5.2 (original)61.0527.3744.21
Qwen3.6-35B-A3B62.2925.6543.97
Qwen3.8-Flash-Next63.9027.1645.53

MOPD

ReviewerAIME 25AIME 26HMMT-Nov.LCB GenOJBenchIFBenchAvg.
Upstream72.8177.6071.2557.4326.7244.5858.40
GLM-5.2 (original)77.7680.4276.7760.7629.7447.7662.20
Qwen3.6-35B-A3B78.3380.9475.6359.5229.3146.6861.74
Qwen3.8-Flash-Next78.4480.9976.2562.8628.0252.3063.14

Both smaller reviewers improve upon their respective upstream checkpoints on all three aggregate evaluations. Qwen3.8-Flash-Next also exceeds the original GLM-5.2-reviewed ROSS in aggregate Math (77.20 vs. 76.98), Code (45.53 vs. 44.21), and MOPD (63.14 vs. 62.20). These results establish that effective ROSS selection is achievable with substantially smaller reviewer models.

This is consistent with the scope of the review task. ROSS asks the reviewer to inspect existing verifier-positive outputs, identify undesirable behavior, and select training targets without writing new solutions. As shown in Section 4.5 and Appendix E, corrected mistakes and redundant exploration together account for 92.0–95.0% of sampled masked Math trajectories and 89.3–92.2% of corresponding Code trajectories. The reviewer still performs substantive correctness and boundary checks, but these observations help explain why suitably capable smaller models can work well in this setting.

2. Decision Agreement and Annotation Quality

Across 225,597 trajectories receiving resolved keep/drop decisions from both GLM-5.2 and Qwen3.8-Flash-Next, the reviewers agree on 93.59% of cases, with Cohen's κ = 0.780. For a more detailed quality assessment, we sampled 200 distinct-prompt trajectories: 70 from Math RL, 70 from Code RL, and 60 from MOPD. All three reviewers produced resolved decisions for each trajectory, yielding 600 reviewer annotations for comparison. Kimi-K3.1-Pro assessed the anonymized annotations without reviewer identities or original rationales. We supplemented this assessment with manual spot checks and iterative refinement of the evaluation procedure.

ReviewerDecisions matching Kimi referenceKeep/drop agreementAcceptable supervision among retained trajectories
GLM-5.2186/20093.0%80.5%
Qwen3.6-35B-A3B178/20089.0%66.0%
Qwen3.8-Flash-Next194/20097.0%91.4%

For an additional comparison on identical retained data, we assessed the 153 trajectories retained by all three reviewers:

ReviewerAcceptable supervision on common retained set
GLM-5.283.7%
Qwen3.6-35B-A3B62.7%
Qwen3.8-Flash-Next91.5%

Qwen3.8-Flash-Next combines high agreement with the original reviewer and high supervision acceptability in the blinded assessment. On commonly retained trajectories, its acceptability exceeds GLM-5.2 by 7.8 percentage points. These results characterize both selection consistency and reviewer-dependent quality, beyond downstream scores alone.

W3/Q3 continues with annotation workload and cost in response (4/6).

Reviewer_24FjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ROSS, an offline supervised fine-tuning procedure that reuses historical self-generated trajectories collected during reinforcement learning or on-policy distillation. ROSS first retains trajectories that pass an outcome verifier, then uses an LLM reviewer to identify suitable imitation targets. The complete original trajectory remains available as context, while the training loss is applied only to selected assistant tokens. The method is evaluated on mathematics, code generation, instruction following, and software engineering.

Critical strengths
  1. The problem is practically and clearly motivated. Historical rollouts constitute a potentially valuable by-product of post-training, which motivates more selective reuse of these data.

  2. The scope of verification is broad, and the implementation details are quite comprehensive. It demonstrates improvements across independent Math RL and Code RL runs as well as MOPD.

Critical weaknesses
  1. The incremental contribution over existing selective-supervision methods remains unclear. SWE-Prime selects useful segments from successful trajectories while retaining full-trajectory context during SFT. STeP and SRFT are also acknowledged as using loss masking to avoid undesirable steps.

  2. The introduction and conclusion emphasize experience accumulated during policy evolution. However, the distinctive value of historical trajectories is not sufficiently isolated. Tables 1 and 3 establish that reviewer-curated historical data improve the policy, but they do not distinguish historical experience from equally curated successful data generated by the final policy. A comparison between historical rollouts and rollouts generated by the final checkpoint would strengthen the claim.

  3. The external reviewer is a substantial source of supervision whose reliability and cost are not characterized in the main paper. The main paper provides no measured annotation precision, agreement across reviewers, or sensitivity to reviewer choice.

Questions for authors
  1. Is earlier historical experience more useful than recent experience after identical curation?
  2. How accurate and expensive are the annotation decisions? What are the total annotation cost, average number of reviewer calls, revision rate, and rejection rate?
  3. Does the semantic mask outperform an equally sparse nonsemantic mask? In one existing setting, comparison against random contiguous-span masking with matched supervised-token counts would be better. This would determine whether the reviewer identifies unusually useful supervision rather than merely changing target density and weighting.
  4. How many agentic trajectories are used, and how does full-response supervision on the identical retained agentic dataset perform?
12 comments
AuthorsOpenReview ↗

Author Response to Reviewer 24Fj (1/7): Contribution and Related Work

We thank the reviewer for recognizing the motivation, broad empirical evaluation, and careful implementation of ROSS. We appreciate the constructive comments on its contribution relative to existing selective-supervision methods, the value of historical rollouts, reviewer reliability and cost, and the role of semantic supervision selection. We address these points with additional experiments and analyses.

W1. Distinction from Existing Selective-Supervision Methods

We thank the reviewer for highlighting the connections between ROSS and SWE-Prime, STeP, and SRFT. These works share methodological elements with ROSS, including selective supervision and loss masking.

SWE-Prime is closely related concurrent work. ROSS was independently initiated in May 2026, before SWE-Prime first appeared on arXiv on August 27, 2026. We submitted our ICLR 2027 abstract in mid-September and the full manuscript on September 26 (Beijing time; September 25 AoE). These timelines reflect substantially overlapping research periods.

More importantly, ROSS originates from a distinct research motivation and addresses a different problem in LLM post-training. SWE-Prime focuses on improving the quality of SFT supervision from successful agent trajectories by selecting valuable trajectories and segments, primarily in software-engineering tasks. In contrast, ROSS is motivated by the opportunity to reuse historical self-generated rollouts accumulated throughout policy evolution.

Reinforcement learning (RL) and on-policy distillation (OPD) naturally produce large amounts of rollout data as by-products of training. As the policy evolves, these historical trajectories are often considered stale and discarded, even though they may preserve useful behaviors discovered by earlier policies that the final policy does not reliably express. Our central research question is whether and how previously accumulated experience can be recovered and consolidated to further improve the final policy. Selective supervision serves as the enabling mechanism for this objective: it allows the model to learn from valuable historical behaviors without indiscriminately imitating errors or redundant exploration.

This motivation leads to two important characteristics of ROSS:

  1. Broad applicability across post-training paradigms. ROSS provides a unified retrospective relearning framework applicable to RL, multi-teacher on-policy distillation (MOPD), and agentic RL. We demonstrate consistent aggregate improvements across mathematics, code generation, instruction following, and software engineering.

  2. Practical reuse of existing training by-products. ROSS can be applied as an optional offline SFT stage after upstream training, directly leveraging already collected rollouts without requiring additional policy rollout generation or teacher-generated replacement responses. It thereby converts previously accumulated, potentially discarded experience into additional training signal.

The central contribution of ROSS is a general retrospective relearning framework for recovering and consolidating useful self-generated experience accumulated during LLM post-training.

Revised manuscript: The research motivation is presented in Section 1, and the distinctions from related selective-supervision methods are articulated in Section 5 (Related Work).

AuthorsOpenReview ↗

Author Response to Reviewer 24Fj (2/7): Historical vs. Final-Policy Rollouts

W2 and Q1. Historical Rollouts versus Final-Checkpoint Rollouts

The reviewer asks whether historical rollouts provide additional value compared with newly generated rollouts from the final policy.

To answer this question, we conducted additional controlled experiments in Math RL, Code RL, and MOPD, comparing ROSS trained on historical trajectories with ROSS trained on trajectories newly generated by the final upstream checkpoint.

1. Experimental Setup: Matched Initial Rollout Budgets

For each setting, we generated new rollouts from the final upstream checkpoint using the same prompt set and the same per-prompt sampling quota as the corresponding historical rollout collection. Both groups therefore contain exactly the same number of initial rollouts before outcome verification. We then applied the same outcome-verification and ROSS annotation procedures to construct the training datasets, followed by SFT initialized from the same upstream checkpoint. This setup directly compares the learning value of historical experience accumulated during policy evolution with that of recent experience generated by the final policy. The data statistics are summarized below.

SettingFinal checkpointHistorical rollout windowInitial rollouts per groupHistorical verifier-positiveFinal-policy verifier-positive
Math RLiter 47r1–4849,15235,70035,484
Code RLiter 187r1–187191,488152,454115,000
MOPDiter 199First 200 steps51,20042,55836,877

2. Detailed Experimental Results

Math RL: Mathematical Reasoning

MethodAIME 2025AIME 2026HMMT-Nov.Math Avg.
Upstream74.8477.5073.2375.19
Final-checkpoint rollouts + ROSS76.3578.3373.5476.07
Historical ROSS76.4679.9074.5876.98
Historical advantage (pp)+0.11+1.57+1.04+0.91

Code RL: Code Generation

MethodLCB GenerationOJBenchCode Avg.
Upstream58.9525.2242.09
Final-checkpoint rollouts + ROSS58.5726.0842.32
Historical ROSS61.0527.3744.21
Historical advantage (pp)+2.48+1.29+1.89

MOPD: Cross-Domain Performance

MethodAIME 25AIME 26HMMT-Nov.LCB GenOJBenchIFBenchAvg.
Upstream72.8177.6071.2557.4326.7244.5858.40
Final-checkpoint rollouts + ROSS76.3578.8073.6558.0029.3145.8360.32
Historical ROSS77.7680.4276.7760.7629.7447.7662.20
Historical advantage (pp)+1.41+1.62+3.12+2.76+0.43+1.93+1.88

3. Analysis

With the same prompt sets and initial rollout sampling budgets, Historical ROSS consistently outperforms ROSS trained on newly generated final-checkpoint rollouts across all evaluated benchmarks in all three settings.

Specifically:

  • Math RL: Historical rollouts improve the average score by 0.91 percentage points.
  • Code RL: Historical rollouts improve the average score by 1.89 percentage points.
  • MOPD: Historical rollouts improve the six-benchmark average by 1.88 percentage points, with particularly notable gains of 3.12 points on HMMT-November and 2.76 points on LCB Generation.

These findings support the central motivation of ROSS:

Historical trajectories accumulated during policy evolution are not merely additional training examples; they can preserve useful behaviors that remain insufficiently consolidated in the final policy.

By selectively relearning from this accumulated experience, ROSS achieves better downstream performance than the evaluated alternative based on newly sampled final-policy trajectories. Moreover, historical rollouts are naturally generated during upstream RL or OPD training. Unlike the final-checkpoint resampling approach, ROSS directly reuses these existing trajectories without requiring additional policy rollout generation. Together, these results highlight both the effectiveness and practical value of historical experience reuse in LLM post-training.

Revised manuscript: Section 4.2 and Table 1 report the comparison; Appendix H (including Table 12) details the rollout sources, matched generation budgets, and evaluation protocol.

AuthorsOpenReview ↗

Author Response to Reviewer 24Fj (3/7): Reviewer Sensitivity and Agreement

W3 and Q2. Reviewer Reliability, Model Sensitivity, and Annotation Cost

We evaluate reviewer sensitivity, annotation quality and agreement, and the complete annotation workload and API-equivalent cost of ROSS supervision.

1. Sensitivity to Reviewer Model Size and Capability

The original ROSS experiments use GLM-5.2, a large mixture-of-experts model with approximately 753B total parameters, operating in high-thinking mode. To investigate whether ROSS requires such a large reviewer, we additionally evaluated two substantially smaller models:

  • Qwen3.6-35B-A3B: approximately 35B total parameters, with 3B activated per token, using thinking mode.
  • Qwen3.8-Flash-Next: approximately 125B main-model parameters plus 51B embedding parameters (approximately 176B in total), with 6B activated per token, using xhigh mode.

We repeated the ROSS annotation and downstream SFT procedures using these alternative reviewers and compared the resulting performance against the original GLM-5.2-reviewed ROSS.

Math RL Results

ReviewerAIME 2025AIME 2026HMMT-Nov.Math Avg.
Upstream74.8477.5073.2375.19
GLM-5.2 (original)76.4679.9074.5876.98
Qwen3.6-35B-A3B75.5279.9574.3876.61
Qwen3.8-Flash-Next77.7180.0573.8577.20

Code RL Results

ReviewerLCB GenerationOJBenchCode Avg.
Upstream58.9525.2242.09
GLM-5.2 (original)61.0527.3744.21
Qwen3.6-35B-A3B62.2925.6543.97
Qwen3.8-Flash-Next63.9027.1645.53

MOPD Results

ReviewerAIME 25AIME 26HMMT-Nov.LCB GenOJBenchIFBenchAvg.
Upstream72.8177.6071.2557.4326.7244.5858.40
GLM-5.2 (original)77.7680.4276.7760.7629.7447.7662.20
Qwen3.6-35B-A3B78.3380.9475.6359.5229.3146.6861.74
Qwen3.8-Flash-Next78.4480.9976.2562.8628.0252.3063.14

Both smaller reviewers improve over their corresponding upstream checkpoints in terms of aggregate Math RL, Code RL, and MOPD performance. Qwen3.8-Flash-Next performs particularly well. Compared with the original GLM-5.2 reviewer, it achieves higher aggregate scores across all three settings:

  • Math RL: 77.20 versus 76.98.
  • Code RL: 45.53 versus 44.21.
  • MOPD: 63.14 versus 62.20.

These results show that ROSS can achieve effective supervision selection with reviewers substantially smaller than GLM-5.2. Qwen3.8-Flash-Next delivers competitive or better downstream performance across the three settings.

2. Why Smaller Reviewers Can Work Effectively

The reviewer-sensitivity results are consistent with the focused nature of ROSS annotation. The reviewer evaluates an existing verifier-positive trajectory and selects supervision targets from the original response. It identifies unsuitable behaviors and preserves useful segments without generating replacement solutions.

Section 4.5 and Appendix E (Figure 6) of the revised manuscript characterize the dominant patterns in excluded supervision. Two categories, corrected mistakes and redundant exploration, account for the vast majority of the sampled excluded-content patterns:

  • Math: 92.0% and 95.0% of sampled masked trajectories in the early and late training windows, respectively.
  • Code: 89.3% and 92.2% in the corresponding windows.

These distributions show that the sampled excluded regions predominantly contain corrected mistakes and exploratory detours within otherwise successful trajectories. Reviewers must still assess local correctness, reasoning completeness, and precise span boundaries, but their task centers on evaluating and selecting existing text. Together with the reviewer-sensitivity results, this pattern is consistent with the strong performance of smaller models such as Qwen3.8-Flash-Next on ROSS annotation.

3. Inter-Reviewer Agreement and Annotation Quality

We further evaluated annotation reliability through large-scale inter-reviewer agreement and an independent quality assessment.

Inter-reviewer agreement.

Among 225,597 trajectories for which both GLM-5.2 and Qwen3.8-Flash-Next produced definitive keep/drop decisions, the two reviewers achieved:

  • 93.59% trajectory-level decision agreement.
  • Cohen's κ = 0.780.

This indicates substantial agreement between the original large reviewer and the smaller Qwen3.8-Flash-Next model.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author Response and Manuscript Revisions

Dear Area Chair and Reviewers,

Thank you for your time and thoughtful feedback throughout the review process. We sincerely appreciate the constructive comments from all three reviewers, which have helped us strengthen our paper.

During the discussion, the reviewers raised several overlapping concerns regarding the value of historical rollouts, the effectiveness of selective supervision, reviewer dependence, and the generality and efficiency of ROSS, along with other questions specific to individual reviewers. We conducted additional experiments and analyses to address these concerns, with the main findings summarized below:

  • Value of historical experience: Under matched initial rollout budgets, ROSS trained on historical rollouts consistently outperforms ROSS trained on newly generated final-policy rollouts across Math RL, Code RL, and MOPD.
  • Effectiveness of selective supervision: Comparisons with token-budget-matched random masking and full supervision on identical agentic trajectories show that selecting informative supervision targets provides benefits beyond simply filtering successful trajectories or reducing supervised tokens.
  • Reviewer robustness and cost: ROSS achieves competitive or better results with smaller reviewer models. Under the specified API pricing, Qwen3.8-Flash-Next also reduces the estimated API-equivalent annotation cost by 77.5% relative to GLM-5.2.
  • Broader applicability: Beyond the main 35B experiments, we also evaluated ROSS on a 4B model in both Math RL and Code RL, observing improvements in both settings. We further applied ROSS to a real-world, large-scale MOPD training pipeline spanning 11 domains. In this setting, ROSS improves 14 of 17 evaluation configurations, including all seven agentic evaluations, demonstrating its practical feasibility in large-scale LLM post-training.

We have incorporated the main experimental findings and corresponding details into the revised manuscript, including the historical rollout comparisons (Section 4.2 and Appendix H), selective supervision ablations (Sections 4.3–4.4 and Appendix I), reviewer robustness and cost analysis (Section 4.6 and Appendix J), and production-stage evaluation (Appendix K). Additional 4B Code RL and trajectory diversity results are provided in our discussion responses.

More broadly, we view ROSS as an exploration of a complementary direction to the commonly used SFT-then-RL post-training pipeline, where selective SFT helps consolidate useful historical experience accumulated across different post-training settings, including RL, on-policy distillation, and agentic RL.

We are grateful that the reviewers indicated our responses and additional experiments had largely addressed their main concerns, and that all three concluded the discussion with positive ratings (3). We sincerely appreciate their careful consideration of our additional evidence and their constructive engagement throughout the process.

Thank you again for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.