- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies a warmup stage for agentic RL in which a smaller model is supervised by a larger teacher on trajectories it generates itself, and is then trained with verifiable rewards after the teacher is removed. The authors report that this warmup reaches good performance sooner than off-policy fine-tuning, rejection-sampled fine-tuning and teacher-free warmup, and give a theoretical argument that connects on-policy distillation to the student's success rate at the start of RL. Experiments are on ALFWorld and ScienceWorld with Qwen3 models, with additional analyses of invalid actions, teacher preference retention and training cost.
Critical strengths
-
The pipeline is described clearly enough to follow: the teacher is used only during warmup and is removed before RL, so a reader can see exactly what is being compared.
-
The reporting is candid. The paper shows task categories where the method is worse and cost settings where it is more expensive, which is not common and made me trust the rest of the tables more.
Critical weaknesses
-
The warmup variants differ in several things at once: where the trajectories come from, how much supervision they receive, and how many updates they get before RL starts. I could not tell which of these produces the gain, so the conclusion that supervision on the student's own states is the key factor does not clearly follow.
-
The main results look like they come from a single run, and I did not find any measure of variation across runs. The theory also seems disconnected from the experiments: I could not find where the quantity it is about is measured.
Questions for authors
-
Among the differences between the warmup variants, which one do you consider the causal factor, and is there a comparison that isolates it?
-
Do the main results come from a single training run, and does the ordering hold across seeds?
-
Since most groups eventually contain only successes, at what stage of training does the reported advantage actually arise?
4 comments
Response to Reviewer mxr5 (Part 2/3)
[ & ] Does the ordering hold across training seeds?
W2. The main results look like they come from a single run, and I did not find any measure of variation across runs. The theory also seems disconnected from the experiments: I could not find where the quantity it is about is measured.
Q2. Do the main results come from a single training run, and does the ordering hold across seeds?
We added three independent training seeds per method in each environment. The original results used one run per configuration. OPW-CE has the highest mean final Unseen performance among the four methods in both environments.
(1) The ordering across seeds depends on the baseline. OPW exceeds Joint in two of three ALFWorld runs and one of three ScienceWorld runs, and exceeds tuned SFT-RS and GRPO warmup + GRPO in all three runs in both environments.
The mean paired differences are +1.06 ± 1.77 percentage points on ALFWorld and +0.26 ± 1.81 task-score points on ScienceWorld. Their nominal paired 95% t intervals are [−3.35, 5.46] and [−4.23, 4.75], respectively. Both include zero.
Table R2. Final Qwen3-4B Unseen performance over the same three additional training seeds (mean ± sample SD).
| Unseen metric | OPW-CE (Ours) | Joint OPD+RL | Tuned SFT-RS | GRPO warmup + GRPO | OPW-CE − Joint |
|---|---|---|---|---|---|
| ALFWorld success (%) | 95.21 ± 1.04 | 94.15 ± 1.16 | 91.14 ± 1.78 | 76.49 ± 1.48 | +1.06 pp |
| ScienceWorld task score | 71.99 ± 1.65 | 71.73 ± 1.92 | 70.44 ± 1.84 | 59.60 ± 2.00 | +0.26 |
Appendix C.5 gives the training settings and individual-run results.
(2) Direct rollouts show initial-success gains at the measured KL values. Mean cumulative reverse KL is 192.28–466.99 nats per trajectory across the two sizes and environments (numerical plug-in lower bound of 0, Appendix C.7). Theorem 3.2 gives sufficient conditions for initial reward discovery. On the same diagnostic tasks, the observed 4B initial-success gains over Base are 17.19 percentage points on ALFWorld and 33.59 on ScienceWorld.
Thank you for requesting these checks. The additional runs strengthen the empirical comparison and let readers assess both mean performance and variation across runs.
Response to Reviewer mxr5 (Part 3/3)
[] When does OPW overtake GRPO warmup?
Q3. Since most groups eventually contain only successes, at what stage of training does the reported advantage actually arise?
On ALFWorld 4B Seen, OPW overtakes GRPO warmup within the first 20 GRPO steps, despite starting lower (Table R3).
Table R3. ALFWorld 4B Seen success during GRPO after warmup.
| GRPO steps after warmup | OPW initialization (%) | GRPO warmup initialization (%) |
|---|---|---|
| 0 | 46.79 | 55.98 |
| 20 | 88.57 | 67.41 |
| 60 | 96.07 | 82.68 |
(1) The early-learning advantage persists when GRPO retains Adam. OPW starts lower but gains 41.8 versus 3.21 percentage points in the first 20 steps and finishes at 96.07% versus 92.50%.
(2) The lead is already visible by GRPO step 20. Over GRPO steps 1–20 after OPW, the all-success and mixed-group shares among eight-rollout groups average 44.7% and 46.9%, respectively. The all-success share averages 72.8% over steps 21–40. Under binary rewards, all-success groups have zero within-group reward contrast. Thank you for asking when the benefit arises. These measurements make the early-learning benefit of OPW clearer.
Follow-Up on Review and Feedback
Dear Reviewer mxr5,
Thank you for your positive recommendation and for recognizing our clear training procedure and candid reporting. On ALFWorld Unseen, average success during GRPO was higher with trajectories from the current student than with teacher trajectories. Across three additional training seeds per method, OPW-CE had the highest mean final Unseen performance on both benchmarks.
We would be glad to hear your thoughts after reading our rebuttal. Please feel free to share any remaining questions or concerns.
Best regards,
The Authors
Response to Reviewer mxr5 (Part 1/3)
Dear Reviewer mxr5,
Thank you for noting that “The pipeline is described clearly enough to follow” and that “The reporting is candid.” We appreciate your recognition of the clear separation between teacher-guided warmup and teacher-free RL, and of our reporting of task and cost differences across methods. Your questions helped us strengthen the evidence for the role of student-generated trajectories through comparisons at equal supervision and update counts. Below, we present those controls, additional training runs, measured theoretical quantities, and evidence on when the learning advantage emerges.
[W1 & Q1] Does the source of warmup trajectories affect subsequent learning?
Current-student trajectories improve average GRPO success under both CE and reverse KL on ALFWorld 4B Unseen.
(1) The average-success gain persists at equal supervision and update counts. The single-run controls hold loss normalization, supervised-token count, and optimizer updates fixed within each objective, with 40 warmup and 60 GRPO steps (Appendix D.4). Average success uses the trapezoidal mean over GRPO steps 0–60, including step 0.
Table R1. Effect of trajectory source at equal supervised-token and optimizer-update counts.
(2) Current-student trajectories improve average success under both objectives. The gains over teacher trajectories are about 3.5 percentage points under CE (95% paired-task interval [1.23, 5.82]) and 2.10 under reverse KL [0.06, 4.09]. Resampling also improves the CE average by 3.90 points [1.51, 6.36] over fixed initial-student trajectories.
The average-success and paired-task analyses were added after reviewing the six conditions' aggregate results. The intervals describe task variation for fixed models. Final-success ordering varies.
Thank you for suggesting these controls. They help isolate trajectory source and resampling and strengthen the evidence for their effect on average GRPO performance.