acceptodds
Under review as a conference paper at ICLR 2027

From Imitation to Reward Discovery: On-Policy Warmup for Agentic RL

Abstract

Reinforcement learning with a verifiable reward (RLVR) offers a scalable approach to training language-model agents, yet sparse outcome rewards can leave early training with little signal for policy improvement. We identify an **On-Policy Acceleration Phenomenon**: in our main comparisons, RLVR initialized with on-policy distillation reaches high performance earlier in training and achieves both higher average performance during subsequent RLVR and higher final performance than the alternative baselines. Motivated by this observation, we study **On-Policy Warmup (OPW)**, a teacher-guided stage in which the student trains with teacher supervision on its own interaction trajectories before transitioning to RLVR. Unlike imitation on fixed teacher-generated trajectories, OPW targets states induced by the student’s own decisions, including imperfect actions and recovery situations. We provide a theoretical explanation by connecting on-policy reverse-KL distillation to trajectory-level distribution matching. Under a competent teacher and sufficiently small population distillation loss, this connection yields a lower bound on initial verifier success and a corresponding bound on reward-discovery complexity. For group-relative RLVR, we further characterize when increased success probability produces more reward-informative groups. Together, our findings support on-policy distillation as an effective warmup for agentic RLVR and identify initial reward discovery as a mechanism that can contribute to the observed acceleration.

Primary area:
reinforcement learning
Keywords:
LLM agents, on-policy distillation, reinforcement learning with verifiable rewards, warm-up training, post-training, learning dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_mxr5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a warmup stage for agentic RL in which a smaller model is supervised by a larger teacher on trajectories it generates itself, and is then trained with verifiable rewards after the teacher is removed. The authors report that this warmup reaches good performance sooner than off-policy fine-tuning, rejection-sampled fine-tuning and teacher-free warmup, and give a theoretical argument that connects on-policy distillation to the student's success rate at the start of RL. Experiments are on ALFWorld and ScienceWorld with Qwen3 models, with additional analyses of invalid actions, teacher preference retention and training cost.

Critical strengths
  1. The pipeline is described clearly enough to follow: the teacher is used only during warmup and is removed before RL, so a reader can see exactly what is being compared.

  2. The reporting is candid. The paper shows task categories where the method is worse and cost settings where it is more expensive, which is not common and made me trust the rest of the tables more.

Critical weaknesses
  1. The warmup variants differ in several things at once: where the trajectories come from, how much supervision they receive, and how many updates they get before RL starts. I could not tell which of these produces the gain, so the conclusion that supervision on the student's own states is the key factor does not clearly follow.

  2. The main results look like they come from a single run, and I did not find any measure of variation across runs. The theory also seems disconnected from the experiments: I could not find where the quantity it is about is measured.

Questions for authors
  1. Among the differences between the warmup variants, which one do you consider the causal factor, and is there a comparison that isolates it?

  2. Do the main results come from a single training run, and does the ordering hold across seeds?

  3. Since most groups eventually contain only successes, at what stage of training does the reported advantage actually arise?

4 comments
AuthorsOpenReview ↗

Response to Reviewer mxr5 (Part 1/3)

Dear Reviewer mxr5,

Thank you for noting that “The pipeline is described clearly enough to follow” and that “The reporting is candid.” We appreciate your recognition of the clear separation between teacher-guided warmup and teacher-free RL, and of our reporting of task and cost differences across methods. Your questions helped us strengthen the evidence for the role of student-generated trajectories through comparisons at equal supervision and update counts. Below, we present those controls, additional training runs, measured theoretical quantities, and evidence on when the learning advantage emerges.

[W1\textcolor{#B91C1C}{\textbf{W1}} & Q1\textcolor{#B45309}{\textbf{Q1}}] Does the source of warmup trajectories affect subsequent learning?

W1. The warmup variants differ in several things at once: where the trajectories come from, how much supervision they receive, and how many updates they get before RL starts. ... the conclusion that supervision on the student's own states is the key factor does not clearly follow.

Q1. Among the differences between the warmup variants, which one do you consider the causal factor, and is there a comparison that isolates it?

Current-student trajectories improve average GRPO success under both CE and reverse KL on ALFWorld 4B Unseen.

(1) The average-success gain persists at equal supervision and update counts. The single-run controls hold loss normalization, supervised-token count, and optimizer updates fixed within each objective, with 40 warmup and 60 GRPO steps (Appendix D.4). Average success uses the trapezoidal mean over GRPO steps 0–60, including step 0.

Table R1. Effect of trajectory source at equal supervised-token and optimizer-update counts.

Trajectory sourceSupervision objectiveAverage Unseen success during GRPO (%)Final success (%)
TeacherTop-1 cross-entropy80.8595.52
Current studentTop-1 cross-entropy84.3495.24
TeacherFull-vocabulary reverse KL81.3994.68
Current studentFull-vocabulary reverse KL83.4995.71
Fixed teacher trajectoriesTop-1 cross-entropy81.5194.22
Fixed initial-student trajectoriesTop-1 cross-entropy80.4495.43

(2) Current-student trajectories improve average success under both objectives. The gains over teacher trajectories are about 3.5 percentage points under CE (95% paired-task interval [1.23, 5.82]) and 2.10 under reverse KL [0.06, 4.09]. Resampling also improves the CE average by 3.90 points [1.51, 6.36] over fixed initial-student trajectories.

The average-success and paired-task analyses were added after reviewing the six conditions' aggregate results. The intervals describe task variation for fixed models. Final-success ordering varies.

Thank you for suggesting these controls. They help isolate trajectory source and resampling and strengthen the evidence for their effect on average GRPO performance.

AuthorsOpenReview ↗

Response to Reviewer mxr5 (Part 2/3)

[W2\textcolor{#B91C1C}{\textbf{W2}} & Q2\textcolor{#B45309}{\textbf{Q2}}] Does the ordering hold across training seeds?

W2. The main results look like they come from a single run, and I did not find any measure of variation across runs. The theory also seems disconnected from the experiments: I could not find where the quantity it is about is measured.

Q2. Do the main results come from a single training run, and does the ordering hold across seeds?

We added three independent training seeds per method in each environment. The original results used one run per configuration. OPW-CE has the highest mean final Unseen performance among the four methods in both environments.

(1) The ordering across seeds depends on the baseline. OPW exceeds Joint in two of three ALFWorld runs and one of three ScienceWorld runs, and exceeds tuned SFT-RS and GRPO warmup + GRPO in all three runs in both environments.

The mean paired differences are +1.06 ± 1.77 percentage points on ALFWorld and +0.26 ± 1.81 task-score points on ScienceWorld. Their nominal paired 95% t intervals are [−3.35, 5.46] and [−4.23, 4.75], respectively. Both include zero.

Table R2. Final Qwen3-4B Unseen performance over the same three additional training seeds (mean ± sample SD).

Unseen metricOPW-CE (Ours)Joint OPD+RLTuned SFT-RSGRPO warmup + GRPOOPW-CE − Joint
ALFWorld success (%)95.21 ± 1.0494.15 ± 1.1691.14 ± 1.7876.49 ± 1.48+1.06 pp
ScienceWorld task score71.99 ± 1.6571.73 ± 1.9270.44 ± 1.8459.60 ± 2.00+0.26

Appendix C.5 gives the training settings and individual-run results.

(2) Direct rollouts show initial-success gains at the measured KL values. Mean cumulative reverse KL is 192.28–466.99 nats per trajectory across the two sizes and environments (numerical plug-in lower bound of 0, Appendix C.7). Theorem 3.2 gives sufficient conditions for initial reward discovery. On the same diagnostic tasks, the observed 4B initial-success gains over Base are 17.19 percentage points on ALFWorld and 33.59 on ScienceWorld.

Thank you for requesting these checks. The additional runs strengthen the empirical comparison and let readers assess both mean performance and variation across runs.

AuthorsOpenReview ↗

Response to Reviewer mxr5 (Part 3/3)

[Q3\textcolor{#B45309}{\textbf{Q3}}] When does OPW overtake GRPO warmup?

Q3. Since most groups eventually contain only successes, at what stage of training does the reported advantage actually arise?

On ALFWorld 4B Seen, OPW overtakes GRPO warmup within the first 20 GRPO steps, despite starting lower (Table R3).

Table R3. ALFWorld 4B Seen success during GRPO after warmup.

GRPO steps after warmupOPW initialization (%)GRPO warmup initialization (%)
046.7955.98
2088.5767.41
6096.0782.68

(1) The early-learning advantage persists when GRPO retains Adam. OPW starts lower but gains 41.8 versus 3.21 percentage points in the first 20 steps and finishes at 96.07% versus 92.50%.

(2) The lead is already visible by GRPO step 20. Over GRPO steps 1–20 after OPW, the all-success and mixed-group shares among eight-rollout groups average 44.7% and 46.9%, respectively. The all-success share averages 72.8% over steps 21–40. Under binary rewards, all-success groups have zero within-group reward contrast. Thank you for asking when the benefit arises. These measurements make the early-learning benefit of OPW clearer.

AuthorsOpenReview ↗

Follow-Up on Review and Feedback

Dear Reviewer mxr5,

Thank you for your positive recommendation and for recognizing our clear training procedure and candid reporting. On ALFWorld Unseen, average success during GRPO was higher with trajectories from the current student than with teacher trajectories. Across three additional training seeds per method, OPW-CE had the highest mean final Unseen performance on both benchmarks.

We would be glad to hear your thoughts after reading our rebuttal. Please feel free to share any remaining questions or concerns.

Best regards,
The Authors

Reviewer_uq62OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to warm up a LLM agent before grpo in multi-turn environments, where sparse binary rewards often produce all-failure groups with zero advantage early in training. They propose On-Policy Warmup (OPW), which runs a short stage of on-policy distillation: the student generates trajectories in the environment, a fixed teacher supplies top-1 token targets at student-visited histories, and after 40 steps training switches to teacher-free grpo. The theoretical part observes that the summed per-step reverse KL on student states equals the trajectory-level KL (Prop. 3.1), and via data processing and Pinsker derives a lower bound on student verifier success, a rollout bound for discovering a first success, and a monotonicity statement for the probability of mixed-reward grpo groups when p is at most 1/2. Experiments on ALFWorld and ScienceWorld compare OPW+grpo against direct grpo, direct opd, and sft, sft with rejection sampling, and grpo warmups of equal length. OPW gives the best final and average performance in most settings (e.g., 96.7% vs. 86.6% on ALFWorld Unseen for 4B) and much higher pass@8. Additional analyses cover training cost, repeated invalid actions, supervision removal, resampling vs. fixed trajectories, retention of teacher preferences, and warmup duration.

Critical strengths

The improvement is obvious. Across two environments, two student sizes, and both splits, OPW followed by grpo gives the highest average performance during grpo.

The analyses are comprehensive. The paper tries to identify where OPW's advantage comes from: the drop in repeated invalid actions and the targeted removal of supervision at those positions compared with random removal, the one-step teacher-target probability change on all-failure groups , the fixed vs. resampled trajectory ablation, and continuation tests at common histories. Not all of these are conclusive, but together they make the mechanism more concrete than a learning curve alone would. The observation that OPW can start lower than grpo warmup yet learn faster is a useful caution against using warmup accuracy as a proxy.

Critical weaknesses
  1. Although this may sounds AI, I still have to mention the seeding problem: in most main tables the runs come from a single seed, the only two-seed result shows the early-gain advantage shrinking from +30.4 to +17.2, and the warmup-duration study uses a different seed for each condition.

  2. The SFT baselines may be under-trained (constant lr 1e-6, one pass over the data, 160 vs. 200 optimizer steps). They barely improve over the base model after warmup, so the gap may come from training budget rather than on-policy supervision.

  3. The theory is fairly standard and does not match the method. It analyzes reverse KL, while training uses top-1 cross-entropy, and the bound is likely vacuous for responses of this length.

  4. Novelty is limited relative to concurrent work such as OPDSearch+ and Sequential Beats Joint, and baselines such as GKD, Guided-OPD, LUFFY and GiGPO are missing.

Questions for authors
  1. How many independent training seeds underlie Tables 2 to 5? Can you report mean and standard deviation over at least three seeds for the main ALFWorld and ScienceWorld comparisons?

  2. Could you tune the SFT and SFT-RS warmups (learning rate, epochs, matched optimizer steps) and report their accuracy right after warmup? If OPW's advantage persists against a properly fitted SFT model, the main claim becomes much stronger.

  3. What is the measured per-trajectory reverse KL between student and teacher after OPW, and does the resulting lower bound l exceed the Base success rate on any task? If not, how should readers interpret the theory?

  4. How does top-1 cross-entropy compare with full-distribution reverse KL, which is the objective the theory analyzes?

  5. Why does direct OPD for 100 steps (53.5% ALFWorld Unseen) stay so far below both the RL-trained students and the warmup stage's own trajectory? Does this indicate the teacher is the bottleneck?

7 comments
AuthorsOpenReview ↗

Response to Reviewer uq62 (Part 1/4)

Dear Reviewer uq62,

Thank you for recognizing the empirical improvements and noting that “The analyses are comprehensive.” We appreciate your attention to the supervision-removal, resampling, and common-history analyses, and your description of the lower-starting, faster-learning observation as “a useful caution against using warmup accuracy as a proxy.” Together, these observations motivate our focus on how warmup affects later learning. Below, we address training variability and duration, stronger SFT baselines, the theoretical connection, and comparisons with related OPD-to-RL methods.

[W1\textcolor{#B91C1C}{\textbf{W1}} & Q1\textcolor{#B45309}{\textbf{Q1}}] Training repetitions and warmup duration

W1. Although this may sounds AI, I still have to mention the seeding problem: in most main tables the runs come from a single seed, the only two-seed result shows the early-gain advantage shrinking from +30.4 to +17.2, and the warmup-duration study uses a different seed for each condition.

Q1. How many independent training seeds underlie Tables 2 to 5? Can you report mean and standard deviation over at least three seeds for the main ALFWorld and ScienceWorld comparisons?

We added three independent seeds per method in each environment and a fixed-seed warmup-duration comparison. Tables 2–5 used one training run per configuration.

Table R1. Final Qwen3-4B Unseen performance over three additional seeds (mean ± SD).

Unseen metricOPW-CE (Ours)Joint OPD+RLTuned SFT-RSGRPO warmup + GRPOOPW-CE − Joint
ALFWorld success (%)95.21 ± 1.0494.15 ± 1.1691.14 ± 1.7876.49 ± 1.48+1.06 pp
ScienceWorld task score71.99 ± 1.6571.73 ± 1.9270.44 ± 1.8459.60 ± 2.00+0.26

(1) OPW-CE has the highest sample mean final Unseen performance among the four methods (Table R1). Appendix C.5 gives the training settings and individual-run results.

(2) Fixing the seed separates warmup-duration effects from changes in seed identity. The Qwen3-4B runs fix training seed 1 and allocate the same 100 steps as 20 + 80, 40 + 60, 60 + 40, or 80 + 20 OPW + GRPO steps, using the original normalization (Appendix G.2). On ScienceWorld Unseen, 20 + 80 scores 73.83 versus 68.23 for 40 + 60, while 60 + 40 achieves the highest repeated task success. The preferred allocation therefore depends on the evaluation metric.

Thank you for drawing attention to training variability and warmup duration. The added runs strengthen the main comparison, while the fixed-seed study makes the effect of duration easier to assess.

AuthorsOpenReview ↗

Response to Reviewer uq62 (Part 2/4)

[W2\textcolor{#B91C1C}{\textbf{W2}} & Q2\textcolor{#B45309}{\textbf{Q2}}] How well does OPW compare with tuned SFT?

W2. The SFT baselines may be under-trained (constant lr 1e-6, one pass over the data, 160 vs. 200 optimizer steps). They barely improve over the base model after warmup, so the gap may come from training budget rather than on-policy supervision.

Q2. Could you tune the SFT and SFT-RS warmups (learning rate, epochs, matched optimizer steps) and report their accuracy right after warmup? If OPW's advantage persists against a properly fitted SFT model, the main claim becomes much stronger.

OPW still leads on ALFWorld 4B Unseen after tuning SFT and SFT-RS and matching warmup optimizer updates. We kept the offline teacher data fixed and swept learning rates for both SFT and SFT-RS using 80 warmup optimizer updates, with additional 160-update and batch-88 comparisons for SFT-RS.

(1) Tuned SFT and SFT-RS achieve 49.02% and 59.77% validation success after warmup. Both methods use 80 optimizer updates. The table reports success on the 64 ALFWorld validation tasks from the GRPO pool used to select the learning rate, 10−510^{-5} for both methods (Appendix C.6).

Warmup methodSuccessful rollouts / totalValidation success (%)
Tuned SFT, batch 64251 / 51249.02
Tuned SFT-RS, batch 64306 / 51259.77

Table R2. Final single-run ALFWorld 4B Unseen success after 60 GRPO steps.

WarmupWarmup Adam updatesWarmup examplesFinal success (%)
Tuned SFT, batch 64805,12085.35
Tuned SFT-RS, batch 64805,12092.07
SFT-RS, 160 updates16010,24092.26
SFT-RS, larger batch of 88807,04093.66
OPW (Ours)805,12096.74

(2) OPW retains its advantage at nearly equal supervised-token counts. Before tuning, SFT and SFT-RS reached 80.32% and 87.78% final Unseen success. At 80 warmup updates, batch-88 SFT-RS uses 9.74M supervised tokens versus OPW's 9.76M and finishes about 3.1 points below OPW. Table R2 uses the original-normalization OPW reference (Appendix C.6).

(3) OPW uses fewer GPU-hours in the original comparison. The original SFT, SFT-RS, and OPW + GRPO runs cost 31.4, 30.9, and 27.3 GPU-hours, including teacher costs (Appendix C.3). DigiRL also compared offline and offline-to-online training at similar trajectory counts [1].

Thank you for encouraging stronger offline baselines. These controls show that OPW's advantage persists against better-fitted SFT models and at nearly equal supervised-token counts.

AuthorsOpenReview ↗

Response to Reviewer uq62 (Part 3/4)

[W3\textcolor{#B91C1C}{\textbf{W3}} & Q3\textcolor{#B45309}{\textbf{Q3}} & Q4\textcolor{#B45309}{\textbf{Q4}}] Does the bound exceed Base, and how do the losses compare?

W3. The theory is fairly standard and does not match the method. It analyzes reverse KL, while training uses top-1 cross-entropy, and the bound is likely vacuous for responses of this length.

Q3. What is the measured per-trajectory reverse KL between student and teacher after OPW, and does the resulting lower bound l exceed the Base success rate on any task? If not, how should readers interpret the theory?

Q4. How does top-1 cross-entropy compare with full-distribution reverse KL, which is the objective the theory analyzes?

We measured reverse KL and compared CE with both full-distribution objectives. Measured mean cumulative KL is 192.28–466.99 nats per trajectory. The plug-in lower bound is 0 on all diagnostic tasks. Theorem 3.2 provides a sufficient condition for improved initial success. For shared initial-state distribution and environment dynamics, student success exceeds Base when pT>p0p_T>p_0 and L<2(pT−p0)2L<2(p_T-p_0)^2, where pTp_T and p0p_0 are stochastic-teacher and Base success probabilities and LL is cumulative population reverse KL. Observed 4B initial success gains over Base are 17.19 percentage points on ALFWorld and 33.59 on ScienceWorld (95% paired-task intervals [8.59, 26.17] and [22.66, 44.92], Appendix C.7).

(1) CE leads on ALFWorld Seen success and ScienceWorld Unseen task score. Reverse KL leads on ALFWorld Unseen success. We compared CE, forward KL, and reverse KL in single runs with the same Qwen3-4B Base model, 32B teacher, task order, response-token normalization, and 40 warmup + 60 GRPO steps (Appendix C.7).

Table R3. Final task performance with common response-token normalization in warmup.

Warmup objectiveALFWorld Seen success (%)ALFWorld Unseen success (%)ScienceWorld Unseen task score
Top-1 cross-entropy98.3094.6873.68
Full-vocabulary forward KL96.4394.8772.85
Full-vocabulary reverse KL96.6196.1873.54

(2) Top-1 CE controls agreement with the greedy teacher. At a shared token history, the student’s probability of choosing a different token is bounded by its CE on the greedy token. With a shared initial state and environment randomness, expected cumulative CE over complete finite student rollouts bounds the probability of any disagreement (Appendix A.3). Thank you for pressing us to connect theory and implementation. The CE analysis and objective comparisons clarify the theoretical connection and which final metrics favor each loss.

Reviewer_w9R2OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the cold-start problem in agentic reinforcement learning with verifiable rewards (RLVR), where sparse binary outcome rewards cause an uninitialized student policy to produce many all-failure rollout groups with zero within-group reward variance and hence no usable policy-gradient signal (Section 1). The authors propose On-Policy Warmup (OPW, Section 3.3), a two-stage training pipeline: in Stage I (40 training steps on a warmup task pool Dw\mathcal{D} _ w), the student policy πθ\pi _ \theta interacts with the environment to generate trajectories under its own state distribution and receives token-level supervision from a fixed teacher πT\pi _ T on assistant-generated tokens; in Stage II (60 training steps on a disjoint RL task pool DRL\mathcal{D} _ {\mathrm{RL}}), the teacher is removed and the warmed-up policy πw\pi _ w is optimized with teacher-free, unclipped GRPO (Eq. (5)).

To motivate OPW theoretically, Section 3.1–3.2 (and Appendix A.1–A.2) show that when the student and teacher share the initial state distribution and environment transition kernel, the trajectory-level reverse KL equals the cumulative on-policy token-level reverse KL, DKL(Pπ∥PT)=LOPD(π)=∑t=1HEst∼dtπ[DKL(π(⋅∣st)∥πT(⋅∣st))]D _ {\mathrm{KL}}(P _ \pi \parallel P _ T) = \mathcal{L} _ {\mathrm{OPD}}(\pi) = \sum _ {t=1}^H \mathbb{E} _ {s _ t \sim d _ t^\pi}[D _ {\mathrm{KL}}(\pi(\cdot \mid s _ t) \parallel \pi _ T(\cdot \mid s _ t))] (Eq. (1), Proposition 3.1), which via the data-processing and Pinsker inequalities yields a verifier success lower bound pπ≥[pT−ε/2]+p _ \pi \ge [p _ T - \sqrt{\varepsilon / 2}] _ + (Theorem 3.2, Eq. (2)), a geometric first-success rollout bound E[Tsucc]≤1/ℓ\mathbb{E}[T _ {\mathrm{succ}}] \le 1/\ell (Corollary 3.3, Eq. (3)), and a monotonicity regime (p≤0.5p \le 0.5) for the binary mixed-group probability mG(p)=1−pG−(1−p)Gm _ G(p) = 1 - p^G - (1 - p)^G (Appendix A.2, Eq. (6)). Empirically, using Qwen3-1.7B and Qwen3-4B students and a Qwen3-32B teacher on ALFWorld and ScienceWorld (Sections 4–5), OPW outperforms direct GRPO, direct OPD, offline SFT, SFT-RS, and teacher-free GRPO warmup—most notably lifting Qwen3-4B ALFWorld Unseen success from 86.6% (Direct GRPO) and 87.8% (SFT-RS) to 96.7% (Tables 2–3)—and is accompanied by diagnostic analyses of invalid-action repetition (Table 6, Appendix D.3 Table 18), fixed versus resampled student trajectories (Table 7, Appendix D.2 Table 17), common-history action interventions (Appendix E Tables 19–20), successful-sequence diversity (Figure 3, Appendix F.2 Table 22), training GPU-hours (Appendix C.3 Table 15), and warmup duration (Figure 4, Appendix G Tables 23–24).

Critical strengths

1. Strong, practically relevant empirical gains on interactive agent benchmarks with a simple teacher-free RL handoff.
OPW requires teacher queries only during the short 40-step warmup stage and then transitions cleanly to standard verifier-driven GRPO without retaining the 32B teacher in memory during RL. On ALFWorld with Qwen3-4B (Tables 2–4, Figure 2), this pipeline improves final task success to 96.1% (Seen) and 96.7% (Unseen), outperforming 100-step Direct GRPO (89.4% / 86.6%), 100-step Direct OPD (45.9% / 53.5%), and all matched 40w+60G warmup baselines (SFT: 83.9% / 80.3%; SFT-RS: 89.0% / 87.8%; GRPO warmup: 82.7% / 76.5%), while boosting 8-attempt consistency pass8\mathrm{pass}^8 on ALFWorld Unseen from 61.9% (Direct GRPO) and 68.7% (SFT-RS) to 88.8%.

2. Unusually thorough behavioral and mechanistic diagnostics across the warmup-to-RL boundary.
Rather than treating warmup as a black-box initialization, Section 5 and Appendices C–G probe what changes during warmup and subsequent RL: decomposing broad coverage (pass@k\mathrm{pass}@k) from repeated-attempt reliability (passk\mathrm{pass}^k) in Table 5, isolating the impact of repeated-invalid-action supervision in Table 6 and Table 18, contrasting fixed first-round student trajectories against online resampled trajectories in Table 7 and Table 17, measuring retention of high-value teacher corrections on shared histories in Tables 19–20, and accounting for teacher rollout/scoring GPU-hours in Table 15.

Critical weaknesses

1. The theoretical reverse-KL framework (Proposition 3.1, Theorem 3.2, Corollary 3.3) does not apply to the implemented teacher-top-1 cross-entropy objective, and no ablation bridges the two.
The entire theoretical motivation in Section 3.1–3.2 relies on controlling the population on-policy reverse KL LOPD(π)=∑t=1HEst∼dtπ[DKL(π(⋅∣st)∥πT(⋅∣st))]\mathcal{L} _ {\mathrm{OPD}}(\pi) = \sum _ {t=1}^H \mathbb{E} _ {s _ t \sim d _ t^\pi}[D _ {\mathrm{KL}}(\pi(\cdot \mid s _ t) \parallel \pi _ T(\cdot \mid s _ t))] (Eq. (1)), whereas the implemented algorithm (ray_trainer.py lines 780–804, Appendix D.1 line 1118) optimizes token-mean cross-entropy −log⁡πθ(vt∣ht)-\log \pi _ \theta(v _ t \mid h _ t) against the teacher's greedy argmax token vt=arg⁡max⁡vπT(v∣ht)v _ t = \arg\max _ v \pi _ T(v \mid h _ t). Because hard-label cross-entropy against the teacher's mode does not bound DKL(πθ∥πT)D _ {\mathrm{KL}}(\pi _ \theta \parallel \pi _ T), the paper has neither a theoretical guarantee for its actual algorithm nor an empirical ablation comparing full-distribution reverse-KL (or top-KK logit distillation) against hard top-1 cross-entropy. Furthermore, on ScienceWorld, where 92.2% of rollout groups already exhibit partial-credit reward variance during warmup (Appendix G.1 Figure 8) and SFT-RS achieves a 5.6×5.6\times larger early Unseen gain ΔR20\Delta R _ {20} than OPW (+19.5 vs. +3.5, Table 11), the binary all-failure reward-discovery mechanism cannot explain OPW's final advantage.

2. Incomplete statistical validation, asymmetric 1.7B vs. 4B experimental coverage, and missing comparisons to the closest OPD-to-RL baselines.

  • Single-seed main tables and halved effect size on the second seed: All main results (Tables 2–7) come from a single training seed (seed: 1). In the only two-seed check (Appendix C.2 Table 14, p. 21, Qwen3-4B ALFWorld 40w+100G), the average curve advantage Rˉ100\bar{R} _ {100} of OPW over GRPO warmup drops by roughly half from Seed 1 to Seed 2 on both Seen (+13.0 →\to +6.6 pp) and Unseen (+20.1 →\to +9.5 pp). Moreover, although verl/experimental/opw/metrics.py (line 385) computes 2,000-repeat task-level bootstrap confidence intervals (task_bootstrap_ci95), no confidence intervals are reported in the paper—which matters on ScienceWorld Unseen, where the pass8\mathrm{pass}^8 margin between OPW (8.5%) and Direct GRPO (7.0%, Table 4) corresponds to just 17 versus 14 tasks out of 200.
  • Asymmetric 1.7B vs. 4B reporting and skipped model scales: Main final tables (Tables 2–6), trajectory-resampling ablations (Table 7), and mechanism interventions (Tables 16–20) are reported only for Qwen3-4B (and Table 21 omits the ALFWorld, Unseen block altogether, leaving Qwen3-1.7B without tabulated ALFWorld, Unseen pass@1 scores), whereas the warmup-duration sweep (20/40/60/80 steps in Figure 4 and Tables 23–24) is reported only for Qwen3-1.7B while mixing different seeds across durations (Appendix G.2, p. 27). Standard intermediate scales (Qwen3-8B) and teacher-size sweeps are also absent.
  • Missing closest OPD-to-RL baselines and optimizer-step parity: Given recent sequential and interactive OPD-then-RL works (Sequential Beats Joint, RL Starts before RL, OPDSearch+, cited in Section 2 Table 1), the paper lacks matched-budget comparisons against joint OPD+RL (e.g., combining environment reward advantages with teacher distillation during warmup) and equal-optimizer-step SFT/SFT-RS (200 vs. 160 optimizer updates in Table 8).
Questions for authors
  1. Bridging the reverse-KL theory (Section 3.1–3.2) and the hard top-1 cross-entropy implementation (ray_trainer.py lines 780–804):
    Can you either (a) report controlled experiments comparing your hard top-1 cross-entropy warmup (ℓt=−log⁡πθ(vt∣ht)\ell _ t = -\log \pi _ \theta(v _ t \mid h _ t)) against full-distribution (or top-KK) on-policy reverse-KL DKL(πθ(⋅∣st)∥πT(⋅∣st))D _ {\mathrm{KL}}(\pi _ \theta(\cdot \mid s _ t) \parallel \pi _ T(\cdot \mid s _ t)) under the same 40-step warmup budget, or (b) provide a formal imitation/coupling bound that applies directly to hard top-1 cross-entropy? In addition, what are the empirical values of teacher_student/sample_kl_s_to_t (already computed in verl/experimental/opw/metrics.py line 548) and the resulting numerical lower bound [pT−ε/2]+[p _ T - \sqrt{\varepsilon / 2}] _ + in Theorem 3.2 at the end of warmup?

  2. Multi-seed reliability, task-bootstrap CIs, and missing Qwen3-1.7B ALFWorld, Unseen numbers:

    • Please provide the missing ALFWorld, Unseen rows for Qwen3-1.7B and Qwen3-4B in Appendix F.1 Table 21, as well as the 2,000-repeat task_bootstrap_ci95 intervals (already implemented in metrics.py) for the main comparisons in Tables 2–4.
    • Since Appendix C.2 Table 14 shows the OPW-over-GRPO gain Rˉ100\bar{R} _ {100} shrinking from +13.0 / +20.1 pp (Seed 1) to +6.6 / +9.5 pp (Seed 2), can you report at least 3-seed mean ±\pm std for OPW, SFT-RS (matched at 80 warmup optimizer updates), GRPO warmup, and a Joint OPD+RL baseline?
  3. Causal test of the reward-discovery mechanism on ScienceWorld:
    In ScienceWorld, 92.2% of task groups already have distinct partial-credit scores at OPW step 15 (Appendix G.1 Figure 8), and SFT-RS achieves a much larger early 20-step Unseen gain ΔR20\Delta R _ {20} than OPW on Qwen3-4B (+19.5 vs. +3.5, Table 11). If you train ScienceWorld Stage II using strictly binary terminal completion rewards Rx(τ)∈{0,1}R _ x(\tau) \in \lbrace 0, 1 \rbrace (suppressing partial scores), or compare OPW and SFT-RS checkpoints matched for initial binary pass@8\mathrm{pass}@8, does OPW's advantage widen as predicted by Corollary 3.3?

8 comments
AuthorsOpenReview ↗

Response to Reviewer w9R2 (Part 1/4)

Dear Reviewer w9R2,

Thank you for highlighting our “Strong, practically relevant empirical gains” and “Unusually thorough behavioral and mechanistic diagnostics.” We appreciate your recognition of both the simple transition to teacher-free RL and the analyses of coverage, repeated invalid actions, teacher corrections, and training cost. These aspects connect the practical warmup recipe to evidence about how it helps subsequent learning. Below, we address the theory–implementation connection, statistical evidence, baseline and model-scale coverage, and your proposed binary-reward test on ScienceWorld.

[W1\textcolor{#B91C1C}{\textbf{W1}} & Q1\textcolor{#B45309}{\textbf{Q1}}] Comparing top-1 cross-entropy with full-distribution distillation

W1. The theoretical reverse-KL framework (Proposition 3.1, Theorem 3.2, Corollary 3.3) does not apply to the implemented teacher-top-1 cross-entropy objective, and no ablation bridges the two.

Q1. Can you either (a) report controlled experiments comparing ... hard top-1 cross-entropy warmup ... against full-distribution ... on-policy reverse-KL ... under the same 40-step warmup budget, or (b) provide a formal imitation/coupling bound ...? ... what are the empirical values of teacher_student/sample_kl_s_to_t ... and the resulting numerical lower bound ... in Theorem 3.2 at the end of warmup?

We provide both the requested 40-step comparison and a coupling bound for top-1 CE. The CE, forward-KL, and reverse-KL runs share response-token normalization and subsequent GRPO training. ALFWorld Unseen final success was 96.18% for reverse KL and 94.68% for CE. ScienceWorld Unseen scores were 73.68, 72.85, and 73.54 in CE, forward-KL, and reverse-KL order.

(1) Direct rollouts show initial-success gains at the measured KL values. Mean cumulative reverse KL is 192.28–466.99 nats per trajectory on the diagnostic tasks (Theorem 3.2 plug-in lower bound of 0). At warmup step 40, teacher_student/sample_kl_s_to_t averages 0.1643 / 0.1052 nats/token on ALFWorld and 0.2578 / 0.1775 on ScienceWorld (1.7B / 4B, Appendix C.7). On the same diagnostic tasks, observed 4B initial-success gains over Base are 17.19 percentage points on ALFWorld and 33.59 on ScienceWorld (95% paired-task intervals [8.59, 26.17] and [22.66, 44.92]).

(2) Top-1 CE bounds disagreement with the greedy teacher. Let pgp_g be its success probability and CπC_\pi the population expectation of cumulative top-1 CE over complete finite rollouts from student sampling policy π\pi. Give both policies the same initial state and environment randomness. Conditional on a shared token prefix, disagreement has probability 1−q≤−log⁡q1-q\le-\log q, where qq is the student’s sampling probability of the greedy token. Summing over possible first disagreements bounds the probability of any disagreement by CπC_\pi. Identical trajectories have identical outcomes, so pπ≥max⁡(0,pg−Cπ)p_\pi\ge\max(0,p_g-C_\pi). Thus, Cπ<pg−p0C_\pi<p_g-p_0 suffices to exceed Base success p0p_0 (Appendix A.3).

Thank you for asking us to bridge this gap. The objective comparisons and CE coupling analysis connect the implemented loss with the theory, while the rollout measurements show the initial-success gains directly.

AuthorsOpenReview ↗

Response to Reviewer w9R2 (Part 2/4)

[W2\textcolor{#B91C1C}{\textbf{W2}} & Q2\textcolor{#B45309}{\textbf{Q2}}] Training seeds and task-level intervals

W2. Single-seed main tables and halved effect size on the second seed: All main results (Tables 2–7) come from a single training seed (seed: 1).

Q2. Please provide the missing ALFWorld, Unseen rows ... in Appendix F.1 Table 21, as well as the 2,000-repeat task_bootstrap_ci95 intervals ... for the main comparisons in Tables 2–4. ... can you report at least 3-seed mean ... std for OPW, SFT-RS ... GRPO warmup, and a Joint OPD+RL baseline?

We added three independent training seeds per method in each environment. The original main results used one run per configuration. OPW-CE has the highest mean final Unseen performance among the four methods (Table R1). Appendix C.5 gives the training settings, individual-run results, and training-seed intervals.

Table R1. Final Qwen3-4B Unseen performance over three additional seeds (mean ± SD).

Unseen metricOPW-CE (Ours)Joint OPD+RLTuned SFT-RSGRPO warmup + GRPOOPW-CE − Joint
ALFWorld success (%)95.21 ± 1.0494.15 ± 1.1691.14 ± 1.7876.49 ± 1.48+1.06 pp
ScienceWorld task score71.99 ± 1.6571.73 ± 1.9270.44 ± 1.8459.60 ± 2.00+0.26

(1) Task-level intervals support gains over direct GRPO in ALFWorld success and ScienceWorld score. Appendix C.4 reports 95% task-level intervals for the original fixed checkpoints in Tables 2–4, covering all six methods and both splits. We use 2,000 paired task-bootstrap resamples, keeping each task's eight rollouts together.

On ALFWorld Unseen, OPW exceeds direct GRPO in pass@1 by 10.2 percentage points [6.3, 14.1]. On ScienceWorld Unseen, the task-score difference is 7.0 points [4.3, 9.8], while the pass^8 difference is 1.5 percentage points [−2.5, 6.0]. Here, pass^8 measures success in all eight rollouts. Thank you for encouraging this broader assessment. The additional runs and task-level intervals strengthen the empirical support for OPW and make the sources of variation explicit.

AuthorsOpenReview ↗

Response to Reviewer w9R2 (Part 3/4)

[W2\textcolor{#B91C1C}{\textbf{W2}} & Q2\textcolor{#B45309}{\textbf{Q2}}] Offline baselines, model scales, and related methods (continued)

W2. Asymmetric 1.7B vs. 4B reporting and skipped model scales: ... Standard intermediate scales (Qwen3-8B) and teacher-size sweeps are also absent. ... the paper lacks matched-budget comparisons against joint OPD+RL ... and equal-optimizer-step SFT/SFT-RS (200 vs. 160 optimizer updates in Table 8).

(2) OPW remains ahead after tuning SFT and SFT-RS and matching OPW's 80 warmup optimizer updates. We keep the offline data fixed, select learning rates on 64 validation tasks from the GRPO pool, and follow each warmup with the same 60 GRPO steps (Table R2, Appendix C.6).

Table R2. Final ALFWorld 4B Unseen success after 60 GRPO steps.

WarmupWarmup Adam updatesWarmup examplesFinal success (%)
Tuned SFT805,12085.35
Tuned SFT-RS805,12092.07
SFT-RS, 160 updates16010,24092.26
SFT-RS, larger batch807,04093.66
OPW (Ours)805,12096.74

With batch 88 and 80 updates, SFT-RS processes 9.74M supervised tokens, close to OPW's 9.76M, and finishes about 3.1 percentage points below the original-normalization OPW reference.

(3) OPW leads the Qwen3-1.7B comparisons on the reported final metrics. On ALFWorld Unseen at the final checkpoint, OPW, tuned SFT-RS, and Joint OPD+RL achieve 82.74%, 81.06%, and 82.65% pass@1, respectively, and 61.94%, 41.79%, and 50.00% pass^8. Their corresponding ScienceWorld Unseen task scores are 63.06, 58.99, and 56.67.

We added the missing ALFWorld Unseen rows to Appendix F.1, Table 31, covering both model sizes and all four original warmups, with the existing evaluation stages and behavioral metrics.

(4) Fixing the seed separates warmup-duration effects from changes in seed identity. The Qwen3-4B runs fix training seed 1 and allocate the same 100 steps as 20 + 80, 40 + 60, 60 + 40, or 80 + 20 OPW + GRPO steps, using the original normalization (Appendix G.2). On ScienceWorld Unseen, 20 + 80 scores 73.83 versus 68.23 for 40 + 60, while 60 + 40 achieves the highest repeated task success. The preferred allocation therefore depends on the evaluation metric.

(5) Larger teachers improve ALFWorld success, while student-scale results depend on the metric. On ALFWorld Unseen, 4B students reach 86.85%, 90.95%, and 94.68% after OPW + GRPO with 8B, 14B, and 32B teachers, respectively, using the same CE normalization. With an 8B student, success is 94.22% for OPW, 92.44% for tuned SFT-RS, and 88.71% for direct GRPO. In each of two training runs per method on ScienceWorld Unseen with an 8B student and a 32B teacher, OPW completes all eight rollouts on more tasks at the final checkpoint, while SFT-RS leads in final mean task score and complete-task success.

(6) OPW has the highest final task score in the common-normalization ScienceWorld comparison. In the single-run ScienceWorld 4B comparison with common response-token normalization, final scores are OPW 73.68, Joint OPD+RL 72.33, Guided-OPD 71.64, OPDSearch+ 71.49, and LUFFY 68.57. Guided-OPD leads on average GRPO performance. LUFFY leads on pass^8 (31.50% versus 15.50%). The single-run ALFWorld comparison favors Joint, while the additional three-seed mean slightly favors OPW. With 40 OPW steps followed by 60 GiGPO steps, ScienceWorld Unseen task score is 71.91 versus 59.76 for 100 direct GiGPO steps. The corresponding pass^8 is 9.50% versus 10.50%, respectively. Thank you for encouraging stronger baselines and broader model coverage. These comparisons show where OPW's gains persist after offline tuning and how the benefits depend on model scale and evaluation metric.

Reviewer_xCa3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes On Policy Warmup, the idea is before GRPO, a small student is trained on its own trajectories with token targets from the teacher model, and then trained with GRPO without the teacher. The theoretical argument links reverse KL OPD to the trajectory level KL, and gives a lower bound on the student's initial success rate. On ALFWorld and ScienceWorld, the paper shows OPW+GRPO beats other common practices.

Critical strengths
  1. Clear and practical recipe with meaningful gains on benchmark comparing to other common practices.
  2. Clean ablation study to show why the warmup works: 1. reusing the initial student's trajectories vs resampling them every step; 2. removing teacher supervision exactly at repeated invalid actions vs at random actions with same token counts
Critical weaknesses
  1. I am not convinced the theory explains the gains. The analysis is about reverse KL to the teacher, but the method is trained with cross entropy on the teacher's top-1 token, so the theorem does not really cover the loss that was used. Moreover, the bound is only informative when the KL summed over the whole trajectory is under about 2 nats, but that seems hard to reach for trajectories of thousands of tokens, and the paper does not report it either. Also the paper's own results do not fit the "reward discovery" story, onn ALFWorld 4B, OPW actually starts below the GRPO warmup, but it gains more points in the first 20 RL steps, the authors also acknowledge that warmup scores do not explain later learning. So OPW seems to help for some other reason, but I think the title and framing should be revised to reflect that.
Questions for authors
  1. What is the estimated summed reverse KL after warmup, and what lower bound does Thm3.2 give compared with the observed initial success? How does the top-1 cross entropy loss relate to the reverse KL objective the theory assumes?
  2. GRPO warmup + GRPO ends below direct GRPO for the same 100 steps, is this caused by the pool switch and optimizer reset? If so, direct GRPO is the fairer teacher-free baseline.
  3. Table 2 and 3 seem to come from one training run per method, and the second seed in Table 14 roughly halves OPW’s advantage over GRPO warmup. Could you report the main results over at least three training seeds, with confidence interval, to see how stable the training is?
5 comments
AuthorsOpenReview ↗

Response to Reviewer xCa3 (Part 1/3)

Dear Reviewer xCa3,

Thank you for recognizing our “Clear and practical recipe with meaningful gains” and the “Clean ablation study.” We appreciate your emphasis on both the practical gains and the evidence from trajectory resampling and targeted supervision at repeated invalid actions. These analyses are central to understanding how a short teacher-guided warmup supports later teacher-free learning. Below, we address the theory–implementation connection, the effects of task-pool and optimizer choices, and variation across additional training runs.

[W1\textcolor{#B91C1C}{\textbf{W1}} & Q1\textcolor{#B45309}{\textbf{Q1}}] What does the measured reverse KL imply?

W1. I am not convinced the theory explains the gains. ... the method is trained with cross entropy on the teacher's top-1 token, so the theorem does not really cover the loss that was used. ... the bound is only informative when the KL summed over the whole trajectory is under about 2 nats ... OPW actually starts below the GRPO warmup, but it gains more points in the first 20 RL steps ... I think the title and framing should be revised to reflect that.

Q1. What is the estimated summed reverse KL after warmup, and what lower bound does Thm3.2 give compared with the observed initial success? How does the top-1 cross entropy loss relate to the reverse KL objective the theory assumes?

We measured cumulative reverse KL and added a top-1 CE bound. Measured mean cumulative KL is 192.28–466.99 nats per trajectory across four settings, giving a numerical plug-in lower bound of 0. Theorem 3.2 gives a sufficient condition for earlier reward discovery. On a fixed task, let pTp_T and p0p_0 denote the stochastic teacher's and Base's success probabilities, and let LL denote cumulative population reverse KL. With a shared initial-state distribution and environment dynamics, pT>p0p_T>p_0 and L<2(pT−p0)2L<2(p_T-p_0)^2 guarantee higher student success than Base and fewer expected independent rollouts to the first success under a fixed policy.

(1) Top-1 CE controls agreement with the greedy teacher. Fix a greedy teacher with success probability pgp_g. Let CπC_\pi be the population expectation of cumulative top-1 CE over complete finite rollouts from student sampling policy π\pi. Couple both policies from the same initial state with shared environment randomness. Conditional on a shared token prefix, disagreement has probability 1−q≤−log⁡q1-q\le-\log q, where qq is the student’s sampling probability of the greedy token. Summing over possible first disagreements bounds the probability of any disagreement by CπC_\pi. Identical trajectories have identical outcomes, so pπ≥max⁡(0,pg−Cπ)p_\pi\ge\max(0,p_g-C_\pi). If Cπ<pg−p0C_\pi<p_g-p_0, this bound exceeds Base (Appendix A.3).

(2) CE and reverse KL lead on different final metrics. The Qwen3-4B runs share the 32B teacher, task order, response-token normalization, 40 warmup steps (80 Adam updates), and 60 subsequent GRPO steps (Appendix C.7).

Table R1. Final task performance with common response-token normalization in warmup.

Warmup objectiveALFWorld Seen success (%)ALFWorld Unseen success (%)ScienceWorld Unseen task score
Top-1 cross-entropy98.3094.6873.68
Full-vocabulary forward KL96.4394.8772.85
Full-vocabulary reverse KL96.6196.1873.54

With CE, forward KL, and reverse KL, respectively, the percentages of Unseen tasks successful in all eight rollouts were 86.57%, 82.84%, and 83.58% on ALFWorld and 15.50%, 9.50%, and 11.00% on ScienceWorld.

(3) OPW improves initial success and task coverage. On 32 diagnostic tasks per environment with eight rollouts each, OPW raises Qwen3-4B single-rollout complete-task success over Base by 17.19 percentage points on ALFWorld and 33.59 on ScienceWorld (95% paired-task intervals [8.59, 26.17] and [22.66, 44.92], Appendix C.7). The percentage of tasks solved at least once rises from 56.25% to 75.00% on ALFWorld and from 34.38% to 90.63% on ScienceWorld.

Table R2. Initial complete-task success and mixed-reward groups before GRPO.

SettingTrajectory success (%), Base → OPWGroups containing both success and failure (%), Base → OPW
ALFWorld, Qwen3-1.7B9.77 → 49.2221.88 → 46.88
ALFWorld, Qwen3-4B30.86 → 48.0543.75 → 53.13
ScienceWorld, Qwen3-1.7B0.78 → 23.053.13 → 59.38
ScienceWorld, Qwen3-4B12.89 → 46.4831.25 → 84.38

Thank you for prompting this closer connection between the implemented loss and our analysis. The added comparisons clarify the initial-success gains from teacher supervision and how different distillation objectives affect the final outcome.

AuthorsOpenReview ↗

Response to Reviewer xCa3 (Part 2/3)

[W1\textcolor{#B91C1C}{\textbf{W1}}] Subsequent learning after warmup (continued)

(4) OPW can learn faster during GRPO even when it starts lower. The theory analyzes initial reward discovery. We have shortened the title to On-Policy Warmup for Agentic RL and clarified the distinction between initial reward discovery and subsequent learning. On ALFWorld 4B Seen, OPW starts 9.2 percentage points below GRPO warmup and leads by 21.2 points after 20 GRPO steps. Removing supervision at repeated invalid actions reduces the early gain by 7.4 points relative to random removal of a similar number of action tokens (Appendix D.3), linking the placement of teacher feedback to subsequent learning.

[Q2\textcolor{#B45309}{\textbf{Q2}}] Do the pool switch and optimizer reset explain the gap?

Q2. GRPO warmup + GRPO ends below direct GRPO for the same 100 steps, is this caused by the pool switch and optimizer reset? If so, direct GRPO is the fairer teacher-free baseline.

Resetting Adam reduces GRPO performance on pool B, while OPW remains ahead of both GRPO baselines. To separate the effects of task-pool and optimizer choices, all four staged controls continue the same 40-step GRPO checkpoint for 60 steps (Table R3).

Table R3. Final ALFWorld 4B Unseen success under task-pool and optimizer choices.

Training pathFinal success (%)
GRPO warmup + GRPO on A, retain Adam62.50
GRPO warmup + GRPO on A, reset Adam71.64
GRPO warmup + GRPO on B, retain Adam87.59
GRPO warmup + GRPO on B, reset Adam76.49
Direct GRPO, 100 steps86.57
OPW (Ours) + GRPO on B96.74

(1) Resetting Adam helps on pool A but hurts on pool B. OPW still exceeds direct GRPO and the strongest staged control by 10.2 and 9.1 percentage points, respectively.

(2) On Seen tasks, OPW starts below the Adam-retained control but learns faster. It gains 41.8 versus 3.21 percentage points in the first 20 GRPO steps and finishes at 96.07% versus 92.50%. Thank you for suggesting these controls. They show that OPW's advantage over teacher-free warmup persists when Adam is retained and help separate the warmup benefit from the optimizer-reset effect.

AuthorsOpenReview ↗

Response to Reviewer xCa3 (Part 3/3)

[Q3\textcolor{#B45309}{\textbf{Q3}}] How stable are the main results?

Q3. Table 2 and 3 seem to come from one training run per method, and the second seed in Table 14 roughly halves OPW’s advantage over GRPO warmup. Could you report the main results over at least three training seeds, with confidence interval, to see how stable the training is?

We added three independent training seeds per method in each environment. The original main tables used one run per configuration. Appendix C.5 gives the training settings and individual-run results.

Table R4. Final Qwen3-4B Unseen performance over three additional training seeds (mean ± sample SD, with differences paired by seed).

Environment and metricMethodMean ± SDNominal 95% t interval
ALFWorld Unseen success (%)OPW-CE (Ours)95.21 ± 1.04[92.63, 97.80]
Joint OPD+RL94.15 ± 1.16[91.28, 97.03]
Tuned SFT-RS91.14 ± 1.78[86.72, 95.56]
GRPO warmup + GRPO76.49 ± 1.48[72.81, 80.17]
OPW-CE − Joint+1.06 ± 1.77 pp[−3.35, 5.46]
ScienceWorld Unseen task scoreOPW-CE (Ours)71.99 ± 1.65[67.90, 76.08]
Joint OPD+RL71.73 ± 1.92[66.97, 76.49]
Tuned SFT-RS70.44 ± 1.84[65.87, 75.01]
GRPO warmup + GRPO59.60 ± 2.00[54.63, 64.57]
OPW-CE − Joint+0.26 ± 1.81[−4.23, 4.75]

OPW has the highest sample mean in both environments. Thank you for requesting these additional runs. They strengthen the evidence for OPW's gains over tuned SFT-RS and GRPO warmup and give a clearer picture of training variability.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all four reviewers for their thoughtful feedback. They recognize our “Clear and practical recipe with meaningful gains” (xCa3) and “Strong, practically relevant empirical gains” (w9R2). They also highlight the “Clean ablation study” (xCa3) and note that “The analyses are comprehensive.” (uq62) and “The reporting is candid.” (mxr5).

(1) Three additional training seeds quantify performance and uncertainty. OPW-CE has the highest final Qwen3-4B Unseen sample mean among the four compared methods, with 95.21 ± 1.04% ALFWorld success and 71.99 ± 1.65 ScienceWorld task score (mean ± sample SD over three additional training seeds per method in each environment) (xCa3 (Q3\textcolor{#B45309}{\textbf{Q3}}), w9R2 (W2\textcolor{#B91C1C}{\textbf{W2}}/Q2\textcolor{#B45309}{\textbf{Q2}}), uq62 (W1\textcolor{#B91C1C}{\textbf{W1}}/Q1\textcolor{#B45309}{\textbf{Q1}}), mxr5 (W2\textcolor{#B91C1C}{\textbf{W2}}/Q2\textcolor{#B45309}{\textbf{Q2}})).

(2) OPW gains persist against stronger offline baselines. After tuning SFT and SFT-RS and matching warmup updates, the larger-batch SFT-RS control with nearly equal supervised-token counts reaches 93.66% final Qwen3-4B ALFWorld Unseen success, versus 96.74% for OPW after the same 60 GRPO steps (w9R2 (W2\textcolor{#B91C1C}{\textbf{W2}}/Q2\textcolor{#B45309}{\textbf{Q2}}) and uq62 (W2\textcolor{#B91C1C}{\textbf{W2}}/Q2\textcolor{#B45309}{\textbf{Q2}})).

(3) Initial discovery and later learning have distinct supporting evidence. We added a top-1 CE coupling bound and full-distribution KL comparisons under common normalization. On the diagnostic tasks, OPW raises Qwen3-4B initial single-rollout success over Base by 17.19 percentage points on ALFWorld and 33.59 on ScienceWorld. In an ALFWorld 4B Seen ablation, removing supervision at repeated invalid actions reduces the first-20-step GRPO gain by 7.4 points relative to removing a similar number of random action tokens (xCa3 (W1\textcolor{#B91C1C}{\textbf{W1}}/Q1\textcolor{#B45309}{\textbf{Q1}}), w9R2 (W1\textcolor{#B91C1C}{\textbf{W1}}/Q1\textcolor{#B45309}{\textbf{Q1}}), uq62 (W3\textcolor{#B91C1C}{\textbf{W3}}/Q3\textcolor{#B45309}{\textbf{Q3}}/Q4\textcolor{#B45309}{\textbf{Q4}} and W4\textcolor{#B91C1C}{\textbf{W4}})).

(4) Trajectory controls identify gains in average GRPO performance. With the objective, supervised-token count, and optimizer updates fixed, current-student trajectories improve Qwen3-4B ALFWorld Unseen average success over teacher trajectories by about 3.5 percentage points under CE (GRPO steps 0–60, 95% paired-task interval [1.23, 5.82]), with a gain also under reverse KL. Resampling improves the CE average by 3.90 points [1.51, 6.36] over fixed initial-student trajectories (mxr5, W1\textcolor{#B91C1C}{\textbf{W1}}/Q1\textcolor{#B45309}{\textbf{Q1}}).

AuthorsOpenReview ↗

Update of Manuscript

Dear Reviewers,

We have revised the manuscript in response to your feedback. Changes are highlighted in blue.

Main Text

  • Theory and implementation [Sections 3 and 4.1]. Clarified the sufficient conditions for initial reward discovery and the distinction between top-1 CE training and population reverse KL.
  • Additional comparisons [Section 4.2]. Summarized results from three additional training seeds and stronger offline baselines.
  • Framing [Title, Abstract, Introduction, and Section 4.2]. Shortened the title to On-Policy Warmup for Agentic RL to emphasize the practical warmup contribution. Aligned the wording with Section 3.1, Theoretical Motivation for On-Policy Warmup, and clarified the scope of the main comparisons, including OPW's competitive performance with the evaluated Joint OPD+RL baseline.

Appendix

  • Theory [Appendix A, including A.3]. Clarified when the success bound is informative and added a coupling bound for top-1 CE.
  • Behavior and warmup duration [Appendix F.1, Table 31 and Appendix G.2]. Added the missing ALFWorld Unseen rows and a fixed-seed Qwen3-4B duration comparison.
  • Uncertainty [Appendix C.4, Tables 16–17 and Appendix C.5, Tables 18–19]. Added task-level intervals and the individual results and statistics for the three additional training seeds.
  • Warmup comparisons [Appendices C.6–C.8, Tables 20–24]. Added offline tuning, objective comparisons, initial-success diagnostics, and task-pool and optimizer controls.
  • CE diagnostic [Appendix C.7]. Added the existing cumulative top-1 CE measurement for the original ALFWorld Qwen3-4B warmup checkpoint alongside the KL diagnostics.
  • Trajectory source [Appendix D.4, Table 28]. Added comparisons with equal supervised-token counts and optimizer updates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.