- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper revisits whether DAgger, a classical imitation learning algorithm, remains effective in the era of LLM agents. The student and teacher are interpolated at the turn or prefix level: each trajectory switches to teacher generation with a certain probability, the teacher then provides per-turn annotations on the states the student actually visits, and these annotations are used for supervised fine-tuning, mitigating the covariate shift problem in which the student drifts into state distributions it is unfamiliar with. On SWE-Gym, 4B and 8B students surpass SFT, GRPO, and on-policy distillation under a fair comparison with the same initialization, scaffold, and data; the 4B model's resolution rate on SWE-bench Verified rises from 23.4% to 27.3%.
Critical strengths
- The problem is well defined and practically valuable. The paper targets a real problem in the early post-training of multi-step agents: when the model is too weak, or the task difficulty mismatches the model's current ability, on-policy methods sample almost entirely failed trajectories, providing neither reward signal nor anything to learn from, and training idles in the early stage. Turn-level teacher interpolation fills exactly this gap, letting the student reach supervised states early on. This pain point is especially relevant for small models with weak reasoning, or when benchmarks are too difficult and early training suffers from a difficulty mismatch.
- The ablations are comprehensive and proactively answer the most natural questions: how to set the student-teacher mixing speed (three linear-annealing schedules swept), whether a smaller teacher works, whether the method transfers to math tasks (gains on GSM8K as well), and which loss function and divergence measure suit the supervision signal best.
Critical weaknesses
- Missing the cold-start-plus-RL comparison. If I understand correctly, at the two extremes of the mixing ratio the method degenerates into two known practices: a constant teacher ratio of 1 approximates SFT (imitating the teacher), and 0 approximates fully on-policy RL training. The paper compares SFT and OPD, but I do not see in the main tables a comparison of SFT cold start followed by GRPO, a common post-training pipeline.
- No principled way to set the mixing ratio β, and the optimal schedule is inconsistent across benchmarks (Table 6). β is simply linearly annealed over 5 iterations with three hand-tuned speeds (slow/base/fast); on Verified-100 fast is best (30.0), while on Holdout the DAgger-style schedule is best with base (19.0). Although the authors acknowledge that the best rate depends on the protocol and benchmark, this hardly forms a general methodology, since the mixing ratio β is precisely what distinguishes the method from an SFT+RL pipeline.
- Insufficient ablation sample sizes (Tables 6, 9): on Verified-100, 1 point equals 1 task; the conclusions in F.1 and F.3 rest on 2-4 point gaps over 100 problems without multiple seeds, while Table 10 shows ±1.7% seed-level fluctuation.
Questions for authors
-
SFT cold start followed by RL is a common post-training practice: can the authors add the SFT-to-GRPO comparison with the budget matched to the mixture strategy? If this combination matches or exceeds the mixture strategy, how should the positioning of the proposed method be adjusted?
-
The mixing ratio β is the key degree of freedom distinguishing this method from "SFT then RL": beyond the three hand-tuned linear-annealing speeds, is there a transferable principle for setting it (e.g., adapting to the student's current success rate or to the state-distribution mismatch)? If not, what empirical conclusions can be drawn?
If the authors can address these concerns, I would consider raising my score.
3 comments
Weakness 2 & Question 2: Performance-Adaptive Schedule for
Thank you for suggesting a performance-adaptive mixing schedule. Following this idea, we designed and evaluated a feedback rule that automatically adjusts based on the student's standalone success rate, under the principle that teacher intervention should decay faster as the student approaches the teacher's capability.
Adaptive Rule and Experimental Setup. Using the same 5-iteration setup, we evaluate the student acting alone (greedy decoding) after each iteration on a fixed probe set of 100 tasks sampled from the SWE-Gym training pool. Let denote the student's resolution rate on this training probe set and denote the teacher's reference rate estimated from the initial teacher-only training batch. We update via:
Under this rule, a weak student retains stronger teacher guidance, whereas a student approaching the teacher's performance triggers a faster transition toward on-policy rollouts. Crucially, is determined solely from training-split feedback without using SWE-Gym Holdout or SWE-Bench Verified scores to update . Across the five iterations, this rule yields the trajectory : , naturally producing a rapid warm-start decay without manual tuning.
Results and Analysis. We compare the adaptive schedule against the fixed schedules and endpoints from Table 6 below:
| Method / Schedule | SWE-Gym Holdout (%) | SWE-Bench Verified-100 (%) |
|---|---|---|
| Iterative SFT () | 15.0 | 23.0 |
| OPD () | 16.0 | 24.0 |
| DAgger-style (fixed slow schedule) | 17.0 | 27.0 |
| DAgger-style (fixed base schedule) | 19.0 | 29.0 |
| DAgger-style (fixed fast schedule) | 17.0 | 30.0 |
| DAgger-style (adaptive schedule) | 19.0 | 30.0 |
The adaptive schedule outperforms iterative SFT and OPD by +4.0/+3.0 points on Holdout and +7.0/+6.0 points on Verified-100. Notably, it resolves the trade-off between the fixed base and fast schedules, simultaneously matching the best fixed result on both SWE-Gym Holdout (19.0%) and SWE-Bench Verified-100 (30.0%) without hand-tuning the decay speed. We have added this adaptive formulation and experiment to Section 3.2 and Appendix F.1 (Table 6, highlighted in blue in the revised manuscript).
Weakness 3: Interpreting Ablation Margins and Evaluation Variability
We clarify how the ablation results in Appendices F.1 and F.3 should be interpreted alongside the multi-seed evaluation in Table 10.
First, our core conclusions rely on consistent, multi-point gains confirmed on the full benchmark. In Table 6, all tested mixture schedules outperform iterative SFT (23.0%) and OPD (24.0%) on SWE-Bench Verified-100, with our base DAgger-style schedule improving over the strongest baseline (SFT-to-OPD, 16.0% / 26.0%) by +3.0 points on both benchmarks (19.0% / 29.0%), and the adaptive schedule reaching 19.0% / 30.0%. Furthermore, as shown in Table 10 (three decoding seeds at temperature 0.7), this advantage holds with low variance on both Verified-100 and the full 466-task SWE-Bench Verified benchmark:
| Method | SWE-Gym Holdout (%) | SWE-Bench Verified-100 (%) | Full SWE-Bench Verified (%) |
|---|---|---|---|
| SFT | |||
| OPD | |||
| Ours (AggreVaTe-style) | |||
| Ours (DAgger-style) |
On the full benchmark, DAgger-style training exceeds OPD by +3.9 points ( vs. ) with low variance across decoding seeds. Accordingly, the role of Table 6 is to show that mixture training consistently outperforms the baselines across a wide range of schedules, rather than to claim a strict statistical ranking between individual schedules (e.g., 29.0% vs. 30.0%) on a 100-task subset.
Second, Appendix F.3 (Table 9) is intended to show compatibility with both supervision losses rather than to claim one loss is universally superior. In Table 9, 4B DAgger-style training achieves 17.0% / 26.0% with pure CE and 19.0% / 26.0% with executor-matched supervision (CE + OPD), both clearly outperforming iterative SFT (15.0% / 23.0%) and OPD (16.0% / 24.0%). Thus, our primary takeaway is that mixture rollouts are effective under either supervision rule (enabling both white-box and black-box teachers), while the exact 0–4 point differences between the two losses depend on the rollout protocol and benchmark. We have updated Appendices F.1 and F.3 (highlighted in blue) to state the scope of these ablation claims explicitly.
Dear Reviewer Eaor,
As the discussion period is ending soon, we want to send a brief reminder that we have posted our responses, the new SFT-to-GRPO and adaptive-schedule experiments, and the revised paper. Please let us know if our updates address your concerns or if any questions remain. Thank you for your time.
We thank the reviewer for recognizing the practical value of our approach and for the constructive suggestions on baselines, mixing schedules, and statistical validation. We address each point and report the requested experiments below.
Weakness 1 & Question 1: SFT–OPD Endpoints and the SFT-to-GRPO Baseline
Below, we clarify that our teacher–student interpolation connects SFT and OPD (rather than reward-based RL), and report the requested SFT-to-GRPO baseline, which remains substantially below mixed-rollout training under a matched online rollout budget.
Endpoints of β (SFT and OPD). Let βi denote the teacher-execution probability at each turn. Under the executor-matched supervision in Eq. (14), βi≡1 corresponds to iterative SFT (at∼πe(⋅∣st) with unit cross-entropy weight wt=1), whereas βi≡0 corresponds to OPD (at∼πθ(⋅∣st) with reverse-KL distillation weight wt=−logπe(at∣st)πθ(at∣st)). Even at βi=0, the student receives dense token-level supervision from the teacher rather than sparse terminal task rewards as in GRPO. As reported in Table 6, the staged SFT-to-OPD baseline reaches 16.0% on SWE-Gym Holdout and 26.0% on SWE-Bench Verified-100, trailing our base DAgger-style schedule (19.0% and 29.0%).
Matched-Budget SFT-to-GRPO. Following the SFT-to-OPD setup in Appendix F.1, we initialize GRPO from the same 4.5K-effective-sample SFT checkpoint (two supervised passes over SWE-Gym) and run five online GRPO iterations with 64 tasks and 8 rollouts per task (5×64×8=2,560 online trajectories, matching the 5×512=2,560 online rollouts of mixture training). Using the same SWE-Gym training pool, OpenHands scaffold, and greedy evaluation, we compare SFT-to-GRPO against the Table 6 baselines below:
While SFT-to-GRPO slightly improves over direct GRPO on SWE-Gym Holdout (10.0% vs. 9.0% in Table 3), it remains below iterative SFT, OPD, SFT-to-OPD, and DAgger-style training on both evaluation sets, trailing DAgger-style training by 9.0 and 12.0 points, respectively (and even at its best intermediate iteration, 13.0% / 26.0%, it does not surpass SFT-to-OPD).
Training diagnostics explain this bottleneck: across the five GRPO iterations, 242 of the 320 task groups (75.6%) produce eight failed trajectories, yielding zero group-relative advantage and no reward-driven policy-gradient signal. Thus, even after an SFT warm start, terminal test rewards remain too sparse on long-horizon SWE tasks for pure RL to sustain progress in our setup, whereas teacher-interleaved rollouts provide dense supervision on every visited state regardless of terminal patch success.
These results confirm that our method serves as a dense, teacher-guided post-training recipe complementary to reward-based RL. We have added this SFT-to-GRPO experiment and discussion to Appendix F.1 (Table 6, highlighted in blue in the revised manuscript).