acceptodds
Under review as a conference paper at ICLR 2027

Revisiting DAgger in the Era of LLM-Agents

Abstract

Long-horizon LM agents learn from multi-turn interaction, where a single early mistake can alter the subsequent state distribution and derail the whole trajectory. Existing recipes fall short in complementary ways: supervised fine-tuning provides dense teacher supervision but suffers from covariate shift because it is trained on off-policy teacher trajectories; while reinforcement learning with verifiable rewards avoids this off-policy mismatch by learning from on-policy rollouts but with only sparse outcome feedback. We address this dilemma by revisiting Dataset Aggregation (DAgger) for multi-turn LM agents: the algorithm collects trajectories through a turn-level interpolation of student and teacher policies, and the student is then trained on these trajectories using supervised labels provided by the teacher. By directly interacting with environments, we expose the model to realistic states likely to be encountered during deployment, thereby effectively mitigating covariate shift. Besides, since the student is learned by mimicking the teacher's behavior, it receives rich feedback during learning. To demonstrate DAgger enjoys the benefits of both worlds, we tested the algorithm to train a software-engineering agent with 4B- and 8B-scale student models. On SWE-bench Verified, our DAgger-style training improves over the strongest post-training baseline by +3.9 points at 4B and +3.6 points at 8B. The resulting 4B agent reaches 27.3, outperforming representative published 8B SWE-agent systems, while the 8B agent achieves 29.8, surpassing SWE-Gym-32B and coming within 5 points of stronger 32B-scale agents. Together with consistent gains on the held-out SWE-Gym split, these results suggest the effectiveness of DAgger for modern long-horizon LM agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Imitation Learning, Dataset Aggregation (DAgger), LLM Post-training, Covariate Shift, Long-Horizon Decision Making, Software Engineering Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EaorOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits whether DAgger, a classical imitation learning algorithm, remains effective in the era of LLM agents. The student and teacher are interpolated at the turn or prefix level: each trajectory switches to teacher generation with a certain probability, the teacher then provides per-turn annotations on the states the student actually visits, and these annotations are used for supervised fine-tuning, mitigating the covariate shift problem in which the student drifts into state distributions it is unfamiliar with. On SWE-Gym, 4B and 8B students surpass SFT, GRPO, and on-policy distillation under a fair comparison with the same initialization, scaffold, and data; the 4B model's resolution rate on SWE-bench Verified rises from 23.4% to 27.3%.

Critical strengths
  • The problem is well defined and practically valuable. The paper targets a real problem in the early post-training of multi-step agents: when the model is too weak, or the task difficulty mismatches the model's current ability, on-policy methods sample almost entirely failed trajectories, providing neither reward signal nor anything to learn from, and training idles in the early stage. Turn-level teacher interpolation fills exactly this gap, letting the student reach supervised states early on. This pain point is especially relevant for small models with weak reasoning, or when benchmarks are too difficult and early training suffers from a difficulty mismatch.
  • The ablations are comprehensive and proactively answer the most natural questions: how to set the student-teacher mixing speed (three linear-annealing schedules swept), whether a smaller teacher works, whether the method transfers to math tasks (gains on GSM8K as well), and which loss function and divergence measure suit the supervision signal best.
Critical weaknesses
  1. Missing the cold-start-plus-RL comparison. If I understand correctly, at the two extremes of the mixing ratio the method degenerates into two known practices: a constant teacher ratio of 1 approximates SFT (imitating the teacher), and 0 approximates fully on-policy RL training. The paper compares SFT and OPD, but I do not see in the main tables a comparison of SFT cold start followed by GRPO, a common post-training pipeline.
  2. No principled way to set the mixing ratio β, and the optimal schedule is inconsistent across benchmarks (Table 6). β is simply linearly annealed over 5 iterations with three hand-tuned speeds (slow/base/fast); on Verified-100 fast is best (30.0), while on Holdout the DAgger-style schedule is best with base (19.0). Although the authors acknowledge that the best rate depends on the protocol and benchmark, this hardly forms a general methodology, since the mixing ratio β is precisely what distinguishes the method from an SFT+RL pipeline.
  3. Insufficient ablation sample sizes (Tables 6, 9): on Verified-100, 1 point equals 1 task; the conclusions in F.1 and F.3 rest on 2-4 point gaps over 100 problems without multiple seeds, while Table 10 shows ±1.7% seed-level fluctuation.
Questions for authors
  1. SFT cold start followed by RL is a common post-training practice: can the authors add the SFT-to-GRPO comparison with the budget matched to the mixture strategy? If this combination matches or exceeds the mixture strategy, how should the positioning of the proposed method be adjusted?

  2. The mixing ratio β is the key degree of freedom distinguishing this method from "SFT then RL": beyond the three hand-tuned linear-annealing speeds, is there a transferable principle for setting it (e.g., adapting to the student's current success rate or to the state-distribution mismatch)? If not, what empirical conclusions can be drawn?

If the authors can address these concerns, I would consider raising my score.

3 comments

We thank the reviewer for recognizing the practical value of our approach and for the constructive suggestions on baselines, mixing schedules, and statistical validation. We address each point and report the requested experiments below.

Weakness 1 & Question 1: SFT–OPD Endpoints and the SFT-to-GRPO Baseline

Below, we clarify that our teacher–student interpolation connects SFT and OPD (rather than reward-based RL), and report the requested SFT-to-GRPO baseline, which remains substantially below mixed-rollout training under a matched online rollout budget.

Endpoints of β\beta (SFT and OPD). Let βi\beta_i denote the teacher-execution probability at each turn. Under the executor-matched supervision in Eq. (14), βi≡1\beta_i \equiv 1 corresponds to iterative SFT (at∼πe(⋅∣st)a_t \sim \pi_e(\cdot\mid s_t) with unit cross-entropy weight wt=1w_t = 1), whereas βi≡0\beta_i \equiv 0 corresponds to OPD (at∼πθ(⋅∣st)a_t \sim \pi_\theta(\cdot\mid s_t) with reverse-KL distillation weight wt=−log⁡πθ(at∣st)πe(at∣st)w_t = -\log \frac{\pi_\theta(a_t\mid s_t)}{\pi_e(a_t\mid s_t)}). Even at βi=0\beta_i = 0, the student receives dense token-level supervision from the teacher rather than sparse terminal task rewards as in GRPO. As reported in Table 6, the staged SFT-to-OPD baseline reaches 16.0% on SWE-Gym Holdout and 26.0% on SWE-Bench Verified-100, trailing our base DAgger-style schedule (19.0% and 29.0%).

Matched-Budget SFT-to-GRPO. Following the SFT-to-OPD setup in Appendix F.1, we initialize GRPO from the same 4.5K-effective-sample SFT checkpoint (two supervised passes over SWE-Gym) and run five online GRPO iterations with 64 tasks and 8 rollouts per task (5×64×8=2,5605 \times 64 \times 8 = 2{,}560 online trajectories, matching the 5×512=2,5605 \times 512 = 2{,}560 online rollouts of mixture training). Using the same SWE-Gym training pool, OpenHands scaffold, and greedy evaluation, we compare SFT-to-GRPO against the Table 6 baselines below:

MethodSWE-Gym Holdout (%)SWE-Bench Verified-100 (%)
Iterative SFT15.023.0
OPD16.024.0
SFT-to-OPD16.026.0
SFT-to-GRPO (New)10.017.0
DAgger-style (base)19.029.0

While SFT-to-GRPO slightly improves over direct GRPO on SWE-Gym Holdout (10.0% vs. 9.0% in Table 3), it remains below iterative SFT, OPD, SFT-to-OPD, and DAgger-style training on both evaluation sets, trailing DAgger-style training by 9.0 and 12.0 points, respectively (and even at its best intermediate iteration, 13.0% / 26.0%, it does not surpass SFT-to-OPD).

Training diagnostics explain this bottleneck: across the five GRPO iterations, 242 of the 320 task groups (75.6%) produce eight failed trajectories, yielding zero group-relative advantage and no reward-driven policy-gradient signal. Thus, even after an SFT warm start, terminal test rewards remain too sparse on long-horizon SWE tasks for pure RL to sustain progress in our setup, whereas teacher-interleaved rollouts provide dense supervision on every visited state regardless of terminal patch success.

These results confirm that our method serves as a dense, teacher-guided post-training recipe complementary to reward-based RL. We have added this SFT-to-GRPO experiment and discussion to Appendix F.1 (Table 6, highlighted in blue in the revised manuscript).

Weakness 2 & Question 2: Performance-Adaptive Schedule for β\beta

Thank you for suggesting a performance-adaptive mixing schedule. Following this idea, we designed and evaluated a feedback rule that automatically adjusts βi\beta_i based on the student's standalone success rate, under the principle that teacher intervention should decay faster as the student approaches the teacher's capability.

Adaptive Rule and Experimental Setup. Using the same 5-iteration setup, we evaluate the student acting alone (greedy decoding) after each iteration ii on a fixed probe set of 100 tasks sampled from the SWE-Gym training pool. Let s^i\hat{s}_i denote the student's resolution rate on this training probe set and se=29.3%s_e = 29.3\% denote the teacher's reference rate estimated from the initial teacher-only training batch. We update βi\beta_i via:

β1=1,βi+1=βimax⁡(0, 1−s^ise).\beta_1 = 1, \qquad \beta_{i+1} = \beta_i \max\left(0,\, 1 - \frac{\hat{s}_i}{s_e}\right).

Under this rule, a weak student retains stronger teacher guidance, whereas a student approaching the teacher's performance triggers a faster transition toward on-policy rollouts. Crucially, βi\beta_i is determined solely from training-split feedback without using SWE-Gym Holdout or SWE-Bench Verified scores to update βi\beta_i. Across the five iterations, this rule yields the trajectory βi\beta_i: 1.000→0.478→0.128→0.052→0.0271.000 \rightarrow 0.478 \rightarrow 0.128 \rightarrow 0.052 \rightarrow 0.027, naturally producing a rapid warm-start decay without manual tuning.

Results and Analysis. We compare the adaptive schedule against the fixed schedules and endpoints from Table 6 below:

Method / ScheduleSWE-Gym Holdout (%)SWE-Bench Verified-100 (%)
Iterative SFT (βi≡1\beta_i \equiv 1)15.023.0
OPD (βi≡0\beta_i \equiv 0)16.024.0
DAgger-style (fixed slow schedule)17.027.0
DAgger-style (fixed base schedule)19.029.0
DAgger-style (fixed fast schedule)17.030.0
DAgger-style (adaptive schedule)19.030.0

The adaptive schedule outperforms iterative SFT and OPD by +4.0/+3.0 points on Holdout and +7.0/+6.0 points on Verified-100. Notably, it resolves the trade-off between the fixed base and fast schedules, simultaneously matching the best fixed result on both SWE-Gym Holdout (19.0%) and SWE-Bench Verified-100 (30.0%) without hand-tuning the decay speed. We have added this adaptive formulation and experiment to Section 3.2 and Appendix F.1 (Table 6, highlighted in blue in the revised manuscript).

Weakness 3: Interpreting Ablation Margins and Evaluation Variability

We clarify how the ablation results in Appendices F.1 and F.3 should be interpreted alongside the multi-seed evaluation in Table 10.

First, our core conclusions rely on consistent, multi-point gains confirmed on the full benchmark. In Table 6, all tested mixture schedules outperform iterative SFT (23.0%) and OPD (24.0%) on SWE-Bench Verified-100, with our base DAgger-style schedule improving over the strongest baseline (SFT-to-OPD, 16.0% / 26.0%) by +3.0 points on both benchmarks (19.0% / 29.0%), and the adaptive schedule reaching 19.0% / 30.0%. Furthermore, as shown in Table 10 (three decoding seeds at temperature 0.7), this advantage holds with low variance on both Verified-100 and the full 466-task SWE-Bench Verified benchmark:

MethodSWE-Gym Holdout (%)SWE-Bench Verified-100 (%)Full SWE-Bench Verified (%)
SFT14.7±0.614.7\pm0.623.7±0.623.7\pm0.623.2±0.723.2\pm0.7
OPD15.7±0.615.7\pm0.624.0±1.024.0\pm1.023.6±0.423.6\pm0.4
Ours (AggreVaTe-style)15.7±2.515.7\pm2.525.3±1.525.3\pm1.524.8±0.424.8\pm0.4
Ours (DAgger-style)16.7±0.616.7\pm0.629.0±1.729.0\pm1.727.5±0.327.5\pm0.3

On the full benchmark, DAgger-style training exceeds OPD by +3.9 points (27.5±0.3%27.5 \pm 0.3\% vs. 23.6±0.4%23.6 \pm 0.4\%) with low variance across decoding seeds. Accordingly, the role of Table 6 is to show that mixture training consistently outperforms the baselines across a wide range of schedules, rather than to claim a strict statistical ranking between individual schedules (e.g., 29.0% vs. 30.0%) on a 100-task subset.

Second, Appendix F.3 (Table 9) is intended to show compatibility with both supervision losses rather than to claim one loss is universally superior. In Table 9, 4B DAgger-style training achieves 17.0% / 26.0% with pure CE and 19.0% / 26.0% with executor-matched supervision (CE + OPD), both clearly outperforming iterative SFT (15.0% / 23.0%) and OPD (16.0% / 24.0%). Thus, our primary takeaway is that mixture rollouts are effective under either supervision rule (enabling both white-box and black-box teachers), while the exact 0–4 point differences between the two losses depend on the rollout protocol and benchmark. We have updated Appendices F.1 and F.3 (highlighted in blue) to state the scope of these ablation claims explicitly.

Dear Reviewer Eaor,

As the discussion period is ending soon, we want to send a brief reminder that we have posted our responses, the new SFT-to-GRPO and adaptive-schedule experiments, and the revised paper. Please let us know if our updates address your concerns or if any questions remain. Thank you for your time.

Reviewer_56yCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper revisits dataset aggregation method to improve the LLM agents over multi-turn interactions. Testing between DAgger, and AggreVaTe, the Dagger outperforms other post training approaches, i.e. GRPO, SFT, and OPD among SWE-gym tasks.

Critical strengths

The paper presents a clear training objective with propor formulation of the teach / student lables. It demonstrates superior performance over the multi-turn LLM agent task on SWE-gym.

Critical weaknesses

The effectiveness of the method is generally robust to the rolout schedule. However, the AggreVaTe-style might not be fully optimized as the best schedule is on the fast end.

Questions for authors
  1. Why Qwen3-4b baseline is significantly better than Qwen3-8B?
  2. Have you considered to test teacher-prefix, student-completion approach? The teach prefix might simplify the multi-turn task and help student to learn gradually.

nit - LM and LLM are mix-used.

4 comments

We thank the reviewer for the positive assessment of our formulation and empirical results, and for the helpful suggestions.

Weakness 1: Schedule Optimization for AggreVaTe-Style Rollouts

We agree that further tuning may improve the AggreVaTe-style variant. Our main finding is that teacher–student mixed rollouts provide effective post-training strategies under simple schedules, yielding consistent gains over the baselines evaluated in the paper.

First, our main results in Table 3 do not rely on a heavily tuned schedule: to keep hyperparameters consistent across variants, Table 3 uses the default base schedule (0→100,+200 \to 100, +20) rather than the best-performing fast schedule. Even under this default setting, our methods consistently outperform SFT, GRPO, and OPD at both 4B and 8B scales.

Second, as summarized from Table 6 below, all three AggreVaTe schedules outperform the iterative-SFT and OPD baselines on SWE-Bench Verified-100:

MethodSWE-Gym Holdout (%)SWE-Bench Verified-100 (%)
Iterative SFT15.023.0
OPD16.024.0
AggreVaTe-style (slow: +10+10/iter)15.025.0
AggreVaTe-style (base: +20+20/iter)16.026.0
AggreVaTe-style (fast: +50+50/iter)17.029.0

Note that under our 5-iteration setup and 100-turn limit, the fast schedule (+50+50 per iteration) already reaches full student rollouts by the third iteration. Moving even faster would reduce the schedule to an abrupt switch from teacher to student rollouts (similar to the staged SFT-to-OPD baseline in Table 6, which reaches only 26.0% on Verified-100). We have clarified these schedule boundaries in Appendix F.1 (highlighted in blue in the revised manuscript).

Question 1: Initial Performance Gap Between the 4B and 8B Base Models

This initial inversion occurs because the two base models come from different release generations, and the expected scaling trend is restored after task-specific post-training.

First, the 4B student uses a newer instruction-tuned checkpoint. As noted in Section 4.1, our 4B student is initialized from Qwen3-4B-Instruct-2507 [1], whereas the 8B student is initialized from the original Qwen3-8B release [2] (as no 2507 update was released at the 8B scale). Compared to the initial Qwen3 release, the 2507 update specifically improves instruction following and tool use [1], giving the 4B base model a stronger out-of-the-box fit for the OpenHands scaffold.

Second, after post-training, the 8B model outperforms the 4B model across all teacher-based methods on SWE-Bench Verified, and on both benchmarks under our mixture-rollout methods. Once task-specific supervision bridges the initial tool-use gap, the larger capacity of the 8B model consistently translates into stronger final performance (Table 3):

MethodQwen3-4B-Instruct-2507 (Holdout / Verified)Qwen3-8B (Holdout / Verified)
Base Model5.0% / 11.2%2.0% / 7.7%
+ SFT15.0% / 22.9%12.0% / 23.4%
+ OPD16.0% / 23.4%16.0% / 26.2%
+ Ours (AggreVaTe-style)16.0% / 24.5%17.0% / 27.3%
+ Ours (DAgger-style)17.0% / 27.3%19.0% / 29.8%

Notably, under DAgger-style training, the 8B student achieves a larger absolute gain (+22.1 points on SWE-Bench Verified vs. +16.1 points at 4B), reaching 29.8%.

References:

[1] Qwen Team. Qwen3-4B-Instruct-2507 Model Card, Hugging Face.

[2] Qwen Team. Qwen3-8B Model Card, Hugging Face.

Question 2: Teacher-Prefix, Student-Completion Rollouts

Thank you for this insightful suggestion. Following this idea, we implemented the teacher-prefix, student-completion protocol as a curriculum baseline and evaluated both its task resolution and rollout behavior. While this strategy works reasonably well, it underperforms both our student-prefix (AggreVaTe-style) and turn-level (DAgger-style) protocols.

Experimental Setup. Using Qwen3-4B-Instruct-2507 under the same 5-iteration budget, we start with a teacher-only iteration and then sample the teacher-prefix length uniformly from {0,…,Ki}\{0, \dots, K_i\} with Ki∈{80,60,40,20}K_i \in \{80, 60, 40, 20\} across iterations 2–5, letting the student complete the remaining turns.

Task-Resolution Comparison. Comparing against the base-schedule results from Table 6 (under the same executor-matched supervision setup), teacher-prefix training reaches 15.0% on SWE-Gym Holdout and 25.0% on SWE-Bench Verified-100, trailing student-prefix by 1.0 point and turn-level DAgger by 4.0 points on both benchmarks:

Rollout ProtocolSWE-Gym Holdout (%)SWE-Bench Verified-100 (%)
Teacher-prefix, student-completion (New)15.025.0
Student-prefix, teacher-completion (AggreVaTe-style)16.026.0
Turn-level mixing (DAgger-style)19.029.0

Behavioral Analysis. To better understand this gap, we analyzed the evaluation trajectories of the teacher-prefix model on SWE-Bench Verified-100 (greedy decoding, no teacher intervention):

Evaluation MetricTeacher-Prefix Model
Non-empty patch submission (finish called)67.0%
Context-overflow termination24.0%
Repetitive-loop termination4.0%
Mean trajectory length (turns)58.1
Task resolution rate25.0%

These diagnostics suggest two plausible reasons why a teacher-prefix curriculum is less effective than student-prefix or turn-level mixing:

First, teacher prefixes shield the student from early mistakes. Because the teacher handles the initial exploration and bug localization during training, the student mostly practices completing trajectories from clean, well-localized states. At test time, however, the student must explore on its own and lacks the recovery skills needed when its early search goes off track.

Second, unguided student suffixes still suffer from long-horizon drift. Once the student takes over the suffix during training, early missteps can lead to lengthy, unproductive exploration (reflected in the 58.1-turn average length and 24.0% context-overflow rate), limiting patch accuracy despite a 67.0% submission rate.

We have added this baseline and behavioral analysis to Appendix F.1 (Table 6, highlighted in blue in the revised manuscript).

Nit: Consistent Terminology

Thank you for pointing this out. We have standardized our terminology across the revised manuscript (highlighted in blue), consistently using "LLM" and "LLM agents" instead of mixing "LM" and "LLM".

Thank for the additional evidence. My concerns are addressed.

Thank you very much for taking the time to review our additional experiments and for your positive feedback. We are glad our responses addressed your concerns, and we have incorporated these updates into the revised paper.

Reviewer_KBwAOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper applies DAgger to long-horizon agent training, aiming to address both the distribution shift problem in SFT and the cold start problem in OPD. The idea is to let the student make progress with help from the teacher, while also learning from teacher supervision on states it is likely to encounter itself. Specifically, at each turn, the method randomly selects either the teacher or student to execute an action, with the student's participation gradually increasing during training. On SWE-bench Verified, the method achieves 27.3% and 29.8% with Qwen3-4B and 8B, respectively, outperforming SFT, GRPO, and OPD. Turn-level mixing also performs better than student-prefix + teacher-completion and the two-stage SFT-to-OPD approach.

Critical strengths
  1. The paper addresses two practical challenges in long-horizon agent training. Training only on teacher trajectories can lead to distribution shift when the student makes mistakes and encounters unseen states during deployment. Relying only on student rollouts can also be problematic when the model is weak, as trajectories may fail early and never reach useful later states. The proposed teacher-student mixed rollout is a reasonable way to address both issues.

  2. The paper compares its method with SFT, GRPO, and OPD, as well as a two-stage SFT-to-OPD approach and a student-prefix + teacher-completion rollout strategy. The method consistently outperforms these baselines on both Qwen3-4B and 8B.

Critical weaknesses
  1. The source of the improvements is not fully analyzed. The paper attributes the benefits of mixed rollout to reducing cold start issues and distribution shift, but the current experiments do not clearly separate the contributions of these two factors. More direct analysis of when teacher intervention happens, how often it helps recover failed trajectories, and how state coverage affects training would make the explanation more convincing.

  2. The teacher intervention strategy is relatively simple. The paper randomly selects the teacher or student based on a fixed probability schedule, without considering the current state. However, some turns may be easy enough for the student to handle independently, while others, such as locating bugs, modifying critical code, or handling test failures, may benefit more from teacher intervention. As a result, random mixing may not make the best use of teacher calls.

Questions for authors
  1. Beyond final success rates and teacher-student KL divergence, does mixed rollout cover more useful states in the later stages of a task? How often does teacher intervention recover a failing trajectory? Are the improvements mainly due to the teacher helping the trajectory move forward, or to the student receiving supervision on its own error states? How does random mixing compare with state-dependent teacher intervention?

  2. Alternating between the teacher and student at every turn may disrupt long-horizon planning. For example, the student might start fixing a bug using one approach, while the teacher switches to a different solution, and the student may then switch back again. Does this switching cause repeated or conflicting code changes? The paper mainly compares the final performance of DAgger-style and AggreVaTe-style rollouts, but does not examine whether such planning conflicts actually happen. How does turn-level random mixing affect planning consistency, and why does it still perform better than student-prefix + teacher-completion?

  3. The paper assumes that the teacher's action at a given state is a suitable supervision target for the student. However, there may be a large capability gap between the two models. For example, a 30B teacher may be able to make a complex code modification, while a 4B student may struggle to imitate that action or continue the task afterward. Have the authors considered filtering or adapting teacher actions to the student's capabilities?

  4. The paper uses CE loss for every turn, but different actions may have very different impacts on the final task outcome. For example, searching an irrelevant file may have little effect, while correctly locating a bug or modifying a critical piece of code could determine whether the task succeeds. Treating all turns equally may cause the model to spend too much training effort on simple operations while giving insufficient attention to important decisions. Have the authors considered weighting the supervision based on each turn's contribution to the final outcome?

8 comments

We thank the reviewer for recognizing the practical motivation and empirical strengths of our work. Below we address each point and report the requested trajectory analyses and experiments.

Weakness 1 & Question 1(a–c): Cold Start, Distribution Shift, and Rollout Behavior

Cold-start mitigation and distribution-shift reduction are two complementary views of the same mixed-rollout mechanism, evaluated relative to OPD (βi≡0\beta_i \equiv 0) and SFT (βi≡1\beta_i \equiv 1), respectively. Teacher execution helps rollouts reach productive trajectories early when student-only exploration fails (mitigating cold start relative to OPD), while student execution exposes training to histories shaped by the learner's own decisions (reducing train–test mismatch relative to SFT). We quantify these effects through three trajectory analyses on matched 512-task batches across iterations (initialized from the same SFT checkpoint).

Early productivity and task-level recovery. Before the first update, student-only rollouts (OPD) resolve only 9.0% of tasks and produce a valid submission on 34.6%, whereas initial teacher-guided collection (β1=1.0\beta_1 = 1.0) reaches 23.8% resolution and 84.6% valid submissions, directly illustrating OPD's cold-start bottleneck. Pairing trajectories by task across all five iterations shows a consistent conditional recovery advantage:

Iteration (βi\beta_i)Resolution Rate: DAgger-style / OPD (%)DAgger Solve Rate Given OPD Fails (%)OPD Solve Rate Given DAgger Fails (%)
Iter 1 (1.0)23.8 / 9.017.61.5
Iter 2 (0.8)27.5 / 13.119.13.0
Iter 3 (0.6)22.3 / 11.114.52.3
Iter 4 (0.4)27.5 / 15.218.95.1
Iter 5 (0.2)19.3 / 14.910.05.1

Across iterations, DAgger-style rollouts resolve 10.0–19.1% of the tasks where the paired OPD rollout fails, whereas OPD resolves only 1.5–5.1% of the tasks where DAgger-style fails.

Coverage across trajectory stages. Because each turn's executor is sampled independently with probability βi\beta_i, teacher support is naturally distributed across the entire horizon without requiring a hand-crafted failure detector. At βi=0.6\beta_i = 0.6, for instance, the observed teacher-execution shares across five equal trajectory-depth quintiles are 60%, 60%, 59%, 60%, and 61%, covering both initial repository exploration and later code editing/debugging.

Late-stage interaction on student-influenced histories. Defining the late stage as all turns from the first successful source-file edit onward, we compare average late-stage turns per trajectory below:

βi\beta_iAll Late-Stage Turns (DAgger-style / OPD)Late-Stage Turns on Resolved Tasks (DAgger-style / OPD)
1.022.6 / 30.44.6 / 2.9
0.824.1 / 23.86.1 / 2.7
0.626.2 / 34.05.5 / 3.3
0.426.1 / 29.56.4 / 4.4
0.228.9 / 33.74.9 / 3.9

While OPD rollouts often spend more total turns after editing begins (due to unproductive trial-and-error), DAgger-style rollouts provide 1.3–2.3$\times$ more late-stage turns on trajectories that actually resolve the task. Moreover, once βi<1\beta_i < 1, over 95% of late-stage turns occur on histories containing prior student actions, ensuring that teacher supervision is applied to student-induced contexts. This aligns with Figure 2 (where SFT's reverse KL rebounds from 0.10 to 0.126 while mixture training stays flat near 0.10) and Table 6 (where DAgger-style reaches 19.0% / 29.0%, outperforming both iterative SFT at 15.0% / 23.0% and OPD at 16.0% / 24.0%).

Weakness 2 & Question 1(d): State-Dependent Teacher Intervention

Thank you for suggesting a state-dependent intervention baseline. To test this, we implemented a disagreement-based gate that prioritizes teacher execution on turns where the student and teacher propose different actions, while matching the expected teacher-call budget of our default schedule.

Experimental Setup and Budget Matching. At each turn, both policies propose an action given the current history. We flag a disagreement whenever the two proposals differ in coarse action type (inspection, editing, bash execution, or finishing), target different files during editing, or when the student output is unparseable. Using the base schedule (βi∈{1.0,0.8,0.6,0.4,0.2}\beta_i \in \{1.0, 0.8, 0.6, 0.4, 0.2\} over five iterations, 512 rollouts/iter) and an online estimate of the disagreement rate q^∈(0,1)\hat{q} \in (0, 1), we set the teacher-execution probabilities on disagreement and agreement turns to:

pdis=min⁡(1,βiq^),pagr=max⁡(0,βi−q^1−q^).p_{\mathrm{dis}} = \min\left(1, \frac{\beta_i}{\hat{q}}\right), \qquad p_{\mathrm{agr}} = \max\left(0, \frac{\beta_i - \hat{q}}{1 - \hat{q}}\right).

This prioritizes teacher takeover under disagreement while satisfying q^pdis+(1−q^)pagr=βi\hat{q} p_{\mathrm{dis}} + (1-\hat{q}) p_{\mathrm{agr}} = \beta_i. The realized teacher-execution fractions across the five iterations are 99.9%, 80.0%, 60.5%, 41.1%, and 20.1%, closely matching the target βi\beta_i budget.

Results and Behavioral Analysis. We compare the state-dependent mixture against the Table 6 baselines below:

MethodSWE-Gym Holdout (%)SWE-Bench Verified-100 (%)
Iterative SFT15.023.0
OPD16.024.0
SFT-to-OPD16.026.0
State-dependent mixture (New)16.027.0
DAgger-style (default random mixture)19.029.0

While state-dependent mixing outperforms iterative SFT, OPD, and SFT-to-OPD on Verified-100 (27.0%), it trails our default random mixture by 3.0 points on Holdout and 2.0 points on Verified-100. Comparing paired training trajectories at βi=0.8\beta_i = 0.8 and 0.6 reveals how the gate alters execution allocation:

βi\beta_iExecution Metric (%)Random MixtureState-Dependent Mixture
0.8Editor modifications among student-executed actions17.3411.45
0.8Teacher-executed share of all editor modifications80.4686.88
0.6Editor modifications among student-executed actions16.5913.05
0.6Teacher-executed share of all editor modifications62.1071.52

Because student–teacher disagreements frequently occur during code edits, the gate disproportionately hands file modifications over to the teacher (e.g., 71.52% vs. 62.10% at βi=0.6\beta_i = 0.6), reducing opportunities for the student to execute its own edits and observe the resulting environment feedback (13.05% vs. 16.59%). Moreover, surface-level action or file disagreement is an imperfect proxy for intervention value. These results validate random turn-level mixing as a simple, effective default that preserves balanced student exposure across action types.

Question 2: Planning Consistency Across Teacher–Student Handoffs

To examine whether turn-level switching causes conflicting edits or plan oscillation, we inspected concrete trajectories from mixed-rollout training.

First, trajectory inspections show test-driven refinement rather than repetitive edit oscillation. In dask/dask #8501, for example, the student changes dtype inference in da.full from type(fill_value) to np.dtype(fill_value), which fixes the reproduction script but fails on a scalar input (5). The teacher rolls back this edit. Rather than blindly reinstating its old change, the student runs diagnostic tests and implements a combined solution (try np.dtype(fill_value) with a TypeError fallback to type(fill_value)), passing all 89 tests and resolving the issue.

Second, the shared execution history maintains continuity across handoffs. Because both policies condition on the same evolving repository state, diffs, and test outputs, each handoff builds on observable progress. In getmoto/moto #7012, the teacher fixes a malformed class declaration introduced by the student while keeping its import, the student then implements the model timestamp and XML response field, the teacher refines the timestamp to UTC-aware datetime handling, and the student runs the test suite (passing 104 tests) to resolve the task.

Third, returning control to the student after teacher corrections helps explain the advantage over student-prefix + teacher-completion. Under AggreVaTe-style (student-prefix) rollouts, once the teacher takes over, the student never acts again in that trajectory. Turn-level DAgger instead allows the student to resume control after a teacher correction (as in the Moto case), exposing the student to post-recovery decision states. This aligns with DAgger-style outperforming AggreVaTe-style under the base schedule (19.0% / 29.0% vs. 16.0% / 26.0% in Table 6).

Question 3: Teacher–Student Capability Gap

Inspecting our trajectories shows that the 4B student can both execute the 30B teacher's local editing patterns and continue productively from teacher-written code.

First, the 4B student can execute the same code-repair patterns as the 30B teacher. For instance, in pandas/pandas #47848 (fixing boolean column indexing in styling functions), both models apply the same fix—replacing numeric column lists with an explicit columns.isin(...) mask. Within the same trajectory, the 30B teacher applies this pattern to background_gradient and text_gradient, and the 4B student subsequently adapts and applies the exact same pattern to bar, verifies both functions with a reproduction script, and resolves the issue.

Second, the student can actively debug and refine teacher-written code. In dask/dask #7073 (handling empty partitions in grouped value_counts), the teacher introduces a helper function for empty chunks. Taking over on subsequent turns, the 4B student fixes an indentation error in the teacher's helper, updates its method binding to self._value_counts_chunk, removes an invalid x.name attribute access, and passes the regression test.

Third, these capabilities transfer to standalone student evaluation. On python/mypy #16781, the trained 4B student acting entirely alone reproduces a missing property-deleter decorator in mypy/stubgen.py, implements .deleter handling, verifies the generated stub across 339 unit tests, and resolves the issue. At the aggregate level (Table 7), supervision from the 30B (and 480B) teacher boosts the 4B student from 5.0% / 13.0% to 17.0% / 27.0% (and 19.0% / 30.0%).

While we do not apply a separate capability-based action filter—relying instead on turn-level teacher labels and gradual βi\beta_i annealing so the student increasingly practices on its own continuations—we agree that explicit capability-aware filtering is an interesting direction for future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of the Rebuttal

We thank all reviewers for their comments. We have uploaded a revised paper, with all changes marked in blue. Paired scores below are resolution rates on SWE-Gym Holdout and SWE-Bench Verified-100. The main changes and additional experiments are listed below.

  1. State-dependent teacher intervention (Reviewer KBwA). We tested a disagreement gate at matched teacher budgets. It reaches 16.0% / 27.0%, below random turn-level mixing (19.0% / 29.0%), because it shifts more code edits to the teacher (71.5% vs. 62.1% at β = 0.6) and leaves students with fewer self-executed edits.

  2. Unfiltered rollout ablation (Reviewer KBwA). Following Reviewer KBwA's follow-up on Appendix E.2, we disabled finish filtering so all rollouts are retained. After two iterations (1,024 trajectories each), unfiltered DAgger reaches 14.0% / 23.0% versus 12.0% / 19.0% for unfiltered OPD while using fewer distilled tokens (10.90M vs. 15.69M). We will add the full five-iteration results to Appendix E.2.

  3. Teacher-prefix, student-completion rollouts (Reviewer 56yC). Appendix F.1 (Table 6) adds this curriculum baseline and its trajectory diagnostics. It reaches 15.0% / 25.0%, below student-prefix AggreVaTe (16.0% / 26.0%) and turn-level DAgger (19.0% / 29.0%).

  4. SFT followed by GRPO (Reviewer Eaor). Appendix F.1 (Table 6) now includes a matched-budget SFT-to-GRPO baseline (2,560 online rollouts). It reaches 10.0% / 17.0%, compared with 19.0% / 29.0% for turn-level DAgger, as 242 of 320 task groups fail all eight rollouts and receive zero reward advantage.

  5. Performance-adaptive mixing schedule (Reviewer Eaor). Section 3.2 and Appendix F.1 (Table 6) add an adaptive rule that updates β using only the student's success rate on 100 training tasks. It reaches 19.0% / 30.0%, matching the best fixed schedule on both benchmarks without manual tuning.

  6. Clarifications and terminology (Reviewers Eaor and 56yC). Appendices F.1 and F.3 clarify the scope of the 100-task ablations alongside the full 466-task results in Table 10, as well as the AggreVaTe schedule boundaries. We also standardized the terminology across the paper to use "LLM" and "LLM agents".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.