- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Experience Distillation, a method for transferring the experience acquired by agents through environment interactions into model parameters. The teacher generates action targets conditioned on historical interaction experience, and the student learns to imitate these actions through supervised fine-tuning, enabling it to perform tasks without access to the original experience context. The authors further introduce experience preprocessing and branch packing to handle interaction histories and improve training efficiency. Experiments evaluate EPD on software engineering and text-adventure tasks.
Critical strengths
- The problem is practically relevant. Agents can accumulate valuable experience through trial and error, and distilling this experience into model parameters could reduce their reliance on historical interaction context at inference time.
- The results show that teacher supervision conditioned on experience can substantially improve student performance. Directly applying SFT to collected experience yields limited gains, whereas generating supervision targets conditioned on experience leads to better results. This provides empirical evidence for further exploring how interaction experience can be converted into learnable agent behaviours.
Critical weaknesses
- The core idea of EPD is to have a teacher generate training targets conditioned on experience context and then transfer the resulting behaviours to a student through supervised fine-tuning. This is closely related to existing context distillation approaches. Experience preprocessing and enhanced teacher reasoning primarily improve target generation, while branch packing mainly reduces training overhead. The paper needs to clarify its distinctive methodological contribution relative to prior work on context distillation and experience-based agent learning.
- In packed sequences, teacher-generated actions are inserted at each position, but the subsequent observations still come from the environment feedback generated by the original recorded actions. Consequently, the history used to generate subsequent teacher actions contains transitions that are inconsistent with the synthetic actions and their corresponding environment feedback. Although the authors acknowledge that branch packing is a practical approximation to the one-step distillation objective and compare it with training on separate branches, the results primarily show comparable performance under the evaluated settings. They do not sufficiently explain why this contextual inconsistency does not substantially affect learning or whether the approximation generalises to more complex tasks.
- EPD combines experience preprocessing, enhanced teacher reasoning, and supervised fine-tuning. Although the authors investigate the effect of enhanced teacher reasoning on the Detective task, systematic evidence across tasks is lacking to establish the respective contributions of experience information, teacher reasoning prompts, and training strategies to the overall gains.
- The paper does not disclose the identities, sizes, specific versions, or architectural details of the base models used for SWE and TaleSuite. This limits reproducibility and makes it difficult to assess how strongly EPD depends on the capabilities or training characteristics of particular models.
- The comparison between teacher-sampled forward KL and student-sampled reverse KL is conducted on only two TaleSuite tasks. While this experiment supports the authors' choice of objective on the evaluated tasks, it is insufficient to establish whether the observed difference generalises more broadly. In addition, the paper measures efficiency in terms of environment samples. This metric captures environment interaction costs but does not fully account for the computational costs of long-context teacher inference, experience preprocessing, target generation, and student training. The reported environment-sample efficiency therefore does not directly establish an advantage in overall training cost.
Questions for authors
- In branch packing, teacher-generated actions and subsequent environment feedback from the original trajectories can be contextually inconsistent. Could the authors further analyse how this inconsistency affects the training objective and final performance, and explain why the current experiments sufficiently support this approximation?
- Could the authors provide the identities, sizes, versions, and architectural details of the base models used for SWE and TaleSuite to facilitate reproducibility and analysis of the results?
5 comments
Rebuttal experimental settings and supporting details for Reviewer wA4g
Experimental settings and supporting details for Reviewer wA4g
Qwen checkpoints and inference settings
| Model | Weight revision | Architecture | Layers / hidden size / heads / KV heads |
|---|---|---|---|
| Qwen/Qwen3-4B-Thinking-2507 | 768f209d9ea81521153ed38c47d515654e938aea | Qwen3ForCausalLM | 36/2560/32/8 |
| Qwen/Qwen3.5-9B | c202236235762e1c871ad0ccb60c8ee5ba337b9a | Qwen3_5ForConditionalGeneration | 32/4096/16/4 |
| Qwen3.8-27B | 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 | Qwen3_5ForConditionalGeneration | 64/5120/24/4 |
We use native chat and thinking templates and supervise full teacher replies. For the two-game comparison, reply limits are 8,192 tokens for 4B/9B and 4,096 for 27B medium. The six-game 27B run uses medium reasoning and a 16,384-token limit, training each game separately and evaluating eight episodes per game. The camera-ready paper will include environment versions, prompts, data construction, loss masks, and training and inference settings.
Performance and cost comparison with OPCD
Task execution and feedback can be costly. We report measured GPU hours as requested. Both use medium reasoning (4,096 reply tokens, up to 100 decisions/episode). Scores average normalized peaks over eight resets/game. Post-collection H20 GPU hours cover 128 EPD and the first 32 OPCD updates, excluding evaluation and unassigned startup/compilation. Preprocessing has no separate timing. EPD sums measured generation and training. OPCD runs 8 new episodes per update. Student rollout (generation, execution, waiting) takes 60.45%/50.30% of update time on Detective/Balances.
| Stage | Detective GPU·h | Balances GPU·h |
|---|---|---|
| EPD: teacher target generation | 2.76 | 5.83 |
| EPD: training stage | 6.84 | 10.51 |
| EPD: total (128 updates) | 9.60 | 16.34 |
| OPCD: student game rollout | 138.43 | 249.79 |
| OPCD: experience context and teacher scoring | 45.63 | 152.61 |
| OPCD: optimizer update | 25.85 | 70.63 |
| OPCD: weight synchronization | 1.71 | 1.65 |
| OPCD: verification, saving, and other gaps | 17.40 | 21.91 |
| OPCD: total (32 updates) | 229.01 | 496.60 |
What experience and teacher supervision contribute
-
How we measure teacher action quality. The teacher independently proposes actions using the collected experience. We sample 128 decisions per model and game, one pair per state under the same remaining budget. For 27B, 49 Detective pairs and 83 Balances pairs changed the action. Extra rollouts serve only to evaluate teacher actions.
-
What EPD and SFT trained on. All use 128 updates and global batch 16. Five-episode runs supervise one reply per example (LR 1e-6, sequence limit 65,536). Failure-only runs use one sequence per episode. On the five-episode set, SFT-all uses all 223 recorded replies, and SFT-success uses 121 from two environment-confirmed wins. The three episodes ended in death and contain 102 decisions, with a best peak of 110/360. EPD uses one teacher reply per decision from those episodes and reflections. SFT learns the recorded reasoning and actions, with loss applied only to those replies. All methods are evaluated in eight independent games from reset without source experience.
-
Preprocessing, reasoning, and training. The SWE experience in Table 1 includes an accepted commit. Preprocessing in Section 3.3 can use Codex-style context compaction or trajectory summarization to retain outcomes and feedback. Table 5 uses the internal model and 4,096 three-step teacher rollouts per condition, with one to five collected histories in context. ICL gain retention is the fraction of the zero-shot-to-ICL improvement retained by distillation.
| Condition | Normalized score (%) | ICL gain retained (%) |
|---|---|---|
| Zero-shot | 31.1 | 0 |
| EPD without enhanced teacher reasoning | 50.7 | 34.9 |
| EPD with enhanced teacher reasoning | 71.8 | 72.5 |
| ICL | 87.2 | 100 |
At similar loss, eight epochs at 1e-6 retained more ICL gain than four at 1e-5 (Figure 4).
Branch packing and closed-loop performance
Both variants use one supervised reply per example, the same five episodes, 128 updates, global batch 16, medium reasoning, and a 4,096-token limit. All models receive the same history and remaining budget at each of 129 states absent from the training episodes. We report normalized final scores from eight continuations per state, weighting states equally. Scores include points earned before handoff (mean 28.42%). In Table 3 of our paper, each packed sequence supervises multiple teacher replies. Packed uses 64 updates and Separate uses 768. We will report the unseen-state results in the camera-ready paper.
Forward and reverse KL
Table 10 averages 16 runs with matched compute and single-task training. In OPD, the student samples without experience and the teacher scores those tokens with experience. Both KL variants use recorded observations and zero extra environment rollouts.
Dear Reviewer wA4g, We appreciate how busy you are with multiple submissions. If you have a moment before the rebuttal deadline, we would be grateful if you could let us know whether our responses have addressed your concerns.
Reply for Reviewer wA4g
Thank you for raising reproducibility and experimental coverage. Our original SWE and TaleSuite experiments used distinct large in-house frontier models with details confidential under company policy. For reproducibility, we tested three open-weight Qwen models. Experience Distillation (EPD) improved all three over zero-shot. Teacher actions could raise final scores. EPD beat supervised fine-tuning (SFT) on full and failed experience, and Packed improved unseen-state continuation. EPD also beat On-Policy Context Distillation (OPCD) on average with about 1/28 of the GPU hours and zero new environment rollouts during distillation.
Results across Qwen models. (New experiments for this rebuttal.) All three Qwen models improved over zero-shot on both games. Scores are mean normalized peaks (zero-shot → EPD, %). Across six games, Qwen3.8-27B rose from 17.34% to 53.42%.
Performance and cost comparison with OPCD. (New experiment for this rebuttal.) EPD beat OPCD on average with fewer GPU hours and zero new rollouts at the reported budgets. Both used Qwen3.8-27B and the same five episodes per game.
EPD approximates trajectory-level context distillation (Eq. (1), §3.1) with one teacher decision per recorded history (Eq. (2), §3.2). OPCD executes student actions during distillation. In Table 2 of our paper, one-step generation retained 83.8% of the in-context learning (ICL) gain across five games, vs. 57.3% for two-step world-model branches.
What experience and teacher supervision contribute
Teacher action quality. (New experiment for this rebuttal.) The 27B teacher’s actions more often raised than lowered final scores in both games. We ran the recorded and proposed actions from the same state, then let the same frozen base model finish. Rates include ties in valid changed-action pairs. Games are weighted equally.
EPD versus SFT on recorded trajectories. (New experiment for this rebuttal.) EPD beat SFT trained on all trajectories or just wins, and when both trained only on failures. For Qwen3.8-27B, we matched updates and batch size. Scores average normalized peaks from Detective resets.
Component ablations. (Existing experiments.) In Table 1 of our paper, experience raises SWE pass@1 from 5.3% to 76.4% and the six-game mean from 18.5% to 45.6%. EPD retains 64.8% and 93.4% of those gains vs. 3.8% and −2.6% for SFT on the same data. Enhanced teacher reasoning raises Detective scores from 50.7% to 71.8% (Table 5). At similar loss, lower learning rates and more epochs retain more ICL gain (Figure 4).
Branch packing and closed-loop performance. (New experiment for this rebuttal.) Packed outperformed Separate from 129 unseen Detective states (79.93% vs. 67.63%, base 51.34%). Packed pairs each observation with its recorded action and marks teacher suggestions separately. Earlier unexecuted suggestions can affect later targets and predictions. Loss applies only to teacher replies. This is how Equation (4), Section 3.3, approximates Equation (2). Table 3 in our paper reports 43.8% for Packed and 43.1% for Separate across six games, with Packed taking under one tenth as long for generation and training.
Forward and reverse KL. (Existing experiment.) On Balances and Detective, teacher-sampled forward KL retains 96.7% and 98.4% of the ICL gain vs. 9.1% and 0.4% for student-sampled reverse KL (OPD, Table 10 in our paper). The large zero-shot–ICL gap and examples in Appendix A.6 suggest that student rollouts without experience seldom contain the teacher’s task-specific decisions, limiting the overlap that helps OPD work (Li et al., 2026, arXiv:2604.13016).
See our follow-up for details.