acceptodds
Under review as a conference paper at ICLR 2027

Sample-Efficient Learning from Agent Experience

Abstract

Real-world agent learning is often constrained by costly environment interactions, such as running time-consuming experiments or obtaining human feedback. Incontext learning offers a highly sample-efficient way for agents to learn from their own interaction histories, but its gains disappear once that experience is removed from the context. Separately, context distillation provides a mechanism for internalizing contextual information into model weights. However, applying it to agents’ interaction histories without sacrificing environment sample efficiency remains underexplored. We term this problem Experience Distillation and develop an implementation that requires no further environment interaction beyond the collected experience. Experiments on 749 curated software-engineering tasks and six text-adventure games show that it retains at least 64.8% of the gains from incontext learning across both domains, whereas direct supervised fine-tuning on the collected experience recovers only 3.8%. Compared with classical reinforcementlearning baselines, in-context learning from trial-and-error experience followed by Experience Distillation matches their performance with at least 9.6× fewer environment samples.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent, LLM, Learning from Experience

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wA4gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Experience Distillation, a method for transferring the experience acquired by agents through environment interactions into model parameters. The teacher generates action targets conditioned on historical interaction experience, and the student learns to imitate these actions through supervised fine-tuning, enabling it to perform tasks without access to the original experience context. The authors further introduce experience preprocessing and branch packing to handle interaction histories and improve training efficiency. Experiments evaluate EPD on software engineering and text-adventure tasks.

Critical strengths
  • The problem is practically relevant. Agents can accumulate valuable experience through trial and error, and distilling this experience into model parameters could reduce their reliance on historical interaction context at inference time.
  • The results show that teacher supervision conditioned on experience can substantially improve student performance. Directly applying SFT to collected experience yields limited gains, whereas generating supervision targets conditioned on experience leads to better results. This provides empirical evidence for further exploring how interaction experience can be converted into learnable agent behaviours.
Critical weaknesses
  • The core idea of EPD is to have a teacher generate training targets conditioned on experience context and then transfer the resulting behaviours to a student through supervised fine-tuning. This is closely related to existing context distillation approaches. Experience preprocessing and enhanced teacher reasoning primarily improve target generation, while branch packing mainly reduces training overhead. The paper needs to clarify its distinctive methodological contribution relative to prior work on context distillation and experience-based agent learning.
  • In packed sequences, teacher-generated actions are inserted at each position, but the subsequent observations still come from the environment feedback generated by the original recorded actions. Consequently, the history used to generate subsequent teacher actions contains transitions that are inconsistent with the synthetic actions and their corresponding environment feedback. Although the authors acknowledge that branch packing is a practical approximation to the one-step distillation objective and compare it with training on separate branches, the results primarily show comparable performance under the evaluated settings. They do not sufficiently explain why this contextual inconsistency does not substantially affect learning or whether the approximation generalises to more complex tasks.
  • EPD combines experience preprocessing, enhanced teacher reasoning, and supervised fine-tuning. Although the authors investigate the effect of enhanced teacher reasoning on the Detective task, systematic evidence across tasks is lacking to establish the respective contributions of experience information, teacher reasoning prompts, and training strategies to the overall gains.
  • The paper does not disclose the identities, sizes, specific versions, or architectural details of the base models used for SWE and TaleSuite. This limits reproducibility and makes it difficult to assess how strongly EPD depends on the capabilities or training characteristics of particular models.
  • The comparison between teacher-sampled forward KL and student-sampled reverse KL is conducted on only two TaleSuite tasks. While this experiment supports the authors' choice of objective on the evaluated tasks, it is insufficient to establish whether the observed difference generalises more broadly. In addition, the paper measures efficiency in terms of environment samples. This metric captures environment interaction costs but does not fully account for the computational costs of long-context teacher inference, experience preprocessing, target generation, and student training. The reported environment-sample efficiency therefore does not directly establish an advantage in overall training cost.
Questions for authors
  • In branch packing, teacher-generated actions and subsequent environment feedback from the original trajectories can be contextually inconsistent. Could the authors further analyse how this inconsistency affects the training objective and final performance, and explain why the current experiments sufficiently support this approximation?
  • Could the authors provide the identities, sizes, versions, and architectural details of the base models used for SWE and TaleSuite to facilitate reproducibility and analysis of the results?
5 comments
AuthorsOpenReview ↗

Reply for Reviewer wA4g

  • Thank you for raising reproducibility and experimental coverage. Our original SWE and TaleSuite experiments used distinct large in-house frontier models with details confidential under company policy. For reproducibility, we tested three open-weight Qwen models. Experience Distillation (EPD) improved all three over zero-shot. Teacher actions could raise final scores. EPD beat supervised fine-tuning (SFT) on full and failed experience, and Packed improved unseen-state continuation. EPD also beat On-Policy Context Distillation (OPCD) on average with about 1/28 of the GPU hours and zero new environment rollouts during distillation.

  • Results across Qwen models. (New experiments for this rebuttal.) All three Qwen models improved over zero-shot on both games. Scores are mean normalized peaks (zero-shot → EPD, %). Across six games, Qwen3.8-27B rose from 17.34% to 53.42%.

    ModelDetectiveBalances
    Qwen/Qwen3-4B-Thinking-25079.72 → 13.544.90 → 7.35
    Qwen/Qwen3.5-9B15.97 → 25.3513.48 → 18.38
    Qwen3.8-27B12.15 → 89.2414.71 → 20.83
  • Performance and cost comparison with OPCD. (New experiment for this rebuttal.) EPD beat OPCD on average with fewer GPU hours and zero new rollouts at the reported budgets. Both used Qwen3.8-27B and the same five episodes per game.

    TaskEPD score (%)EPD GPUhEPD · Env sample cost during distillation (episodes)OPCD score (%)OPCD GPUhOPCD · Env sample cost during distillation (episodes)
    Detective89.249.60055.56229.01256
    Balances20.8316.34022.06496.60256
    Mean55.0312.97038.81362.81256

    EPD approximates trajectory-level context distillation (Eq. (1), §3.1) with one teacher decision per recorded history (Eq. (2), §3.2). OPCD executes student actions during distillation. In Table 2 of our paper, one-step generation retained 83.8% of the in-context learning (ICL) gain across five games, vs. 57.3% for two-step world-model branches.

  • What experience and teacher supervision contribute

    • Teacher action quality. (New experiment for this rebuttal.) The 27B teacher’s actions more often raised than lowered final scores in both games. We ran the recorded and proposed actions from the same state, then let the same frozen base model finish. Rates include ties in valid changed-action pairs. Games are weighted equally.

      27B outcomeDetective (%)Balances (%)Mean (%)
      Higher34.6920.4827.59
      Lower20.416.0213.22
    • EPD versus SFT on recorded trajectories. (New experiment for this rebuttal.) EPD beat SFT trained on all trajectories or just wins, and when both trained only on failures. For Qwen3.8-27B, we matched updates and batch size. Scores average normalized peaks from Detective resets.

      Experience for EPDExperience for SFTEPD (%)SFT (%)
      Five episodesAll five episodes89.247.29
      Five episodesTwo wins from those episodes89.2455.90
      Three failed episodesSame three failed episodes27.0811.46
    • Component ablations. (Existing experiments.) In Table 1 of our paper, experience raises SWE pass@1 from 5.3% to 76.4% and the six-game mean from 18.5% to 45.6%. EPD retains 64.8% and 93.4% of those gains vs. 3.8% and −2.6% for SFT on the same data. Enhanced teacher reasoning raises Detective scores from 50.7% to 71.8% (Table 5). At similar loss, lower learning rates and more epochs retain more ICL gain (Figure 4).

  • Branch packing and closed-loop performance. (New experiment for this rebuttal.) Packed outperformed Separate from 129 unseen Detective states (79.93% vs. 67.63%, base 51.34%). Packed pairs each observation with its recorded action and marks teacher suggestions separately. Earlier unexecuted suggestions can affect later targets and predictions. Loss applies only to teacher replies. This is how Equation (4), Section 3.3, approximates Equation (2). Table 3 in our paper reports 43.8% for Packed and 43.1% for Separate across six games, with Packed taking under one tenth as long for generation and training.

  • Forward and reverse KL. (Existing experiment.) On Balances and Detective, teacher-sampled forward KL retains 96.7% and 98.4% of the ICL gain vs. 9.1% and 0.4% for student-sampled reverse KL (OPD, Table 10 in our paper). The large zero-shot–ICL gap and examples in Appendix A.6 suggest that student rollouts without experience seldom contain the teacher’s task-specific decisions, limiting the overlap that helps OPD work (Li et al., 2026, arXiv:2604.13016).

See our follow-up for details.


AuthorsOpenReview ↗

Rebuttal experimental settings and supporting details for Reviewer wA4g

Experimental settings and supporting details for Reviewer wA4g

Qwen checkpoints and inference settings

ModelWeight revisionArchitectureLayers / hidden size / heads / KV heads
Qwen/Qwen3-4B-Thinking-2507768f209d9ea81521153ed38c47d515654e938aeaQwen3ForCausalLM36/2560/32/8
Qwen/Qwen3.5-9Bc202236235762e1c871ad0ccb60c8ee5ba337b9aQwen3_5ForConditionalGeneration32/4096/16/4
Qwen3.8-27B1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0Qwen3_5ForConditionalGeneration64/5120/24/4

We use native chat and thinking templates and supervise full teacher replies. For the two-game comparison, reply limits are 8,192 tokens for 4B/9B and 4,096 for 27B medium. The six-game 27B run uses medium reasoning and a 16,384-token limit, training each game separately and evaluating eight episodes per game. The camera-ready paper will include environment versions, prompts, data construction, loss masks, and training and inference settings.

Performance and cost comparison with OPCD

Task execution and feedback can be costly. We report measured GPU hours as requested. Both use medium reasoning (4,096 reply tokens, up to 100 decisions/episode). Scores average normalized peaks over eight resets/game. Post-collection H20 GPU hours cover 128 EPD and the first 32 OPCD updates, excluding evaluation and unassigned startup/compilation. Preprocessing has no separate timing. EPD sums measured generation and training. OPCD runs 8 new episodes per update. Student rollout (generation, execution, waiting) takes 60.45%/50.30% of update time on Detective/Balances.

StageDetective GPU·hBalances GPU·h
EPD: teacher target generation2.765.83
EPD: training stage6.8410.51
EPD: total (128 updates)9.6016.34
OPCD: student game rollout138.43249.79
OPCD: experience context and teacher scoring45.63152.61
OPCD: optimizer update25.8570.63
OPCD: weight synchronization1.711.65
OPCD: verification, saving, and other gaps17.4021.91
OPCD: total (32 updates)229.01496.60

What experience and teacher supervision contribute

  • How we measure teacher action quality. The teacher independently proposes actions using the collected experience. We sample 128 decisions per model and game, one pair per state under the same remaining budget. For 27B, 49 Detective pairs and 83 Balances pairs changed the action. Extra rollouts serve only to evaluate teacher actions.

  • What EPD and SFT trained on. All use 128 updates and global batch 16. Five-episode runs supervise one reply per example (LR 1e-6, sequence limit 65,536). Failure-only runs use one sequence per episode. On the five-episode set, SFT-all uses all 223 recorded replies, and SFT-success uses 121 from two environment-confirmed wins. The three episodes ended in death and contain 102 decisions, with a best peak of 110/360. EPD uses one teacher reply per decision from those episodes and reflections. SFT learns the recorded reasoning and actions, with loss applied only to those replies. All methods are evaluated in eight independent games from reset without source experience.

  • Preprocessing, reasoning, and training. The SWE experience in Table 1 includes an accepted commit. Preprocessing in Section 3.3 can use Codex-style context compaction or trajectory summarization to retain outcomes and feedback. Table 5 uses the internal model and 4,096 three-step teacher rollouts per condition, with one to five collected histories in context. ICL gain retention is the fraction of the zero-shot-to-ICL improvement retained by distillation.

ConditionNormalized score (%)ICL gain retained (%)
Zero-shot31.10
EPD without enhanced teacher reasoning50.734.9
EPD with enhanced teacher reasoning71.872.5
ICL87.2100

At similar loss, eight epochs at 1e-6 retained more ICL gain than four at 1e-5 (Figure 4).

Branch packing and closed-loop performance

Both variants use one supervised reply per example, the same five episodes, 128 updates, global batch 16, medium reasoning, and a 4,096-token limit. All models receive the same history and remaining budget at each of 129 states absent from the training episodes. We report normalized final scores from eight continuations per state, weighting states equally. Scores include points earned before handoff (mean 28.42%). In Table 3 of our paper, each packed sequence supervises multiple teacher replies. Packed uses 64 updates and Separate uses 768. We will report the unseen-state results in the camera-ready paper.

Forward and reverse KL

Table 10 averages 16 runs with matched compute and single-task training. In OPD, the student samples without experience and the teacher scores those tokens with experience. Both KL variants use recorded observations and zero extra environment rollouts.

Dear Reviewer wA4g, We appreciate how busy you are with multiple submissions. If you have a moment before the rebuttal deadline, we would be grateful if you could let us know whether our responses have addressed your concerns.

Reviewer_1fkmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to distill agents’ interaction experience into model weights without additional environment interaction. Its method generates experience-conditioned teacher decisions at recorded histories and trains a student without the experience context, using branch packing for efficiency. Experiments on software-engineering tasks and text-adventure games show substantial retention of ICL gains, alongside improvements on held-out SWE tasks.

Critical strengths
  1. The method addresses a useful constraint. Generating improved decisions at recorded histories avoids both additional environment interaction and world-model rollout errors.

  2. The empirical gains are substantial. EPD strongly outperforms direct SFT, while ablations support branch packing and additional experiments demonstrate held-out transfer and repeated collect-and-distill cycles.

Critical weaknesses
  1. Teacher targets are not validated as better decisions. Access to complete experience provides hindsight, but generated replacement actions are never executed or checked for improved downstream return. The paper therefore leaves unclear how reliably its supervision represents policy improvement.

  2. Closed-loop distribution shift remains insufficiently examined. Student actions can lead away from recorded histories, where subsequent behavior is unsupervised. One-step branching avoids world-model errors but does not address compounding policy errors; branch packing additionally introduces previous unexecuted teacher decisions into training histories.

Questions for authors
  1. When executed from recoverable recorded states, how often do teacher-generated actions improve downstream return relative to the original actions?
  2. How does performance change as student trajectories diverge from recorded histories? Does branch packing affect this robustness?
  3. How does EPD compare with SFT restricted to successful actions or trajectory segments, under matched training budgets?
  4. Why is GRPO charged 448 samples in the efficiency comparison when its reported best score is reached at 240?
5 comments
AuthorsOpenReview ↗

Reply for Reviewer 1fkm

  • Thank you for asking us to validate teacher actions, test generalization beyond training histories, and clarify the baseline budgets. Among valid comparisons where the action changed, the 27B teacher’s proposed actions raised final scores in 27.59% of cases and lowered them in 13.22%, weighting Detective and Balances equally. From 129 unseen Detective states, Experience Distillation (EPD) with packed sequences scored 79.93%, ahead of both supervised fine-tuning (SFT) baselines. From the start of the game, EPD also outperformed success-only SFT at the same training budget (89.24% vs. 55.90%). We will report 240 episodes per task for GRPO’s best checkpoint and 448 for its full run.

  • Teacher action quality. (New experiment for this rebuttal.) The 27B teacher’s proposed actions led to higher final scores in 27.59% of valid pairs where the action changed and lower scores in 13.22%, with both games weighted equally. The rest tied. At each saved state, we ran one continuation starting with the recorded action and one starting with the teacher’s proposal. The same frozen base model finished both episodes. As shown below, the analysis uses the 490 valid pairs in which the action changed, including ties.

    TeacherDetective: higher / lower (%)Balances: higher / lower (%)Mean: higher / lower (%)
    Qwen/Qwen3-4B-Thinking-250720.65 / 15.224.04 / 2.0212.35 / 8.62
    Qwen/Qwen3.5-9B33.85 / 29.232.94 / 2.9418.39 / 16.09
    Qwen3.8-27B34.69 / 20.4120.48 / 6.0227.59 / 13.22
  • Closed-loop generalization and branch packing. EPD raised pass@1 from 4.62% to 8.84% on 494 software-engineering tasks excluded from experience collection and distillation (Table 4 in our paper).

    • Unseen-state continuation. (New experiment for this rebuttal.) As shown below, Packed EPD scored 79.93% from 129 unseen Detective states, ahead of Separate EPD (67.63%) and both SFT baselines. SFT-all trained on all collected trajectories, and SFT-success trained only on successful ones. The states were absent from the five training episodes. At each state, all models received the same episode history and controlled the rest of the game under the same remaining budget. We averaged eight final scores per state and weighted states equally. Scores are percentages of the game maximum of 360 and include the points earned before handoff, which averaged 28.42% of the maximum.

      Base modelSFT-allSFT-successEPD (Separate)EPD (Packed)
      51.34%63.99%64.91%67.63%79.93%
  • Full-game comparison with success-only SFT. (New experiment for this rebuttal.) As shown below, EPD outperformed success-only SFT from the start of Detective under the same training budget (89.24% vs. 55.90%). We evaluated the same EPD (Packed), SFT-success, and SFT-all checkpoints as in the unseen-state test. Here we average the highest score reached in each of eight independent games and normalize by the game maximum.

    MethodMean normalized episode peak (%)
    EPD89.24
    SFT-success55.90
    SFT-all7.29
  • GRPO sample accounting. (Figure 1, Table 8, and Appendix B.1 in our paper.) GRPO reached its best score at 240 episodes per task. Figure 1 paired that score with the full budget of 448 episodes per task. We will use 240 for the best-checkpoint comparison. Across the same six TaleSuite games, EPD used 47 episodes in total (7.83 per task). As shown below, GRPO’s best checkpoint used 30.6 times as many environment samples as EPD.

    Method and evaluation pointEnvironment samples per taskSix-task mean (%)
    EPD7.8343.8
    GRPO: best observed model, update 1624029.9
    GRPO: final model44826.0

We provide the experimental settings and supporting details in our follow-up comment.


AuthorsOpenReview ↗

Rebuttal experimental settings and supporting details for Reviewer 1fkm

Teacher action quality

  • Which decisions we tested. For each model and game, we randomly sampled 128 recorded decisions from the five episodes selected for distillation, covering the episodes and their early, middle, and late stages. The 4B, 9B, and 27B models scored 18.40%, 30.21%, and 84.72% with in-context learning (ICL) on Detective, respectively.

  • How we measure teacher action quality. At each saved state, the teacher independently proposes an action from the collected experience. We separately execute the recorded and teacher-proposed actions from that state. The same frozen base model then controls the rest of both episodes under the same remaining decision budget, up to 100 decisions for the full episode. It sees only the current episode’s actions and environment feedback. We compare final scores, with death, zero scores, and budget exhaustion all valid outcomes.

  • How we counted higher, tied, and lower scores. Each state contributes one pair. Of 768 selected states, 754 yield valid pairs, including 490 with changed actions and 264 with unchanged actions. Fourteen teacher targets were incomplete. The table uses the 490 changed-action pairs, keeping ties in the denominator. The Detective and Balances counts are 92 and 99 for 4B, 65 and 102 for 9B, and 49 and 83 for 27B. Extra interactions are used only to evaluate the teacher’s actions.

Generalization to held-out tasks (Existing experiment: Table 4 in our paper.)

EPD improved pass@1 both on tasks in new repositories and on new tasks in previously seen repositories. Using the paper’s internal model, we trained EPD on experience from 749 software-engineering tasks and compared the zero-shot and distilled models on 494 held-out tasks. The held-out tasks were excluded from experience collection and distillation.

Task subsetTasksZero-shot pass@1 (%)EPD pass@1 (%)
Full OOD set4944.628.84
Cross-repository OOD2784.728.87
Within-repository OOD2164.498.80

Unseen-state continuation

  • Which states we tested. The pool comes from earlier continuations by both SFT and EPD variants, along with collection, ICL, and EPD evaluation trajectories. We select 43 states from each third of the source episode’s actual length, prioritizing existing inputs and then a fixed hash order. We deduplicate using Jericho’s hash of game objects and game-specific memory and exclude all 117 hashes from the five training episodes.

  • How models continued from those states. Each model completed 1,032 valid continuations, eight per state. Each continuation starts with the same saved episode history and remaining budget, and uses medium reasoning and a 4,096-token reply limit. We will include this evaluation in the camera-ready paper.

  • What Packed changes in training. Packed keeps earlier unexecuted teacher suggestions in the training input. Each observation remains paired with the recorded action that produced it.

Full-game comparison with success-only SFT

  • Same training budget and different supervision. The trained models use the same Qwen3.8-27B initialization, five source episodes, 128 updates, and global batch 16. Each example supervises one complete reasoning-and-action reply, with loss applied only to that reply. SFT-all learns all 223 recorded replies. SFT-success learns 121 replies from the two environment-confirmed wins. EPD trains on new teacher replies generated using the same five episodes of experience. The training run uses learning rate 1e-6 and a 65,536-token sequence limit. Full-game evaluation uses medium reasoning and a 4,096-token reply limit.

  • Failure-only comparison. With only the first three failed episodes, EPD scored 27.08% with one teacher reply per decision and 28.82% with four, versus 11.46% for SFT on the recorded replies. EPD used new replies at all 102 decisions. All three used 128 updates and global batch 16. Scores are normalized mean peaks over eight independent episodes from the start without source experience.

GRPO sample accounting (Existing experiment: Figure 1, Table 8, and Appendix B.1 in our paper.)

Method and evaluation pointEnvironment samples per taskSix-task mean (%)
EPD7.8343.8
GRPO: best observed model, update 1624029.9
GRPO: final model44826.0

The best-checkpoint ratio uses the unrounded EPD average, 240 ÷ (47/6) = 30.6. The previous 57.2-fold ratio describes full-run expenditure. We will distinguish the best-checkpoint and full-run budgets in the figure and text.

Dear Reviewer 1fkm, We appreciate how busy you are with multiple submissions. If you have a moment before the rebuttal deadline, we would be grateful if you could let us know whether our responses have addressed your concerns.

Reviewer_tfgTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method that transfers the benefits of experience-conditioned decision-making into a model’s parameters by training a student to imitate a teacher that leverages accumulated experience. The goal is to retain the benefits of ICL without repeatedly providing extensive experience at inference time. Experiments on software engineering and text-based adventure tasks show that EPD outperforms standard SFT and approaches experience-conditioned ICL in several settings. The paper also reports favorable sample-efficiency comparisons with selected reinforcement learning baselines. However, the evidence for the broader sample-efficiency claims is less conclusive: the results are tied to a limited set of tasks and baselines, and it remains unclear how much of the gain comes from the distillation objective itself versus the teacher’s experience and training protocol.

Critical strengths
  1. EPD tackles a practical problem: ICL relies on providing past experience at inference time, which can be costly. By distilling experience into the model, EPD avoids the need for extra environment interactions during training the student, making the approach potentially more efficient.
  2. The experiments show that EPD outperforms standard SFT and can approach ICL in several settings. The paper also compares EPD with RL baselines, giving a better picture of its performance and sample efficiency than results against a single baseline would.
  3. The discussion was not limited to few-shot single task learning, but expanded to multi-task and continual learning cases, which I believe is one of the most important issues.
Critical weaknesses
  1. The experiments compare SFT, ICL, and RL baselines but lack comparisons with existing context distillation and in-context reinforcement learning methods, such as Test-Time Training, OPCD or DICP. These comparisons would help clarify how EPD differs from existing approaches and whether it offers better environment-sample efficiency.
  2. EPD seems to rely on a capable teacher and useful experience data, but it is unclear how well it works when the teacher has weak ICL capabilities or the collected trajectories contain many failures. Does EPD still help in these cases, or does it mainly work when sufficient successful trajectories are available? Understanding this would help clarify how practical EPD is in real-world agent settings.
Questions for authors

I believe t there are bunches of missing citations related to Test-Time Training (TTT), please consider add discussion about this topic.

3 comments
AuthorsOpenReview ↗

Reply for Reviewer tfgT

  • Thank you for your feedback. Experience Distillation (EPD) outscored On-Policy Context Distillation (OPCD) on average at the reported budgets, using about 1/28 as many GPU hours and zero additional environment samples during distillation. Models with weaker in-context learning (ICL) also benefited from EPD. The 27B teacher improved final scores most often when changing an action. EPD also outperformed supervised fine-tuning (SFT) when both used only failed Detective episodes.

  • Experimental comparison with OPCD

    • Performance and cost. (New experiment for this rebuttal.) As shown below, EPD outperformed OPCD on average across Detective and Balances at the reported budgets (55.03% vs. 38.81%). EPD used about 1/28 of OPCD’s GPU hours (12.97 vs. 362.81) and required zero additional environment rollouts during distillation, compared with 256 per game for OPCD. Both methods started from the same Qwen3.8-27B checkpoint and used the same five episodes of experience per game. EPD generated teacher replies at recorded decisions. OPCD executed student actions and collected new feedback.

      TaskEPD score (%)EPD GPUhEPD · Env sample cost during distillation (episodes)OPCD score (%)OPCD GPUhOPCD · Env sample cost during distillation (episodes)
      Detective89.249.60055.56229.01256
      Balances20.8316.34022.06496.60256
      Mean55.0312.97038.81362.81256

      Scores are normalized mean peaks over eight independent episodes per game. The mean weights both games equally.

  • EPD with weaker ICL teachers and failure-only experience

    • EPD gains with weaker ICL teachers. (New experiment for this rebuttal.)

      Yes. EPD helped models with weaker ICL, and the larger models with stronger ICL gained more after distillation. On Detective, adding each model’s own experience raised scores over zero-shot by 8.68, 14.24, and 72.57 percentage points for 4B, 9B, and 27B. The EPD gains below follow the same ordering on both games.

      ModelDetective · Zero-shotDetective · after EPD trainingBalances · Zero-shotBalances · after EPD training
      Qwen/Qwen3-4B-Thinking-25079.72%13.54% (+3.82 pp)4.90%7.35% (+2.45 pp)
      Qwen/Qwen3.5-9B15.97%25.35% (+9.38 pp)13.48%18.38% (+4.90 pp)
      Qwen3.8-27B12.15%89.24% (+77.08 pp)14.71%20.83% (+6.13 pp)
    • Teacher action quality. (New experiment for this rebuttal.)

      Teachers with higher Detective ICL scores produced better actions more often on average (12.35% for 4B, 18.39% for 9B, and 27.59% for 27B). With 27B, final scores improved in 27.59% of valid changed-action pairs and fell in 13.22%, averaged equally across the games. The rest tied. For each of the 490 valid changed-action pairs, we tried both actions from the same recorded state, using the same frozen base model and remaining budget to finish the episode.

      ModelDetective (higher-return % / lower-return %)Balances (higher-return % / lower-return %)Average
      Qwen/Qwen3-4B-Thinking-250720.65% / 15.22%4.04% / 2.02%12.35% / 8.62%
      Qwen/Qwen3.5-9B33.85% / 29.23%2.94% / 2.94%18.39% / 16.09%
      Qwen3.8-27B34.69% / 20.41%20.48% / 6.02%27.59% / 13.22%
    • Failure-only experience. (New experiment for this rebuttal.)

      EPD outperformed SFT using only failed Detective episodes (27.08% vs. 11.46%) under the same training budget. Both methods used Qwen3.8-27B and the same first three episodes, all ending in death, with a best peak of 110/360.

      MethodNormalized score (%)
      SFT11.46
      EPD (1 teacher reply per decision)27.08
  • Test-Time Training, TTT-Discover, and DICP. We will cite and discuss these methods in Sections 5.1–5.2 of the camera-ready paper. Section 5.1 already cites OPCD (Ye et al., 2026b), and Section 5.2 discusses in-context reinforcement learning. EPD uses recorded interactions to generate supervision and transfers the resulting gains into weights. Test-Time Training (Sun et al., 2020) updates weights via self-supervision on unlabeled test inputs. TTT-Discover (Yuksekgonul et al., 2026) updates weights using rewards from executing new candidates at test time. DICP (Son et al., 2025) learns dynamics and planning from interaction histories and uses new experience in context with fixed weights at test time. An EPD-trained student can act without the trajectories and reflections used for distillation.

We provide the experimental settings and supporting details in our follow-up comment.


AuthorsOpenReview ↗

Rebuttal experimental settings and supporting details for Reviewer tfgT

Experimental settings and supporting details for Reviewer tfgT

Performance and cost comparison with OPCD

  • Same model and experience. EPD and OPCD start from the same Qwen3.8-27B checkpoint and the same five episodes of experience per game. EPD keeps earlier teacher replies in the generation context and supervises one teacher reply per training example. In the paper’s packed training, a sequence can supervise multiple teacher replies.

  • Evaluation. Both methods use medium reasoning, a 4,096-token reply limit, and at most 100 decisions per episode. We average eight independent evaluations from the start of each game.

  • Training budget and GPU hours. The reported results use 128 updates for EPD and 32 for OPCD. OPCD collects eight new episodes per update. We count H20 GPU hours for target generation and training after shared experience collection, excluding independent evaluation and unassigned startup. EPD adds separately measured target-generation and student-training costs.

EPD gains with weaker ICL teachers

  • Experience collection and ICL scores. We compare three Qwen models with different measured ICL gains on Detective and Balances. Each uses its own trial-and-error experience and reflections from five consecutive attempts for ICL and EPD. On Detective, ICL scores were 18.40%, 30.21%, and 84.72% for 4B, 9B, and 27B, respectively.

  • How we trained with teacher replies. The teacher keeps earlier teacher replies in context when generating targets. We train each model separately on one teacher reply per example for 128 updates, with global batch 16 and learning rate 1e-6.

  • Evaluation. We use each model’s native template and evaluate eight independent episodes from the start of the game, with no cross-episode experience. The reply limit is 8,192 tokens for 4B and 9B and 4,096 tokens for 27B with medium reasoning. We average episode peaks and normalize by the game maximum.

Teacher action quality

  • Which decisions we tested. For each model and game, we randomly sample 128 decision points from the five episodes selected for distillation, covering the episodes and their early, middle, and late stages.

  • How we measure teacher action quality. At each recorded state, the teacher independently proposes an action using the collected experience. We execute the recorded action and the proposed action separately from the same state. The same frozen base model then continues both episodes under the same remaining decision budget, up to 100 decisions for the full episode. Its history contains only the current episode’s actions and environment feedback. We compare final game scores, treating death, zero scores, and budget exhaustion as valid outcomes.

  • How we counted higher, tied, and lower scores. Each state contributes one pair. Of 768 selected states, 754 yield valid pairs, including 490 with changed actions and 264 with unchanged actions. Fourteen teacher targets were incomplete. The table uses the 490 changed-action pairs and retains ties in the denominator. The Detective and Balances denominators are 92 and 99 for 4B, 65 and 102 for 9B, and 49 and 83 for 27B. Extra environment interactions are used only for this evaluation.

Failure-only experience

  • Which failed episodes we used. We use the same first three failed Detective episodes for EPD and SFT. They contain 102 decisions and all end in death. Their peak scores are 110, 20, and 90, with a best peak of 30.56% of the game maximum of 360.

  • What EPD and SFT trained on. EPD trains on one reply per decision, generated by the teacher after reading the episodes and reflections. SFT trains on all recorded reasoning and action replies, with the loss applied to those replies only.

  • Same training budget and evaluation. Both start from Qwen3.8-27B and use 128 updates with global batch 16. We evaluate each in eight independent episodes from the start without the source experience and report the mean peak score normalized by the game maximum.

Dear Reviewer tfgT, We appreciate how busy you are with multiple submissions. If you have a moment before the rebuttal deadline, we would be grateful if you could let us know whether our responses have addressed your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary and New Experimental Results

Dear Reviewers and Area Chairs, thank you for your thoughtful feedback. To complement the original experiments on in-house models, we ran new experiments with three open-weight Qwen models (4B, 9B, and 27B). The results are consistent with our paper’s findings on Experience Distillation (EPD), showing performance gains across model sizes and sample efficiency in our comparison with On-Policy Context Distillation (OPCD).

On TaleSuite games, EPD outperformed supervised fine-tuning (SFT) trained on all collected trajectories or on successful trajectories alone. This advantage also held when both methods used only failed trajectories and in a separate evaluation from unseen states. In our comparison with OPCD on TaleSuite games, EPD achieved a higher mean score at the reported budgets while using substantially fewer GPU hours and requiring no additional environment samples during distillation. The reviewers’ questions guided the comparisons and analyses we added in this rebuttal.

Reviewer tfgT (Weak Accept) asked about related methods, weaker teachers, and failed experience. We compared EPD directly with OPCD, discussed test-time training and DICP, and found that EPD outperformed SFT when both used only failed trajectories.

Reviewer 1fkm (Weak Reject) questioned teacher action quality, generalization, and baseline fairness. We measured downstream returns after executing teacher-generated actions, tested continuation from unseen states, added a success-only SFT baseline under a matched update budget, and clarified how we counted GRPO’s environment samples.

Reviewer wA4g (Weak Reject) raised concerns about the contribution, branch packing, teacher reasoning, reproducibility, and cost. We clarified the method, evaluated branch packing on unseen Detective states, summarized the teacher reasoning ablation in Table 5 of our paper, and documented the versions and configurations of the Qwen models used in the new experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.