acceptodds
Under review as a conference paper at ICLR 2027

Learning Without Remembering: Agent Self-Improvement through Experience Compilation

Abstract

Synthetic interaction data provide a scalable resource for training tool-using agents, yet successful demonstrations may not meet the learning needs of the current student. We investigate how feedback accumulated during generation and learning can be translated into requirements for constructing future training data. We introduce Learning Without Remembering (LWR), an agent learning framework that compiles stage-level experience into synthesis policies. LWR jointly analyzes generation successes and failures alongside changes in student performance and behavior before and after training, distilling synthesis guidelines with explicit applicability conditions that can be reused across examples. These guidelines are translated into generation instructions, workflow allocation, and interaction plans. With the generator and controller weights held fixed, the updated policy guides the generation of complete interaction trajectories, which undergo environment verification before being used for continued supervised fine-tuning of the student. Subsequent generation and learning feedback supports further policy revision. Past experience is thus consolidated into generation requirements and incorporated into student parameter learning through new training data, without retrieving past cross-task trajectories at test time. Experiments on and AppWorld with two student model sizes demonstrate improved task completion. Compared with fixed synthesis, LWR improves macro-average success rates by 5.00 and 6.67 percentage points for the 8B and 30B students, respectively, and yields additional gains on most AppWorld metrics.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLM agents, self-improving agents, continual learning, synthetic data generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qXy1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Learning Without Remembering (LWR), a framework that improves tool-using agents by adapting how their training data are generated. It combines generation successes and failures with changes in student performance and behavior to produce reusable synthesis guidelines with explicit applicability conditions. These guidelines shape subsequent trajectory generation. The teacher and controller weights remain fixed, while the student continues supervised fine-tuning on verified trajectories. At test time, the student does not retrieve past cross-task trajectories. Experiments on τ³ and AppWorld use two Qwen3 student models and compare LWR with several data-construction methods and a fixed synthesis policy. Relative to Fixed synthesis, LWR improves τ³ macro-average success by 5.00 and 6.67 percentage points and improves most AppWorld metrics. A paired analysis of a representative AppWorld run shows both tasks gained and tasks lost relative to Fixed synthesis, as well as partial progress within scenarios.

Critical strengths
  1. The idea is clear and practical. LWR uses feedback from data generation and student learning to improve how future training examples are constructed. This is a useful way to adapt synthetic data to the students' learning needs.

  2. The experiments include useful analysis. The authors evaluate two benchmarks and two model sizes. They report both improvements and regressions, including tasks that LWR solves and tasks it fails to solve compared with Fixed synthesis. This gives a more balanced view of the results.

Critical weaknesses
  1. The experimental setup needs to be clearer. It is unclear how many stages were run, whether the 1,024 trajectories are used per stage or across the whole experiment, and how much maintenance data are added or reused. I would also like to know whether LWR and the baselines use comparable amounts of training data and generation resources. For example, APIGen-MT uses more trajectories on τ³, while AgentRefine is tested on AppWorld without in-domain training. A short comparison table in the main text showing the stages, data amounts, and training and generation budgets would make the results easier to interpret.

  2. The ablation does not show which parts of the method matter most. The comparison with Fixed synthesis suggests that updating the synthesis policy helps, but it does not tell us whether the gains come from generation feedback, student feedback, or particular changes to the generation strategy. Results after each stage would also help show whether repeated updates are useful beyond a single update. A few targeted ablations would make the benefits of these design choices clearer.

Questions for authors
  1. How many stages were run? Please clarify this for each benchmark and model, and whether the main results come from the final stage or a selected checkpoint. Results after each stage would help show whether performance keeps improving.

  2. Are the data and training budgets comparable? Does the 1,024-trajectory count refer to each stage or the whole experiment? Does it include maintenance data, and are those examples reused? A short table in the main text comparing data amounts, training budgets, and generation attempts across methods would help.

  3. How do you decide whether a generated trajectory is suitable for training? Do you only check whether it completes the task correctly, or also whether it shows the behavior that the student is supposed to learn? How many generation attempts do LWR and Fixed synthesis need to collect the same number of accepted trajectories, and how do their costs compare?

  4. Which parts of the method contribute most to the gains? Could you compare generation feedback only, student feedback only, and updating teaching guidance while keeping workflow allocation fixed? Comparing a single policy update with repeated updates under the same total budget would also help show whether multiple stages are needed.

4 comments
AuthorsOpenReview ↗

Response Part 1/4

Q1. How many stages were run, and which stage's results are reported?

We thank the reviewer for the suggestions. The added ALFWorld experiments clarify the stages, budgets, acceptance rules, and design contributions.

The new ALF experiments contain two student-training stages and one feedback-based policy revision:

Direct → train M1 on 1024 shared demonstrations → collect M1's own TRAIN feedback → revise the policy and generate second-stage demonstrations → continue training from M1 to obtain M2.

M1 is trained for 1024 steps and M2 for 1216 steps, with one epoch per stage. For each run, we report the final checkpoint after the prescribed number of training steps, rather than selecting an intermediate checkpoint using independent evaluation scores. KEEP, naive student feedback, and LWR all continue training from the same M1 on their second-stage demonstrations.

Table 1: Stage-wise results on ALFWorld

ModelMethodSeenUnseenOverall
8BBase model8.5713.4310.95
8BShared M172.1477.2474.64
8BKEEP87.7488.0687.90±0.76
8BNaive student feedback87.6287.6987.65±0.28
8BLWR90.8394.1592.46±0.28
30BBase model12.1411.9412.04
30BShared M183.2183.9683.58
30BKEEP90.9591.5491.24±0.63
30BNaive student feedback92.6294.2893.43±0.80
30BLWR92.8696.6494.71±1.32

Scores are percentages. Overall is the success rate across the combined 140 Seen and 134 Unseen tasks. The base-model score is a single-run reference from an earlier evaluation, and the shared M1 is the common first-stage checkpoint. KEEP, naive student feedback, and LWR report second-stage M2 results using training seeds 100, 17, and 57. Each checkpoint is evaluated twice; we first average the two evaluations, then compute the mean and sample standard deviation across the three seeds. The shared M1 and each method's demonstrations are fixed across seeds, without regenerating data.

LWR increases Overall from the shared M1's 74.64% and 83.58% to 92.46% and 94.71% for 8B and 30B, respectively. A more direct policy comparison is between second-stage KEEP and LWR: both continue training and receive new demonstrations, while LWR achieves gains of 4.56 and 3.47 percentage points, positive in all three seeds at both model sizes. The gains therefore go beyond the difference between continued training and no further training. We revise the policy only once in these experiments and then keep it fixed, without further iterations; this single revision already yields the gains above.

AuthorsOpenReview ↗

Response Part 2/4

Q2. How are data volume, maintenance, and cumulative training budgets counted?

In ALF, 1024 refers to the stage-specific demonstrations in each stage, excluding maintenance data. The first stage uses 1024 shared demonstrations. The second uses another 1024 stage-specific demonstrations plus 192 maintenance demonstrations selected from the M1 training data. Maintenance demonstrations are exposed once more in the second stage and are identical, example by example, across methods at each model size.

Table 2: Cumulative training budgets

ModelMethodSecond-stage supervised tokensCumulative supervised tokens
8BAPIGen-MT156,488257,067
8BDataEnvGym151,001251,580
8BAuto Evol-Instruct116,177216,756
8BAgent-World152,700253,279
8BKEEP112,100212,679
8BNaive student feedback113,803214,382
8BLWR110,180210,759
30BAPIGen-MT156,488257,067
30BAuto Evol-Instruct116,177216,756
30BKEEP113,366213,945
30BNaive student feedback115,126215,705
30BLWR106,562207,141

All entries are supervised-token counts. The cumulative column includes the 100,579 supervised tokens used to train the shared M1. Every group uses 1024 demonstrations and 1024 steps in the first stage, followed by 1024 stage-specific demonstrations plus 192 maintenance demonstrations and 1216 steps in the second stage, for a cumulative total of 2240 steps. The optimizer is reset at the start of the second stage. Initialization, sample count, maintenance data, and training steps are matched; actual supervised-token counts are reported above. KEEP, naive student feedback, and LWR also use the same second-stage training worlds.

LWR's 8B stage-specific training set consists of 907 demonstrations generated by its own policy and 117 same-world KEEP fallbacks. For 30B, the corresponding counts are 918 generated demonstrations and 106 fallbacks. Both final sets contain 1024 demonstrations.

The two public baselines highlighted by the reviewer do have different settings. In the original τ³ experiments, APIGen-MT uses 4103 released demonstrations, no maintenance data, and 513 training steps, serving as a reference for SFT on released data. In the original AppWorld experiments, AgentRefine directly evaluates a checkpoint trained on τ data, without in-domain AppWorld training, and is therefore a cross-environment transfer reference. We will label these settings explicitly. The new ALF comparisons are reported separately, using a shared M1 and the same demonstration counts and maintenance budget.

AuthorsOpenReview ↗

Response Part 3/4

Q3. How are demonstrations accepted, and does generation success imply effective teaching?

The new ALF demonstrations are first executed in the actual environment, where success is confirmed by the official task-completion signal. We then check task and interaction records, trajectory completeness, supervisable agent targets, and content duplication. KEEP and LWR use the same basic acceptance and training-example construction rules.

Generation guidance intended only for the teacher is removed before training. Invalid actions for which the environment returns "Nothing happens" are excluded from the supervision loss, although the failure history may remain in the context of subsequent correct actions. Task success and student benefit are assessed through environment validation and independent student evaluation, respectively. Whether the criteria are reflected in the demonstrations is examined through the actual policy text and demonstration action records.

Table 3: Generation opportunities and API requests

ModelMethodGeneration opportunitiesAcceptedAcceptance rateAPI requests
8BKEEP1536116675.91%32,975
8BNaive student feedback1536125781.84%29,380
8BLWR102490788.57%13,719
8BDataEnvGym1560103666.41%45,396
30BNaive student feedback1142103090.19%18,251
30BLWR102491889.65%12,085

One generation opportunity corresponds to an attempt to produce a complete candidate trajectory; API requests include failures and retries. For 8B, KEEP and naive student feedback each make one attempt in the same 1536 candidate worlds. We then select 1024 worlds with accepted demonstrations from both methods in a frozen order. LWR generates 907 accepted demonstrations in these worlds, and the remaining 117 positions reuse KEEP demonstrations from the same worlds. For 30B, the corresponding counts are 918 newly generated demonstrations and 106 fallbacks.

Generating the shared M1 training data required an additional 29,675 API requests. The second-stage data for the 8B KEEP, naive student-feedback, and LWR comparisons required 76,074 requests in total, or 105,749 including M1. The newly generated naive-feedback and LWR data for 30B required 30,336 requests in total, while reusing the recorded M1 and KEEP sources.

Within the same candidate pool, the 8B naive-feedback baseline raises generation acceptance from KEEP's 75.91% to 81.84%, yet student Overall changes from 87.90% to 87.65%. The final training worlds and demonstration counts are the same. This shows that a higher teacher task-completion rate does not automatically produce demonstrations that are more useful for student learning; post-training student performance must also be examined.

We also examined actual decision changes in the demonstrations and student trajectories. Across the same 1024 worlds for 8B, LWR and KEEP differ in their complete action sequences in 718 demonstrations. The number of demonstrations containing invalid actions decreases from 325 to 201. A further comparison against naive feedback with matched fallback reveals the following changes in cooling tasks.

Table 4: Decision changes in 8B cooling demonstrations

MetricNaive student feedback + matched fallbackLWR
Demonstrations with successful cooling immediately after opening the fridge1980
Demonstrations containing invalid actions6011
Total invalid actions6618
Mean step of first picking up the target object6.414.89

The analysis covers 106 pairs of cooling demonstrations whose action sequences differ after matching the 117 KEEP fallback positions. The first three rows count demonstrations or actions. The final row reports the mean action step at which the object subsequently cooled is first picked up.

On 46 independent cooling tasks, the seed100 student solves 4 more tasks than the matched-fallback control, with the same outcome in both evaluation runs. The key progress is first identifying and finding the target object that the control did not pick up, then completing cooling and placement. These records show concrete changes in both the demonstrations and the student's decisions.

AuthorsOpenReview ↗

Response Part 4/4

Q4. Which design choices contribute to the gains, and are repeated updates necessary?

We have added a naive-feedback control that is more direct than Fixed: it constructs a reusable teaching plan from M1's TRAIN task outcomes and trajectories, then appends the plan to the initial policy. LWR instead combines student records with training-side KEEP teacher demonstrations to recompile the full generation policy, specifying applicable states, key actions, and recovery and stopping conditions.

The core comparison fixes M1, the 1024 second-stage training worlds, task-family composition, maintenance data, and training steps. What changes is the teaching guidance and generation process within the same worlds. LWR's mean paired gain over naive student feedback is 4.81±0.11 percentage points for 8B and 1.28±2.01 percentage points for 30B, with positive gains in three and two seeds, respectively.

To separate the effect of fallback, we replace the naive-feedback demonstrations at the same positions where LWR falls back with the identical KEEP demonstrations, yielding the following three-seed results.

Table 5: Fallback ablation

ModelNaive student feedbackMatched fallbackLWRPaired gain
8B87.65±0.2889.17±0.6492.46±0.28+3.28±0.48
30B93.43±0.8091.97±0.9794.71±1.32+2.74±2.03

"Matched fallback" replaces the naive-feedback demonstrations at exactly the positions where LWR falls back with the same KEEP demonstrations, leaving all other positions unchanged. This replaces 117 and 106 positions for 8B and 30B, respectively. The three method columns report Overall success rates in percent. Paired gain is LWR minus matched fallback, in percentage points. All values are means ± sample standard deviations across three seeds; paired gains are computed within each seed before aggregation.

At both model sizes, LWR outperforms the matched-fallback control in all three seeds. Fallback helps naive student feedback at 8B but lowers its score at 30B; fallback alone therefore cannot explain LWR's gains. In addition, removing old maintenance data at 8B still leaves a gain of 3.47±0.63 percentage points over KEEP. Replacing maintenance positions with repeated exposures to new demonstrations from the same task families, while retaining 1216 steps, yields a gain of 4.14±0.53 percentage points. Both settings show positive gains in all three seeds. These controls support the overall contribution of feedback compilation, distinguish the effects of fallback and maintenance, and demonstrate LWR's effectiveness.

We will include the stage settings, full training and generation budgets, acceptance rules, and these ablations in the revised manuscript to clarify how demonstrations are generated and used for training, and which comparisons support the method's contribution.

Reviewer_tmvkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Learning Without Remembering (LWR), a framework for generating SFT data for tool-using agents. The method involves a controller, a teacher, and a student. The controller uses generation results and changes in student performance to update the data generation guidelines. The teacher follows these guidelines to produce new interaction trajectories, and the student learns from the verified trajectories through SFT. The authors evaluate LWR on τ3\tau^3 and AppWorld with two student model sizes and compare it with several data generation methods.

Critical strengths
  1. The paper is clearly written and well organized.
  2. The experiments are extensive, including multiple benchmarks, model scales, ablation studies, and analyses of synthetic data quality.
Critical weaknesses
  1. It is unclear whether the observed gains reflect transferable agent capabilities rather than adaptation to benchmark-specific task structures. LWR trains the student through SFT on teacher-generated trajectories, supplemented with maintenance data. This likelihood-based imitation objective does not directly optimize task outcomes under student-generated rollouts. Since the main implementation generates demonstrations within existing training tasks, the improvements could partly reflect learning recurring workflows shared between training and evaluation, rather than acquiring more general interaction and problem-solving capabilities. This is a possibility that requires investigation, not evidence that SFT cannot generalize. The authors should analyze train--test similarity in task structure and isolate the contribution of maintenance data through an ablation, including disclosure of its source, size, and composition.

  2. The experiments do not sufficiently establish whether the comparisons are matched in cumulative training budget and total data-generation cost. Fixed synthesis already generates fresh trajectories or environments under an unchanged policy, and both settings organize synthetic data into sets of 1,024 trajectories; therefore, the concern is not that Fixed necessarily reuses a frozen dataset. Rather, equal trajectory counts per set do not establish equal cumulative exposure across stages. The paper specifies one epoch per stage and mentions separate tracking of maintenance data and supervised tokens, but does not provide a complete accounting of cumulative training or controller and feedback-collection costs. A stronger comparison would allow baseline methods to refresh and expand their datasets through additional teacher rollouts while matching LWR's cumulative student-training budget and maintenance-data allocation. A separate equal-total-cost comparison should allow simpler methods to redirect the resources spent on feedback analysis and controller inference toward additional sampling and training. These controls would help establish whether LWR produces more useful training data, rather than gains that could also be obtained by scaling a simpler pipeline.

  3. The evaluation covers only τ3\tau^3 and AppWorld, leaving the applicability of the method to other agent tasks insufficiently explored. Additional experiments should include tool-using mathematical reasoning and repository-level software engineering, with explicit evaluation on held-out problem families, repositories, or unfamiliar tool compositions. Relevant mathematical reasoning settings include [1,2], which combine reasoning with external computational tools or interactive code execution. Relevant coding settings include [3] where agents navigate repositories, edit files, and execute tests, and [4] which provides executable software-engineering training environments and evaluates trained agents on SWE-bench. Extending LWR to these settings, with explicitly controlled train--test separation, would provide stronger evidence that its feedback-driven synthesis improves generalization beyond the workflows represented in its current training environments.

[1] ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving

[2] ReTool: Reinforcement Learning for Strategic Tool Use in LLMs

[3] SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering

[4] Training Software Engineering Agents and Verifiers with SWE-Gym

Questions for authors
  1. The maintenance data are insufficiently specified. Could the authors clarify their source, composition, size, and mixing ratio with newly generated trajectories? Why are they needed, and how much do they contribute to the reported performance? A controlled ablation with and without maintenance data would help distinguish their contribution from that of feedback-driven synthesis.
  2. See weaknesses.
5 comments
AuthorsOpenReview ↗

Response Part 1/4

Q1. Do the gains merely reflect adaptation to recurring benchmark workflows?

We thank the reviewer for the suggestions. We added ALFWorld results at both model sizes, structural analysis, maintenance ablations, and budget accounting.

For each model size, we train a shared M1 on 1024 demonstrations and collect its feedback on 120 TRAIN worlds. All second-stage methods start from that M1 and train for 1216 steps on 1024 stage-specific demonstrations plus the same 192 maintenance demonstrations. KEEP generates new demonstrations without changing its policy.

Table 1: Main results on ALFWorld

ModelMethodSeenUnseenOverall
8BBase model8.5713.4310.95
8BShared M172.1477.2474.64
8BAPIGen-MT81.6792.7987.10±0.56
8BDataEnvGym80.0087.8183.82±0.86
8BAuto Evol-Instruct89.2990.6789.96±0.00
8BAgent-World79.5290.1784.73±0.59
8BKEEP87.7488.0687.90±0.76
8BNaive student feedback87.6287.6987.65±0.28
8BLWR90.8394.1592.46±0.28
30BBase model12.1411.9412.04
30BShared M183.2183.9683.58
30BAPIGen-MT81.4392.0486.62±1.32
30BAuto Evol-Instruct92.8695.1593.98±0.80
30BKEEP90.9591.5491.24±0.63
30BNaive student feedback92.6294.2893.43±0.80
30BLWR92.8696.6494.71±1.32

Scores are percentages over 140 Seen and 134 Unseen tasks; Overall pools all 274 tasks. Overall is reported as mean ± sample SD across training seeds 100, 17, and 57, after averaging two evaluations per checkpoint. M1 and each method's data are fixed across seeds. Base is an earlier single-run reference; M1 is a shared checkpoint, so neither has an across-seed SD. All second-stage methods share M1, demonstration count, maintenance data, and steps. KEEP, naive feedback, and LWR also share the same 1024 worlds; public baselines use different world pools.

LWR's Unseen gains over KEEP are 6.09 and 5.10 percentage points for 8B and 30B. Its paired Overall gains are 4.56±1.02 and 3.47±1.93 points, positive in all three seeds at both sizes. Mean gains over naive feedback are 4.81 and 1.28 points; over Auto Evol-Instruct, 2.49 and 0.73 points.

SFT updates the student; feedback changes how new demonstrations are generated. We also audited structural reuse:

Table 2: Structural overlap between training and evaluation for 8B

Evaluation splitTasksDuplicate game filesTasks with seen composition templatesTasks in seen rooms
Seen1400130140
Unseen13401160

Here, "seen" means present in cumulative training. M1 and second-stage data span 2048 disjoint TRAIN worlds and 508 templates; maintenance reuses M1 data. A template is "task family + target-object type + intermediate receptacle + destination type," excluding room and world IDs. We check game files by SHA and rooms by official IDs. TRAIN120 feedback worlds also share no identical game files with the 274 evaluation tasks.

KEEP and LWR share training worlds and template coverage. Using this fixed definition, we stratified the 8B evaluations:

Table 3: 8B results stratified by task-composition template

Task groupTasksKEEPLWRPaired gain
Seen compositions24687.60±0.5492.01±0.31+4.40±0.85
Unseen compositions2890.48±4.4996.43±0.00+5.95±4.49

Scores are percentages, and paired gains are in percentage points. All values are means ± sample standard deviations across three training seeds. The unseen-composition group includes 10 Seen tasks and 18 Unseen tasks; grouping is based on the fixed task fields.

LWR improves on both seen and unseen compositions despite identical template coverage. The gains therefore extend to new worlds and cannot be explained solely by covering more evaluation templates.

AuthorsOpenReview ↗

Response Part 2/4

Q2. What are the maintenance data, and how much do they contribute?

Maintenance data come from the accepted demonstrations used to train the shared M1. We select 192 complete task trajectories in proportion to the task-family frequencies in M1's training data and mix them with the 1024 second-stage demonstrations. These are not evaluation data and require no additional teacher generation. They have already been used in the M1 stage and are exposed once more in the second stage.

Table 4: Second-stage data composition

DataDemonstrationsProportion
Stage-specific demonstrations102484.21%
Maintenance demonstrations19215.79%
Total1216100%

Proportions are based on trajectory counts. Each demonstration is a complete task interaction trajectory. KEEP, naive student feedback, and LWR use identical maintenance demonstrations at each model size, with the same mixing procedure and training settings.

We have completed three-seed maintenance ablations for 8B. One removes the old maintenance demonstrations entirely. The other replaces the original 192 maintenance positions with repeated exposures to new demonstrations from the same task families, retaining 1216 training steps. Both changes are applied symmetrically to KEEP and LWR.

Table 5: Maintenance-data ablation for 8B

SettingTraining stepsKEEPLWRPaired gain
Old-data maintenance121687.90±0.7692.46±0.28+4.56±1.02
No maintenance102488.08±0.5691.55±0.21+3.47±0.63
New-data replay121688.75±0.6992.88±0.18+4.14±0.53

Scores are Overall success rates in percent. Paired gain is LWR minus KEEP under the same setting, in percentage points. All values are means ± sample standard deviations across three training seeds. All settings fix the shared M1, each method's 1024 stage-specific demonstrations, and their relative order. "Old-data maintenance" adds 192 M1 demonstrations; "no maintenance" removes them; "new-data replay" repeats new demonstrations from the same task families at the original 192 maintenance positions. These repeated exposures do not count as additional independent data.

LWR outperforms KEEP in all three seeds under all three settings. Its gain remains 3.47 points without maintenance and 4.14 points when new-data replay matches the 1216-step budget.

For LWR, old-data maintenance exceeds no maintenance by 0.91 points, but removing maintenance also changes the step count and learning-rate schedule. Same-step new-data replay exceeds old-data maintenance by 0.43 points. Thus, LWR's gains do not depend on old maintenance data.

AuthorsOpenReview ↗

Response Part 3/4

Q3. Are cumulative training budgets and total generation costs comparable?

The new ALF experiments use a shared M1 and explicitly align both the second-stage training schedule and the cumulative schedule starting from Direct:

Direct → train M1 on 1024 shared demonstrations → collect M1's own TRAIN feedback → generate second-stage demonstrations → continue training from the same M1.

The first stage uses 1024 demonstrations and 1024 steps. The second uses 1024 stage-specific demonstrations plus 192 shared maintenance demonstrations and 1216 steps, for a cumulative total of 2240 steps. Each stage trains for one epoch, and the optimizer is reset at the start of the second stage. KEEP also generates new demonstrations for the second stage. We revise the policy once based on feedback and then keep it fixed for generation.

Table 6: Cumulative student-training budgets

ModelMethodSecond-stage supervised tokensCumulative supervised tokens
8BAPIGen-MT156,488257,067
8BDataEnvGym151,001251,580
8BAuto Evol-Instruct116,177216,756
8BAgent-World152,700253,279
8BKEEP112,100212,679
8BNaive student feedback113,803214,382
8BLWR110,180210,759
30BAPIGen-MT156,488257,067
30BAuto Evol-Instruct116,177216,756
30BKEEP113,366213,945
30BNaive student feedback115,126215,705
30BLWR106,562207,141

"Cumulative" includes the 100,579 supervised tokens used to train the shared M1. All groups share the same initialization, number of stages, demonstration count, maintenance data, and training steps. Supervised-token counts are reported as actually measured.

Thus, LWR receives neither more cumulative training steps nor more supervised tokens, yet achieves higher mean independent-evaluation scores at both model sizes. Together with the maintenance ablations, this supports the additional learning value of its new demonstrations under this student-training budget.

On the generation side, we count opportunities and API requests from actual request logs, including failures, format repairs, and retries. The main generation sources are listed below.

Table 7: Generation opportunities and API-request budgets

SourceMethodGeneration opportunitiesAcceptedPolicy compilationAPI requests
SharedShared M113681027029,675
SharedAPIGen-MT28261026077,036
8BDataEnvGym15601036045,396
SharedAuto Evol-Instruct134410325928,862
8BAgent-World16091026648,614
8B sourceKEEP15361166032,975
8BNaive student feedback153612571129,380
8BLWR1024907713,719
30BNaive student feedback114210301018,251
30BLWR1024918712,085

One generation opportunity corresponds to an attempt to produce a complete candidate trajectory. "Accepted" counts demonstrations passing each method's acceptance checks; these are subsequently deduplicated and selected in a frozen order. For example, among the 1026 accepted APIGen-MT demonstrations, 2 are duplicates after compilation, leaving 1024 for training. "Policy compilation" is a subset of API requests; 0 means no separately recorded policy-compilation requests. "Shared" sources are reused across the two model sizes or multiple seeds, and only requests that actually occurred are counted. The paired KEEP demonstrations for 30B also reuse the recorded KEEP source.

For 8B, KEEP and naive student feedback each make one attempt in the same 1536 candidate worlds. We then select 1024 worlds with accepted demonstrations from both methods for the core comparison. LWR produces 907 accepted demonstrations in these worlds, and the remaining 117 positions use KEEP demonstrations from the same worlds. For 30B, we use its own M1 feedback and obtain 918 newly generated demonstrations and 106 same-world KEEP fallbacks. Every final stage-specific training set contains 1024 demonstrations; fallbacks do not count as successful LWR generations.

Feedback collection uses each model's M1 on TRAIN120, producing 2898 and 2283 local environment action steps for 8B and 30B, respectively. These counts measure student execution. Repeating training with different seeds or evaluating the same checkpoint does not regenerate demonstrations. We align the cumulative student-training schedule and report the actual differences in supervised tokens and generation requests.

Reviewer_AqadOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LWR uses generation outcomes and changes in a student's behavior to revise the requirements for its next training data. A fixed controller turns this feedback into reusable synthesis guidelines, and a fixed teacher generates environment-verified trajectories for student fine-tuning. The experiments use two Qwen3 students on tau3 and AppWorld, comparing LWR with fixed synthesis and adapted data-generation methods. At test time, the student uses its learned parameters without retrieving earlier cross-task trajectories.

Critical strengths

1. Some gains over fixed synthesis are substantial. For the 8B student, AppWorld Normal task completion rises from 23.8% to 47.0%; for 30B, tau3 macro success rises from 45.0% to 51.67%. These results make feedback-guided demonstration construction worth studying, even though the gains vary across models and metrics.

2. The training/test separation is clear: policy feedback comes from TRAIN tasks, and generation-only guidance is removed before student training and evaluation. The paper also reports domain regressions and task/scenario completion separately, which helps show both the benefits and the tradeoffs.

Critical weaknesses

1. The added contribution over earlier feedback-driven synthesis is not established experimentally. DataEnvGym already separates planning from generation in an iterative teacher-student loop, and CoEvolve uses behavioral feedback to guide validated task generation. LWR's joint feedback and scoped guidelines are plausible additions, but Fixed removes all feedback-driven policy updates. It does not show that these additions outperform a simpler student-feedback teaching plan.

2. I couldn't reconstruct the learning schedule or total budget. Appendix A reports 1024 accepted trajectories and one epoch per stage, but not how many stages ran or the maintenance-data sizes. Cumulative supervised-token and generation-call totals are also missing. This makes it hard to check whether LWR and Fixed receive comparable total training, or what repeated policy revision contributes.

3. The DataEnvGym comparison also depends on adapter choices. LWR reuses family-level plans and falls back to Fixed when a plan is invalid; DataEnvGym uses per-example proposals, and invalid proposals consume generation opportunities (Appendix C.1). These choices may affect the accepted data and their generation cost. The paper gives no invalid-output rates or comparison under a shared fallback rule.

4. The three-seed means come without per-seed results or uncertainty estimates. This matters for the smaller, mixed results: 30B AppWorld Normal TGC improves by 3.5 points over Fixed, while Challenge TGC drops by 1.9 points. Figure 3 covers one representative 8B run, so it doesn't establish how consistently these gains and tradeoffs occur.

Questions for authors

1. With the same training and generation budget, how does LWR compare with a reusable plan built from student feedback alone? What changes when generation feedback or pre/post-training differences are removed?

2. How many complete synthesis-training-feedback stages produced each result? Please give per-stage performance and data sizes, maintenance-data counts, total supervised tokens, and teacher/controller calls for LWR and Fixed. Does a plan compiled once and then kept fixed perform similarly?

3. How often are LWR plans and DataEnvGym proposals invalid, and how many generation opportunities produce the 1024 accepted trajectories? Does their AppWorld comparison change under the same retry/fallback rule and generation budget?

4. Could you report per-seed LWR and Fixed results with uncertainty estimates, using paired differences where appropriate? Do the 30B Normal gain, Challenge decline, and Retail/Telecom tradeoff persist across seeds?

7 comments
AuthorsOpenReview ↗

Response Part 1/4

Q1. Does LWR provide additional benefits over reusable plans based only on student feedback?

We thank the reviewer for the suggestions. We added ALFWorld experiments at both model sizes, with naive-feedback and fallback controls and three training seeds.

We first train a shared M1 for each model size on 1024 demonstrations, then collect feedback from that M1 on TRAIN tasks. In the second stage, all methods start from the same M1 for that model size and train for 1216 steps on 1024 stage-specific demonstrations plus exactly the same 192 maintenance demonstrations.

Table 1: Main results on ALFWorld

ModelMethodSeenUnseenOverall
8BBase model8.5713.4310.95
8BShared M172.1477.2474.64
8BAPIGen-MT81.6792.7987.10±0.56
8BDataEnvGym80.0087.8183.82±0.86
8BAuto Evol-Instruct89.2990.6789.96±0.00
8BAgent-World79.5290.1784.73±0.59
8BKEEP87.7488.0687.90±0.76
8BNaive student feedback87.6287.6987.65±0.28
8BLWR90.8394.1592.46±0.28
30BBase model12.1411.9412.04
30BShared M183.2183.9683.58
30BAPIGen-MT81.4392.0486.62±1.32
30BAuto Evol-Instruct92.8695.1593.98±0.80
30BKEEP90.9591.5491.24±0.63
30BNaive student feedback92.6294.2893.43±0.80
30BLWR92.8696.6494.71±1.32

All scores are percentages. Overall is the success rate across the combined 140 Seen and 134 Unseen tasks. The value after "±" is the sample standard deviation across three training seeds; for each seed, we first average two evaluations of the same checkpoint. The base-model score is a single-run reference from an earlier evaluation, and M1 is a shared checkpoint, so neither has an across-seed standard deviation. All second-stage methods use the same M1 initialization, number of demonstrations, maintenance data, and training steps. KEEP, naive student feedback, and LWR also use the same 1024 training worlds; the world pools used by the public baselines are not identical to this set.

The naive student-feedback baseline constructs a reusable teaching plan from M1's task outcomes and execution trajectories, then appends it to the initial generation policy. KEEP leaves the initial policy unchanged while generating new demonstrations. LWR combines student execution records with teacher demonstrations from the training split to recompile the generation policy, specifying applicable states, key actions, and recovery and stopping conditions.

For 8B and 30B, LWR improves over naive student feedback by an average of 4.81 and 1.28 percentage points, respectively, and over KEEP by 4.56 and 3.47 percentage points. LWR achieves higher mean scores than these controls at both model sizes, and exceeds Auto Evol-Instruct by 2.49 and 0.73 percentage points, respectively. These results demonstrate an additional benefit from LWR.

AuthorsOpenReview ↗

Response Part 2/4

Q2. How many stages were actually run, and what were the maintenance-data and cumulative training budgets?

The ALF experiments have two training stages and one policy revision: Direct → train M1 on shared demonstrations → collect M1's own TRAIN feedback → revise the policy and generate second-stage demonstrations → continue training from M1 to obtain M2. The first stage uses 1024 demonstrations and 1024 steps. The second uses 1024 stage-specific demonstrations plus 192 maintenance demonstrations and 1216 steps, for a cumulative total of 2240 steps. Each stage trains for one epoch, and the optimizer is reset at the start of the second stage. Maintenance demonstrations are selected from the M1 training data in proportion to task-family frequencies and are identical, example by example, across methods at each model size.

Table 2: Supervised-token budgets

ModelMethodSecond stageCumulative
8BKEEP112,100212,679
8BNaive student feedback113,803214,382
8BLWR110,180210,759
30BKEEP113,366213,945
30BNaive student feedback115,126215,705
30BLWR106,562207,141

All entries are supervised-token counts. "Cumulative" includes the 100,579 supervised tokens used to train the shared M1 in the first stage. The methods are matched on initialization, sample count, maintenance data, and update steps; their actual supervised-token counts are reported above.

We revise the policy only once in these experiments and then keep it fixed, without further iterations. This single revision already yields the gains reported above. We will add multi-stage experiments for further comparison.


AuthorsOpenReview ↗

Response Part 3/4

Q3. Do invalid outputs, generation budgets, and fallback rules affect the comparison?

We re-examined AppWorld generation records and added ALFWorld fallback experiments. In AppWorld, DataEnvGym received additional generation opportunities; LWR did not actually use Fixed fallback.

Initially, each method was allocated 2040 generation opportunities at each student model size. After environment validation, length filtering, and deduplication, LWR and DataEnvGym produced 1125 and 883 accepted demonstrations for 8B, corresponding to yields of 55.15% and 43.28%. For 30B, the corresponding counts were 1185 and 812, with yields of 58.09% and 39.80%. Thus, under the same initial opportunity budget, LWR produced more accepted demonstrations at both model sizes and already met the target of 1024 new training demonstrations.

During planning, 403/2040 and 541/2040 DataEnvGym proposals were rejected for 8B and 30B, respectively, corresponding to rejection rates of 19.75% and 26.52%. These invalid proposals still counted against the opportunity budget. LWR compiled plans at the task-family level, and all 30 plans at each model size passed validation. Fallback for invalid plans was therefore never triggered.

To avoid disadvantaging DataEnvGym because of its shortfall in accepted demonstrations, we provided additional generation opportunities, adding 141 and 212 accepted demonstrations for 8B and 30B, respectively, to reach 1024 new training demonstrations in each case. For 30B, this required 525 additional opportunities, bringing the total to 2565. LWR, by contrast, selected 1024 demonstrations directly from the accepted pool produced within its original 2040 opportunities, without additional generation. All supplementary demonstrations were generated by DataEnvGym, without Fixed fallback or duplicate examples used to fill the shortfall.

The completed comparison therefore uses DataEnvGym's expanded 1024-demonstration set, matching the final training-set size while giving DataEnvGym additional generation opportunities.

As a supplementary experiment, we directly compared DataEnvGym, the original LWR setting, and LWR without fallback on ALFWorld with the 8B model. All three start from the same M1 and train for 1216 steps on 1024 stage-specific demonstrations plus the same 192 maintenance demonstrations. The no-fallback variant retains the original 907 LWR demonstrations and generates additional accepted demonstrations in the remaining 117 worlds, replacing the KEEP demonstrations previously used as fallback.

Table 3: Supplementary ALFWorld 8B results

ModelMethodSeenUnseenOverall
8BDataEnvGym81.0787.3184.12
8BLWR, original fallback setting90.7194.4092.52
8BLWR, all 1024 generated, no fallback90.0094.0391.97

Table 3 reports results for training seed 100, whereas Table 1 reports averages across three training seeds. All scores are percentages; each score in Table 3 averages two complete evaluations of the same checkpoint. LWR without fallback achieves an Overall score of 91.97%, still 7.85 percentage points above DataEnvGym, with a difference of -0.55 percentage points relative to the original fallback setting. The two LWR settings achieve similar scores, and both retain an advantage over DataEnvGym.

We will add these generation statistics and supplementary results to clarify the comparison conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.