- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Learning Without Remembering (LWR), a framework that improves tool-using agents by adapting how their training data are generated. It combines generation successes and failures with changes in student performance and behavior to produce reusable synthesis guidelines with explicit applicability conditions. These guidelines shape subsequent trajectory generation. The teacher and controller weights remain fixed, while the student continues supervised fine-tuning on verified trajectories. At test time, the student does not retrieve past cross-task trajectories. Experiments on τ³ and AppWorld use two Qwen3 student models and compare LWR with several data-construction methods and a fixed synthesis policy. Relative to Fixed synthesis, LWR improves τ³ macro-average success by 5.00 and 6.67 percentage points and improves most AppWorld metrics. A paired analysis of a representative AppWorld run shows both tasks gained and tasks lost relative to Fixed synthesis, as well as partial progress within scenarios.
Critical strengths
-
The idea is clear and practical. LWR uses feedback from data generation and student learning to improve how future training examples are constructed. This is a useful way to adapt synthetic data to the students' learning needs.
-
The experiments include useful analysis. The authors evaluate two benchmarks and two model sizes. They report both improvements and regressions, including tasks that LWR solves and tasks it fails to solve compared with Fixed synthesis. This gives a more balanced view of the results.
Critical weaknesses
-
The experimental setup needs to be clearer. It is unclear how many stages were run, whether the 1,024 trajectories are used per stage or across the whole experiment, and how much maintenance data are added or reused. I would also like to know whether LWR and the baselines use comparable amounts of training data and generation resources. For example, APIGen-MT uses more trajectories on τ³, while AgentRefine is tested on AppWorld without in-domain training. A short comparison table in the main text showing the stages, data amounts, and training and generation budgets would make the results easier to interpret.
-
The ablation does not show which parts of the method matter most. The comparison with Fixed synthesis suggests that updating the synthesis policy helps, but it does not tell us whether the gains come from generation feedback, student feedback, or particular changes to the generation strategy. Results after each stage would also help show whether repeated updates are useful beyond a single update. A few targeted ablations would make the benefits of these design choices clearer.
Questions for authors
-
How many stages were run? Please clarify this for each benchmark and model, and whether the main results come from the final stage or a selected checkpoint. Results after each stage would help show whether performance keeps improving.
-
Are the data and training budgets comparable? Does the 1,024-trajectory count refer to each stage or the whole experiment? Does it include maintenance data, and are those examples reused? A short table in the main text comparing data amounts, training budgets, and generation attempts across methods would help.
-
How do you decide whether a generated trajectory is suitable for training? Do you only check whether it completes the task correctly, or also whether it shows the behavior that the student is supposed to learn? How many generation attempts do LWR and Fixed synthesis need to collect the same number of accepted trajectories, and how do their costs compare?
-
Which parts of the method contribute most to the gains? Could you compare generation feedback only, student feedback only, and updating teaching guidance while keeping workflow allocation fixed? Comparing a single policy update with repeated updates under the same total budget would also help show whether multiple stages are needed.
4 comments
Response Part 2/4
Q2. How are data volume, maintenance, and cumulative training budgets counted?
In ALF, 1024 refers to the stage-specific demonstrations in each stage, excluding maintenance data. The first stage uses 1024 shared demonstrations. The second uses another 1024 stage-specific demonstrations plus 192 maintenance demonstrations selected from the M1 training data. Maintenance demonstrations are exposed once more in the second stage and are identical, example by example, across methods at each model size.
Table 2: Cumulative training budgets
| Model | Method | Second-stage supervised tokens | Cumulative supervised tokens |
|---|---|---|---|
| 8B | APIGen-MT | 156,488 | 257,067 |
| 8B | DataEnvGym | 151,001 | 251,580 |
| 8B | Auto Evol-Instruct | 116,177 | 216,756 |
| 8B | Agent-World | 152,700 | 253,279 |
| 8B | KEEP | 112,100 | 212,679 |
| 8B | Naive student feedback | 113,803 | 214,382 |
| 8B | LWR | 110,180 | 210,759 |
| 30B | APIGen-MT | 156,488 | 257,067 |
| 30B | Auto Evol-Instruct | 116,177 | 216,756 |
| 30B | KEEP | 113,366 | 213,945 |
| 30B | Naive student feedback | 115,126 | 215,705 |
| 30B | LWR | 106,562 | 207,141 |
All entries are supervised-token counts. The cumulative column includes the 100,579 supervised tokens used to train the shared M1. Every group uses 1024 demonstrations and 1024 steps in the first stage, followed by 1024 stage-specific demonstrations plus 192 maintenance demonstrations and 1216 steps in the second stage, for a cumulative total of 2240 steps. The optimizer is reset at the start of the second stage. Initialization, sample count, maintenance data, and training steps are matched; actual supervised-token counts are reported above. KEEP, naive student feedback, and LWR also use the same second-stage training worlds.
LWR's 8B stage-specific training set consists of 907 demonstrations generated by its own policy and 117 same-world KEEP fallbacks. For 30B, the corresponding counts are 918 generated demonstrations and 106 fallbacks. Both final sets contain 1024 demonstrations.
The two public baselines highlighted by the reviewer do have different settings. In the original τ³ experiments, APIGen-MT uses 4103 released demonstrations, no maintenance data, and 513 training steps, serving as a reference for SFT on released data. In the original AppWorld experiments, AgentRefine directly evaluates a checkpoint trained on τ data, without in-domain AppWorld training, and is therefore a cross-environment transfer reference. We will label these settings explicitly. The new ALF comparisons are reported separately, using a shared M1 and the same demonstration counts and maintenance budget.
Response Part 3/4
Q3. How are demonstrations accepted, and does generation success imply effective teaching?
The new ALF demonstrations are first executed in the actual environment, where success is confirmed by the official task-completion signal. We then check task and interaction records, trajectory completeness, supervisable agent targets, and content duplication. KEEP and LWR use the same basic acceptance and training-example construction rules.
Generation guidance intended only for the teacher is removed before training. Invalid actions for which the environment returns "Nothing happens" are excluded from the supervision loss, although the failure history may remain in the context of subsequent correct actions. Task success and student benefit are assessed through environment validation and independent student evaluation, respectively. Whether the criteria are reflected in the demonstrations is examined through the actual policy text and demonstration action records.
Table 3: Generation opportunities and API requests
| Model | Method | Generation opportunities | Accepted | Acceptance rate | API requests |
|---|---|---|---|---|---|
| 8B | KEEP | 1536 | 1166 | 75.91% | 32,975 |
| 8B | Naive student feedback | 1536 | 1257 | 81.84% | 29,380 |
| 8B | LWR | 1024 | 907 | 88.57% | 13,719 |
| 8B | DataEnvGym | 1560 | 1036 | 66.41% | 45,396 |
| 30B | Naive student feedback | 1142 | 1030 | 90.19% | 18,251 |
| 30B | LWR | 1024 | 918 | 89.65% | 12,085 |
One generation opportunity corresponds to an attempt to produce a complete candidate trajectory; API requests include failures and retries. For 8B, KEEP and naive student feedback each make one attempt in the same 1536 candidate worlds. We then select 1024 worlds with accepted demonstrations from both methods in a frozen order. LWR generates 907 accepted demonstrations in these worlds, and the remaining 117 positions reuse KEEP demonstrations from the same worlds. For 30B, the corresponding counts are 918 newly generated demonstrations and 106 fallbacks.
Generating the shared M1 training data required an additional 29,675 API requests. The second-stage data for the 8B KEEP, naive student-feedback, and LWR comparisons required 76,074 requests in total, or 105,749 including M1. The newly generated naive-feedback and LWR data for 30B required 30,336 requests in total, while reusing the recorded M1 and KEEP sources.
Within the same candidate pool, the 8B naive-feedback baseline raises generation acceptance from KEEP's 75.91% to 81.84%, yet student Overall changes from 87.90% to 87.65%. The final training worlds and demonstration counts are the same. This shows that a higher teacher task-completion rate does not automatically produce demonstrations that are more useful for student learning; post-training student performance must also be examined.
We also examined actual decision changes in the demonstrations and student trajectories. Across the same 1024 worlds for 8B, LWR and KEEP differ in their complete action sequences in 718 demonstrations. The number of demonstrations containing invalid actions decreases from 325 to 201. A further comparison against naive feedback with matched fallback reveals the following changes in cooling tasks.
Table 4: Decision changes in 8B cooling demonstrations
| Metric | Naive student feedback + matched fallback | LWR |
|---|---|---|
| Demonstrations with successful cooling immediately after opening the fridge | 19 | 80 |
| Demonstrations containing invalid actions | 60 | 11 |
| Total invalid actions | 66 | 18 |
| Mean step of first picking up the target object | 6.41 | 4.89 |
The analysis covers 106 pairs of cooling demonstrations whose action sequences differ after matching the 117 KEEP fallback positions. The first three rows count demonstrations or actions. The final row reports the mean action step at which the object subsequently cooled is first picked up.
On 46 independent cooling tasks, the seed100 student solves 4 more tasks than the matched-fallback control, with the same outcome in both evaluation runs. The key progress is first identifying and finding the target object that the control did not pick up, then completing cooling and placement. These records show concrete changes in both the demonstrations and the student's decisions.
Response Part 4/4
Q4. Which design choices contribute to the gains, and are repeated updates necessary?
We have added a naive-feedback control that is more direct than Fixed: it constructs a reusable teaching plan from M1's TRAIN task outcomes and trajectories, then appends the plan to the initial policy. LWR instead combines student records with training-side KEEP teacher demonstrations to recompile the full generation policy, specifying applicable states, key actions, and recovery and stopping conditions.
The core comparison fixes M1, the 1024 second-stage training worlds, task-family composition, maintenance data, and training steps. What changes is the teaching guidance and generation process within the same worlds. LWR's mean paired gain over naive student feedback is 4.81±0.11 percentage points for 8B and 1.28±2.01 percentage points for 30B, with positive gains in three and two seeds, respectively.
To separate the effect of fallback, we replace the naive-feedback demonstrations at the same positions where LWR falls back with the identical KEEP demonstrations, yielding the following three-seed results.
Table 5: Fallback ablation
| Model | Naive student feedback | Matched fallback | LWR | Paired gain |
|---|---|---|---|---|
| 8B | 87.65±0.28 | 89.17±0.64 | 92.46±0.28 | +3.28±0.48 |
| 30B | 93.43±0.80 | 91.97±0.97 | 94.71±1.32 | +2.74±2.03 |
"Matched fallback" replaces the naive-feedback demonstrations at exactly the positions where LWR falls back with the same KEEP demonstrations, leaving all other positions unchanged. This replaces 117 and 106 positions for 8B and 30B, respectively. The three method columns report Overall success rates in percent. Paired gain is LWR minus matched fallback, in percentage points. All values are means ± sample standard deviations across three seeds; paired gains are computed within each seed before aggregation.
At both model sizes, LWR outperforms the matched-fallback control in all three seeds. Fallback helps naive student feedback at 8B but lowers its score at 30B; fallback alone therefore cannot explain LWR's gains. In addition, removing old maintenance data at 8B still leaves a gain of 3.47±0.63 percentage points over KEEP. Replacing maintenance positions with repeated exposures to new demonstrations from the same task families, while retaining 1216 steps, yields a gain of 4.14±0.53 percentage points. Both settings show positive gains in all three seeds. These controls support the overall contribution of feedback compilation, distinguish the effects of fallback and maintenance, and demonstrate LWR's effectiveness.
We will include the stage settings, full training and generation budgets, acceptance rules, and these ablations in the revised manuscript to clarify how demonstrations are generated and used for training, and which comparisons support the method's contribution.
Response Part 1/4
Q1. How many stages were run, and which stage's results are reported?
We thank the reviewer for the suggestions. The added ALFWorld experiments clarify the stages, budgets, acceptance rules, and design contributions.
The new ALF experiments contain two student-training stages and one feedback-based policy revision:
Direct → train M1 on 1024 shared demonstrations → collect M1's own TRAIN feedback → revise the policy and generate second-stage demonstrations → continue training from M1 to obtain M2.
M1 is trained for 1024 steps and M2 for 1216 steps, with one epoch per stage. For each run, we report the final checkpoint after the prescribed number of training steps, rather than selecting an intermediate checkpoint using independent evaluation scores. KEEP, naive student feedback, and LWR all continue training from the same M1 on their second-stage demonstrations.
Table 1: Stage-wise results on ALFWorld
Scores are percentages. Overall is the success rate across the combined 140 Seen and 134 Unseen tasks. The base-model score is a single-run reference from an earlier evaluation, and the shared M1 is the common first-stage checkpoint. KEEP, naive student feedback, and LWR report second-stage M2 results using training seeds 100, 17, and 57. Each checkpoint is evaluated twice; we first average the two evaluations, then compute the mean and sample standard deviation across the three seeds. The shared M1 and each method's demonstrations are fixed across seeds, without regenerating data.
LWR increases Overall from the shared M1's 74.64% and 83.58% to 92.46% and 94.71% for 8B and 30B, respectively. A more direct policy comparison is between second-stage KEEP and LWR: both continue training and receive new demonstrations, while LWR achieves gains of 4.56 and 3.47 percentage points, positive in all three seeds at both model sizes. The gains therefore go beyond the difference between continued training and no further training. We revise the policy only once in these experiments and then keep it fixed, without further iterations; this single revision already yields the gains above.