- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that agents which store abstractions of past experience (rules, strategies, skills) tend to reproduce their own errors: a bad abstraction shapes future trajectories, which are then abstracted again. As an alternative it proposes ANTI-PERSISTENCE. Memory keeps only raw trajectories and outcomes, and guidance for each new task is generated on the fly and then thrown away. A bandit-style selector picks one of four guidance modes per task. Experiments on ALFWorld, WebShop and ScienceWorld with three open models compare the method against ReAct, Reflexion, ExpeL, AutoRefine and trajectory, abstraction and joint memory variants. It has the best average success rate, and discarding guidance beats keeping it in a direct comparison. Ablations cover the mode selector and its hyperparameters.
Critical strengths
-
The paper tests a clear design question under a controlled protocol: keep inferred guidance, or rebuild it from stored trajectories for each task. All methods share fixed train/held-out partitions and memory is frozen at evaluation. Under this setup the method beats trajectory and abstraction memory in all 9 settings and joint memory in 8 (Table 1), and abstraction and joint memory trail trajectory memory in 7 of 9.
-
The mode selector is ablated sensibly. In Fig. 6 the bandit beats fixed modes and random selection on Qwen, and the best fixed mode differs by benchmark. Fig. 8 shows the zero-cost none mode is used in every setting.
Critical weaknesses
-
The error-reproduction claim in the title and Contribution 3 is not tested in the main text. Contribution 3 promises "associations between earlier and subsequently generated abstractions", but Section 4 defers this to Sections E.3 and F. Nothing in Section 4 tracks whether an error in one abstraction reappears in a later one.
-
The baselines seem to be under-specified. Section 4.1 only names the four agent baselines. ExpeL on Llama/ALFWorld scores 7.2 against 29.3 for ReAct, which suggests a setup problem. The main text does not say how Reflexion, the strongest baseline at 52.9 on average, works on held-out tasks when memory is frozen during evaluation.
Questions for authors
- How were gamma = 0.998 and lambda = 0.05 chosen, and on which split were Figs. 5 and 7 computed? If on the held-out sets, please redo Table 1 with values picked on a split of the training tasks and add seed-level significance tests. A gain that holds up would raise my soundness score.
- What evidence in Section 4 shows error reproduction specifically, rather than a general benefit of on-demand synthesis? For example, seed a known faulty rule, delete it, and count how often the error reappears in later guidance under persistent memory vs. anti-persistence.
- In Table 2, how is persisted guidance stored, indexed and retrieved, and how does Persist differ from Abs. Mem.? Why does Persist on Llama/ALFWorld (26.7) fall below ReAct (29.3)?
- For Llama/ScienceWorld, what fraction of episodes end early under each memory design, and why (parse errors, context length, invalid actions)? If most of the joint-memory drop comes from such failures, the result is not about error reproduction.
- Does Reflexion get multiple trials per held-out task, and does its reflection memory carry across tasks during evaluation? Why does ExpeL on Llama/ALFWorld (7.2) score far below ReAct (29.3)?
3 comments
To Reviewer NWKv (2/3)
Q3: How were gamma = 0.998 and lambda = 0.05 chosen, and on which split were Figs. 5 and 7 computed? If on the held-out sets, please redo Table 1 with values picked on a split of the training tasks and add seed-level significance tests. A gain that holds up would raise my soundness score.
A3: We selected γ = 0.998 and λ = 0.05 through preliminary tests on a small-scale dataset, then fixed them before held-out evaluation and used the same values across all Table 1 settings. The former gradually discounts older feedback; the latter mildly penalizes extra synthesis calls. Figures 5 and 7 are subsequent held-out sensitivity analyses and did not determine the defaults. We will clarify the pilot dataset and selection procedure in the revised manuscript.
We also ran additional experiments with DeepSeek-V4-Flash, fixing the code defaults γ = 0.995 and λ = 0.05 without further tuning. Both used three seeds and a single final test evaluation with memory frozen. AppWorld used 20 training and 50 test tasks; WebWalkerQA used 300 distinct training tasks and 150 test tasks from unseen websites.
| Memory configuration | AppWorld success (%) | WebWalkerQA success (%) |
|---|---|---|
| Trajectory Memory | 88.0 | 40.2 |
| Abstraction Memory | 93.3 | 45.1 |
| Joint Memory | 96.0 | 45.3 |
| Anti-persistence | 92.7 | 48.2 |
We added two-sided exact seed-level paired sign tests, comparing success-rate differences within each seed and excluding ties:
| Baseline | AppWorld wins/losses (p) | WebWalkerQA wins/losses (p) |
|---|---|---|
| Trajectory Memory | 2/0 (0.50) | 3/0 (0.25) |
| Abstraction Memory | 1/2 (1.00) | 2/0 (0.50) |
| Joint Memory | 1/2 (1.00) | 2/0 (0.50) |
With three seeds, the smallest attainable two-sided p-value is 0.25, so none reached significance. On WebWalkerQA, Anti-persistence matched or exceeded each memory baseline in every seed. Full results and statistical details will be included in the revised manuscript.
Q4: What evidence in Section 4 shows error reproduction specifically, rather than a general benefit of on-demand synthesis? For example, seed a known faulty rule, delete it, and count how often the error reappears in later guidance under persistent memory vs. anti-persistence.
A4: To examine specific error content, we extended the Useful/Toxic/None paired design in Appendix F.1 into a controlled lineage experiment, which we will report in Section 4. It checks whether the same target errors recur in later executions and appear in newly generated guidance.
Using DeepSeek-V4-Flash on AppWorld, we compared 12 incorrect rules (Toxic), their matched corrections (Useful), and no injected rules (None). Each pair differed in a single fact. All conditions used the same 30 tasks, three random seeds, and three rounds of skill evolution. Initial rules remained available during execution. The skill evolver received execution traces and existing skill names, with the rule text withheld.
Dedicated detectors checked the generated code for each target error, while a judge applied the same criteria across conditions to check whether newly generated rules restated that error.
| Condition | Execution error recurrence (round 0 → round 3) | New rules restating a target error |
|---|---|---|
| None: no injected rules | 42% → 24% | 1.4% |
| Useful: matched corrected rules | 31% → 27% | 2.8% |
| Toxic: incorrect rules | 70% → 57% | 4.6% |
After three rounds, execution error recurrence was 30 percentage points higher in Toxic than in Useful. Some newly generated rules also restated the same target errors. These checks examine both execution behavior and specific content in new abstractions, providing evidence alongside the success-rate comparisons. The text-recurrence rates are reported descriptively.
To Reviewer NWKv (3/3)
Q5: In Table 2, how is persisted guidance stored, indexed and retrieved, and how does Persist differ from Abs. Mem.? Why does Persist on Llama/ALFWorld (26.7) fall below ReAct (29.3)?
A5: Table 2 is a direct ablation under a shared configuration. Discard and Persist use identical trajectory stores, bandits, cost weights, synthesis prompts, and execution settings. They differ in whether generated guidance is retained and reused.
During training, Persist appends fresh guidance to JSONL with task information, mode, outcome, iteration, and task embedding. Guidance from successful and failed episodes is retained with outcome labels. From the second iteration onward, whenever synthesis is invoked, Persist retrieves the top-2 items of the same task type from earlier iterations by cosine similarity and prepends them to the fresh guidance. Only newly generated text is stored, and the store remains frozen during evaluation.
| Persist | Abs. Mem. | |
|---|---|---|
| Stored content | Task-specific guidance | General skills extracted by an evolver |
| Retrieval | Top-2 when synthesis is invoked | Top-5 every episode |
| Fresh task-time synthesis | Yes | No |
| Trajectory store retained | Yes | No |
On Llama/ALFWorld, Persist and ReAct share the backbone and base evaluation settings, with a mean difference of 2.6 percentage points (26.7% vs. 29.3%). Guidance from earlier tasks, including failed episodes, may interfere with current decisions. The direct ablation shows a larger difference: 54.7% for Discard versus 26.7% for Persist, a mean advantage of 28.0 percentage points, supporting discarding synthesized guidance after use in this setting.
Q6: For Llama/ScienceWorld, what fraction of episodes end early under each memory design, and why (parse errors, context length, invalid actions)? If most of the joint-memory drop comes from such failures, the result is not about error reproduction.
A6: A6. Assuming at least one LLM call per environment step and a maximum of 50 steps, we calculated the following bounds on budget exhaustion from Tables 1 and 6.
| Memory configuration | Success rate | Budget exhaustion |
|---|---|---|
| Trajectory Memory | 20.0% | ≤7.50% |
| Abstraction Memory | 18.9% | ≤2.66% |
| Joint Memory | 7.2% | ≤2.30% |
| Anti-persistence | 30.6% | ≤7.12% |
Under these assumptions, Joint Memory’s budget-exhaustion rate is at most 2.30%, so most failures occur before the step budget is exhausted, consistent with the possibility of early failure discussed in §4.3. The aggregate metrics cannot distinguish incorrect completion from termination caused by parsing errors, context limits, or invalid actions. These results alone therefore cannot establish whether error reproduction caused the performance decline.
Q7: Does Reflexion get multiple trials per held-out task, and does its reflection memory carry across tasks during evaluation? Why does ExpeL on Llama/ALFWorld (7.2) score far below ReAct (29.3)?
A7: All methods share the model configuration, task partitions, base agent prompt, and per-step output budget (§4.1, Appendix D), while retaining their respective memory and reflection procedures.
Reflexion. Each held-out task permits up to three attempts. New reflections are used only for retries on that task and cleared afterward. They do not carry across tasks, and persistent memory accumulated during training remains frozen during evaluation.
ExpeL. Its 7.2% on Llama/ALFWorld and ReAct’s 29.3% were obtained under the same base configuration. In Table 1, ExpeL outperforms ReAct on ALFWorld with the other two backbones, showing backbone-dependent benefits. The low score alone does not establish a configuration error.
To Reviewer NWKv (1/3)
Q1: The error-reproduction claim in the title and Contribution 3 is not tested in the main text. Contribution 3 promises "associations between earlier and subsequently generated abstractions", but Section 4 defers this to Sections E.3 and F. Nothing in Section 4 tracks whether an error in one abstraction reappears in a later one.
A1: We will add the controlled lineage experiment to the main text and update Contribution 3 accordingly. Building on the Useful/Toxic/None paired design in Appendix F.1, the experiment checks whether specific errors recur in later executions and appear in newly generated rules.
Using DeepSeek-V4-Flash on AppWorld, we compared 12 incorrect rules (Toxic), their matched corrections (Useful), and no injected rules (None). Each pair differed in a single fact. All conditions used the same 30 tasks, three random seeds, and three rounds of skill evolution. Initial rules remained available during execution. The skill evolver received execution traces and existing skill names, with the rule text withheld.
Dedicated detectors checked the generated code for each target error, while a judge applied the same criteria across conditions to check whether newly generated rules restated that error.
After three rounds, execution error recurrence was 30 % higher in Toxic than in Useful. Some newly generated rules also restated the same target errors. The main text will present both execution-level and content-level checks, with the text-recurrence rates reported descriptively.
Q2: The baselines seem to be under-specified. Section 4.1 only names the four agent baselines. ExpeL on Llama/ALFWorld scores 7.2 against 29.3 for ReAct, which suggests a setup problem. The main text does not say how Reflexion, the strongest baseline at 52.9 on average, works on held-out tasks when memory is frozen during evaluation.
A2: Our comparisons use shared base execution settings while preserving each baseline’s algorithmic procedures. All methods use the same backbone configuration, training and held-out partitions, training tasks per iteration, base agent prompt, and per-step output budget. Each episode permits at most 50 environment steps. These settings are specified in §4.1 and Appendix D.
Reflexion’s task-local reflection and frozen memory operate at different scopes. Each held-out task allows up to three attempts: the initial attempt plus two retries. Reflections generated after failure are used only for subsequent attempts on that task and cleared afterward. Cross-task memory accumulated during training remains frozen throughout evaluation. New trajectories, outcomes, and reflections do not carry across held-out tasks. Task-local reflection is therefore compatible with frozen cross-task memory.
ExpeL’s 7.2% is the result obtained under these shared settings. Beyond base execution, ExpeL extracts rules from experience and uses those rules alongside retrieved examples on later tasks, making its benefits dependent on the backbone’s ability to process experience. As Table 1 shows, ExpeL underperforms ReAct on Llama/ALFWorld but outperforms it with Qwen3-Coder-Next and Gemma-4-31B-it. This demonstrates backbone-dependent performance. A low score in one setting alone does not establish a configuration error.