acceptodds
Under review as a conference paper at ICLR 2027

After the Fix: How Corrected Agent Histories Transfer to Related Tasks

Abstract

Does repairing an episode make its experience a better memory for the next task? We transfer the same failed source before and after accepted repair to a fixed target, alongside independent execution. Our 3,300 runs cover 100 ThinkingBox pairs and the same 100 APEX pairs with and without source-state inheritance, under eleven conditions. ThinkingBox's Full/Skill/Hybrid correction gains are 44/29/32 percentage points, with corrected performance 25/22/18 points above independence; inference weakens at the task-family level. Yet 12 of Full's 15-point larger correction gap over Skill come from worse uncorrected performance, not better corrected memory. Moreover, 22 of Full's 46 upward transitions restore observed baseline success. Neither APEX regime establishes comparable aggregate correction benefits. Action evidence connects workflow gains with reusable obligations and convention conflicts with source-local choices. Text APEX's accepted execution reaches 52% versus its summary's 40%, without robust global/group-level superiority or an estab- lished advantage over independence. Smaller handoffs reduce input but increase calls. The value of repairing experience is therefore distinct from the value of reusing it: memory updates require both a previous-version reference and a fresh-start reference.

Primary area:
causal reasoning
Keywords:
LLM agent memory, experience transfer, negative transfer, context compression, counterfactual evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9dvnOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This empirical study asks whether repairing a failed source episode improves its usefulness for a different target task. It compares the same source before and after accepted repair, multiple handoff representations, and independent target execution. The 3,300 runs cover 100 ThinkingBox pairs and 100 APEX pairs evaluated in two handoff regimes. ThinkingBox shows large correction gains, while neither APEX regime establishes comparable aggregate benefits. The analysis distinguishes improvement over an uncorrected memory from benefit over a fresh start, and shows that a larger correction gap need not identify the best corrected representation. Smaller handoffs also reduce input tokens without necessarily reducing execution calls.

Critical strengths
  1. The same-source, fixed-target design yields a useful methodological contribution to evaluating memory updates. Comparing corrected experience with both its predecessor and independent execution prevents a misleading conclusion from a large C-U gap alone. The ThinkingBox Full/Skill example makes this consequential: the correction gaps are 44 versus 29 points, but corrected performance differs by only three points; twelve of the fifteen points come from the weaker uncorrected Full condition. This changes how one should interpret and select memory revisions, beyond the already-known observation that memory can help or hurt.

  2. The paper supports its main distinction with unusually careful outcome accounting. It reports paired transitions, multiplicity adjustments, group-level sensitivity, all-target denominators, and action-level audits, while acknowledging that the two APEX regimes share pairs and are not a clean state-only intervention. The 22 baseline-restoring cases among Full's 46 upward ThinkingBox transitions and the input-token/model-call tradeoff are informative even without a new memory algorithm. These concrete evaluation lessons are the main basis for my positive recommendation.

Critical weaknesses
  1. Repeat-run reliability is unresolved. There is one execution per condition and target, so the transition decomposition combines treatment-associated differences with stochastic execution and, in APEX, judging variation. Appendix C.3 itself reports that the two B-only executions disagree on 17 of the same 100 APEX targets. Family/world dependence also weakens the inferential support: no primary contrast survives the group-level global-24 adjustment (Appendix B.2), although the ThinkingBox effect sizes and cluster intervals remain encouraging. The paper generally states these limitations correctly, but they prevent interpreting individual restoration/harm counts as stable task-specific effects or the APEX null results as equivalence.

  2. The scope of the transferable lesson is still limited by cohort construction and the bundled intervention. Sources are selected for initial failure and successful repair, ThinkingBox includes pilot sources, and feedback has privileged diagnostic access. Repair simultaneously changes procedural content, amount of evidence, and approval cues; selected trajectory audits cannot isolate which component transfers. In particular, the strong ThinkingBox lifecycle improvement might partly reflect supplying a generic benchmark workflow obligation. Without a generic-checklist or carefully matched feedback-content control, the results support the conditional value of the complete repair package more strongly than a general rule about which corrections should enter memory.

Questions for authors
  1. Robustness evidence: Are repeated B-only/U/C executions available for any subset? If so, can you report condition-level success probabilities and the stability of the restoration/new-success decomposition, rather than only one observed outcome path per target?

  2. Cohort sensitivity: Does the ThinkingBox conclusion persist after excluding the 13 pilot sources, and under equal-family weighting? Please distinguish this from the already reported leave-one-family-out analysis.

  3. Mechanism clarification: How many source repairs explicitly introduce the shared ticket-lifecycle obligation? Is there evidence comparing corrected handoffs with a target-independent checklist containing that obligation, to separate source-specific correction transfer from generic workflow instruction?

3 comments

Dear reviewer, as the discussion period ends soon, could you please let us know if our response and revisions above have resolved your concerns? Thank you!

Dear reviewer, as the discussion period ends soon, could you please let us know if our response and revisions above have resolved your concerns? Thank you!

Dear reviewer, as the rebuttal period ends soon, could you please let us know if our response and revisions above have resolved your concerns? Thank you!

Reviewer_sqSkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper compares the experience of the same failed agent episode before and after an accepted repair, as memory for a related target task. It hands this experience to the target in several formats and compares both versions with running the target without any experience. On ThinkingBox, the repaired experience clearly beats both alternatives, while on APEX-Agents it shows no comparable gain. Further analyses separate restored from new successes, look at action traces, and compare reading and execution costs.

Critical strengths
  1. Handing the same failed episode to a fixed target before and after an accepted repair, and also running that target with no handoff, is a well-controlled and well-chosen way to evaluate a memory update.
  2. Provid a great Informative breakdown of the gains: Figure 3 separates targets the agent already solves with no handoff from those it solves only with the repaired memory, and shows that much of the apparent repair gain is of the first kind.
  3. The Cost accounting is useful: the paper reports how much input the target reads and how many model calls it makes as separate costs, and shows that a shorter handoff reduces the first but not the second (Section 6.3, Figure 5).
Critical weaknesses

(1) While the title and conclusion state the observation as a general principle, but the positive effect appears on one benchmark with one acting model, and each condition runs once per target (Section 4, Appendix B.1). APEX shows no comparable gain (Section 5.1), and with no handoff at all the two APEX regimes already disagree on about one target in six (Appendix C.3), so I cannot tell the APEX transitions discussed in Section 5.2 apart from run-to-run variation.

(2) I am not convinced that the ThinkingBox gain reflects corrected experience in general rather than the target's grading rules. The simulated feedback reads the executed trace and the validator differences, sources and targets come from the same workflow family (Appendix A), and the improvement concentrates on the family's shared ticket-lifecycle rule (Table 18); some repaired summaries even state field values that the target validator checks (Appendix D). Since this is the only setting with a positive result, it is unclear to me whether the central claim survives once this route is ruled out.

(3) The paper describes a package with the paired-analysis scripts, source identifiers, cohort construction, settings and prompts (Reproducibility Statement, Appendix A), but no supplementary material was found. Without it I could not check how sources were selected, what the repair feedback said, or whether the reported statistics reproduce.

On the angle of presentations, the abstract packs many results and abbreviations into single sentences, and settings that decide how the results should be read, such as what the repair feedback can see, appear only in Appendix A. I had to move back and forth between the main text and the appendix to understand what the main comparison holds fixed.

On a side note, I noticed some work that may be missing as references. Earlier studies report that agents copy similar retrieved experiences and that erroneous or outdated memories mislead later tasks [1], which is close to this paper's finding that unrepaired memory can hurt the target; that abstract procedural memories and detailed trajectories transfer differently [2], which bears directly on the comparison of handoff formats; and that procedural memory can be updated through reflection and validation [3], which resembles the repair step itself. It would be interesting to see how the authors view these works in relation to their own, for example whether they serve as background that this paper builds on, and what this paper adds beyond them.

References [1] Zidi Xiong et al. How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behavior. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL), 2026. [2] Qisheng Hu, Quanyu Long, and Wenya Wang. When Continual Learning Moves to Memory: A Study of Experience Reuse in LLM Agents. arXiv preprint arXiv:2604.27003, 2026. [3] Runnan Fang et al. Memp: Exploring Agent Procedural Memory. In Findings of the Association for Computational Linguistics: ACL 2026, 2026.

Questions for authors
  1. Does the ThinkingBox result hold with a different acting model, and how often do outcomes change when the same condition is rerun on a target? The latter would show whether the APEX transitions discussed in Section 5.2 exceed run-to-run variation.
  2. How much of the ThinkingBox gain remains for pairs whose feedback and repaired handoffs do not mention rules that the target validator checks (Appendix A, Table 18)?
  3. Could you share the package the paper refers to, in particular the feedback prompt, the feedback texts that the repaired branches received?
  4. How do you see [1], [2] and [3] in relation to your work, for example as background that it builds on, and what does your paper add beyond them?

References [1] Zidi Xiong et al. How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behavior. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL), 2026. [2] Qisheng Hu, Quanyu Long, and Wenya Wang. When Continual Learning Moves to Memory: A Study of Experience Reuse in LLM Agents. arXiv preprint arXiv:2604.27003, 2026. [3] Runnan Fang et al. Memp: Exploring Agent Procedural Memory. In Findings of the Association for Computational Linguistics: ACL 2026, 2026.

3 comments

Dear reviewer, as the discussion period ends soon, could you please let us know if our response and revisions above have resolved your concerns? Thank you!

Dear reviewer, as the discussion period ends soon, could you please let us know if our response and revisions above have resolved your concerns? Thank you!

Dear reviewer, as the rebuttal period ends soon, could you please let us know if our response and revisions above have resolved your concerns? Thank you!

Reviewer_4AjJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether repairing an episode makes its experience a better memory for the next task. It transfers the same failed source before and after accepted repair to a fixed target, alongside independent execution. Across 3,300 runs on ThinkingBox and APEX, correction improves transfer on ThinkingBox, while neither APEX regime establishes comparable aggregate benefits. A larger correction gain can reflect worse uncorrected performance rather than better corrected memory. Smaller handoffs reduce input tokens but increase model calls. The paper concludes that the value of repairing experience is distinct from the value of reusing it: memory updates require comparison against both the previous version and a fresh start.

Critical strengths
  • Very interesting research question. Does accepting a correction on task A make that experience useful for task B?

  • Including independent execution is a useful control. In ThinkingBox, 22 of Full’s 46 upward transitions restore successes already observed without memory. This distinction helps separate fixing a misleading memory from creating additional capability.

Critical weaknesses
  • The paper was hard to read and understand. Some sections like L45-49 were very dense and read like AI generated.

  • It is unclear how much of the ThinkingBox improvement requires corrected task experience rather than a generic workflow checklist. Section 5.4 shows that ticket-lifecycle violations fall from 55 to two under Full and from 40 to seven under Skill. This suggests that learning shared operational requirements accounts for an important part of the improvement. Would a short instruction to open the ticket, perform the action, and close it recover similar gains without transferring the source episode?

  • Several comparisons change multiple components together, limiting the conclusions about what should be retained. Corrected versus uncorrected memory changes the trajectory, feedback, accepted outcome, length, and potentially the authority conveyed by user approval. It does not isolate which component helps transfer. Similarly, APEX-state and APEX-text differ in textual construction as well as source-file inheritance (Appendix A, L756–757). The paper acknowledges these boundaries, but an identical-text state/no-state comparison and controls for approval markers would make the findings more actionable.

Questions for authors
  • Can a generic workflow checklist or an unrelated corrected source from the same family reproduce the ThinkingBox gains?

  • Which component contributes most: corrective feedback, the accepted execution, or the indication that the source was approved?

3 comments

Dear reviewer, as the discussion period ends soon, could you please let us know if our response and revisions above have resolved your concerns? Thank you!

Dear reviewer, as the discussion period ends soon, could you please let us know if our response and revisions above have resolved your concerns? Thank you!

Dear reviewer, as the rebuttal period ends soon, could you please let us know if our response and revisions above have resolved your concerns? Thank you!

Comments on the paper

1 comment

Dear Reviewers, Area Chair, and Senior Area Chair,

Now that the author rebuttal period has closed, we would like to request a reassessment of our submission in light of our responses and the additional supporting evidence provided during the discussion.

In our rebuttal, we provided detailed clarifications, raw experimental logs, and the corresponding experiment scripts to facilitate verification of our experimental procedures and reported results. We respectfully ask the reviewers to consider these materials and update their assessments and scores accordingly. In particular, where our responses and supporting evidence have resolved concerns underlying the initial evaluation, we ask that reviewers revise their scores upward to reflect those resolutions.

We would greatly appreciate the AC’s and SAC’s assistance in reminding the reviewers to complete this reassessment and update their reviews and scores as appropriate before the final recommendations are made. If any substantive concerns remain unresolved, we would appreciate reviewers identifying them explicitly in their updated assessments.

Thank you for your time and effort, and for helping ensure that the final evaluation takes our rebut

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.