- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces ReHarness, a framework for agentic continual learning that keeps the underlying LLM fixed while updating external memory. After each task, it uses the observed trajectory and outcome to revise previously retrieved reflections, with revisions constrained by their source trajectories, and stores a new reflection for future use. Experiments across three benchmarks and three LLM backbones demonstrate improved overall performance over memory, reflection, prompt optimization, and raw-context baselines.
Critical strengths
-
The method is clearly motivated and easy to understand, directly addressing how overgeneralized or outdated reflections can repeatedly misguide an agent.
-
The empirical evaluation includes diverse baselines, particularly strong raw-context methods, and useful ablations that support the effectiveness of the proposed framework.
Critical weaknesses
-
The role of downstream feedback is not sufficiently isolated. Current ablation remove the entire reflection operation, it does not establish whether the gains come from feedback-driven correction or simply from giving the model another opportunity to reconsider the original evidence. A control with the same source access and a comparable inference budget, but without downstream feedback, would help distinguish these explanations.
-
The stability analysis does not directly establish retention. Since ReHarness overwrites earlier reflections, it remains unclear whether these revisions preserve useful knowledge or increasingly tailor memory to the most recent distribution. An A→B→A diagnostic, or evaluation on the earlier distribution after adaptation, would provide more direct evidence for the claimed stability–plasticity balance.
Questions for authors
-
Revision selectivity. A failure may result from execution errors or environmental randomness rather than an incorrect historical reflection. How does the revision agent avoid damaging valid reflections in such cases? Also, every retrieved reflection is passed to the revision agent, although some may already be correct or may have been retrieved despite having little relevance to the current task. Is there an explicit criterion for deciding whether a reflection needs revision or should remain unchanged?
-
Retrieval robustness. Retrieval affects both the current decision and the selection of reflections for subsequent revision. The paper does not specify the detail retrieval operation. When retrieval returns many irrelevant or inapplicable reflections, could this both impair the current decision and trigger inappropriate changes to the memory bank?
6 comments
Response to Reviewer ePP6 - Part II
Q1: Revision selectivity and robustness to noisy feedback
We agree that a low reward does not necessarily indicate an incorrect reflection, particularly in stochastic environments such as Exploitable Poker. As discussed in Sec. 4.2, poker provides noisy credit-assignment signals because realized outcomes depend on hidden information and chance. Consequently, ReHarness achieves only marginal gains in this setting.
Rather than applying a fixed reward threshold to determine which reflections should be revised, ReHarness delegates the extent of revision to the revision agent, which independently examines each retrieved reflection using its original source trajectory and the current trajectory as feedback. The revision agent can preserve valid information while refining unsupported hypotheses or overly broad applicability conditions.
To examine whether revisions are selective, we measure character-level edit distance by retrieval rank, averaged across the six CL-Bench tasks:
| Backbone | Rank 1 | Rank 2 | Rank 3 | Rank 4 | Rank 5 |
|---|---|---|---|---|---|
| Claude | 1355.33 | 1213.77 | 1148.33 | 1118.34 | 1139.08 |
| GPT | 1735.28 | 1326.08 | 1198.28 | 1076.95 | 1110.99 |
| Gemini | 1054.80 | 1030.45 | 933.38 | 872.63 | 962.04 |
Revision magnitude generally decreases with retrieval rank, suggesting that the revision agent makes larger changes to more relevant reflections. This is consistent with the intuition that highly relevant reflections offer more opportunities to refine their applicability based on downstream feedback.
This also explains why ReHarness does not require an explicit binary revise-or-retain criterion: the revision agent determines how much each retrieved reflection should change, rather than applying the same modification indiscriminately.
Q2: Retrieval implementation and robustness
As specified in Lines 239–241, ReHarness uses Qwen3-Embedding-8B for retrieval. Given the current query , we compute cosine similarity between its embedding and those of previous queries , then retrieve the reflections associated with the top- most similar queries, where .
We agree that irrelevant or inapplicable retrievals can both misguide the current decision and introduce noise into subsequent revision. ReHarness partially mitigates this through source-grounding constraints and by allowing the revision agent to refine applicability conditions.
However, like other experience-based continual learning methods, ReHarness assumes that the task stream contains reusable structure across instances. When previous experiences are largely unrelated to the current task, retrieval and backward revision become less informative. We will clarify this assumption and the potential sensitivity to retrieval quality.
We thank the reviewer again for the insightful suggestions. We hope our additional ablations and explanations address the main concerns, and we would be grateful if the reviewer could consider raising the score.
Gentle Follow-Up on Our Rebuttal
We thank the reviewer again for their time and consideration. As the rebuttal period is approaching its end, we wanted to kindly follow up and ask whether our response has addressed your concerns or if there are any remaining questions we could clarify.
We would greatly appreciate your feedback and consideration of our additional results.
Response to Reviewer ePP6 - Part I
We thank the reviewer for the constructive feedback and positive assessment of our work. We address each concern below and will update the manuscript accordingly.
W1: Isolating the contribution of downstream feedback
We conduct three additional ablations on CL-Bench with Claude Opus 4.8 to isolate the contribution of downstream feedback from merely reconsidering the original evidence:
We compare these settings against ReHarness and the no-revision baseline. Results are reported as normalized gain:
Simply flagging potentially ineffective reflections yields only a modest improvement over no revision (0.161 vs. 0.144). More importantly, both rewriting variants underperform ReHarness (0.182 and 0.171 vs. 0.217), despite revisiting the same source trajectories and performing additional rewriting.
These results suggest that the benefits of backward revision cannot be explained solely by additional reflection or rewriting. Downstream feedback provides a useful signal for correcting previously accumulated experience.
W2: Preserving knowledge under distribution shifts
We agree that an A→B→A evaluation would provide more direct evidence of retention. Our current stability analysis measures transfer at distribution boundaries, rather than explicitly testing whether earlier knowledge remains recoverable after adaptation.
ReHarness prioritizes adapting reflections to the current distribution while retaining source-grounded information from earlier trajectories. Its revision mechanism can also readapt reflections when the environment returns to a previously encountered distribution.
We did not perform retrospective evaluation on previously completed instances because our strictly online protocol processes each instance only once. Replaying previously observed instances after their outcomes have informed memory updates would introduce a different evaluation setting and potential information leakage. However, an A→B→A stream with newly sampled instances from distribution A would be a valid and useful retention diagnostic.
We will distinguish the observed stability gains from direct evidence of long-term knowledge retention in the revised manuscript.