acceptodds
Under review as a conference paper at ICLR 2027

HABEAS: Evidence-Gated Reversible Memory Management for LLM Agents

Abstract

External memory lets LLM agents reuse experience, but its value depends on which records remain available. History-based deletion permanently removes memories with low average utility on retrieved tasks. We identify two evidence gaps in this rule: deficient support under deterministic retrieval leaves many task–memory effects unidentified, and task outcomes confound a memory's contribution with difficulty, context, and execution noise. We propose HABEAS, an evidence-gated reversible lifecycle that separates suspension from verdict. Poor historical feedback quarantines a memory; paired audits re-execute completed tasks with and without it; restoration or deletion requires agreement across tasks; and restored memories remain under review. On two benchmarks (CICIoT2023 traffic classification and LAB-DB SQL tasks) and two models, HABEAS raises mean task success over history-based deletion in all sixteen model–benchmark–condition cells. In the primary CICIoT2023 Natural condition, gains are 6.13 and 2.67 percentage points, while the correct-memory deletion rate falls from up to 24.54% to zero. Ablations show that immediate quarantine accounts for most of the gain, the evidence gate reduces incorrect restorations relative to a simple reversible policy, and post-restoration review adds a smaller improvement. HABEAS creates the same-task evidence that retrieval logs cannot supply and uses it to govern irreversible memory decisions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Memory Management, Reversible Memory Management, Paired Auditing, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_cgzgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HABEAS manages a fixed memory bank through active, quarantined, and deleted states. Poor historical feedback triggers quarantine, while paired executions of completed tasks with and without a memory inform restoration or deletion. Restored memories undergo further review. Experiments on traffic classification and SQL tasks compare the approach with history-based deletion and several lifecycle controls.

Critical strengths
  1. Separating immediate containment from permanent deletion is a useful idea. The same-task inclusion/exclusion comparison also addresses a real weakness of assigning a task’s outcome directly to the retrieved memory.

  2. The evaluation includes relevant controls beyond history-based deletion, particularly Simple-Reversible, uniform audits, and lifecycle ablations. Reporting service availability alongside deletion rates and audit costs makes the tradeoffs easier to assess.

Critical weaknesses
  1. Quarantine can become irreversible after one negative audit. Equation 3 requires zero negative outcomes within an episode for restoration, while Appendix B.1 allows a new episode only after an active memory is quarantined again. Once a quarantined memory receives a negative audit, it therefore cannot recover under the stated rules, even if subsequent evidence is positive. Requiring two negatives for deletion protects the stored record, but not its service eligibility. This matters because roughly 17% of correct memories remain unavailable in CICIoT Natural despite zero deletion (Table A11). The design retains a form of irreversible removal based on potentially noisy evidence.

  2. Historical-answer reuse may reintroduce outcome-selection bias. Quarantine depends on poor service outcomes, yet some audits reuse those historical with-target responses. Conditioning on this selection can make a reused response pessimistic relative to a fresh execution, even when the reuse rule itself does not inspect correctness. Matching the task and context is insufficient to establish the unbiased contrast in Equation 2 under this adaptive protocol. The fresh-versus-historical variability in Table A2 makes this worth checking directly; aggregate comparisons of nonrandomly assigned fresh and reused audits do not resolve it.

Questions for authors

NA

3 comments

Thank you for checking Eq. 3 against Appendix B.1 and for the precise reading of the reuse rule. We tested both concerns directly.

W1. One negative audit blocks restoration

You are correct: one valid negative prevents restoration within the current quarantine episode, and the submitted policy cannot reopen an episode while the memory stays quarantined. The submission states the zero-negative condition (Eq. 3) but not this consequence. We will state it in §4.3 and define "reversible" as "content retained and restorable on evidence", not as a guaranteed return to service.

How much of the unavailability it explains. In CICIoT Natural, 25 (Qwen) and 20 (GPT) source-correct memories are still quarantined at the end of the three seeds. Three per model had received a negative in their final episode. The largest groups, 17 and 15 entries, had received only ties and still had audit allowance left. A single negative can therefore block recovery, but inconclusive audits explain most of the observed cases.

Is a single negative reliable? Not fully, as you suspect. When we re-executed both sides of 738 directional audits twice, 62.80% of the 1,476 comparisons kept the sign, 33.94% became ties and 3.25% reversed. This is why no single audit can trigger an irreversible action: deletion requires negatives on two tasks and two nearest-neighbor groups, and ties are inconclusive (§4.3). A single negative only keeps a memory out of service, with its content retained.

What relaxing the rule does.

  • Positives may offset a negative. This is the net-vote gate of Simple-Reversible, already in §6.2: 6/20 incorrect restorations against 2/2, 2.98%/6.11% of source-correct memories permanently deleted against none, and 0.88/2.96 pp lower success (Tables A20–A21).
  • A negative counts only if a second, fresh audit confirms it. 20 of 60 negatives are neutralized in Natural and 21 of 68 in Stress20; success changes by −1.23 pp [−4.95, 2.49] and +0.20 pp [−0.30, 0.70].
  • An episode may reopen 100 service tasks after its last negative. In Qwen/Natural, source-correct memories unavailable at the end fall from 25 to 21, but 2 source-incorrect memories are restored and then selected 89 times (none under the submitted rule), and mean success goes from 73.80% to 72.43% (Stress20: −0.53 pp).

Reopening thus gives a real availability benefit without a consistent gain in success, and the strict rule trades availability for restoration precision. We keep it as the default and will report both relaxations. We agree that zero deletion is not zero availability cost: in the main Natural runs HABEAS ends with 17.09%/17.11% of source-correct memories in quarantine, where HM has deleted 24.54%/20.51% (Table A11).

W2. Reuse of historical answers and outcome selection

You identify a genuine selection dependency. All 141 historical with-target answers reused in quarantine audits had contributed to the feedback that triggered that quarantine. Eq. 2 defines a contrast between two specified executions. For a reused answer to estimate its side without bias, its expected score must match that of a fresh answer after conditioning on audit selection, and matching the task and context does not establish this.

  • Same-event re-execution. We re-executed the reused side of all 1,731 reuse events on its original input, holding the other side fixed. In the 141 trigger-linked cases, correct answers rose from 16 to 21. Over all with-target reuses (quarantine audits and post-restoration reviews) the count rose from 93 to 106 of 264 (+4.92 pp); the without-target count went from 751 to 749 of 1,466. This supports your concern that selected historical with-target answers can be pessimistic. Execution timing and generation variability prevent us from attributing the whole difference to selection.
  • Effect on decisions. Substituting the fresh answer changes the direction category of Δ\Delta in 14.5% of pairs and the local restore, delete or re-quarantine action in 25 of 1,731 (1.4%).
  • Runs with no reuse (Qwen, three seeds). With both sides freshly executed on every audit, success in Natural is 73.50%, against 73.80% for the submitted policy and 67.67% for HM; in Stress20 the three values are 72.63%, 72.93% and 72.57%. The Natural advantage over HM persists when reuse is removed entirely; in Stress20 fully fresh HABEAS and HM are nearly tied. Fresh auditing costs 5–6% more tokens. Forbidding only the reuse of pre-trigger with-target answers gives −3.07 pp [−11.33, 5.19] and −0.07 pp [−0.83, 0.69] relative to the submitted policy.

The revision will state the scope of Eq. 2 explicitly: pairs with both sides fresh estimate it without bias, and pairs with a reused side are a cost-saving approximation, reported with its measured deviation. Fully fresh auditing is the unbiased option. We did not randomize reuse among concurrent candidates; our evidence is same-event re-execution plus whole-run variants.

Thank you for the detailed response and your efforts to address my concerns. The additional analyses and fully fresh audit experiments largely address my concern about historical-answer reuse, and I appreciate the clearer discussion of the availability–precision tradeoff. However, I remain concerned that a single potentially noisy negative audit can permanently block a memory’s return to service under the default policy, even with subsequent positive evidence. For now, I am inclined to maintain my original rating, but I would be open to raising it if additional evidence alleviates this remaining concern.

Thank you for the careful reading and for pressing this point. Your concern is well founded, and we agree with it. Under the default rule, one negative audit during quarantine blocks restoration, and nothing in the default rule lifts the block later. The submission stated the rule but did not examine this consequence. Below we report how often it occurs and a new experiment that tests whether later positive evidence supports restoration.

How often the block applies. A memory reaches this point after two signals against it: poor task feedback triggered its quarantine, and a paired audit then returned a negative. To answer your concern we counted these cases in the logs. In Natural, 6 of the 267 source-correct memories (2.25%) end the run in quarantine with a blocking negative; over all 24 main CICIoT runs the count is 22 of 879 (2.5%). We do not want to overstate what quarantine gives these memories. While a memory is blocked it does not serve, and within a run that is the same as losing it. On that basis the loss is still much smaller than under the baseline: in Natural, HM permanently deletes 24.54% (Qwen) and 20.51% (GPT) of correct memories. What quarantine adds is that the decision stays correctable. The content and the audit record are kept, so a restoration step can be added, as the experiment below does. We agree that the default rule does not yet use this for memories with a negative.

What the existing runs show.

  • Re-execution. When the negative is replaced by the result of re-executing the same audit, 3 of the 22 would have been restored at that point. A noisy negative therefore did cost restorations, for 3 of 879 correct memories.
  • Later evidence. In the main experiment and the five independent task pools added during this rebuttal at Reviewer f4hq's request (§6.4), 812 quarantine episodes received a first negative. 64 of them were audited once more and none twice, so these runs do not show whether two later tasks would have given positives. We therefore designed and ran the new experiment below.

New experiment: two real chances after a first negative. The rule follows your sentence that a single noisy negative can block a memory "even with subsequent positive evidence". The original negative is kept. Available audit slots go first to memories with a first negative, and a memory is restored if it then obtains positives on two different tasks from two different nearest-neighbor groups. Task order, memory bank and audit budget are those of the default runs (Qwen, Natural and Stress20, three paired seeds).

Of the 89 episodes with a first negative, 74 received at least one further audit. The first further audit returned 44 ties, 30 negatives and no positive. 38 episodes completed two further audits in two groups: 29 were tie/tie, 8 tie/negative and 1 tie/positive. One episode therefore showed positive evidence on a later task, none showed it on two tasks, and no memory was restored through the new path. Only 2 of the 38 belong to correct memories, both tie/tie, so for correct memories the sample is very small. Success changes by −2.57 pp [−8.78, 3.65] in Natural and −0.43 pp [−2.74, 1.87] in Stress20 relative to the default. In this experiment the block therefore made no practical difference.

Why we keep the rule for now. Audits raise success from 66.67% to 73.80% (Qwen, Natural; Table 3), but their outcomes are noisy in both directions. During quarantine, 22 of the 531 negatives belong to correct memories, and 74 of the 133 positives belong to incorrect ones, because most quarantined memories are incorrect. A rule can therefore only choose which error the noise produces, and rules that let positives outweigh a negative return incorrect memories to service. In the five policy-comparison blocks, the net vote of Simple-Reversible restores 14 correct and 6 incorrect memories in Natural (13 and 20 in Stress20), against 7 and 2 (8 and 2) for HABEAS, and HABEAS has the highest success of the policies compared (Tables A20–A21).

What can be improved. You are right that the design can be improved here, and we are grateful that you pointed it out. The three cases above suggest a first step: a negative that would block a memory which already has positive evidence should be re-executed before it takes effect. This repeats only the decisive audits, unlike the blanket confirmation in our first reply. A second step is to let the block expire on evidence, with a stricter requirement than for a first restoration, so that later positives can lift it without admitting incorrect memories. We have not validated either step and do not claim them. The revised Appendix I will state the block as a limitation and describe these directions as future work, and Appendix M will report the counts and the new experiment. Thank you again for the careful review.

Reviewer_2sadOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of memory deletion in LLM agents. Existing history-based deletion policies permanently discard memories associated with poor task outcomes, which often conflates memory quality with task difficulty and execution noise. The authors propose HABEAS, an evidence-gated reversible memory lifecycle. When historical feedback degrades, memories are quarantined rather than deleted. HABEAS then conducts counterfactual paired audits on completed tasks (re-executing with and without the memory) to isolate true causal utility before deciding on permanent deletion or probation-based restoration. Experiments show consistent task success gains and eliminate the false deletion of correct memories.

Critical strengths

1 Reframing memory deletion as a counterfactual evaluation problem cleanly separates genuine memory deficiencies from task difficulty and environmental noise.

2 Decoupling suspension (quarantine) from final deletion eliminates the premature removal of useful memories, reducing the correct-memory deletion rate from up to 24.5% to zero.

3 The ablations convincingly demonstrate that early quarantine drives the bulk of performance gains, while paired audits prevent erroneous restorations.

Critical weaknesses

1 Re-executing completed tasks with and without a memory in paired audits introduces substantial computational and token costs, but the paper offers limited analysis on cost-accuracy trade-offs under strict inference budgets.

2 Experiments rely on traffic classification and SQL tasks where environment states can be reset and re-executed easily. It remains unclear how paired audits handle real-world interactive environments with irreversible side effects or stochastic tool responses.

3 Limited Scope of Benchmarks: Testing is confined to two relatively narrow benchmarks; evaluating on diverse, long-horizon interactive agent tasks (e.g., WebArena or SWE-bench) would better substantiate generalizability.

Questions for authors

How does HABEAS handle environments with irreversible side effects (e.g., API calls, external database writes) where exact paired re-execution is prohibited or expensive?

3 comments

Thank you for the careful review and for the concrete questions.

W1. Cost–accuracy trade-off under strict inference budgets

The audit budget is an explicit, capped parameter (Table A3), and Appendix D reports calls, tokens and success for the no-audit, uniform and targeted configurations. We added budget points to obtain a cost–accuracy curve (Qwen/Natural, three seeds, uniform allocation without semantic ranking; only the budget changes; tokens relative to no audits):

Audit pairs per 1,000 tasks02550100200
Extra tokens0+3%+6%+12%+21%
Success (%)66.6770.1071.1073.9776.67

The 0 and 100 points are in the submission (Table A12); history-based deletion (HM) is at 67.67%. The submitted configuration, which adds semantic ranking, reaches 73.80% at +36% tokens. The 200-pair uniform setting is 2.87 pp higher with 11% fewer tokens; the 25- and 50-pair settings use 24% and 22% fewer tokens for 3.70 and 2.70 pp less success. Under a strict budget we therefore recommend dropping semantic ranking and spending the budget on audits. Two caveats: with three seeds, the paired intervals against the 100-pair setting include zero; and a smaller budget leaves more source-correct memories waiting in quarantine at the end of a run (30, 28 and 24 of 151 at 25, 50 and 200 pairs), although none is permanently deleted at any budget. On LAB, total tokens differ from the no-audit configuration by −3.15% to +2.69%. The curve will be added to the main text.

W2 and Question. Irreversible side effects and stochastic tool responses

Thank you for raising this; the submission did not state the requirement explicitly. A paired audit requires that both sides start from the same initial state and that replaying does not repeat a real side effect.

  • State-changing tasks. An audit replays a completed task from its saved initial state in an isolated copy, never in the live environment (Appendix B.1). LAB-DB is not read-only: 89 of the 189 audit pairs in the LAB runs replay tasks that write to the database.
  • Tasks that cannot be replayed, such as production payments or external writes without a trusted test interface, are excluded from audit. The flagged memory is still withheld from service and waits for evidence from other eligible tasks. If none arrives, it stays in quarantine unresolved and nothing is deleted. Limited replayability slows restoration; it cannot cause a wrong deletion.
  • Measured. On CICIoT we restricted audit eligibility to a fixed 50% or 25% of completed tasks: success in Qwen/Natural goes from 73.80% to 73.37% and 68.57% (Stress20: +0.17 and −0.10 pp), with no source-correct memory deleted. On LAB-DB we admitted only read-only (SELECT) tasks to audits: 12 runs produced 74 audit pairs and no restoration or deletion, with success within 1.3 pp of unrestricted audits. Service continues when replayable tasks are scarce, but too few audit opportunities cost success and leave memories unresolved.
  • Stochastic responses. A single pair records only the two outcomes actually obtained; where the interface allows, exogenous inputs can be fixed or sampling repeated. When we re-executed both sides of 738 directional audits twice, 62.80% of the 1,476 comparisons kept the sign, 33.94% became ties and 3.25% reversed. HABEAS therefore treats ties as inconclusive and requires agreement on two tasks and two nearest-neighbor groups (§4.3). We also flipped each side's audit outcome with probability 0.1 or 0.2 (Qwen/Natural, three seeds): no source-correct memory was permanently deleted, the gate admitted 4 and 6 incorrect restorations against 20 and 23 for a single-positive gate, and success fell from 73.80% to 70.73% and 67.33%.

We will state replayability as an explicit requirement and limitation.

W3. Scope of benchmarks

We agree; Appendix I lists benchmark and model breadth as a limitation, and we did not add WebArena or SWE-bench during the rebuttal period. Traffic classification tests a long sequence of retrieval decisions, and LAB-DB (from LifelongAgentBench) adds multi-turn SQL execution and a controllable database state; both let us observe how one disposal decision affects later service. WebArena and SWE-bench add what these lack: the agent's own choice of actions, external state changes, and a high cost of obtaining a second comparable outcome. Our evidence therefore applies to task streams with replayable audit opportunities; performance on the two benchmarks you name remains to be verified. Within that scope we added five independent CICIoT task pools: HABEAS is ahead of HM in 4 of 5 blocks in each of three settings, by 1.3–3.2 pp on average (task-level 95% intervals exclude zero; block-level intervals over five blocks include it). This indicates that the result does not depend on a single task pool.

Thank you again for your thoughtful review. We have posted our point-by-point response and uploaded the revised manuscript. We would appreciate your feedback on whether the revisions address your concerns.

Thanks for your response.

Reviewer_2e3dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Is a piece of memory or retrieved context useful or not? This is a central question for RAG. When what could be retrieved is potentially malicious, this is serious and prior work has looked at it. But what about when it is not potentially evil, just ordinarily not helpful? The standard baseline is a variant of "three strikes and you're out" --- whereby a piece of information that when retrieved, doesn't result in task success repeatedly, is simply marked as bad and blocked from being retrieved again. The paper identifies a key problem: failing on a task could be due to the task being hard instead of the information being generically not helpful.

The paper gives a natural approach to dealing with this: temporary quarantine suspect memories (but don't delete yet), and then spend some extra effort in doing paired tests where we see if tasks succeed with-and-without the memory. If the memory seems to systematically make things worse, then delete it. But if it actually seems to help, then remove it from quarantine.

Experiments say that this approach to reduce the impact of inherent false-alarms in identifying bad memories is helpful.

Critical strengths

I haven't done a systematic exploration of the literature here and so I do not know the extent to which this is truly novel or the extent to which the current state of the art is genuinely the hard aggressive pruning approach that they are improving upon. However, assuming that it is both novel and that the baseline is indeed well chosen, this feels relevant. Context management is a truly important task.

Their conceptual contributions --- that hard aggressive pruning is guaranteed to get things wrong and that one should view a pruning decision as a testable hypothesis to be investigated --- feel important. They might be obvious in hindsight, but most truly practical insights are.

Critical weaknesses

The most critical weakness is that the main paper (I only skimmed the rest in the sense of scrolling very fast on the screen, whereas I actually printed out and read the main paper) does not have any sense of iso-compute type of comparisons. These audits cost token use, etc. The easiest comparison is to simply do variants of test-time scaling (best-of-N) for the baseline approach and to tune its hyperparameters accordingly. Without that, one can't see if this is a reasonable way of spending the extra compute.

The next most important weakness is that the paper is really badly written --- I'm sure it makes sense to superhuman LLMs but for me as a human reader, this was on the bad side. Not the worst written paper in my pile, but objectively bad in my mind. Their AI assistant clearly did a number on the writing. In the abstract alone we encounter "separates suspension from verdict." I can see the metaphor but before that phrase, there's nothing in the writing to prepare us. And the metaphors are mixed willy nilly. It gets worse in the paper. "This is the support problem of off-policy evaluation, which combines a reward model with a correction for the logging policy and in which deficient support linits what logged feedback identifies unless smoothness assumptions tighten partial-identification bounds outside overlap." Phew! That is grammatical and even makes sense, but is exhausting.

Now, to be fair, I think that as a community we are going to have to figure out how we move in a world in which papers are largely going to be read by AI agents and coding agents. How do we treat them as the sample code that they probably are instead of as human artifacts (which they are not)? But I am reading as a person because I was instructed to do so and am reporting the weakness accordingly.

Questions for authors
  1. What is the compute footprint of this intervention?
  2. What is the natural iso-compute best comparison class?
  3. What are the hyperparameters that need to be tuned (probabilities, thresholds, amount retrieved, etc.) and how did you ensure that the baseline was optimally tuned? Especially in a way that held compute constant?
  4. The RAG problem is natural to look at, but what is the counterpart of this in agentic contexts? Where the agent can choose to look at more or to investigate more deeply?
2 comments

We greatly respect and value your careful review, and thank you for reading the main text so closely.

W1 / Q1–Q3. Compute footprint and iso-compute comparison

You are right: the main text had no iso-compute comparison and left costs to the appendix. We ran the comparison you propose; both will move into the main text.

Q1. Footprint (Appendix D, Table A14). Per 1,000 CICIoT tasks, HABEAS makes about 1,180 model calls against 1,000 without audits and uses 34–38% more tokens in every cell. Two-thirds of that is the semantic ranking that orders audits; without it the overhead is 12–13% (Appendix I). On LAB the total differs by −3.15% to +2.69%.

Q2. Iso-compute comparison. We gave the baseline, history-based deletion (HM), the extra compute as test-time scaling. Best-of-N needs a test-time verifier, which classification lacks, so we used its verifier-free form: three samples and a majority vote. Five new CICIoT blocks with Qwen, each with its own bank and 1,000 tasks; values are Natural / Stress20.

MethodTokens per run (M)Success (%)Correct memories deleted (of 211 / 167)
HM2.17 / 2.1665.92 / 62.9449 / 36
HM, tuned separately2.17 / 2.1562.16 / 63.7856 / 41
HM + vote on 18% of tasks (matched tokens)2.95 / 2.9465.38 / 63.6654 / 34
HM + vote on all tasks6.50 / 6.4762.02 / 62.8851 / 39
HABEAS3.01 / 2.9667.24 / 66.100 / 0

With voting on all tasks, the 96 / 132 tasks (of 5,000) without a majority count as failures. HABEAS has the highest mean success: 1.86/2.44 pp above the token-matched baseline and 5.22/3.22 pp above the one using 2.2 times its tokens. The 95% intervals over the five blocks include zero ([−1.10, 4.82] and [−4.42, 9.30] for the token-matched case), so the claim is limited: the same compute spent on test-time scaling does not close the gap, and it does not reduce deletion errors.

Success also rises with the audit budget, from 66.67% at 0 pairs to 76.67% at 200 pairs per 1,000 tasks (Qwen/Natural).

Q3. Hyperparameters. In the main experiments HM and HABEAS were tuned jointly and share their parameters: bank, task stream, Top-3 retrieval and risk trigger (≥ 5 uses, mean centered feedback ≤ −0.2, checked every 10 tasks; §5.2). HM deletes on this trigger and HABEAS quarantines on it, so neither method has a private setting that favors it. HABEAS's other parameters (evidence gate, audit budget, review rule; Table A3) are fixed throughout and ablated in §6.3 and Appendix D. We then tuned HM on its own, at unchanged compute: minimum uses {3, 5, 8} × threshold {−0.1, −0.2, −0.3} on three development blocks disjoint from the test blocks, with the choice (3, −0.2) frozen before testing. The separately tuned rule is 3.76 pp below the shared setting in Natural and 0.84 pp above it in Stress20. Separate tuning gives no consistent improvement over the baseline we reported, which indicates that the reported HM results are representative.

Q4. The counterpart in agentic settings

In our experiments retrieval is a fixed Top-k rule, so the agent cannot choose to look at more; on LAB-DB it acts over up to three SQL turns, but memory is still retrieved for it. The lifecycle does not depend on who retrieves: it needs a per-memory record of use and outcome, and the ability to replay a completed task with and without a memory. Under agent-directed retrieval, quarantine removes a record from what the agent can find, and the coverage problem of §3.1 remains: the agent's own choices decide which memories receive feedback. A natural extension lets the agent decide when a suspicious memory is worth an audit, turning our fixed audit budget into an action. We have not evaluated this and will list it as future work.

W2. Writing

We apologize for the state of the writing. You are right, and it is our responsibility. We also agree with your point about AI assistance: as our AI use statement says, we used AI tools to draft and polish parts of the text, and we did not edit the result strictly enough for readers. A paper is written for people, and prose that exhausts its reader has failed. We are now going through the manuscript sentence by sentence to check and simplify the wording: metaphors such as "verdict" are removed, each term is defined at first use, and long sentences are split. The two passages you quote now read:

  • "separates suspension from verdict" → "HABEAS first stops retrieving a suspicious memory without deleting it. It decides whether to restore or delete the memory only after further tests."
  • The off-policy sentence in §3.1 → "Logged feedback covers only the task–memory pairs that retrieval selected. This is the support problem of off-policy evaluation (Dudík et al., 2011): where a memory is never selected, the logs cannot reveal its effect unless effects are assumed to vary smoothly across tasks (Sachdeva et al., 2020; Khan et al., 2024)."

Thank you again for your thoughtful review. We have posted our point-by-point response and uploaded the revised manuscript. We would appreciate your feedback on whether the revisions address your concerns.

Reviewer_f4hqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HABEAS, a way for an LLM agent to manage its long-term memories. When a memory starts to look unreliable, HABEAS does not delete it immediately. Instead, it temporarily stops using the memory, reruns some of the same tasks with and without it, and then decides whether to restore or remove it. Restored memories can be checked again later.

The motivation is simple: a memory may look bad because it was used on difficult tasks, and many memories are not used often enough to receive reliable feedback. The authors test HABEAS on CICIoT2023 and LAB-DB with two models. Compared with standard history-based deletion, HABEAS improves the reported success rate in every tested setting. The ablations suggest that immediately blocking suspicious memories provides most of the benefit, while evidence-based restoration and later review provide additional gains. The main limitations are that each setting uses only three seeds and that the extra audits are expensive, especially on CICIoT.

Critical strengths

The core idea is clear and practical. The paper separates “this memory looks suspicious” from “this memory should be permanently deleted,” which is a sensible design for systems where deletion is hard to undo. The same-task comparison with and without a memory is also more informative than judging a memory only from the success or failure of tasks where it happened to be retrieved.

The evaluation is more complete than a simple method in terms of baseline comparison. The authors include no-memory, unmanaged retrieval, and oracle references, as well as lifecycle ablations, a simpler reversible baseline, audit-allocation comparisons, and explicit model-call and token costs. The paper also reports important failure modes instead of hiding them.

Critical weaknesses

The results are promising, but the experiments are still too small to support a strong claim of consistent improvement. Each setting uses only three seeds, and all three seeds share the same task pool. Some gains are also quite unstable: for Qwen on CICIoT Natural, the improvements over HM are +13.1, 0.0, and +5.3 percentage points across the three seeds. In several LAB settings, the gain is only about one percentage point. I would describe the results as encouraging but preliminary, rather than as reliable improvements across all settings.

The audit is a useful idea, but the causal comparison is not as clean as the paper suggests. In the main CICIoT experiments, most audit pairs reuse one side from an earlier service execution, and reused and freshly executed pairs were not randomly assigned. The memory bank may also have changed before the audit. Moreover, the audit measures whether a memory helps on one task relative to one particular replacement; it does not measure whether the memory is generally useful. HABEAS also leaves many memories quarantined or unresolved and increases token use by roughly 34–38% on CICIoT.

These are important practical costs. The novelty claim should also be narrowed. Controlled deletion/re-answer audits and feedback-based memory utility have already been explored in Hindsight Memory-PRM, MemAudit, and MemRL, while CoEvo-Mem also studies the feedback loop between retrieval and memory updates. HABEAS’s clearest new contribution is the persistent quarantine–restore–delete lifecycle with post-restoration review, rather than the audit or causal-identification idea itself.

Questions for authors
  1. The main performance claims are based on only three seeds that share the same task pool. Could the authors report paired confidence intervals and statistical tests, and clarify whether the gains persist on independent task pools? In particular, how should readers interpret the Qwen CICIoT differences of +13.1, 0.0, and +5.3 points, as well as the sub-one-point gains in several LAB settings?

  2. How sensitive are the results to the audit implementation? Please report results separately for reused and freshly executed audit pairs, and provide an ablation with randomized fresh paired audits. It would also be helpful to quantify how often the memory-bank state changes between the original service task and the later audit, and to test the effect of the “one fresh audit every ten pairs” rule and the prescribed backfill memory.

  3. What is the precise contribution of HABEAS beyond recent methods such as Hindsight Memory-PRM, MemAudit, MemRL, and CoEvo-Mem? Can the authors provide direct comparisons or controlled ablations that separate the effects of immediate quarantine, paired auditing, the evidence threshold, and post-restoration review? This would clarify whether the main novelty is the audit itself or the complete reversible lifecycle.

2 comments

Thank you for the constructive review.

W1 / Q1. Statistics and independent task pools

We agree; §6.1 already notes that the three seeds share one task pool. The missing piece was a comparison with HM on independent pools, which we ran on five new CICIoT blocks (each with its own bank and 1,000 tasks; submitted settings).

  • Success over HM: HABEAS is ahead in 4 of 5 blocks in each of the three settings, on average by +1.32 pp (Qwen/Natural), +3.16 pp (Qwen/Stress20) and +1.70 pp (GPT/Natural). Task-level paired bootstrap 95% intervals exclude zero: [0.44, 2.22], [2.12, 4.20], [0.50, 2.90] (McNemar p = 0.004, < 0.001, 0.007). Block-level paired t intervals over the five blocks include zero: [−2.79, 5.43], [−2.49, 8.81], [−5.62, 9.02]. The gain is about 1–3 points on these pools, smaller in Natural than in the main runs; five blocks cannot establish its stability across pools.
  • Deletions: HABEAS deleted no source-correct memory in any of the 15 runs; HM deleted 49 of 211, 36 of 167 and 24 of 184.
  • By class: the low-accuracy attack classes (TCP, HTTP, PSHACK floods) gain in both designs (+5.6/+2.6/+7.7 pp on the new pools); BenignTraffic gains only in the main runs.
  • +13.1/0.0/+5.3 and LAB: seed-to-seed variation around a positive mean of uncertain size. LAB gains of 0.34–0.67 pp are one or two tasks per run, indistinguishable from zero.

The revision will restate the "all sixteen cells" claim as a comparison of means and report the independent pools with both intervals and the class breakdown.

W2 / Q2. Audit implementation

The submission states these caveats itself (§4.2, Appendix D).

  • Reused vs fresh pairs: already reported separately in Appendix D.
  • Re-executing all reused sides: the direction of Δ\Delta changes in 14.5% of pairs and the resulting decision in 1.4%; reused with-target answers were somewhat pessimistic (4.9 pp more often correct when re-executed).
  • No reuse at all: success changes by −0.30 pp in Natural and in Stress20 ([−1.68, 1.08], [−2.04, 1.44]) for 5–6% more tokens. This is the limit of the every-tenth rule; other rates were not tested.
  • Bank state: it changed before the audit for 1,017 pairs, but reuse requires an identical task, ordered context and settings, so reused inputs are unchanged.
  • Backfill: you are right that an audit measures the target against the memory that would take its slot, not general usefulness; the method section will say so. This is what a deletion changes, and why decisions need agreement across tasks and groups (§4.3). Changing the backfill alters the direction in 19–21% of pairs (14.5% with the context unchanged) and the decision in 1.4–1.6%.
  • Quarantine and cost: at the end of Natural runs HABEAS holds 17.09%/17.11% of source-correct memories in quarantine and deletes none; HM has deleted 24.54%/20.51% (Table A11). Of 1,284 entries still quarantined, 1,129 are source-incorrect. On the new pools the two methods leave similar numbers unavailable (quarantined vs deleted). Two-thirds of the token overhead is semantic ranking, which shows no consistent advantage (12–13% without it; Appendix I).

W3 / Q3. Contribution and components

We agree with your characterization and adopt it: the contribution is the persistent quarantine–restore–delete lifecycle with post-restoration review, not the audit or the feedback-gap observation, which §2 credits to the works you list. The abstract and contribution list will be rewritten to match.

The four separations are already in the submission (Qwen/Natural, relative to HABEAS; §6.2–6.3, Tables A12, A19–A21): no immediate quarantine −16.23 pp; no audits −7.13 pp; single-positive restoration −1.57 pp, with 3 incorrect restorations against 0; net-vote gate −0.88/−2.96 pp, with 6/20 against 2/2; no post-restoration review −4.07 pp. The gate and review differences are smaller and vary across seeds. Audits alone add 0.14 pp over unmanaged retrieval and 7.13 pp inside the lifecycle, so the gain belongs to the lifecycle.

Direct comparisons: HM in all sixteen cells, and Audit-Delete, where the same audit drives deletion without containment (52.18%/52.48% against 67.32%/68.20%, Table A20). An online adaptation of MemAudit that we ran separately reaches 61.27%/55.27% against 73.80%/72.93% for HABEAS; it will not go into the main text, since it tests a policy we built on MemAudit's scores, not the authors' method.

Hindsight Memory-PRM, MemRL and CoEvo-Mem address a different problem and do not stand in a baseline relation to HABEAS. As described in §2, MemRL and CoEvo-Mem learn utility estimates or retrieval policies from feedback, and Hindsight Memory-PRM trains a manager whose actions exclude deletion; none decides at runtime whether to withhold, restore or delete a memory, the only decision HABEAS makes. Any comparison would need a deletion rule of our own design and would test our construction, not their method, so we have not run one.

Thank you again for your thoughtful review. We have posted our point-by-point response and uploaded the revised manuscript. We would appreciate your feedback on whether the revisions address your concerns.

Reviewer_aMfQOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates memory management for LLM agents, focusing on two limitations of history-based memory deletion: insufficient feedback coverage caused by selective retrieval and unreliable memory attribution based on task-level outcomes. The authors present theoretical analyses of these issues and propose HABEAS, an evidence-gated reversible memory management framework. Instead of permanently deleting memories based on poor historical feedback, HABEAS first quarantines potentially harmful memories and conducts paired audits by executing the same task with and without the target memory. Restoration and deletion decisions are based on evidence collected across distinct tasks and retrieval groups, while restored memories remain subject to further review. The framework also incorporates budgeted audit allocation using semantic ranking and prediction-distribution disagreement. The author evaluates HABEAS on CICIoT2023 classification and LAB-DB SQL tasks using two LLMs under natural and controlled memory-error conditions. The results show improvements in task success and reductions in incorrect memory deletion compared with history-based deletion, alongside ablations and comparisons with simpler reversible memory management policies.

Critical strengths
  1. The paper evaluates the proposed framework across two benchmarks, two LLMs, multiple memory-corruption settings, and multiple random seeds. It also includes component-wise ablations, comparisons with simpler reversible memory management strategies, and analyses of audit allocation and computational costs. These experiments provide a relatively detailed empirical characterization of the proposed framework under the considered settings.
Critical weaknesses
  1. Limited methodological novelty and insufficiently motivated heuristic design. The proposed framework primarily combines several intuitive engineering mechanisms, including reversible quarantine, counterfactual auditing, evidence-based decisions, and audit scheduling. While the specific implementations may differ from existing approaches, the paper does not clearly establish what fundamental methodological insight or novel design principle underlies their combination. Many of the components admit multiple reasonable alternatives, yet the necessity and distinctive advantages of the proposed choices are not sufficiently justified. The extensive experiments do not fully address these concerns about methodological novelty and design rationale.

  2. Disconnect between the theoretical analysis and the proposed method, resulting in an incomplete resolution of the motivating problems. The theoretical results mainly characterize relatively intuitive failure modes under simplified assumptions, but provide limited insight into or guidance for the actual algorithm. More importantly, the proposed method does not fully address the fundamental problems highlighted in the introduction. In particular, the feedback coverage issue remains largely unresolved for memories that are never retrieved, while the complexities underlying outcome-based attribution are only partially addressed through an through an expected reward contrast, without sufficient analysis of their implications for reliable decisions. Consequently, the theoretical analysis and methodological design do not form a sufficiently coherent or convincing solution to the stated research problems.

Questions for authors
  1. What is the fundamental methodological insight of HABEAS beyond combining several intuitive memory management mechanisms? In particular, how does the proposed framework differ conceptually from existing counterfactual memory auditing approaches, and what justifies its specific design choices over other reasonable alternatives? Could the authors provide stronger comparisons beyond the Simple-Reversible baseline to better establish the necessity of these choices?

  2. How does HABEAS address the feedback coverage and support deficiency problem emphasized in the introduction and Propositions 1–2? Since the proposed auditing procedure primarily targets memories that have already accumulated sufficient historical feedback to trigger quarantine, memories that are consistently excluded from retrieval may remain unobserved and unaudited. Does the proposed framework actually resolve this problem, or does it mainly address unreliable deletion decisions for previously retrieved memories?

  3. How do the theoretical results inform the design of the proposed algorithm? In particular, Propositions 2–3 analyze simplified retrieval and deletion rules that differ from the actual management procedure. Moreover, the proposed restoration and deletion criteria are not derived from these analyses. Can the authors clarify what concrete theoretical insights justify the algorithm's decision rules and whether these rules provide any guarantees on decision reliability?

  4. What supports the reliability and effectiveness of the proposed heuristic audit allocation strategy? The prediction-disagreement criterion relies on weighted historical predictions, whose estimation accuracy may be affected by limited, biased, or correlated observations. Have the authors examined whether this criterion reliably identifies informative audit pairs? Furthermore, given the additional computational cost and the inconsistent improvements over uniform allocation, what justifies the use of semantic ranking and prediction disagreement?

  5. What is the theoretical significance of Proposition 3 and the optimal-ordering analysis in Appendix A.4? Under the first-failure deletion rule, the eventual deletion of all good memories is a rather straightforward consequence of their nonzero failure probabilities. Similarly, placing bad memories before good ones naturally delays their deletion, but this optimal ordering relies on oracle knowledge and ignores task performance. Rather than focusing on such static results, wouldn't it be more meaningful to analyze the dynamic interactions among memory selection, feedback, and management decisions, particularly those involved in HABEAS?

  6. Are the experimental baselines sufficient to establish the advantages of HABEAS's specific design choices? The Simple-Reversible baseline relies on a relatively basic voting rule and may not provide a sufficiently strong comparison. The paper discusses related works such as MemAudit and Hindsight Memory-PRM, which explore alternative approaches to counterfactual memory evaluation. Why are relevant mechanisms from these works not adapted as alternative auditing or decision-making strategies under comparable experimental settings and budgets? Without stronger comparisons, it remains unclear whether the proposed heuristic designs offer meaningful advantages over existing alternatives.

4 comments
AuthorsOpenReview ↗

Response to Reviewer aMfQ (1/3): novelty and coverage (W1–W2, Q1–Q2)

Thank you for the detailed questions. Soundness is rated 1, while the listed weaknesses concern novelty and the theory–method link; if a specific proof step or experiment is incorrect, please point us to it.

W1 / Q1. Novelty and design rationale

Where we see this differently. We agree that paired counterfactual audits are not new, and the revision does not claim them as a contribution. The contribution is the runtime lifecycle: hold a memory on weak evidence, and take an irreversible action only on same-task evidence. Most of the evidence asked for is already in the submission, mainly in §6.2–6.3 and Appendices D–F.

  1. Components do not act independently (Qwen/Natural; Tables 2, A12, A15). Unmanaged retrieval 57.43%; audits without quarantine 57.57%; quarantine without audits 66.67% (history-based deletion, HM: 67.67%); both 73.80%. Audits add 0.14 pp without quarantine and 7.13 pp with it. Neither component alone exceeds HM.
  2. The obvious alternative fails (§6.3, Table A19). "Audit before deleting, but keep serving the memory meanwhile" is the w/o-quarantine ablation: 16.23 pp below HABEAS.
  3. The standard rule makes the error our design removes (Table A11). In CICIoT Natural, HM permanently deletes 24.54%/20.51% of source-correct memories; HABEAS deletes none.
  4. Prior audits answer a different question (§2.2). MemAudit ranks removals after a poisoning incident, A-MemGuard validates actions before execution, and Hindsight Memory-PRM derives training supervision without destructive updates. None runs a persistent runtime policy in which paired evidence governs both restoration and deletion.
  5. Other reviewers. Reviewers f4hq, 2e3d, 2sad and cgzg each list this separation or the same-task comparison as a strength. We agree the idea is simple; "hold, then decide" is standard in sequential testing (Wald, 1945). To our knowledge it had not been formulated or measured for agent memory.

On "many components admit alternatives." Six of the eight alternatives below were already compared in the submission; two are new.

Design choiceAlternativeSuccess vs HABEASSource
Immediate quarantineKeep serving while audited−16.23 pp§6.3, Table A19
Paired auditsQuarantine only−7.13 pp on Qwen/Natural; lower in 6 of 8 CICIoT cellsTable A12
Two-task/two-group gateSingle positive−1.57 pp; 3 incorrect restorations vs 0§6.3, Table A19
Same gateNet vote−0.88/−2.96 pp; 6/20 incorrect restorations vs 2/2§6.2, Tables A20–A21
Targeted allocationUniformHigher in 4 of 8 CICIoT cells, lower in 4Table A12, Appendix I
Post-restoration reviewRemoved−4.07 pp; 5 correct memories re-quarantined vs 0§6.3, Table A19
Semantic rankingRemoved+0.50/−0.93 pp; 16–18% fewer tokensNew
Zero-negative ruleRe-check; reopeningNo consistent gain; reopening admits incorrect restorationsNew

Immediate quarantine accounts for most of the gain; audits add to it in most CICIoT cells and are the only route to restoration and confirmed deletion. As §6.3 notes, the success differences for the gate and for post-restoration review are smaller and vary across seeds; the gate's clear effect is fewer incorrect restorations. Semantic ranking is not needed. The two new comparisons and a summary of this table will be added to the paper.

W2 / Q2. Does HABEAS resolve the coverage gap?

Thank you for pointing this out; our abstract and introduction did not state the scope clearly. HABEAS addresses part of both gaps, for memories that have been retrieved. Propositions 1–2 motivate two design choices: poor historical feedback triggers only a reversible quarantine, and permanent deletion requires corroborating evidence from comparisons on the same tasks with and without the memory.

Coverage. Audits observe memory–task pairs that retrieval did not select. Across 24 matched CICIoT runs they added 1,461 distinct same-class pairs, raising the observed pairs of flagged memories from 10,139 to 11,600 (+14.4%).

Attribution. Outcome averages misjudge useful memories: on CICIoT Natural, HM permanently deletes 24.54% (Qwen) and 20.51% (GPT) of source-correct memories. HABEAS decides by same-task comparison and deletes none of them (Table A11).

What remains. A never-retrieved memory cannot trigger an audit (§4.1), and most relevant pairs remain unobserved (Table 1). We will state this scope in the abstract and introduction and add the coverage analysis to the appendix. Spending part of the audit budget on unselected pairs is what we plan to study next.

W2's remaining points (theory-to-algorithm link, decision reliability) are answered with Q3 in the next comment.

AuthorsOpenReview ↗

Response to Reviewer aMfQ (2/3): theory and audit allocation (Q3–Q4)

Q3 (and the remaining points of W2). Theory, decision rules and guarantees

We disagree that the analysis gives limited guidance to the algorithm. The simplified assumptions in Propositions 2–3 isolate two failure modes of history-based deletion that call for different decisions. Proposition 2 shows that repeated success with one memory can keep another useful memory out of retrieval indefinitely. HABEAS therefore requires actual use before poor feedback can trigger quarantine (§4.1); a never-retrieved memory can remain unevaluated. Proposition 3 shows that even a beneficial memory with per-use success probability p<1p < 1 faces deletion risk 1−ph1-p^h over hh opportunities under first-failure deletion. This motivates reversible quarantine when service history raises concern, and corroborating evidence before permanent deletion. The propositions motivate this distinction; the two-task, two-nearest-neighbor-group threshold is a design choice evaluated by ablation (§4.3, §6.3).

Outcome attribution poses a further problem: a low service success rate can reflect difficult tasks (§3.2), and even a same-task audit measures the target's effect relative to the specified replacement and surrounding context. We tested how reproducible a single audit judgment is. For 738 originally directional audits, we reran both sides twice. Of 1,476 fresh comparisons, 62.80% retained the original sign, 33.94% became ties, and 3.25% reversed sign. These figures measure repeatability; the original sign is not ground truth. They support the existing rules: HABEAS treats ties as inconclusive, requires corroboration across tasks and retrieval groups, and keeps unresolved cases in quarantine. This gives a concrete procedural guarantee: a single negative audit cannot authorize permanent deletion. Cross-group corroboration does not establish independence or a calibrated decision-error rate (Appendix A.2). Its practical value is measurable in the submission, however. Relaxing the gate to a single positive admits 3 restorations of source-incorrect memories against 0 (Natural, §6.3). A net-vote gate admits 6 against 2 in Natural and 20 against 2 in Stress20, and permanently deletes 2.98%/6.11% of source-correct memories against none (§6.2, Tables A20–A21). Across Natural and Stress10/20/40, HABEAS permanently deletes no source-correct memory in any CICIoT cell (Table A13).

The repeatability measurement will be added to the appendix, and this mapping to the method section.

Q4. The allocation signals

Two premises of this question do not match the submission.

  • The signals cannot affect decision reliability. Semantic ranking and prediction disagreement only order which eligible pairs are tested first. They never count as evidence; every state change requires audit outcomes (§4.5). An unreliable signal costs efficiency only.
  • The comparison with uniform allocation does not test prediction disagreement. The two signals do different jobs. Semantic ranking chooses which quarantined memory to audit and costs a model call; prediction disagreement chooses the task on which to audit it and costs none (§4.5, Appendix B.1). Uniform allocation also selects contexts by disagreement. The "additional computational cost" is therefore that of semantic ranking alone, about two-thirds of the overhead (Appendix H).

The submission already analyzes this. Appendix D compares no-audit, uniform and targeted allocation in all sixteen cells (Tables A12–A14). That targeted allocation "has no consistent advantage over uniform allocation" is our own statement (Appendix I). The submission also already presents semantic ranking as dispensable, with its trade-off: targeted allocation gives the lowest terminal correct-memory unavailability in 6 of 8 CICIoT cells, its semantic ranking costs about half a million tokens per run, HABEAS runs without it on LAB (Appendix B.1), and the configuration without it is reported at 12–13% overhead (Appendix H).

Does the disagreement score identify informative pairs? We analyzed the audit logs of the main CICIoT runs. Among the 1,806 audit pairs with a pre-audit score and a valid outcome, the share that returned a direction (exactly one side correct) is 13.68% / 32.81% / 38.42% for score bins [0, 0.25) / [0.25, 0.5) / [0.5, 0.75) (95 / 1,521 / 190 pairs). The high bin stays above the low bin when only audits with both sides freshly executed are counted. This shows ranking value among audited pairs; the score is not a probability and says nothing about candidates that were not audited.

Semantic ranking alone. Removing it while keeping disagreement lowers tokens by 17.56% (Natural) and 15.76% (Stress10) and changes success by +0.50 and −0.93 pp. The low-cost disagreement signal is worth keeping on CICIoT; semantic ranking remains the optional component.

Both analyses will be added to the appendix, and this discussion moved into the main text.

AuthorsOpenReview ↗

Response to Reviewer aMfQ (3/3): Proposition 3 and baselines (Q5–Q6)

Q5. Proposition 3 and the ordering analysis in Appendix A.4

We agree that the limit is easy to see: with continuing use and a non-zero failure probability, first-failure deletion eventually removes every beneficial memory. Appendix A.4 answers a different, finite-horizon question, and two characterizations of it are not accurate.

  • "Relies on oracle knowledge and ignores task performance." The oracle is deliberate, and it is granted to the deletion rule, not used by our method. A.4 asks: with first-failure deletion held fixed, how much protection can reordering alone give beneficial memories? To make the answer as favorable as possible, the ordering may know each memory's type and optimizes only their retention. Comparing rankings on the same random outcome sequences, placing all bad entries first delays every beneficial deletion and attains the minimum at every finite deadline (A.4.4). Even so, in the example with 50 beneficial entries the minimum expected number of deletions rises from 10.63 to 34.11 as tasks per class go from 10 to 20 (A.4.7). The ordering buys this protection only by delaying the service of beneficial memories. This is the argument for changing the deletion rule itself. The derivation was never meant to optimize task performance, as A.4.4 states.
  • "Static results." The analysis is not static: it follows selection, stochastic feedback and deletion over a task stream and is solved by an exact recursion (A.4.5).

Dynamics of HABEAS. The submission tests this feedback loop in actual runs. On CICIoT Natural with Qwen, letting flagged memories stay in retrieval while they await audit raises the share of retrieval slots filled by source-incorrect memories from 24.67% to 46.70% and lowers task success from 73.80% to 57.57% (Table A15). Restoration shows the opposite path: four source-correct memories that HM deletes and HABEAS restores are then selected on 102, 103, 129 and 105 later tasks (Figure A1). These results show how one disposal decision changes later retrieval and which memories can still receive feedback. A formal characterization of this adaptive process remains open; we will state it as a limitation.

Q6. Baselines

We disagree that the baselines cannot establish the design choices. The submission compares HABEAS with eleven alternatives and references; each design choice has its own control:

Design choiceControlWhere
Using and managing memoryNo-memory, RAG, HM, OracleTables 2, A6
Immediate quarantinew/o quarantine; Audit-Delete§6.3; §6.2
Evidence gateSingle-positive restoration; Simple-Reversible§6.3; §6.2
Post-restoration revieww/o post-restoration management§6.3
Paired auditsB0 (no audits)Appendix D
Audit allocationUniformAppendix D
  • Simple-Reversible is the simplest reversible policy on purpose, and the comparison favors it. It shares HABEAS's audits, eligibility rules and budget caps. Its vote threshold was selected on three development blocks, while HABEAS was frozen and not retuned (§5.3, Appendix F). HABEAS is still 0.88/2.96 pp higher (Natural/Stress20), permanently deletes no source-correct memory against 2.98%/6.11%, and restores 2 source-incorrect memories per condition against 6/20 (Tables A20–A21).
  • MemAudit and Hindsight Memory-PRM are not baselines for this problem. MemAudit ranks removals after a poisoning incident. Hindsight Memory-PRM trains a manager whose actions are write, merge and no-op, with destructive updates excluded (§2.2). Neither defines a runtime rule for holding, restoring or deleting a memory. Their shared mechanism, counterfactual removal and re-answering, is our paired audit, which the submission already tests as a deletion driver without containment: Audit-Delete reaches 52.18%/52.48% against 67.32%/68.20% for HABEAS (Table A20).
  • An online adaptation of MemAudit. In a separate experiment we connected its counterfactual removal replay and structural-anomaly score to online service (three seeds; same Qwen model, initial banks and task order as the main runs). Success is 61.27%/55.27% (Natural/Stress20), against 73.80%/72.93% for HABEAS and 67.67%/72.57% for HM (Table 2). We do not plan to add this result to the main text. It evaluates an online policy that we built on MemAudit's scores, not a method its authors proposed or claimed. The published algorithm and HABEAS do not stand in a baseline relation, so the comparison is not meaningful as one.
  • Hindsight Memory-PRM. A comparison would require us to invent a rule that turns its learned scores into restoration or deletion decisions, match the audit budget and account for training cost. We have not evaluated such an adaptation; it would again test our construction, not their method.

We will bring the overview of controls into the main text.

Thank you again for your thoughtful review. We have posted our point-by-point response and uploaded the revised manuscript. We would appreciate your feedback on whether the revisions address your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes in the revised paper

We thank all five reviewers. We have uploaded the revised paper. The main text is 10 pages. New results appear in §§6.3–6.5 and Appendices J–N. Sections through §6.3 retain their numbering.

What HABEAS does (aMfQ, f4hq)

  • The abstract and §1 explain which memories HABEAS can manage. A memory that is never retrieved receives no feedback or audit (Appendix I). For a retrieved memory, the outcome of the whole task does not tell us how much that memory helped.
  • Prior work identified these two gaps and introduced paired audits. Each audit compares outcomes on the same task with and without the memory. Our analysis explains how these gaps can lead to the permanent deletion of a useful memory. HABEAS quarantines a suspect memory, then uses audits to decide whether to restore or delete it. Restored memories remain under review.
  • §3.2 explains how Propositions 2–3 motivate this sequence. Requiring evidence from two tasks in two groups is a design choice. We group tasks by which memory ranks first under the fixed retrieval rule.

How the method works (aMfQ, f4hq, 2sad, cgzg)

  • §4.2 explains replay from a saved starting point. Reusing an answer approximates the comparison in Eq. 2. In 141 quarantine audits, we reused a with-memory answer from the feedback that triggered quarantine. Fresh with-memory runs gave 21 correct answers, up from 16 (Appendix L).
  • §4.3 explains why quarantine is reversible. The memory is kept and can return to service. If the task succeeds without it and fails with it, that negative result prevents restoration for the rest of that quarantine episode.
  • §4.5 explains two ways to choose what to audit. Uniform selects a memory from a shortlist. Targeted asks the model to rank candidates. Both use prediction disagreement to help select a task. We compare the stored labels in two sets of memories, one with the target and one without it. Memories more similar to the task receive more weight. The score is the total variation distance between the two label distributions and requires no new model call. Restoration or permanent deletion requires supporting audit results on different tasks (§4.3).
  • §5.2 lists the settings that HABEAS shares with history-based deletion (HM), including the rule that triggers quarantine in HABEAS and deletion in HM.

Results

  • §1 compares mean success across sixteen settings (f4hq). On CICIoT Natural, HM permanently deleted 24.54% (Qwen) and 20.51% (GPT) of source-correct memories (three-seed means). HABEAS permanently deleted no source-correct memory in the 24 main CICIoT runs.
  • §6.3 and Table 3 give controls for individual design choices, including two ways to relax the restoration rule after a negative audit and a version without model-based ranking (aMfQ, cgzg).
  • §6.4 and Table 4 compare HABEAS with HM on five independent task pools (f4hq). Mean gains were 1.32 points for Qwen Natural, 3.16 for Qwen Stress20 and 1.70 for GPT Natural. All three pool-level 95% intervals included zero. The task-level analysis pairs outcomes within each pool and keeps the five runs fixed.
  • §6.5 varies the audit budget (2e3d, 2sad). At similar token budgets, HABEAS averaged 1.86 points above HM+vote (matched) in Natural and 2.44 in Stress20 (Table 5). Both paired 95% pool-level intervals included zero. It permanently deleted 54 and 34 correct memories, respectively. HABEAS deleted none.

Scope and supporting analyses

  • §7 and Appendix I explain the limits (aMfQ, 2e3d, 2sad, cgzg). Audits require tasks that can be repeated safely. A quarantined memory is unavailable while it waits for evidence, and one negative audit can block its restoration. We tested two benchmarks with fixed retrieval. A formal analysis of how retrieval, feedback and management affect one another remains open.
  • Appendices H and I discuss the findings and their limits. Appendix J gives results by pool and task class. Appendix K details the voting baselines and budgets.
  • Appendix L examines answer reuse, runs with fresh answers, the choice of replacement memory and audit repeatability. Appendix M tests restricted replay, noisy audits and a third restoration rule (f4hq, cgzg, 2sad). After a first negative, two positive audits on different tasks and in different groups allow restoration. None of the 38 episodes with both follow-ups qualified. Two involved source-correct memories. Appendix N reports how much coverage audits add and examines the signals used to choose tests. Appendices B.1 and D clarify the Uniform policy.

Writing and comparisons

  • We thank Reviewer 2e3d for pointing out two passages that were difficult to follow. We revised both as described in our response and read through the full manuscript sentence by sentence, looking for places where we could make our explanations clearer for readers.
  • Our reply to aMfQ explains why we report the online MemAudit adaptation separately. The separately tuned HM is reported in our reply to 2e3d. Both comparisons remain in the replies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.