- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HABEAS manages a fixed memory bank through active, quarantined, and deleted states. Poor historical feedback triggers quarantine, while paired executions of completed tasks with and without a memory inform restoration or deletion. Restored memories undergo further review. Experiments on traffic classification and SQL tasks compare the approach with history-based deletion and several lifecycle controls.
Critical strengths
-
Separating immediate containment from permanent deletion is a useful idea. The same-task inclusion/exclusion comparison also addresses a real weakness of assigning a task’s outcome directly to the retrieved memory.
-
The evaluation includes relevant controls beyond history-based deletion, particularly Simple-Reversible, uniform audits, and lifecycle ablations. Reporting service availability alongside deletion rates and audit costs makes the tradeoffs easier to assess.
Critical weaknesses
-
Quarantine can become irreversible after one negative audit. Equation 3 requires zero negative outcomes within an episode for restoration, while Appendix B.1 allows a new episode only after an active memory is quarantined again. Once a quarantined memory receives a negative audit, it therefore cannot recover under the stated rules, even if subsequent evidence is positive. Requiring two negatives for deletion protects the stored record, but not its service eligibility. This matters because roughly 17% of correct memories remain unavailable in CICIoT Natural despite zero deletion (Table A11). The design retains a form of irreversible removal based on potentially noisy evidence.
-
Historical-answer reuse may reintroduce outcome-selection bias. Quarantine depends on poor service outcomes, yet some audits reuse those historical with-target responses. Conditioning on this selection can make a reused response pessimistic relative to a fresh execution, even when the reuse rule itself does not inspect correctness. Matching the task and context is insufficient to establish the unbiased contrast in Equation 2 under this adaptive protocol. The fresh-versus-historical variability in Table A2 makes this worth checking directly; aggregate comparisons of nonrandomly assigned fresh and reused audits do not resolve it.
Questions for authors
NA
3 comments
Thank you for the detailed response and your efforts to address my concerns. The additional analyses and fully fresh audit experiments largely address my concern about historical-answer reuse, and I appreciate the clearer discussion of the availability–precision tradeoff. However, I remain concerned that a single potentially noisy negative audit can permanently block a memory’s return to service under the default policy, even with subsequent positive evidence. For now, I am inclined to maintain my original rating, but I would be open to raising it if additional evidence alleviates this remaining concern.
Thank you for the careful reading and for pressing this point. Your concern is well founded, and we agree with it. Under the default rule, one negative audit during quarantine blocks restoration, and nothing in the default rule lifts the block later. The submission stated the rule but did not examine this consequence. Below we report how often it occurs and a new experiment that tests whether later positive evidence supports restoration.
How often the block applies. A memory reaches this point after two signals against it: poor task feedback triggered its quarantine, and a paired audit then returned a negative. To answer your concern we counted these cases in the logs. In Natural, 6 of the 267 source-correct memories (2.25%) end the run in quarantine with a blocking negative; over all 24 main CICIoT runs the count is 22 of 879 (2.5%). We do not want to overstate what quarantine gives these memories. While a memory is blocked it does not serve, and within a run that is the same as losing it. On that basis the loss is still much smaller than under the baseline: in Natural, HM permanently deletes 24.54% (Qwen) and 20.51% (GPT) of correct memories. What quarantine adds is that the decision stays correctable. The content and the audit record are kept, so a restoration step can be added, as the experiment below does. We agree that the default rule does not yet use this for memories with a negative.
What the existing runs show.
- Re-execution. When the negative is replaced by the result of re-executing the same audit, 3 of the 22 would have been restored at that point. A noisy negative therefore did cost restorations, for 3 of 879 correct memories.
- Later evidence. In the main experiment and the five independent task pools added during this rebuttal at Reviewer f4hq's request (§6.4), 812 quarantine episodes received a first negative. 64 of them were audited once more and none twice, so these runs do not show whether two later tasks would have given positives. We therefore designed and ran the new experiment below.
New experiment: two real chances after a first negative. The rule follows your sentence that a single noisy negative can block a memory "even with subsequent positive evidence". The original negative is kept. Available audit slots go first to memories with a first negative, and a memory is restored if it then obtains positives on two different tasks from two different nearest-neighbor groups. Task order, memory bank and audit budget are those of the default runs (Qwen, Natural and Stress20, three paired seeds).
Of the 89 episodes with a first negative, 74 received at least one further audit. The first further audit returned 44 ties, 30 negatives and no positive. 38 episodes completed two further audits in two groups: 29 were tie/tie, 8 tie/negative and 1 tie/positive. One episode therefore showed positive evidence on a later task, none showed it on two tasks, and no memory was restored through the new path. Only 2 of the 38 belong to correct memories, both tie/tie, so for correct memories the sample is very small. Success changes by −2.57 pp [−8.78, 3.65] in Natural and −0.43 pp [−2.74, 1.87] in Stress20 relative to the default. In this experiment the block therefore made no practical difference.
Why we keep the rule for now. Audits raise success from 66.67% to 73.80% (Qwen, Natural; Table 3), but their outcomes are noisy in both directions. During quarantine, 22 of the 531 negatives belong to correct memories, and 74 of the 133 positives belong to incorrect ones, because most quarantined memories are incorrect. A rule can therefore only choose which error the noise produces, and rules that let positives outweigh a negative return incorrect memories to service. In the five policy-comparison blocks, the net vote of Simple-Reversible restores 14 correct and 6 incorrect memories in Natural (13 and 20 in Stress20), against 7 and 2 (8 and 2) for HABEAS, and HABEAS has the highest success of the policies compared (Tables A20–A21).
What can be improved. You are right that the design can be improved here, and we are grateful that you pointed it out. The three cases above suggest a first step: a negative that would block a memory which already has positive evidence should be re-executed before it takes effect. This repeats only the decisive audits, unlike the blanket confirmation in our first reply. A second step is to let the block expire on evidence, with a stricter requirement than for a first restoration, so that later positives can lift it without admitting incorrect memories. We have not validated either step and do not claim them. The revised Appendix I will state the block as a limitation and describe these directions as future work, and Appendix M will report the counts and the new experiment. Thank you again for the careful review.
Thank you for checking Eq. 3 against Appendix B.1 and for the precise reading of the reuse rule. We tested both concerns directly.
W1. One negative audit blocks restoration
You are correct: one valid negative prevents restoration within the current quarantine episode, and the submitted policy cannot reopen an episode while the memory stays quarantined. The submission states the zero-negative condition (Eq. 3) but not this consequence. We will state it in §4.3 and define "reversible" as "content retained and restorable on evidence", not as a guaranteed return to service.
How much of the unavailability it explains. In CICIoT Natural, 25 (Qwen) and 20 (GPT) source-correct memories are still quarantined at the end of the three seeds. Three per model had received a negative in their final episode. The largest groups, 17 and 15 entries, had received only ties and still had audit allowance left. A single negative can therefore block recovery, but inconclusive audits explain most of the observed cases.
Is a single negative reliable? Not fully, as you suspect. When we re-executed both sides of 738 directional audits twice, 62.80% of the 1,476 comparisons kept the sign, 33.94% became ties and 3.25% reversed. This is why no single audit can trigger an irreversible action: deletion requires negatives on two tasks and two nearest-neighbor groups, and ties are inconclusive (§4.3). A single negative only keeps a memory out of service, with its content retained.
What relaxing the rule does.
Reopening thus gives a real availability benefit without a consistent gain in success, and the strict rule trades availability for restoration precision. We keep it as the default and will report both relaxations. We agree that zero deletion is not zero availability cost: in the main Natural runs HABEAS ends with 17.09%/17.11% of source-correct memories in quarantine, where HM has deleted 24.54%/20.51% (Table A11).
W2. Reuse of historical answers and outcome selection
You identify a genuine selection dependency. All 141 historical with-target answers reused in quarantine audits had contributed to the feedback that triggered that quarantine. Eq. 2 defines a contrast between two specified executions. For a reused answer to estimate its side without bias, its expected score must match that of a fresh answer after conditioning on audit selection, and matching the task and context does not establish this.
The revision will state the scope of Eq. 2 explicitly: pairs with both sides fresh estimate it without bias, and pairs with a reused side are a cost-saving approximation, reported with its measured deviation. Fully fresh auditing is the unbiased option. We did not randomize reuse among concurrent candidates; our evidence is same-event re-execution plus whole-run variants.