- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Previous:
MemGate reranks retrieved memories before they enter an agent’s context. A query-conditioned gate adjusts memory embeddings to favor memories appropriate for the current request, rather than merely related to it. The reported results show reduced memory-induced failures with broadly preserved utility, although the gains vary across risk categories.
Updated revirew:
The proopsed MemGate method reranks the selected memories before used by agents. the query-conditioned gate update memory embeddings to select memories that are appropriate for the current user query, rather than related to it. The experimenetal results show reduced memory-induced failures with broadly preserved utility, although the gains vary across risk categories.
Critical strengths
Previous:
The distinction between relevant and appropriate memories is useful. The method can be added without changing the backbone LLM or rewriting the memory store. The reductions in cross-domain leakage and jailbreak ASR are meaningful, and the cross-family re-evaluation strengthens the evidence. The reported retrieval overhead is modest and requires no additional LLM calls.
Update:
The motivation is sound: the distinction between relevant and appropriate memories is useful. More importantly, this method can be added without changing the backbone LLM or rewriting the memories, which is meaningful for cross-domain leakage and jailbreak. Beisdes, the reported retrieval overhead is modest and doesnot need the LLMs.
Critical weaknesses
Previous:
The gains on sycophancy remain small: failure rates improve by only 1.5–3 percentage points across the three judges. Adaptive attacks also expose a limitation. Under white-box oracle injection, MemGate’s ASR is 23.7%, compared with 41.6% for native retrieval; the lower 11.2% result requires the separately trained MemGate-AT variant. The MemTrapBench results are encouraging, but cover two categories close to the paper’s threat model. They support transfer to a different benchmark, not yet broad generalization to substantially different risks.
Uppdate:
The improvements on sycophancy is modest and the adaptive attacks is limitated. In the white-box oracle injection settings, MemGate’s ASR is lower, requiring the separately trained for MemGate-AT. Although the MemTrapBench results are encouraging, but cover two categories close to the paper’s threat model.
Questions for authors
- Do the conclusions hold when training/testing with a judge from a different model family, or with human labels?
- Can an adaptive attacker who knows the gate craft poisoning text that evades it?
- Is one gate call per candidate memory acceptable in latency/cost for production systems?
- How does MemGate compare to simple baselines such as LLM-as-judge filtering or perplexity filtering?
4 comments
Official Comment by Authors (2/2)
Q4. Latency and cost. MemGate makes no per-candidate model call: all candidates go through a 9M-parameter MLP in one batched forward pass. For 50 candidates this costs about 0.9 GFLOPs, roughly 1/18 of one output token from an 8B model, with no API cost, and the cost scales with rather than with the memory store. Measured memory-search latency rises by 0.02–0.12 s at P50 and at most 0.19 s at P90 (Table 5), versus 1.4–4.9 s for every LLM-judge filter (Table 2). With memory embeddings cached at write time, MemGate adds only 6 ms on GPU or 31 ms on CPU.
Q5. Baselines. Table 2 already includes four LLM-as-judge filters (pointwise and listwise, each with GPT-4o-mini and GPT-5.1). We add a perplexity filter: candidates whose perplexity under Qwen3-0.6B exceeds the 95th percentile of benign memories are removed, and the top- is taken by similarity.
| Method | Cross-D. FR↓ | Syco FR↓ | Tool-call FR↓ | ASR↓ | LoCoMo↑ | Extra calls | P50 (s)↓ |
|---|---|---|---|---|---|---|---|
| Native retrieval | 27.0 | 33.5 | 62.9 | 16.8 | 38.9 | 0 | 1.47 |
| Perplexity filter (new) | 26.5 | 33.5 | 61.9 | 16.4 | 38.7 | 0 | 1.62 |
| Pointwise LLM judge (GPT-5.1) | 8.5 | 27.5 | 56.2 | 6.9 | 36.7 | 50 | 4.61 |
| Listwise LLM judge (GPT-5.1) | 7.0 | 27.0 | 55.2 | 6.1 | 40.1 | 1 | 6.38 |
| MemGate | 3.5 | 31.5 | 54.3 | 4.4 | 40.8 | 0 | 1.59 |
The perplexity filter barely changes any metric, likely because inappropriate memories here are fluent text rather than perplexity outliers. It is also slower than MemGate.
Thank you again for your time and thoughtful feedback. We have posted our responses to your comments. Please let us know whether any concerns remain so that we can address them before the discussion period ends.
Thanks authors for the additional experiments and clarifications. The cross-family re-evaluation, batched-scoring explanation, and baseline comparisons address several of my concerns. The lightweight design and demonstrated reductions in memory-induced failures are valuable, but the remaining robustness and generalization is still limited. Therefore, I will keep my score.
Official Comment by Authors (1/2)
We thank the reviewer for the constructive questions. Unless noted, new experiments use the Table 2 setting (OpenClaw, GPT-4o-mini, Nc=50, k=20, 5 runs).
Q1. Judges from other model families. Two metrics already avoid the GPT-4o-mini judge: ASR is judged by Llama-Guard, and LoCoMo F1 uses no LLM judge. MemGate improves both in every setting. For the other three metrics, we re-judged the saved responses with judges from two other model families, using the same rubric and threshold.
All three judges show similar reductions: 21–25 points for cross-domain FR, 1.5–3 for sycophancy, and 7.6–8.6 for tool-call drift. Our conclusions do not depend on the judge.
Q2. Training-set size and unseen risk categories. We test generalization on MemTrapBench (Wang et al., 2026), which played no part in our training, data generation, or design, and differs from our training data in format and content. Each instance is an 18–40-turn dialogue in which an early prior is buried under unrelated turns and then invalidated by a final query that changes when it applies. We evaluate the two categories closest to our threat model:
Scores follow the official protocol (average of four dimensions, judged by GPT-5.2; higher is better). Because each dialogue yields only a few dozen memory units, all memory-enabled methods use k=5.
Without retraining, MemGate closes 67%–98% of the gap between native retrieval and the memory-free setting. Although it is trained on only 1,640 triples, it transfers to unseen dialogues. This suggests that it learns the pattern these failures share, a memory that matches the query in topic but not in scope, rather than memorizing training templates.
Q3. Adaptive attacker. We attacked the gate directly, starting from the PS-Bench planted memories, with two attacks:
Native retrieval faces the same attacks, with cosine similarity as the target. MemGate-AT is a variant trained with attack memories, generated on training queries, added as negatives.
Adaptive attacks raise ASR for every method, but MemGate stays well below native retrieval. The white-box attack is the strongest (23.7% vs. 41.6%), and adversarial training roughly halves it (11.2%). Under the more realistic black-box attack, MemGate reaches only 8.3% (5.6% with adversarial training), versus 27.5% for native retrieval.