- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper targets "rare-but-critical one-shot memories", constraints that are stated once but whose omission can cause unsafe or non-compliant actions, and argues that recurrence- and salience-based admission systematically misses them. ROMC uses a teacher LLM to annotate each utterance with a five-dimensional ordinal risk vector and with trigger/boundary/non-trigger probes carrying remember-vs-forget deviation and severity scores. These are aggregated (Eqs. 3–5) into a PROMOTE/PROBATION/DISCARD decision and distilled into linear heads over a frozen BGE-M3 embedding, with teacher fallback near decision boundaries. A separate logistic-regression activation head controls read-time hard/soft injection (Table 1). Evaluation covers a 90-item write-side benchmark (NatRisk, Table 2), a 1,000-task end-to-end benchmark (Final-1000, Table 3), and extensive appendix analyses.
Critical strengths
Problem framing and evaluation design. Separating raw retention from durable hardening (App. B.3), and pairing trigger tasks with single-entity confusion tasks (Sec. 4.4, App. C.1), exposes a trade-off that memory QA hides. Broad extractors reach high recall only with FPR 0.683–0.950 (Table 2) and OTR of 0.126–0.182 (Table 3). This protocol is reusable independently of the method. Deployability. The student takes about 18 ms per utterance versus 12.5 s for the teacher (App. E.1) and escalates about 16% of items (App. C.6). Downstream, it injects 3.99 memories per task versus 5–48 for the baselines, and it has the lowest operational tokens of any deployable method.
Critical weaknesses
The "counterfactual cost of forgetting" is asserted, not measured, and its contribution is not isolated. Eq. 2 is described as rollouts of agent F with and without m_t. In practice, the teacher sees only the isolated sentence (Fig. 1). In the same call that produces the structural ratings, it writes hypothetical with/without-memory behaviors and assigns them rubric scores (App. F, Steps 5–6), and M_t never enters. At deployment, S_probe is a linear regression from the same utterance embedding as S_struct (Sec. 3.3). The ablation (Sec. 4.3) is scored against teacher labels computed by the full Eq. 4–5 formula, so the "complementarity" result is partly circular. Even so, Fig. 25 shows Structure-only exceeding Full in F1 (0.783 vs. 0.762), and "no transient gate" identical to Full (0.762 / 0.089). This contradicts the text (FPR 0.095 / 0.122, and "Full achieves the best F1"). The evidence supports a distilled LLM risk rubric, not consequence-aware consolidation.
Downstream claims are overstated and not attributable to write-side admission. On Final-1000, ROMC's Strict CR (0.718) is below A-MAC (0.740) and RecMem-full (0.792). Its Overall lead (0.824 vs. 0.808, i.e., 16 of 1,000 tasks, single run, no CIs) comes entirely from lower OTR, because Overall weights trigger and confusion tasks equally. That weighting runs against the paper's asymmetric-cost motivation: with a 2:1 weight on trigger tasks, RecMem-full ranks first. The OTR advantage plausibly comes from the read-side head, which is trained on teacher probes built the same way as the confusion tasks, while the baselines inject a fixed top-k (Mean Inject exactly 5, 10, 25 or 48). No component-swap study is reported, and App. C.4 itself attributes the ranking to the read policy. On the write side, ROMC still misses 13 of 30 critical items (Recall 0.567, Table 2), which is exactly the failure mode the paper targets. Finally, RecMem-full achieving the best deployable Strict CR suggests that the claimed recurrence "consolidation gap" is largely an artifact of durable-only accounting.
Questions for authors
1: Are any with/without-memory behaviors in Eq. 2 obtained by executing a downstream agent? If not, please compare probe scores from real rollouts against the structural rubric alone, at matched teacher cost, using ground-truth rather than teacher-derived labels.
2:Table 2 reports 548 online tokens for 600 turns, whereas ~16% escalation with a teacher prompt of over 2k tokens implies far more. Was fallback active in Tables 2–3, and how are PROBATION records counted under durable-only scoring?