- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses when a long-horizon LLM agent should trust compressed long-term memory. At each decision checkpoint, a frozen scorer ranks the sufficiency of the retrieved memory route. A TRUST/RECOVER threshold is then calibrated by so-called conformal risk control (i.e. CRC) over statistical units like episodes, personas, sources, or books. The controller guarantees that the expected unit-averaged false-safe loss stays at or below a user-set budget alpha.
Critical strengths
Solid and well-controlled empirical methodology: Using paired fixed-state checkpoints, disjoint split units, pre-frozen scorers, and unit-level bootstrap resampling, the experimental setup is clearly thought through. Some findings are particularly useful for practitioners working with agent trajectories.
Critical weaknesses
W1. The controlled risk is a proxy label, disconnected from what deployment needs. The guarantee bounds the failure rate of a workload-specific operational label R_mem. For Retail, the label is "the proposed tool name matches a gold action": tool arguments and task completion are explicitly not required. ALFWorld's label is explicitly "not final episode success." Meanwhile, on-policy Retail shows zero final task success for every policy, including ReCoverMem. So the paper's own numbers demonstrate that a perfectly controlled false-safe rate does not translate into end-task utility. The guarantee is also marginal over units, not conditional on TRUST like "when I trust, how often am I wrong?".
W2. The trade the controller sells is weak in most evaluated regimes, and the guarantee does not hold where it is deployed. For coverage: at alpha=.10 only 10–22% of decisions are trusted; the geometric bound min(1, π+α) allows .38–.55, so roughly half of the attainable coverage is lost. For cost: on Retail the trust/recover cost ratio is ~0.95, so even perfect ranking saves at most ~2.4%, and candidate-based routing costs 1.78x until candidate generation is removed; at alpha=.05 only one backbone saves anything (3.9%). For guarantee: CRC assumes exchangeability between calibration and future units under a frozen scorer and frozen decision-generation protocol. But the deployed controller's own routing changes the states the agent subsequently visits — the policy violates its own assumption by construction. The paper demotes on-policy results to "deployment diagnostics," which is honest. But it concedes that the guarantee holds only in the offline paired protocol — the setting where the guarantee is least needed, since offline results can be re-evaluated at will. No distribution-shift experiment of any kind is reported.
W3. The RECOVER side is entirely uncontrolled, and the data suggest it is not a safe default. The false-safe loss is defined only on the TRUST branch. A RECOVER decision can fail — or actively damage an otherwise successful memory-route decision — with no constraint from alpha. The paper's own paired diagnostic quantifies this: on Retail, recovery repairs 6.6% of memory-route failures but harms 15.8% of memory-route successes — harm exceeds repair by more than 2x. The paired R_rec labels needed to see this are collected but used only for diagnostics, not inside the controller. The routing policy may therefore push decisions into a worse branch that does not count as risk, and the "risk-controlled" framing silently assumes recovery is the safe option. A three-way loss (trust-failure + recover-failure) or an explicit harm bound may close this gap, but the paper does not attempt it.
Questions for authors
Q1. The paired diagnostic shows recovery repairs 6.6% of memory-route failures but harms 15.8% of memory-route successes on Retail. If recovery is net-harmful in a standard regime, the TRUST/RECOVER dichotomy loses its premise that recovery is the safe default. Have you measured or considered a three-way controller — TRUST, RECOVER, ABSTAIN — with both failure modes inside the controlled loss?
Q2. Can you add any experiment on the guarantee's behavior under shift? A synthetic covariate shift between calibration and test units would suffice — e.g., episodes re-ordered, persona distribution re-weighted, or a host-memory swap with the scorer NOT refit. Table 3 refits and recalibrates per host, so it does not test transfer.
Q3. The oracle diagnostic implies ranking quality — not geometry — is the binding constraint on coverage. Yet the scorer is the least-explored component (logistic regression on hand-designed features; no learned or LLM-based scorer in the ablations). Could you give one stronger scorer to show whether the coverage gap is a scorer artifact or a fundamental limit of the protocol.
6 comments
W2: The guarantee where it is deployed
On-policy measurement on all 40 official Retail test episodes. Each policy executes only its own routed action and continues from the state it reaches, so routing changes every later state. We evaluate every executed TRUST decision with the frozen evaluator, replaying the gold-action tracker over each rollout's own action trace (472 controlled decisions per policy). The ReCoverMem trajectories differ from the logging trajectory in 39 of 40 episodes, with the first divergence at step 1–6, so this is a genuine policy-induced state distribution.
| Policy (, 40 episodes) | FS on matched offline states | FS on-policy [95% CI] | Coverage on-policy |
|---|---|---|---|
| Always Trust | .618 | .802 | 1.000 |
| Random + CRC | .072 | .135 | .171 |
| ReCoverMem + CRC | .075 | .056 | .121 |
Routing-induced states shift the failure rate sharply for an uninformed policy: Always Trust rises from .62 to .80. ReCoverMem's FS stays below and does not increase (paired change ), while CRC on a random ranking moves to .135, with its whole interval above . This is an empirical result on one cohort, not an extension of the fixed-state guarantee; it is consistent with the learned score trusting decisions that remain sufficient after the trajectory diverges.
Scope. PersonaMem, PrefEval and NarrativeXL are static-query workloads, so routing does not change future queries and the fixed-state guarantee is the deployment guarantee, provided units are exchangeable and the generation and evaluation protocols stay fixed. For sequential agents, offline calibration certifies a policy before deployment, which is exactly when no paired recovery outcome exists. The on-policy result above and the shift study below complement it empirically.
Coverage and cost. Coverage under CRC is set by how well the score ranks memory-sufficient decisions (Q3). The cost condition predicts Retail's ≤2.4% headroom ex ante. Where branch costs differ, a 3-fold rotation over all 150 NarrativeXL books gives cross-fitted token savings of 9.6% / 11.7% / 15.2% at , with FS ≤ in every cell.
Q1 / W3: A three-way controller that also targets recovery failures
On the premise. Whether recovery is a safe default depends on the workload. On Retail it is not: Always Recover reaches lower routed success than Always Trust (.362 vs. .382). On NarrativeXL it is: recovery repairs 73% of memory-route failures, and Always Recover succeeds on .78–.79 of questions versus .30–.46 for Always Trust. The three-way rule below makes this workload-dependent choice explicit instead of assuming recovery is safe.
- Rule. TRUST iff (the frozen ReCoverMem threshold). Otherwise RECOVER iff , else ABSTAIN. Here is a recovery-success score fit on with the same features.
- Loss. The loss counts executed failures of both kinds, . For fixed it is monotone in , and we select with the unit-level CRC procedure for a budget .
| , | Abstain | Executed-failure loss (test) | Executed accuracy (three-way vs. two-way) |
|---|---|---|---|
| Retail | .459 | .249 (≤ in 4/4) | .538 vs. .457 |
| PersonaMem | .514 | .270 (4/4) | .439 vs. .427 |
| NarrativeXL | .028 | .256 (3/3) | .737 vs. .734 |
The rule learns where recovery is a safe default and where it is not:
- Retail: executed accuracy rises by up to 16 points (Llama), and recovery harm inside the executed set is 1.6–5.1%.
- NarrativeXL: recovery repairs 73% of memory failures, and the rule trusts recovery accordingly.
- Stronger recovery: on the same Retail checkpoints, a stronger recovery operator lifts routed success from .356 to .472–.491 at unchanged FS (fixed-state comparison).
The executed-failure loss stays below on the test set in every configuration. Because and share the calibration split, we report these as realized test results rather than as a finite-sample guarantee; calibrating on a held-out split is a direct extension.
We thank the reviewer for a deep and precise review. Every experiment the reviewer asks for could be run offline, because we stored paired (Rmem,Rrec) labels at every checkpoint, and we ran all of them. We also bring forward an on-policy measurement that speaks directly to the deployment concern. All numbers use the frozen main-table data and match the revised paper (Appendix I).
Highlights