acceptodds
Under review as a conference paper at ICLR 2027

ReCoverMem: Risk-Controlled Trust in Compressed Agent Memory

Abstract

Long-horizon LLM agents compress growing interaction histories into long-term memory, but always trusting compressed memory risks acting on insufficient evidence, while always recovering additional evidence forgoes selective memory use. We introduce ReCoverMem, a Trust/Recover controller that ranks decisions with a learned memory-sufficiency score and applies unit-level conformal risk control. Our audit shows that treating dependent decisions as independent examples weakens the finite-sample correction. Across three workloads and four downstream backbones, always trusting memory yields false-safe risk of 45–72%. At , ReCoverMem routes 10–22% of decisions to Trust, achieving lower false-safe risk and lower exceedance in all 12 configurations. When recovery is substantially more costly, selective trust also reduces execution cost: on NarrativeXL, ReCoverMem saves 8.5–13.6% of executed-branch tokens at and 3.9% at .

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent memory, long-term memory, conformal risk control, selective prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aTrKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses when a long-horizon LLM agent should trust compressed long-term memory. At each decision checkpoint, a frozen scorer ranks the sufficiency of the retrieved memory route. A TRUST/RECOVER threshold is then calibrated by so-called conformal risk control (i.e. CRC) over statistical units like episodes, personas, sources, or books. The controller guarantees that the expected unit-averaged false-safe loss stays at or below a user-set budget alpha.

Critical strengths

Solid and well-controlled empirical methodology: Using paired fixed-state checkpoints, disjoint split units, pre-frozen scorers, and unit-level bootstrap resampling, the experimental setup is clearly thought through. Some findings are particularly useful for practitioners working with agent trajectories.

Critical weaknesses

W1. The controlled risk is a proxy label, disconnected from what deployment needs. The guarantee bounds the failure rate of a workload-specific operational label R_mem. For Retail, the label is "the proposed tool name matches a gold action": tool arguments and task completion are explicitly not required. ALFWorld's label is explicitly "not final episode success." Meanwhile, on-policy Retail shows zero final task success for every policy, including ReCoverMem. So the paper's own numbers demonstrate that a perfectly controlled false-safe rate does not translate into end-task utility. The guarantee is also marginal over units, not conditional on TRUST like "when I trust, how often am I wrong?".

W2. The trade the controller sells is weak in most evaluated regimes, and the guarantee does not hold where it is deployed. For coverage: at alpha=.10 only 10–22% of decisions are trusted; the geometric bound min(1, π+α) allows .38–.55, so roughly half of the attainable coverage is lost. For cost: on Retail the trust/recover cost ratio is ~0.95, so even perfect ranking saves at most ~2.4%, and candidate-based routing costs 1.78x until candidate generation is removed; at alpha=.05 only one backbone saves anything (3.9%). For guarantee: CRC assumes exchangeability between calibration and future units under a frozen scorer and frozen decision-generation protocol. But the deployed controller's own routing changes the states the agent subsequently visits — the policy violates its own assumption by construction. The paper demotes on-policy results to "deployment diagnostics," which is honest. But it concedes that the guarantee holds only in the offline paired protocol — the setting where the guarantee is least needed, since offline results can be re-evaluated at will. No distribution-shift experiment of any kind is reported.

W3. The RECOVER side is entirely uncontrolled, and the data suggest it is not a safe default. The false-safe loss is defined only on the TRUST branch. A RECOVER decision can fail — or actively damage an otherwise successful memory-route decision — with no constraint from alpha. The paper's own paired diagnostic quantifies this: on Retail, recovery repairs 6.6% of memory-route failures but harms 15.8% of memory-route successes — harm exceeds repair by more than 2x. The paired R_rec labels needed to see this are collected but used only for diagnostics, not inside the controller. The routing policy may therefore push decisions into a worse branch that does not count as risk, and the "risk-controlled" framing silently assumes recovery is the safe option. A three-way loss (trust-failure + recover-failure) or an explicit harm bound may close this gap, but the paper does not attempt it.

Questions for authors

Q1. The paired diagnostic shows recovery repairs 6.6% of memory-route failures but harms 15.8% of memory-route successes on Retail. If recovery is net-harmful in a standard regime, the TRUST/RECOVER dichotomy loses its premise that recovery is the safe default. Have you measured or considered a three-way controller — TRUST, RECOVER, ABSTAIN — with both failure modes inside the controlled loss?

Q2. Can you add any experiment on the guarantee's behavior under shift? A synthetic covariate shift between calibration and test units would suffice — e.g., episodes re-ordered, persona distribution re-weighted, or a host-memory swap with the scorer NOT refit. Table 3 refits and recalibrates per host, so it does not test transfer.

Q3. The oracle diagnostic implies ranking quality — not geometry — is the binding constraint on coverage. Yet the scorer is the least-explored component (logistic regression on hand-designed features; no learned or LLM-based scorer in the ablations). Could you give one stronger scorer to show whether the coverage gap is a scorer artifact or a fundamental limit of the protocol.

6 comments

We thank the reviewer for a deep and precise review. Every experiment the reviewer asks for could be run offline, because we stored paired (Rmem,Rrec)(R_\text{mem},R_\text{rec}) labels at every checkpoint, and we ran all of them. We also bring forward an on-policy measurement that speaks directly to the deployment concern. All numbers use the frozen main-table data and match the revised paper (Appendix I).

Highlights

  • The risk control holds on-policy in our cohort. On all 40 official Retail test episodes run on-policy (39/40 trajectories diverge from the logging trajectory), ReCoverMem's realized FS is .056 [.031,.085][.031,.085], below α\alpha and below its offline FS on the same episodes (.075). Under the same policy-induced shift, Random + CRC rises to .135, above α\alpha over its entire interval, and Always Trust rises from .62 to .80.
  • Q1, three-way control. Adding ABSTAIN, with both failure modes in the loss, raises Retail executed accuracy from .457 to .538, keeps the executed-failure rate within budget in every configuration, and abstains on only 3% of NarrativeXL decisions, where recovery is reliable.
  • Q2, shift without refitting. Backbone swaps keep FS ≤ α\alpha with the transferred threshold, and two of three host swaps need no target labels at all. Online adaptation (ACI) holds long-run FS at .098–.102 under every PrefEval backbone shift.
  • Q3, stronger scorer. A Qwen3.8-27B judge under the same CRC raises coverage from .164 to .195 and lowers trusted error from .51 to .41, with realized FS still near α\alpha.

W2: The guarantee where it is deployed

On-policy measurement on all 40 official Retail test episodes. Each policy executes only its own routed action and continues from the state it reaches, so routing changes every later state. We evaluate every executed TRUST decision with the frozen evaluator, replaying the gold-action tracker over each rollout's own action trace (472 controlled decisions per policy). The ReCoverMem trajectories differ from the logging trajectory in 39 of 40 episodes, with the first divergence at step 1–6, so this is a genuine policy-induced state distribution.

Policy (α=.10\alpha=.10, 40 episodes)FS on matched offline statesFS on-policy [95% CI]Coverage on-policy
Always Trust.618.802 [.739,.860][.739,.860]1.000
Random + CRC.072.135 [.102,.165][.102,.165].171
ReCoverMem + CRC.075.056 [.031,.085][.031,.085].121

Routing-induced states shift the failure rate sharply for an uninformed policy: Always Trust rises from .62 to .80. ReCoverMem's FS stays below α\alpha and does not increase (paired change −.019-.019), while CRC on a random ranking moves to .135, with its whole interval above α\alpha. This is an empirical result on one cohort, not an extension of the fixed-state guarantee; it is consistent with the learned score trusting decisions that remain sufficient after the trajectory diverges.

Scope. PersonaMem, PrefEval and NarrativeXL are static-query workloads, so routing does not change future queries and the fixed-state guarantee is the deployment guarantee, provided units are exchangeable and the generation and evaluation protocols stay fixed. For sequential agents, offline calibration certifies a policy before deployment, which is exactly when no paired recovery outcome exists. The on-policy result above and the shift study below complement it empirically.

Coverage and cost. Coverage under CRC is set by how well the score ranks memory-sufficient decisions (Q3). The cost condition predicts Retail's ≤2.4% headroom ex ante. Where branch costs differ, a 3-fold rotation over all 150 NarrativeXL books gives cross-fitted token savings of 9.6% / 11.7% / 15.2% at α=.10\alpha=.10, with FS ≤ α\alpha in every cell.

Q1 / W3: A three-way controller that also targets recovery failures

On the premise. Whether recovery is a safe default depends on the workload. On Retail it is not: Always Recover reaches lower routed success than Always Trust (.362 vs. .382). On NarrativeXL it is: recovery repairs 73% of memory-route failures, and Always Recover succeeds on .78–.79 of questions versus .30–.46 for Always Trust. The three-way rule below makes this workload-dependent choice explicit instead of assuming recovery is safe.

  • Rule. TRUST iff s>τs>\tau (the frozen ReCoverMem threshold). Otherwise RECOVER iff g>λg>\lambda, else ABSTAIN. Here gg is a recovery-success score fit on RrecR_\text{rec} with the same features.
  • Loss. The loss counts executed failures of both kinds, 1{TRUST,Rmem=0}+1{RECOVER,Rrec=0}\mathbb{1}\{\text{TRUST},R_\text{mem}=0\}+\mathbb{1}\{\text{RECOVER},R_\text{rec}=0\}. For fixed τ\tau it is monotone in λ\lambda, and we select λ\lambda with the unit-level CRC procedure for a budget β\beta.
α=.10\alpha=.10, β=.30\beta=.30AbstainExecuted-failure loss (test)Executed accuracy (three-way vs. two-way)
Retail.459.249 (≤ β\beta in 4/4).538 vs. .457
PersonaMem.514.270 (4/4).439 vs. .427
NarrativeXL.028.256 (3/3).737 vs. .734

The rule learns where recovery is a safe default and where it is not:

  • Retail: executed accuracy rises by up to 16 points (Llama), and recovery harm inside the executed set is 1.6–5.1%.
  • NarrativeXL: recovery repairs 73% of memory failures, and the rule trusts recovery accordingly.
  • Stronger recovery: on the same Retail checkpoints, a stronger recovery operator lifts routed success from .356 to .472–.491 at unchanged FS (fixed-state comparison).

The executed-failure loss stays below β\beta on the test set in every configuration. Because τ\tau and λ\lambda share the calibration split, we report these as realized test results rather than as a finite-sample guarantee; calibrating λ\lambda on a held-out split is a direct extension.

Reviewer_DP89OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

ReCoverMem addresses a practical question in memory-augmented agents: when is compressed memory sufficient for the current decision, and when should the agent recover information from its source? A frozen scorer ranks memory sufficiency, and conformal risk control calibrates a TRUST/RECOVER threshold using episodes or sources as statistical units. The guarantee bounds expected unit-averaged false-safe loss under exchangeability. Evaluation spans multiple memory workloads and backbones, alternative host memories with refitting and recalibration, and candidate-based and candidate-free execution. The paper studies both the statistical risk-coverage trade-off and whether selective recovery actually reduces inference cost.

Critical strengths
  1. The statistical formulation takes dependence and the meaning of the guarantee seriously. Calibrating on episode/source units avoids treating correlated decisions as independent calibration examples. The paper also defines the false-safe loss explicitly, states the exchangeability requirement, and separates marginal risk from error conditional on TRUST. These distinctions are consequential for agent-memory evaluation: a low false-safe rate can otherwise be obtained simply by trusting memory rarely, and a large checkpoint count can obscure a small number of independent sources. The unit-aware formulation provides a more defensible basis for comparing memory-trust policies.

  2. The operational evaluation reveals useful limitations rather than presenting calibration alone as sufficient. Paired memory/recovery outcomes, alternative host memories, and the candidate-generation ablation help explain when a seemingly attractive routing policy fails to save resources. Table 4 is particularly informative: candidate-based execution has normalized input cost about 1.781, whereas the separately fitted candidate-free variant is about .995. The paper explicitly distinguishes the intervening accounting counterfactual from an executable candidate-free controller. This cost decomposition is useful knowledge for practitioners designing selective memory retrieval.

Critical weaknesses
  1. The additional value of learning the sufficiency ranking remains uneven across workloads. At alpha=.10 in the primary fixed-state experiments, learned ranking has higher coverage point estimates than Random+CRC in 10 of 12 configurations; three paired 95% coverage-difference intervals exclude zero, although achieved false-safe risk also differs between policies. However, the candidate-free NarrativeXL experiment is less conclusive. In Table A12, all paired learned-minus-random intervals for coverage and success include zero across the three evaluated backbones. These results leave open whether the learned scorer provides a practically material improvement in the long-history regime, rather than whether calibration itself is useful. The scorer diagnostics in Table A11 are informative, but ranking metrics alone do not establish a better calibrated quality-cost operating point. Since learned memory sufficiency is a central component of the proposed controller, the paper would benefit from a clearer account of which workload properties make that component worthwhile.

  2. The reported operating points leave the application-level value dependent on the use case. At alpha=.10 in the three primary fixed-state workloads, unit-equal TRUST coverage is 9.6–21.6%. Appendix D already reports unit-first TrustedError of approximately .075/.182=.41 for Retail/Llama; this is a disclosed property of the operating point, not a contradiction of the marginal guarantee. On the fixed-question NarrativeXL evaluation at alpha=.10, the 8.5–13.6% reductions in book-equal mean executed-branch input-plus-output tokens relative to Always Recover accompany paired reductions of 3.0–7.7 percentage points in routed-answer success. At alpha=.05, Gemma saves 3.9% of tokens (95% CI [2.5, 5.3]), with a routed-answer success difference of −0.5 percentage points (95% CI [−1.5, 0.5]); this is a promising milder trade-off, although the interval does not establish noninferiority. In addition, executed-branch token cost excludes maintenance, scorer training, and calibration and does not measure end-to-end latency. The remaining practical question is which combination of trusted error, task success, and fully accounted cost justifies selective recovery.

My main reservation concerns practical significance: how much the learned controller adds at an operating point that offers a useful task-quality and cost trade-off.

Questions for authors
  1. What explains the stronger learned-ranking gains in Table A8 and the inconclusive learned-versus-random differences in Table A12? Which existing results best establish an advantage after considering achieved false-safe risk, rather than equal nominal alpha alone?

  2. Building on the TrustedError definition and example already in Appendix D, which reported operating point would the authors recommend for a concrete use case? Please consider coverage, unit-first trusted error, task success, and cost together, including their uncertainty where available.

  3. Building on Eq. (1), could you report candidate-free scorer latency for an existing configuration and estimate the incremental amortized calibration and maintenance cost relative to Always Recover? How do these measured overheads change the cost break-even condition under the actual branch-cost distribution? A cost breakdown for an existing configuration would help assess the proposed controller's deployment value.

3 comments

We thank the reviewer for the thoughtful review and for highlighting the unit-aware formulation and the cost decomposition. Below we answer the three questions with new analyses on the frozen main-table data, using the same scores, labels, splits and thresholds. All numbers below match the revised paper (Appendix I).

Q1: Where the learned ranking pays off after accounting for achieved FS

Why Table A8 and Table A12 differ. The main factor is calibration size. Table A12 calibrates on 50 books; subsampling PrefEval's 400 calibration sources to ncal=50n_\text{cal}=50 spreads the coverage gain over about ±.10\pm.10 from threshold variability alone, while Δ\DeltaTE stays between −.126-.126 and −.139-.139 at every ncaln_\text{cal} (Appendix I.2). The A12 coverage intervals are therefore wide mainly because of where calibration places each threshold, not because the ranking advantage disappears. Across configurations, the matched-FS coverage gain also tracks ranking quality (Spearman ρ=.85\rho=.85 with AUROC). With 3× the test books, the Gemma coverage gain is resolved (+4.8+4.8 pp, [+2.3,+7.4][+2.3,+7.4]) and TE is significantly lower on all three backbones.

We added three comparisons that do not rely on equal nominal α\alpha. Error among trusted decisions (TE = FS/Cov). Under any label-independent ranking, the expected TE equals the base failure rate 1−π^1-\hat\pi. TE therefore isolates how much the learned score purifies the trusted set.

  • At α=.10\alpha=.10, learned TE is lower than random TE in 14 of the 15 configurations of Tables 1–2.
  • Pooled over the 12 main configurations, the reduction is −10.1-10.1 pp ([−17.3,−3.4][-17.3,-3.4]).
  • On NarrativeXL/Gemma, TE drops from .462 to .312 (Δ\DeltaTE 95% CI [−.261,−.046][-.261,-.046]). At α=.05\alpha=.05 it drops from .45 to .19 on Gemma, and from .79 to .47 on Qwen.

Coverage at matched FS. At FS^≤.10\widehat{\mathrm{FS}}\le.10, the learned score retains significantly more coverage on:

ConfigurationCoverage gain at matched FS
Retail/Qwen+8.3+8.3 pp
PrefEval/Mistral+5.7+5.7 pp
NarrativeXL/Gemma+5.2+5.2 pp
Retail/Llama+4.7+4.7 pp

Lower FS and higher coverage than Random + CRC at the same α\alpha. In Tables 1–2 this holds for all three NarrativeXL backbones, PrefEval/Llama (.091/.201 vs. .108/.185) and PrefEval/Mistral (.095/.216 vs. .112/.181).

Pooled test (12 configurations). At α=.10\alpha=.10, ReCoverMem retains +3.7+3.7 pp more coverage than Random + CRC ([+1.4,+6.1][+1.4,+6.1], p=.002p=.002), at equal realized FS. We used a hierarchical paired bootstrap that resamples shared units jointly across backbones.

What drives the gain. Across the 14 configurations with paired logs, the coverage gain at matched FS tracks ranking quality at the top of the score distribution: Spearman ρ=.85\rho=.85 with AUROC and ρ=−.71\rho=-.71 with top-qq enrichment. A controlled score-quality experiment maps AUROC to the expected coverage gain (Figure A3, Appendix I.8): at AUROC ≈.60\approx.60 the expected gain is .03.03–.06.06, and gains of ≥.10\ge.10 require AUROC ≳.75\gtrsim.75, while the TE gap is already −.12-.12 to −.17-.17 at AUROC ≈.60\approx.60. A 3-fold rotation over all 150 NarrativeXL books triples the test size; with it the Gemma coverage gain is significant at every α\alpha (+4.8+4.8 pp at α=.10\alpha=.10), and TE is significantly lower for all three backbones. Practical rule: the learned component pays off when the scorer enriches the top of the ranking and calibration has on the order of 10210^2 units; with fewer units, its advantage shows up first in TE rather than in coverage.

Q2: Recommended operating point

Success loss is bounded by α\alpha (new proposition). In fixed-state routing, unit by unit, routed−AR=E[1{TRUST}(Rmem−Rrec)] ≥ −FS.\text{routed}-\text{AR}=\mathbb{E}[\mathbb{1}\{\text{TRUST}\}(R_\text{mem}-R_\text{rec})]\ \ge\ -\mathrm{FS}. CRC therefore guarantees an expected success loss relative to Always Recover of at most α\alpha, with both branches scored by the same operational criterion.

Use caseOperating pointCov.TEΔ\DeltaSucc. vs. ARToken savingLatency saving
Long-history QA, quality firstNXL/Gemma, α=.05\alpha=.05.052.194−0.5-0.5 pp [−1.5,+0.5][-1.5,+0.5]3.9% [2.5, 5.3]3.5% [2.3, 4.8]
Long-history QA, savings firstNXL/Gemma, α=.10\alpha=.10.182.312−3.0-3.0 pp [−5.2,−1.0][-5.2,-1.0]13.6% [11.3, 16.3]12.4% [10.3, 14.8]
Short-history agent (Retail)Retail/Llama, α=.10\alpha=.10.182.412−0.6-0.6 pp [−1.9,+0.0][-1.9,+0.0]≤\le 2.4% (ceiling)--
  • Quality first (recommended): NarrativeXL/Gemma at α=.05\alpha=.05. Four in five trusted answers come from sufficient memory, accuracy stays within [−1.5,+0.5][-1.5,+0.5] pp of Always Recover, and tokens and latency both drop.
  • Savings first: the same configuration at α=.10\alpha=.10. It saves 13.6% of tokens, with the success change inside the α\alpha bound.
  • Short-history agents (Retail). Branch costs are nearly equal, so the cost condition caps the saving at about 2.4%. The value is safer action from memory: trusted error .38–.41 vs. .47–.63 for Random + CRC on Retail with Llama, Gemma and Qwen; Mistral is the exception (.56 vs. .41).

Q3: Latency, amortized costs and the break-even condition (NarrativeXL/Gemma)

  • Scorer. The candidate-free scorer makes no LLM call. It runs in 7.5 ms p50 on one CPU (2.6 ms with a GPU encoder), which is 1.9% of Trust-branch and 0.6% of Recover-branch latency.
  • Branches. The Trust branch uses 2,153 tokens and the Recover branch 8,548 (r=.252r=.252). We replayed the exact logged per-question prompt and completion lengths on vLLM (Gemma-3-12B, one RTX 5090, batch 1): 398 ms (Trust) vs. 1,256 ms (Recover) end to end. Routed latency savings are 3.5% at α=.05\alpha=.05 and 12.4% at α=.10\alpha=.10; these exclude the scorer, which lowers them by at most 0.6 points.
  • Branch-cost distribution. The cost ratio on the trusted set is rT=.253r_T=.253, against a mean of .252.252. The score is uncorrelated with recovery cost (Spearman .05), so Eq. (1) evaluated with the mean cost ratio reproduces the exact saving (13.6%).
  • One-time calibration and fitting cost. Fitting and calibration use about 7.7M tokens on 600 training and 600 calibration questions (12.8M if both branches are also collected on the calibration questions for diagnostics, as we did). Scorer fitting is CPU-only.
  • Maintenance cost relative to Always Recover. If the host already maintains memory, ReCoverMem adds none, since it reads but never modifies that memory. Relative to an Always-Recover-only system that keeps no memory, the Mem0 writer adds about 1.74M tokens per book.
  • Amended break-even. C(δ)Crec=Cov rT+(1−Cov)+s+mCrec\dfrac{C(\delta)}{C_\text{rec}}=\mathrm{Cov}\,r_T+(1-\mathrm{Cov})+\dfrac{s+m}{C_\text{rec}}, with N⋆=CfitCov(1−rT)Crec−s−mN^\star=\dfrac{C_\text{fit}}{\mathrm{Cov}(1-r_T)C_\text{rec}-s-m}. ReCoverMem reads but never modifies host memory, so mm is not incremental for a host that already maintains memory, and s=0s=0 tokens. Routing saves 1,164 tokens per decision at α=.10\alpha=.10, and the one-time fitting cost is recovered after 6.6K–11K decisions.

We have added the operating-point table (Table 4), the cost ledger and the break-even analysis (Appendix I.4) to the paper.

Reviewer_6E4GOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReCoverMem, a controller that decides whether to trust compressed agent memory or recover additional historical evidence. It combines a learned sufficiency score with unit-level conformal risk control and evaluates the resulting risk–coverage and cost trade-offs across several workloads.

Critical strengths

The problem is well motivated, and the treatment of dependent calibration decisions is careful. I particularly appreciate the analysis of effective calibration-unit counts, the distinction between false-safe risk and downstream success, and the transparent reporting of negative results.

Critical weaknesses

My main concern is the incremental contribution over existing work. TierMem already uses learned sufficiency routing between compressed memory and raw evidence, while the risk guarantee here follows directly from standard conformal risk control. The unit-level formulation is sensible, but the theoretical results do not appear to introduce much beyond this application.

The practical gains are also somewhat limited. Learned scoring improves coverage over Random + CRC in some settings, but the difference is statistically resolved in only three of the twelve main configurations. On NarrativeXL, where the token savings are most meaningful, the learned-versus-random coverage differences are not resolved, and the savings come with reduced answer accuracy.

Finally, the controlled risk does not directly translate into reliable agent actions. It is an unconditional false-safe measure, not the error rate conditional on TRUST. In Retail, the operational success label only checks the tool name, recovery rarely repairs memory-route failures, and the on-policy experiments achieve zero final task success. These limitations are acknowledged, but they leave the practical benefit of risk-controlled memory trust less convincing.

Questions for authors

Beyond applying CRC to an existing memory-sufficiency router, what is the main new capability established by ReCoverMem? In particular, can the authors clarify where the learned controller provides a meaningful advantage over simpler CRC-based routing?

2 comments

We thank the reviewer for the careful review and for recognizing the treatment of dependent calibration decisions and the effective-unit analysis. The central question is what ReCoverMem adds beyond putting CRC on an existing router, and where the learned controller beats simpler CRC routing. We ran exactly that comparison, together with a pooled significance test and a larger NarrativeXL evaluation. All numbers use the frozen main-table data and match the revised paper (Appendix I).

The main new capability

ReCoverMem turns "should the agent act on compressed memory?" into a user-set, finite-sample-guaranteed risk budget across dependent agent decisions, and it comes with three components that existing routers lack:

  1. Unit-level calibration, shown to be necessary. With decision-level counting, duplicating decisions twice pushes test FS from .060 to .106 (> α\alpha). Unit-level CRC stays at .060 from 1$\times$ to 10$\times$ duplication.
  2. A guarantee on utility. We add the proposition routed−AlwaysRecover≥−FS\text{routed}-\text{AlwaysRecover}\ge-\mathrm{FS}, unit by unit, with both branches scored by the same operational criterion. Under CRC, the expected success loss is therefore at most α\alpha. Empirically, the inequality holds in all 171 cells we evaluated: every memory-backbone × recovery-backbone pair × 3 risk targets on Retail, PersonaMem and PrefEval (4×4×3 each) and NarrativeXL (3×3×3).
  3. Ex-ante diagnostics. The geometry (Cov≤π+α\mathrm{Cov}\le\pi+\alpha) and the cost condition predict where selective trust pays off before deployment.

TierMem + CRC: the learned controller vs. CRC on an existing router

We re-ran the released TierMem router on its native prompts, turned its decision into a continuous score P(S)P(\texttt{S}), and applied the identical unit-level CRC on each backbone's calibration labels. The re-run reproduces the archived TierMem decisions on 97–98% of test decisions.

α=.10\alpha=.10 (mean over 4 backbones)TierMem + CRC, best scoring mode (FS / Cov)ReCoverMem + CRC (FS / Cov)
Retail.054 / .126.069 / .161
PersonaMem.045 / .078.073 / .126
PrefEval.106 / .164.101 / .205
  • Coverage. Pooled over the 12 configurations, ReCoverMem retains +4.2 to +6.7 pp more coverage than TierMem + CRC (p≤.004p\le.004). It is higher in 11 of 12 configurations under either scoring mode, and in the native mode the gain is individually significant in 9.
  • Cost. ReCoverMem needs no extra LLM call. TierMem adds a 1.3k–2.9k-token router call per decision.
  • Quality of the trusted set. At α=.10\alpha=.10, mean trusted error is .509 for ReCoverMem + CRC vs. .556 for TierMem + CRC, and ReCoverMem's trusted sets are cleaner than the base failure rate in 11 of 12 configurations (8 of 12 for TierMem + CRC).

TierMem was trained to detect whether summaries contain an explicit answer for long-context QA. ReCoverMem's learned operational-sufficiency score is what converts the same risk budget into usable trust.

Practical gains

Pooled over the 12 configurations. We used a hierarchical paired bootstrap that resamples shared units jointly across backbones. At α=.10\alpha=.10, ReCoverMem vs. Random + CRC gives:

  • coverage +3.7+3.7 pp ([+1.4,+6.1][+1.4,+6.1], p=.002p=.002; higher in 10/12 configurations);
  • trusted error −10.1-10.1 pp ([−17.3,−3.4][-17.3,-3.4]);
  • statistically identical realized FS.

Adding the three NarrativeXL configurations (15 in total), the coverage gain is +3.3+3.3 pp and trusted error drops by 10.4 pp (p<.001p<.001).

NarrativeXL with 3× the test size. A 3-fold book rotation over all 150 books (scorer refit and threshold recalibrated per fold; fold 1 reproduces Table 2 exactly) gives, at α=.10\alpha=.10:

  • FS ≤ α\alpha on all backbones (.073–.078);
  • token savings of 9.6% / 11.7% / 15.2% (Qwen / Llama / Gemma);
  • Gemma coverage gain over random of +4.8+4.8 pp ([+2.3,+7.4][+2.3,+7.4]), significant at every α\alpha;
  • trusted error significantly lower for all three backbones (−9.6-9.6, −10.9-10.9, −15.0-15.0 pp).

Accuracy is a controlled quantity. By the proposition above, the success change vs. Always Recover is bounded by α\alpha; in the rotation it is −4.4-4.4 to −6.7-6.7 pp at α=.10\alpha=.10. At α=.05\alpha=.05, NarrativeXL/Gemma saves 3.9% of tokens and 3.5% of latency with accuracy within [−1.5,+0.5][-1.5,+0.5] pp of Always Recover.

Scorer-agnostic gains. Under the same CRC, a stronger zero-shot scorer (Qwen3.8-27B judge) raises coverage from .164 to .195. Its gain over Random + CRC is significant in 9 of 12 configurations (Appendix I, Figure A3).

From risk control to agent actions

  • Error given TRUST. TrustedError joins FS and coverage in the paper (Appendix I.2). ReCoverMem's trusted sets are cleaner than the base rate in 11/12 configurations, versus 5/12 for Random + CRC.
  • Operational labels. CRC controls any bounded monotone loss, so the label is an interface. Through the success-loss proposition, the guarantee transfers to whatever operational label is used for both branches.
  • Stronger recovery, same routing. Keeping the Llama memory route (FS = .075) and plugging in a stronger recovery operator lifts Retail routed success from .356 to .472–.491 (+12 to +14 pp), and the repair rate from 6.6% to 24–30%. This is a fixed-state comparison: trust decisions and recovery quality compose.
  • On-policy. On all 40 Retail test episodes run on-policy (39/40 trajectories diverge from the logging trajectory), ReCoverMem's realized FS is .056 [.031,.085][.031,.085], below α\alpha, while Random + CRC rises to .135 with its whole interval above α\alpha. End-task success is low for every policy, including Always Recover (0/20 on the original cohort), while all policies complete 52–56% of gold actions (tool names only), so routing is not what limits it (Appendix I.5).

We have added the TierMem + CRC baseline, the pooled test, the NarrativeXL rotation, the success-loss proposition and TrustedError to the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.