acceptodds
Under review as a conference paper at ICLR 2027

Stop When Memory Suffices: Evidence-Conditioned Progressive Execution for LLM Agents

Abstract

Long-horizon LLM agents must answer queries over histories whose supporting evidence may be localized or dispersed across interaction steps. Lightweight retrieval can miss decisive dependencies, whereas full memory reconstruction incurs unnecessary cost when the retrieved context already suffices. The required processing depth therefore depends on the evidence recovered for a query, not on query complexity alone. We introduce Router-Mem, an evidence-conditioned progressive execution framework that learns whether to stop after a shared retrieval prefix or continue with broader memory processing. To distinguish sufficiency from relevance, we construct supervision by retaining or removing supporting evidence and discarding candidate negatives that remain answerable. We train a lightweight router with binary label supervision and rationale-conditioned representation distillation to make a single-token termination decision without generating rationales at inference time. When continuation is selected, retrieval hits anchor block expansion, parallel memory analysis, and aggregation instead of restarting memory search from scratch. With DeepSeek-V4-Flash at a routing threshold of 0.5, Router-Mem achieves scores of 55.17% on AMA-Bench and 38.77% on BEAM, reducing online memory-processing time by 27.3% and 25.5% relative to full completion, respectively. Code is available at https://anonymous.4open.science/r/Router-Mem-E0A5.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Agent, Memory, Router

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xzPDOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the trade-off between retrieval accuracy and inference overhead in long-horizon LLM agent memory systems. Existing memory approaches typically rely either on lightweight retrieval or pre-compression, which risks losing fine-grained dependencies, or on expensive query-time memory reconstruction and deep analysis, which incurs substantial compute and latency overhead when initial retrieval is already sufficient. To tackle this issue, the authors introduce the Router-Mem framework. Instead of relying solely on static query complexity, this framework models early termination as an evidence-conditioned progressive execution process grounded in the actual sufficiency of the retrieved evidence.

Critical strengths

The paper presents a clear problem motivation and a well-founded paradigm, breaking down long-horizon agent memory access into an evidence-sufficiency-conditioned progressive pipeline.

Critical weaknesses
  1. This paper forces a 1.7B student without reasoning tokens to mimic a teacher's hidden state derived from explicit CoT rationales creates a severe information bottleneck. The student likely learns surface statistical correlations rather than true causal sufficiency reasoning.

  2. On the million-token BEAM benchmark, Router-Mem drops 4.5% compared to Full Completion. This shows a noticeable false-positive exit rate where reasoning terminates prematurely despite missing key dependencies.

Questions for authors

Please see weaknesses

2 comments
Reviewer_xzPDOpenReview ↗

Response by Reviewer xzPD

Thank you for the clarifications and additional ablations.

However, I remain concerned about the reliability of its sufficiency judgments when key dependencies are missing. I understand that the drop in accuracy does not directly tell us how often the router stops too early. Still, I am not convinced that the efficiency gains justify the 4.49-point drop on BEAM. Overall, the response does not sufficiently change my assessment, and I maintain my original score.

Thank you for engaging with our responses — your follow-up directly prompted the paired deletion experiment in our most recent reply, which we believe is the most direct test of exactly your concern: reliability when key dependencies are missing.

A two-line summary, in case a compact form helps:

Removing the dependency-carrying entries flips 39/47 (83%) of the router's early exits to continuation; removing the same number of unrelated entries flips none (47/47 retained; paired Δ = +0.440, p = 1e-16; AUC 0.861). The two surface signals — the retrieval-score mean (AUC 0.548) and the untrained Qwen3-1.7B yes/no confidence (0.555) — both fail this test, so the router tracks whether the required evidence is present, not how much the context changed. On the 4.49-point concern: τ = 0.9 narrows the gap to 1.17 (42.09 vs. 43.26) at an 8.1% latency reduction — τ = 0.5 is one operating point, not a universal recommendation. The window closes within hours, so if any part of this remains unconvincing, we would be glad to clarify before then. We fully respect your assessment either way and a one-line reaction would mean a lot to us.

Reviewer_nfuMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Router-Mem, an evidence-conditioned progressive execution framework that learns whether to stop after a shared retrieval prefix or continue with broader memory processing. When continuation is selected, retrieval hits anchor block expansion, parallel memory analysis, and aggregation instead of restarting memory search from scratch. To distinguish sufficiency from relevance, the authors construct supervision by retaining or removing supporting evidence and discarding candidate negatives that remain answerable. They train a lightweight router with binary label supervision and rationale-conditioned representation distillation to make a single-token termination decision without generating rationales at inference time. The paper evaluates Router-Mem on AMA-Bench and BEAM, examining the trade-off between answer quality and online memory-processing time.

Critical strengths
  1. The progressive execution framework has practical value, supported by consistent AMA-Bench results.
  2. The supervision construction directly addresses the distinction between relevant and sufficient evidence.
Critical weaknesses
  1. The benefit of evidence-conditioned query selection remains insufficiently established on BEAM. Table 2 demonstrates a quality–latency trade-off, but does not adequately separate effective selection from reducing the frequency of slow-path execution. It reports fast-only and slow-only scores of 30.02% and 43.26%, with 26.6% early termination at τ=0.5. Conditional on fixed per-query endpoint outputs and scores, uniformly random routing at this rate has an expected score of (0.266(30.02%)+0.734(43.26%)=39.74%), compared with the reported 38.77%. This reference calculation does not establish inferiority or statistical significance, but makes a matched-rate random-routing control important. Comparisons using common per-query endpoint outputs, repeated random assignments, measured computation costs, and conversation-level uncertainty estimates would clarify the selection benefit. The stronger AMA-Bench results are encouraging, but the evidence does not yet establish that this benefit transfers consistently to BEAM.
  2. The independent contribution of rationale-conditioned representation distillation is unclear.
Questions for authors

None.

1 comment

Thank you again for a constructive review — the matched-rate control you proposed ended up motivating the core of our new analysis.

With the discussion window closing in a few hours, we wanted to ask whether the per-query-type diagnosis and the lightweight sufficient ∧ low-integration-burden gate we proposed address your concern about the selection benefit on BEAM, and whether the matched-budget ablation (+1.85 vs. +1.33 over matched random) sufficiently isolates the distillation contribution. A one-line reaction would be very helpful to us; we are of course happy to clarify anything before the deadline.

Reviewer_HgQ9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper works on long-term memory in LLM agents. They propose an evidence conditioned progressive execution framework that learns whether to stop after a shared retrieval prefix or continue with broader memory processing. And shows the performance on several benchmarks.

Critical strengths

The research question about long horizon memory is important. The fast and slow paths are reasonably controlled. This makes the quality–efficiency tradeoff easier to interpret. The experiment results are encouraging.

Critical weaknesses

The main concern is that the current BEAM results do not show that the learned router allocates computation better than a simple random allocation baseline. The paper does not sufficiently isolate the contribution of its most distinctive training components. Could we see classification-only training versus classification plus representation distillation, as well as ablations of evidence-intervention supervision and false-negative filtering.

Questions for authors

Could you report random routing and simple retrieval-score/confidence routing at matched early-stop rate or matched computational cost? Can per-example paired fast/slow result show that the learned router identifies examples for which the slow path provides larger marginal benefit? How much does rationale-conditioned representation distillation improve over a classification loss only router at the same computational budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.