acceptodds
Under review as a conference paper at ICLR 2027

Memory Control Signals Emerge Before Action in Long Horizon Agents

Abstract

Long horizon language model agents continuously accumulate interaction history, increasing computational cost while making relevant information harder to preserve and reuse. Existing context management methods mainly focus on how to compress or retrieve history, but largely leave open whether the model itself already represents the need for these memory operations before they occur. We study the hidden state immediately before each agent action and find that compression and recall needs are already encoded in the model's internal representations. These signals cannot be explained by simple context length or interaction progress, and they exhibit distinct formation patterns across model depth. We further show that most memory decision information is preserved in a compact recent context, while selectively restored historical evidence complements the long range dependencies that recent context misses. Based on these findings, we propose Preaction Memory with Evidence Retrieval(PaMER), which combines state guided compression with external evidence retrieval. PaMER+ further introduces step level evidence selection to recover only the historical information required by the current task. Experiments on WorkBuddyBench, across multiple context management baselines and model backbones, show that our framework substantially reduces context consumption while maintaining competitive task performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-horizon agents, memory management, context compression, memory retrieval, hidden representations, adaptive context management

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_F5fGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether long-horizon LLM agents’ hidden states encode the need to compress context or recall historical information. Using probes trained on LLM-annotated decision points, the authors find predictive signals beyond context length and interaction progress, with compression signals generally strengthening across layers. Based on these findings, they propose PaMER, which uses lightweight hidden-state classifiers to trigger compression and embedding-based retrieval to recover stored summaries. PaMER+ selectively restores relevant original interaction steps.

Critical strengths
  1. The paper investigates whether preaction hidden states encode compression and recall needs, offering an interesting connection between internal model representations and agent memory control.

  2. The experiments examine layerwise signals, observable and text-based controls, compact-context variants, and ablations, alongside comparisons with baselines and evaluations across model backbones.

  3. PaMER+ reduces total token usage by 71.7% on the full WorkBuddyBench benchmark while maintaining competitive task performance.

Critical weaknesses
  1. The downstream evaluation is limited to WorkBuddyBench.
  2. The probes predict LLM-annotated compression and recall labels, which do not establish whether these operations actually improve task performance.
Questions for authors
  1. Can you validate the predicted memory needs through controlled interventions that compare task performance with and without compression or recall at the same decision point?
5 comments
AuthorsOpenReview ↗

Response to Weakness 1: Evaluation Beyond WorkBuddyBench

Dear Reviewer,

First, please accept our sincere thanks for the valuable time and effort you have devoted to reviewing our work with such care and rigor. We are especially grateful for your positive assessment of the research question and experimental analysis: your recognition of the connection between preaction hidden states and agent memory control, and of the systematic layerwise analysis, controls, baseline comparisons, and cross-backbone evaluation, is greatly encouraging. We also appreciate your recognition of PaMER+'s quality-token trade-off on the full WorkBuddyBench.

Your comments on evaluation scope and the effectiveness of memory operations helped us identify evidence that needed strengthening and improve our additional experiments and conclusions. We are deeply grateful for both your encouragement and your efforts to improve this work. Below, we address the two weaknesses you identified, followed by your question.

Weakness 1: The downstream evaluation is limited to WorkBuddyBench.

Thank you very much for recognizing the existing baseline and cross-model evaluations, and for encouraging us to examine other task environments. Your positive assessment encouraged us, while your suggestion clarified how to further assess applicability. We therefore added three benchmarks beyond WorkBuddyBench.

We extended evaluation to τ-bench, SWE-bench Verified, and Terminal-Bench 2.0, covering interactive tool use, software repair, and terminal tasks, respectively.

Within each benchmark, the base agent, PaMER, and PaMER+ use the same acting model and inference parameters; they differ only in context management. Exact model versions, inference parameters, and runtime configurations are provided in the revised experimental setup.

The table includes τ-bench overall and domain results, together with the SWE-bench Verified and Terminal-Bench 2.0 base-agent results already reported in our aggregate cost table.

BenchmarkScopeMethodSuccessful tasks/totalMean time (s/task)
τ-benchOverallBase agent113/16571.14
τ-benchOverallPaMER113/16573.28
τ-benchOverallPaMER+110/16578.92
τ-benchAirlineBase agent27/5073.11
τ-benchAirlinePaMER27/50102.80
τ-benchAirlinePaMER+28/50114.13
τ-benchRetailBase agent86/11570.28
τ-benchRetailPaMER86/11560.45
τ-benchRetailPaMER+82/11563.61
SWE-bench VerifiedOverallBase agent84/100820.52
SWE-bench VerifiedOverallPaMER92/100776.41
SWE-bench VerifiedOverallPaMER+91/100843.54
Terminal-Bench 2.0OverallBase agent48/89932.07
Terminal-Bench 2.0OverallPaMER61/891,117.60
Terminal-Bench 2.0OverallPaMER+52/891,567.74

SWE-bench Verified uses a fixed 100-task subset shared by all methods. τ-bench includes 165 tasks (50 Airline and 115 Retail); Terminal-Bench 2.0 includes 89. Each method runs once per task, and all tasks are included in performance and E2E statistics. PaMER+ retains the original implementation.

The results vary across environments:

  • On τ-bench, PaMER matches Base at 113/165 with slightly higher mean latency; PaMER+ achieves 110/165.
  • On Retail, PaMER preserves the success count while reducing latency from 70.28 to 60.45 s. On Airline, its success count is unchanged but latency increases.
  • On SWE-bench Verified, PaMER and PaMER+ solve 92/100 and 91/100 tasks, versus 84/100 for Base.
  • On Terminal-Bench 2.0, they solve 61/89 and 52/89 tasks, versus 48/89 for Base, but E2E latency increases.

These results extend the evidence beyond WorkBuddyBench and characterize the quality-cost trade-offs observed across the additional task environments.

Thank you again for helping us broaden the evaluation. The revision will report both benefits and costs and clearly specify each benchmark's task scope and settings.

AuthorsOpenReview ↗

Response to Weakness 2: Label Predictability and Operational Benefits

Weakness 2: The probes predict LLM-annotated compression and recall labels, which do not establish whether these operations actually improve task performance.

Thank you very much for recognizing the research connection between hidden states and memory needs. We also greatly value your caution that predictable labels do not establish actual task benefits. This observation helped strengthen our reasoning. We therefore added operational interventions and online recall gating to examine predictive signals and downstream benefits separately.

We agree that predicting LLM-annotated memory needs does not imply that compression or recall necessarily improves downstream agent performance.

1. Paired interventions on actual compression and recall

On the fixed Pilot40 list, we created three branches from the same decision-point snapshot of the environment, conversation, and external memory:

  • A: Retain the original history.
  • B: Compress, but skip the designated recall injection.
  • C: Compress and perform the designated recall injection.

The experiment covers 40 independent tasks and 120 terminal trajectories. Mean Reward for A, B, and C is 70.07%, 69.90%, and 74.58%, respectively.

The paired B-A difference is -0.175 percentage points, with a 95% CI of [-1.14, +0.79]. C-B is +4.680 percentage points, with a 95% CI of [+3.25, +6.11]. C-A is +4.505 percentage points, with a 95% CI of [+2.80, +6.21].

These comparisons examine the downstream effects of actual compression and recall operations. The compression CI includes zero and does not establish lossless compression; the mean recall benefit does not imply that all tasks benefit. Full settings and costs are reported in Response 3 below.

2. Online recall-probe gating

Separately from the same-state intervention, we conducted an online recall-gating experiment. Original PaMER follows its existing embedding-based retrieval procedure. The gated version first uses the frozen recall probe to decide whether retrieval should be triggered, invoking the same Qwen3-Embedding-8B retriever when needed.

MetricOriginal PaMERProbe-triggered Recall
Independent tasks4040
Mean Reward (%)69.9273.79
E2E time (s/task)709.72639.19

The gated version improves mean Reward by 3.87 percentage points and reduces mean E2E latency by 9.94%.

Online gating statisticResult
Total decision states286
Retrieval triggered109
Retrieval skipped177
Retrieval rate38.11%

This experiment shows that the learned recall signal can participate in actual retrieval decisions and improve mean Reward on this task set. However, the trigger rate alone does not establish that every skipped retrieval was correct, and average gains do not guarantee benefits for every task.

Thank you again for helping us clarify the relationship between these forms of evidence. The revision will distinguish probe predictability of memory-need labels, the effects of actual memory operations, and their impact on final task performance. We will not interpret probe AUROC directly as operational safety or optimality, but will use actual interventions and online ablations as complementary evidence.

AuthorsOpenReview ↗

Response to Question 1: Same-State Compression and Recall Interventions

Question 1: Can you validate the predicted memory needs through controlled interventions that compare task performance with and without compression or recall at the same decision point?

Thank you very much for recognizing the systematic experimental analysis and for suggesting controlled interventions at the same decision point. Your suggestion provided a clear, actionable direction for moving from label prediction to actual memory operations and their downstream effects. We greatly value this guidance and added same-state paired compression and recall experiments accordingly.

We conducted the interventions on the fixed WorkBuddyBench Pilot40 list: 10 tasks each from Code, Office, Security, and Web, totaling 40 independent tasks and 120 terminal trajectories.

Each task first reaches a predetermined memory decision point. We save the environment, conversation history, and external-memory snapshot, then create three branches from that same snapshot:

ConditionOperation at the designated decision point
A: Retain historySkip the designated compression and retain the original history
B: Compression onlyCompress, but skip the designated recall injection
C: Compression + recallCompress and perform the designated recall injection

Only the designated memory operations differ; subsequent behavior follows the original agent policy. A means retaining history at the designated point, not guaranteeing that the entire subsequent trajectory never compresses.

MetricA: Retain historyB: Compression onlyC: Compression + recall
Independent tasks404040
Mean Reward (%)70.0769.9074.58
Reward SD (pp)10.379.6010.22
API tokens/task1,062,969820,433905,749
9B input tokens/task398,415407,521422,296
8B input tokens/task0179,579280,097
9B CUDA s/task33.8936.6435.72
8B CUDA s/task017.6028.24
E2E s/task910.43640.19699.73

B still embeds and stores summaries; it skips only the designated query injection, so its 8B cost is nonzero.

Paired comparisonReward difference (pp)95% CITasks improved
B-A: Compression-0.175[-1.14, +0.79]19/40
C-B: Recall+4.680[+3.25, +6.11]33/40
C-A: Combined+4.505[+2.80, +6.21]32/40

Compression alone yields a small mean Reward difference with a CI including zero, while reducing API tokens by 22.82% and E2E latency by 29.68% relative to retaining history. This means that no clear average quality difference was observed; it does not establish lossless compression or non-inferiority.

Adding recall after compression increases mean Reward by 4.68 percentage points, indicating a positive effect of historical-information recovery on this task set. Average improvement does not exclude harm to individual tasks or establish that every probe-predicted memory operation is correct or optimal.

These paired interventions provide complementary evidence about operational effectiveness. Their conclusions remain limited by the task set, acting model, environment configuration, and randomness. We will retain these limitations in the revision and identify broader task coverage and repeated runs as future work.

Once again, we sincerely thank you for recognizing the research value, experimental analysis, and system results. Your positive assessment is an important encouragement, for which we are deeply grateful. Your specific suggestions on evaluation scope and operational effectiveness have also helped us identify and address shortcomings, making the evidence and conclusions clearer and better supported.

We truly appreciate the valuable time and effort you have devoted to understanding, evaluating, and improving this work. Thank you for recognizing its contributions while identifying directions for further improvement. Please accept our sincere gratitude and our wishes for continued success, good health, and a fulfilling, enjoyable, and intellectually stimulating academic exchange at this year's ICLR.

Reviewer_jYE4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates whether the hidden states of long-horizon language-model agents contain information about when to compress context or recover historical evidence. From 72,912 LLM-labeled decision points, the authors train lightweight probes on features extracted by a frozen Qwen3.5-9B model, reporting AUROCs of 0.831 for compression and 0.765 for recall. They then introduce PaMER, which uses hidden-state-based compression decisions together with summary storage and embedding retrieval, and PaMER+, which adds step-level evidence selection. On the full 260-task WorkBuddyBench evaluation, PaMER+ reduces token consumption by 71.7% while increasing the average score from 69.9 to 71.6.

Critical strengths

The paper addresses a practical challenge in long-running agents: growing interaction histories impose costs and can obscure relevant evidence. The probing analysis is relatively systematic, covering layer depth, last-token versus mean-pooled representations, recent-context budgets, and historical evidence recovery. Question-grouped data splits reduce the risk of leakage between highly correlated decision points. The system evaluation also includes multiple baselines and acting models, and reports performance regressions rather than only favorable results.

Critical weaknesses

My primary concern is that the proposed controller addresses when to compress, but not what should be compressed or preserved. PaMER retains the task prefix and two most recent interaction blocks, then summarizes older blocks when compression is triggered. Its compression probe reads the prefix and recent blocks, not the older material it is about to remove. Consequently, it cannot directly distinguish indispensable historical evidence from redundant content. Although content-aware compression methods are included as baselines, the comparisons focus on aggregate reward and token usage; they do not establish whether PaMER better preserves task-critical information under matched budgets.

The interpretation of the probing results also warrants caution. The target labels come from an LLM annotator, so predicting these labels does not necessarily mean identifying compression decisions that are safe or optimal for downstream success. Furthermore, although recall signals are analyzed extensively, the deployed system does not use the learned recall probe to trigger retrieval; it relies on embedding-based retrieval instead. This weakens the connection between the paper's two-signal motivation and its actual implementation.

Finally, the frozen 9B feature extractor introduces additional inference costs that are not adequately characterized by token-count reductions alone. PaMER+ also differs from PaMER in routing and other implementation details, making the isolated contribution of step-level selection difficult to assess. Improvements are not uniform across backbones.

Questions for authors

Can the authors evaluate the retention of critical evidence and the downstream effects of false-positive compression decisions?

How does PaMER compare with content-aware filtering at matched token budgets and with full end-to-end latency and compute accounting?

Can a controlled ablation isolate step-level selection from other PaMER+ implementation changes?

7 comments
AuthorsOpenReview ↗

Response to Weakness 1: Critical-Evidence Retention

Dear Reviewer,

We sincerely thank you for the valuable time and effort you have devoted to reviewing our work with such care and rigor. We have carefully considered each of your comments, particularly your specific suggestions on critical-evidence retention, interpretation of the probing results, full system costs, and component contributions. These comments have helped us identify aspects of the original manuscript requiring clarification and stronger evidence, and have provided clear guidance for additional experiments. We greatly appreciate your efforts to improve this work. Below, we address the three main weaknesses you identified, followed by your three questions.

Weakness 1: The controller addresses when to compress, but not what to preserve.

Thank you very much for your careful analysis of critical-evidence retention. Your distinction between compression timing and content preservation helped us clarify the method's scope and supplement its evaluation.

The compression probe uses the task prefix and the two most recent complete interaction blocks to decide compression timing. Historical content is handled by the summarizer and retrieval mechanism, whose preservation of task constraints, code interfaces, and tool results we evaluate through the fact audit and paired interventions below.

1. Downstream performance and cost

On the fixed Pilot40 list (10 tasks each from Code, Office, Security, and Web), we evaluated 40 independent tasks and 120 terminal trajectories.

ConditionOperation at the designated decision point
A: Retain historySkip the designated compression and retain the original history
B: Compression onlyCompress, but skip the designated recall injection
C: Compression + recallCompress and perform the designated recall injection

For each task, we saved the environment, conversation, and external-memory state at a predetermined memory decision point, then created A/B/C branches from that snapshot. Only the designated memory operations differ; subsequent behavior follows the original agent policy. A does not mean that the entire subsequent trajectory never compresses.

MetricABC
Independent tasks404040
Mean Reward (%)70.0769.9074.58
Reward SD (pp)10.379.6010.22
API tokens/task1,062,969820,433905,749
9B input tokens/task398,415407,521422,296
8B input tokens/task0179,579280,097
9B CUDA s/task33.8936.6435.72
8B CUDA s/task017.6028.24
E2E s/task910.43640.19699.73

B still embeds and stores summaries; only the designated recall injection is skipped, so its 8B cost is nonzero.

Paired comparisonReward difference (pp)95% CITasks improved
B-A: Compression-0.175[-1.14, +0.79]19/40
C-B: Recall+4.680[+3.25, +6.11]33/40
C-A: Combined+4.505[+2.80, +6.21]32/40

B reduces API tokens by 22.82% and E2E latency by 29.68% relative to A. Its Reward CI includes zero: this does not establish lossless compression or non-inferiority. Adding recall improves mean Reward by 4.68 pp, but average gains do not rule out harm to individual tasks.

2. Critical-evidence retention audit

We audited summaries from all 40 Pilot40 tasks, extracting 5-7 candidate facts per task, totaling 240. We checked each fact against the original interaction history and verified its necessity for the corresponding task. All 240 passed both checks, giving a final audit set of N=240. We then classified each fact's retention in the generated summary.

CategoryCountPercentage
Fully retained20384.58%
Partially retained208.33%
Missing156.25%
Contradictory20.83%
Total240100%

Of the 240 verified task-critical facts, 223 (92.92%) are fully or partially retained, while 15 are missing and 2 are contradictory. The audit quantifies both retention and the remaining information loss.

3. Scope and future work

Task interventions assess downstream outcomes, while the LLM-based fact audit identifies specific omissions and contradictions in the summaries. These complementary evaluations help locate opportunities to improve evidence preservation. We plan to strengthen audit reliability through multiple auditors and human spot checks.

We plan to combine timing decisions with content-level evidence selection: traceable retention of code interfaces, exact values, and task constraints; original-text retention or delayed compression for uncertain information; and multiple auditors plus human spot checks to expand paired analyses of omissions and task harm.

Thank you again for helping us clarify this boundary. We will describe PaMER as improving quality-cost trade-offs, without claiming lossless preservation of critical evidence.

AuthorsOpenReview ↗

Response to Weakness 2: Probe Interpretation and Online Recall

Weakness 2: Probe results require cautious interpretation, and the recall probe is not used online.

Thank you very much for your careful consideration of probe interpretation and its connection to the deployed system. Your comments helped us distinguish label predictability from actual operational benefits and provided clear guidance for the recall-gating experiment.

We distinguish label predictability from downstream utility: the probes quantify the former, while the same-state interventions and online recall-gating ablation evaluate the latter. The revision will present these as complementary forms of evidence connecting the probing analysis to task performance.

To address the deployment gap, we compared:

  • Original PaMER: retrieval follows the original embedding-based procedure.
  • Probe-triggered Recall: the frozen recall probe first decides whether retrieval is needed; the same retriever is invoked only when the gate is positive.

Both use Qwen3-Embedding-8B, 1,024-dimensional representations, cosine similarity, Top-3 retrieval, and the same summary-storage rules. Their memory stores evolve independently along their respective trajectories. This evaluates the end-to-end online gating policy, rather than an isolated causal effect with a fixed memory store.

1. Task quality and paired results

MetricOriginal PaMERProbe-triggered Recall
Independent tasks4040
Reward (%)69.9273.79
Reward SD (pp)13.0215.08
Tasks improved—29/40
Tasks declined—11/40
Paired Reward difference (pp)—+3.87
Paired 95% CI—[1.73, 6.01]

2. Component costs and end-to-end latency

All 40 tasks are included; usage and timing values are per-task averages.

MetricOriginal PaMERProbe-triggered Recall
API input tokens/task928,191724,606
API output tokens/task77,75047,490
API cache-read tokens/task788,602616,862
API input + output tokens/task1,005,941772,096
Local 9B input tokens/task397,551438,347
Local 8B input tokens/task313,415276,119
9B CUDA s/task36.02657.674
8B CUDA s/task33.46033.196
E2E s/task709.72639.19
Mean 9B peak allocated memory (GiB)18.8018.93
Mean 8B peak allocated memory (GiB)15.1815.35

Gating reduces API input + output tokens by 23.25% and mean E2E latency by 9.94%. Local 9B input tokens increase by 10.26%, and 9B CUDA time increases; 8B CUDA time is approximately unchanged, and component peak memory is similar. Cache-read tokens are part of input tokens and are not counted twice. Local model tokens are reported separately using their respective tokenizers.

3. Retrieval-trigger statistics

MetricResult
Total decision states286
Retrieval triggered109
Retrieval skipped177
Retrieval rate38.11%

These results show that the learned recall signal can participate in online decisions and improve average Reward on this task set. The trigger rate alone does not establish that every skipped retrieval was unnecessary, and average gains do not imply that every task benefits.

A possible explanation is reduced unnecessary retrieval and history injection, which lowers subsequent context processing and interference from irrelevant history. An additional classification head need not itself require extra 9B forward passes, since its input representation can be shared with the compression probe.

We sincerely appreciate your guidance in connecting the probing analysis to the deployed system. The revision will use online ablations as evidence of system benefits while retaining the limitations of label-based probing.

AuthorsOpenReview ↗

Response to Weakness 3: System Costs and Component Contributions

Weakness 3: System costs and component contributions are insufficiently characterized, and gains vary across backbones.

Thank you very much for your specific suggestions on system costs, component attribution, and cross-model stability. They helped us improve cost accounting and design a controlled test of step-level evidence selection.

We agree that acting-model token reductions alone do not characterize the frozen 9B extractor's additional inference cost. Response 5 provides itemized acting and auxiliary token usage, local model processing, component time, E2E latency, and the available GPU memory measurements, with explicit indications of incomplete baseline accounting.

We also agree that PaMER and PaMER+ differ in routing and other implementation details. Their overall performance difference therefore cannot isolate the contribution of step-level selection.

We conducted an ablation within the same PaMER+ implementation, keeping the frozen probe, LLM Router, acting model, tool environment, and task budget unchanged. We disabled only step-level evidence selection, replacing selected historical steps with their complete parent blocks.

Both configurations were evaluated on the same 40 tasks, starting from the same task initialization for each paired task, yielding 80 terminal trajectories.

ConfigurationTasksReward (%)API tokens/task9B CUDA s/taskE2E s/task
Full Parent Block4070.691,930,76561.80876.55
Step-level Evidence Selection4075.251,560,51265.951,055.51

Across the 40 shared tasks, step-level selection increases mean Reward by 4.56 percentage points and reduces external API input + output tokens by 19.18%. However, local feature-processing time increases, and mean E2E latency rises by 20.42%.

Step-level selection offers a clear trade-off on these tasks: higher Reward and lower API token usage, with additional local computation and E2E latency.

This controlled ablation provides evidence of the component's contribution, while larger task sets and repeated runs are needed to assess stability. We will also retain the uneven results across acting backbones and avoid claiming universal improvements.

Thank you again for helping us present system benefits, computational costs, and applicability more clearly. Your suggestions have strengthened the attribution of component effects and the boundaries of our conclusions.

Reviewer_mtSSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper discusses if the agent's hidden state already contains enough signals to make it compress or recall memory before the next action is generated. The authors collect 72,912 LLM-annotated decision points from 2,521 trajectories covering 600 questions. The author builds a linear probes on a frozen Qwen3.5-9B and reaches AUROC 0.831 for compression and 0.765 for recall, which performed better than only using metadata such as context length, turn, and tool type (AUROC 0.758 for compression and 0.549 for recall). This compression signal becomes stronger in later layers, which recall signal peaks at block 16. Based on these findings, the author built PaMER, which uses hidden state prob to trigger compression and embedding similarity to retrieve stored summaries. The author also built PAMER+, which selects evidence at each step. On Pilot40, the base agent scores 67.5, PaMER scores 68.0, and PaMER+ scores 68.9; PaMER+ uses 0.98M API tokens instead of 1.89M for the base agent, a 48.1% reduction. On the full 260-task WorkBuddy benchmark, PaMER+ changes the reported average from 69.9 to 71.6 while reducing API tokens by 71.7%. Across five additional acting models on Pilot40, PaMER+ reduces tokens for all five and improves reward for three.

Critical strengths
  1. The probing results is well controlled. With question-grouped splits, the hidden state predicts recall much better than metadata baseline, (AUROC 0.765 vs. 0.549) and also improves compression prediction (AUROC 0.831 vs. 0.758). In addition, the author also provides insights that these signals appear at different depths: compression becomes stronger towards the final layer, while recall peaks at block 16. These findings provide clear evidence that hidden state contains information about upcoming memory operations better than simple metadata.

  2. Cross-model validation also shows the methods work on different backbone models. On Pilot40, PaMER+ reduces tokens for all five additional backbones in Table 5 and improves the reward for three of them. This shows that the controller can transfer across models, although the reward gain depends on the acting models.

Critical weaknesses
  1. The reported savings are not end-to-end costs. The token results count the acting model's API tokens but exclude the frozen Qwen3.5-9B extractor and Qwen3-Embedding-8B embedder. The paper does not report total latency, peak GPU memory, or monetary cost. Therefore, it is unclear whether PaMER is more efficient after including the controller itself. So far only benchmark data from WorkbuddyBench tested, try to see at least one more benchmark.

  2. The probing analysis is only partly connected to the final system. The paper studies two hidden-state signals: when the agent should compress and when it should recall earlier information. However, PaMER uses only the compression probe online. When retrieving earlier information, it selects stored summaries by embedding similarity, regardless of the recall probe's prediction. Therefore, the recall AUROC and the block-16 result do not show that the deployed system benefits from the recall signal. Also, the compact-context design choices are supported by only five splits, which is too small to establish them reliably. An ablation that adds the recall probe to PaMER would better connect the analysis to the system.

  3. The system evaluation is limited to one benchmark suite. The paper evaluates PaMER only on WorkBuddyBench, and most baseline, ablation, and cross-model comparisons use Pilot40, with only 10 tasks per domain. WorkBuddyBench covers several domains, but all results still come from the same benchmark and agent harness. It is unclear whether the same quality-cost trade-off holds on other long-horizon agent benchmarks.

Questions for authors
  1. Could you rerun the base agent, PaMER and PaMER+ on the same Pilot40 tasks and measure the full system cost?

  2. Could you run a controlled recall ablation with two PaMER variants: the current embedding-based retrieval and a version where the recall probe decides whether retrieval is triggered, while the same embedder selects the stored summaries?

  3. Could you run the evaluation on another benchmark rather than only WorkBuddyBench?

4 comments
AuthorsOpenReview ↗

Response to Q1: End-to-End System Cost on Pilot40

Dear Reviewer,

Thank you for your careful review and constructive feedback. We appreciate your recognition of our controlled probing analysis and cross-model evaluation. We are particularly grateful for your suggestions regarding end-to-end system costs, the connection between the recall probe and online retrieval, and evaluation beyond WorkBuddyBench. These comments have provided clear guidance for our additional experiments and helped us clarify the scope and limitations of our findings. Below, we respond to your concerns and questions point by point.

In response to Q1, we re-evaluated the base agent, PaMER, and PaMER+ on the same fixed Pilot40 task list. We now report API usage, local model input volumes, CUDA active time, end-to-end (E2E) latency, and peak GPU memory. All values are per-task averages over all 40 tasks.

MetricBase agentPaMERPaMER+
Reward (%)67.569.9271.83
API tokens/task≈1.89M1,005,9411,127,691
Local 9B input tokens/task—397,551580,740
Local 8B input tokens/task—313,415—
9B CUDA time (s/task)—36.02693.403
8B CUDA time (s/task)—33.460—
Full-task E2E latency (s/task)909.74709.72866.76
Mean 9B peak allocated memory (GiB)—18.8019.99
Mean 8B peak allocated memory (GiB)—15.18—

The auxiliary models perform representation extraction: Qwen3.5-9B computes hidden states through forward passes, followed by a lightweight decision head; Qwen3-Embedding-8B encodes text into vectors for summary retrieval. Neither component requires autoregressive text generation, so both avoid the sequential decoding overhead associated with generating output tokens. They still incur input-processing, GPU memory, and model-residency costs. Their local input tokens measure computational workload and do not constitute additional API-billed tokens. We report them separately because tokenizers and per-token computational costs differ across models.

Both auxiliary models can be deployed locally on a single machine with sufficient GPU memory, and their instances can be reused across tasks. This avoids separate per-token charges from an external API provider for these components and allows deployment costs to be amortized across tasks. Actual monetary cost depends on hardware pricing, utilization, and amortization. This non-generative description applies specifically to feature extraction and embedding; summary-generation and Router calls remain generative components whose costs must also be accounted for.

PaMER+ uses a separate LLM Router and a structured summary index to locate candidate history and select original steps. It does not use Qwen3-Embedding-8B or embedding-based retrieval, so its 8B metrics are marked with a dash. CUDA time denotes active local-component computation. Memory values are averages of per-task observed PyTorch peak allocated memory, including resident weights and caches.

In this Pilot40 re-evaluation, PaMER's 9B and 8B components required 36.026 and 33.460 CUDA-active seconds per task, respectively. Including auxiliary overhead, PaMER reduced mean E2E latency from 909.74 to 709.72 seconds (21.99%), while PaMER+ reduced it to 866.76 seconds (4.72%). Both also reduced acting-model API token usage. Thus, auxiliary computation did not offset the observed E2E latency savings in this setting.

We distinguish this re-evaluation from the original results: the original Reward values for PaMER and PaMER+ were 68.0 and 68.9, whereas the re-evaluated values are 69.92 and 71.83. All costs in the table correspond to the re-evaluation and are not combined with measurements from the original run.

AuthorsOpenReview ↗

Response to Q2: Recall-Probe Gating Ablation

Dear Reviewer,

We sincerely thank you for this precise and insightful experimental suggestion. Your observation about the connection between the recall-probe analysis and the deployed system identifies a key aspect of the paper that needs to be strengthened and has provided a clear direction for assessing the practical value of the recall signal. In response to your second question, we have conducted a recall-gating ablation on the fixed Pilot40 task list, comparing the original PaMER with a version in which the recall probe decides whether to trigger retrieval, while keeping the embedding retriever and retrieval rules unchanged.

The two configurations are:

  • Original PaMER: retrieval follows the original embedding-based procedure.
  • Probe-triggered Recall: the frozen recall probe first determines whether historical information is needed; the original retrieval procedure is invoked only when the gate is positive.

Both configurations use the same Qwen3-Embedding-8B, 1,024-dimensional representations, cosine similarity, Top-3 retrieval, and summary-storage rules. Their memory stores evolve independently along their respective interaction trajectories. This experiment therefore evaluates the end-to-end effect of the online gating policy, rather than the isolated causal effect of gating with a fixed memory store.

All 40 tasks are included in the statistics below.

MetricOriginal PaMERProbe-triggered Recall
Independent tasks4040
Reward (%)69.9273.79
API input tokens/task928,191724,606
API output tokens/task77,75047,490
API input + output tokens/task1,005,941772,096
Local 9B input tokens/task397,551438,347
Local 8B input tokens/task313,415276,119
9B CUDA time (s/task)36.02657.674
8B CUDA time (s/task)33.46033.196
E2E latency (s/task)709.72639.19
Mean 9B peak allocated memory (GiB)18.8018.93
Mean 8B peak allocated memory (GiB)15.1815.35

The gated version improves mean Reward by 3.87 percentage points, with a paired 95% confidence interval of [1.73, 6.01]. Of the 40 tasks, 29 improve and 11 decline. Acting-model API token usage decreases by 23.25%, and mean end-to-end latency decreases by 9.94%. These benefits come with increased local 9B computation; 8B CUDA time remains approximately unchanged, and peak allocated memory is similar for both components.

Across 286 decision states, the probe triggers retrieval in 109 states and skips it in 177, yielding a retrieval rate of 38.11%. These results show that the recall probe can participate in actual retrieval decisions and improve average task quality and API efficiency on this task set. Task-level outcomes are heterogeneous: Reward improves on 29 tasks and declines on 11, alongside lower aggregate API usage and E2E latency.

One possible explanation is that gating reduces unnecessary retrieval and history injection, lowering subsequent context-processing costs and limiting interference from irrelevant history.

Thank you for highlighting this point. The five-split results provide support for the compact-context design within the evaluated settings. We will make this scope explicit in the revision and note that additional splits and task distributions would help assess its broader robustness.

AuthorsOpenReview ↗

Response to Q3: Evaluation Beyond WorkBuddyBench

Dear Reviewer,

We sincerely thank you for your careful consideration of the scope of our evaluation and the applicability of our conclusions. Your observation that results obtained within a single benchmark and agent framework are insufficient to establish performance in other task environments has helped us strengthen our experimental evaluation. In response to your third question, we have extended the evaluation to SWE-bench Verified, τ-bench, and Terminal-Bench 2.0 to examine quality-cost trade-offs in software repair, interactive tool use, and terminal tasks.

Within each benchmark, the base agent, PaMER, and PaMER+ use the same acting model and inference parameters; the methods differ only in how they manage context. The exact model versions, inference parameters, and runtime configurations are provided in the experimental setup section of the revised manuscript.

All methods use the same task list, fixed in advance for each benchmark: a 100-task subset of SWE-bench Verified, 165 tasks from τ-bench (50 Airline and 115 Retail tasks), and 89 tasks from Terminal-Bench 2.0. Each method is run once per task, and all tasks are included in the task-performance and end-to-end latency statistics.

BenchmarkMethodSuccessful tasks/totalAPI tokens/taskE2E latency (s/task)
SWE-bench VerifiedBase agent84/1001,364,732820.52
SWE-bench VerifiedPaMER92/100853,878776.41
SWE-bench VerifiedPaMER+91/100925,447843.54
τ-benchBase agent113/16597,70171.14
τ-benchPaMER113/16567,73773.28
τ-benchPaMER+110/16573,46578.92
Terminal-Bench 2.0Base agent48/891,010,511932.07
Terminal-Bench 2.0PaMER61/89783,8231,117.60
Terminal-Bench 2.0PaMER+52/89812,5931,567.74

Both methods reduce acting-model API token usage on all three additional benchmarks, while changes in task quality and end-to-end latency vary across environments. On SWE-bench Verified, both methods solve more tasks than the base agent. On τ-bench, PaMER matches the base agent's success count, whereas PaMER+ solves slightly fewer tasks. On Terminal-Bench 2.0, both methods solve more tasks, but their end-to-end latency also increases.

These additional benchmarks extend the evaluation to software repair, interactive tool use, and terminal tasks, showing how the observed quality-cost trade-off varies across environments. The revised manuscript will report both the benefits and costs, and will explicitly state that the SWE-bench Verified results are based on a fixed 100-task subset.

Once again, we sincerely thank you for the valuable time and effort you have devoted to reviewing our work with such care and rigor. Your suggestions have helped us identify and address shortcomings in our experimental evaluation and reasoning, making the paper's conclusions better supported and more clearly presented. We are deeply grateful for your contribution to improving this work. We wish you every success in your work, happiness with your family, good health, and a fulfilling, enjoyable, and intellectually stimulating experience at this year's ICLR.

Dear Reviewer,

As the discussion period draws to a close, we would like to kindly follow up and ensure that we have adequately addressed your previous questions and concerns.

We have also completed the revisions to our manuscript and uploaded the updated PDF version. In preparing this revision, we have carefully considered your valuable feedback, and we sincerely hope that these changes provide clearer and more comprehensive responses to your concerns while further improving the quality of our work.

If you have any remaining questions, concerns, or additional suggestions, we would greatly appreciate your feedback and would be more than happy to provide further explanations or clarifications. Your insights are truly valuable to us and play an important role in helping us refine and strengthen our research.

Once again, we sincerely thank you for the valuable time and effort you have devoted to reviewing our paper, as well as for your thoughtful and constructive comments. We deeply appreciate your dedication and contributions to the academic community.

We also wish you good health, happiness, and continued success in your research and professional endeavors! May your academic journey at ICLR be filled with inspiring discussions, exciting discoveries, and meaningful connections, bringing you a truly rewarding, fortunate, and enjoyable experience!

With our sincere gratitude and warmest wishes,

The Authors

Comments on the paper

1 comment

We have substantially revised the paper in response to the reviewers' valuable feedback. In particular, we added end-to-end system cost evaluations, including auxiliary model computation, GPU memory, and latency; introduced an online recall-probe gating experiment and same-state compression and recall interventions; and conducted a critical-evidence retention audit. We further extended our evaluation to SWE-bench Verified, τ-bench, and Terminal-Bench 2.0, and added a controlled ablation to isolate the contribution of step-level evidence selection. We have also clarified the limitations and scope of our findings.

We sincerely thank all three reviewers for their patient reading, thoughtful comments, and constructive feedback. We are also grateful to the Area Chairs and Senior Area Chairs for their time, careful consideration, and guidance throughout the review process. Their feedback has helped us strengthen the experimental evidence, improve the clarity of the presentation, and better articulate the contributions and limitations of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.