- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates whether long-horizon LLM agents’ hidden states encode the need to compress context or recall historical information. Using probes trained on LLM-annotated decision points, the authors find predictive signals beyond context length and interaction progress, with compression signals generally strengthening across layers. Based on these findings, they propose PaMER, which uses lightweight hidden-state classifiers to trigger compression and embedding-based retrieval to recover stored summaries. PaMER+ selectively restores relevant original interaction steps.
Critical strengths
-
The paper investigates whether preaction hidden states encode compression and recall needs, offering an interesting connection between internal model representations and agent memory control.
-
The experiments examine layerwise signals, observable and text-based controls, compact-context variants, and ablations, alongside comparisons with baselines and evaluations across model backbones.
-
PaMER+ reduces total token usage by 71.7% on the full WorkBuddyBench benchmark while maintaining competitive task performance.
Critical weaknesses
- The downstream evaluation is limited to WorkBuddyBench.
- The probes predict LLM-annotated compression and recall labels, which do not establish whether these operations actually improve task performance.
Questions for authors
- Can you validate the predicted memory needs through controlled interventions that compare task performance with and without compression or recall at the same decision point?
5 comments
Response to Weakness 2: Label Predictability and Operational Benefits
Weakness 2: The probes predict LLM-annotated compression and recall labels, which do not establish whether these operations actually improve task performance.
Thank you very much for recognizing the research connection between hidden states and memory needs. We also greatly value your caution that predictable labels do not establish actual task benefits. This observation helped strengthen our reasoning. We therefore added operational interventions and online recall gating to examine predictive signals and downstream benefits separately.
We agree that predicting LLM-annotated memory needs does not imply that compression or recall necessarily improves downstream agent performance.
1. Paired interventions on actual compression and recall
On the fixed Pilot40 list, we created three branches from the same decision-point snapshot of the environment, conversation, and external memory:
- A: Retain the original history.
- B: Compress, but skip the designated recall injection.
- C: Compress and perform the designated recall injection.
The experiment covers 40 independent tasks and 120 terminal trajectories. Mean Reward for A, B, and C is 70.07%, 69.90%, and 74.58%, respectively.
The paired B-A difference is -0.175 percentage points, with a 95% CI of [-1.14, +0.79]. C-B is +4.680 percentage points, with a 95% CI of [+3.25, +6.11]. C-A is +4.505 percentage points, with a 95% CI of [+2.80, +6.21].
These comparisons examine the downstream effects of actual compression and recall operations. The compression CI includes zero and does not establish lossless compression; the mean recall benefit does not imply that all tasks benefit. Full settings and costs are reported in Response 3 below.
2. Online recall-probe gating
Separately from the same-state intervention, we conducted an online recall-gating experiment. Original PaMER follows its existing embedding-based retrieval procedure. The gated version first uses the frozen recall probe to decide whether retrieval should be triggered, invoking the same Qwen3-Embedding-8B retriever when needed.
| Metric | Original PaMER | Probe-triggered Recall |
|---|---|---|
| Independent tasks | 40 | 40 |
| Mean Reward (%) | 69.92 | 73.79 |
| E2E time (s/task) | 709.72 | 639.19 |
The gated version improves mean Reward by 3.87 percentage points and reduces mean E2E latency by 9.94%.
| Online gating statistic | Result |
|---|---|
| Total decision states | 286 |
| Retrieval triggered | 109 |
| Retrieval skipped | 177 |
| Retrieval rate | 38.11% |
This experiment shows that the learned recall signal can participate in actual retrieval decisions and improve mean Reward on this task set. However, the trigger rate alone does not establish that every skipped retrieval was correct, and average gains do not guarantee benefits for every task.
Thank you again for helping us clarify the relationship between these forms of evidence. The revision will distinguish probe predictability of memory-need labels, the effects of actual memory operations, and their impact on final task performance. We will not interpret probe AUROC directly as operational safety or optimality, but will use actual interventions and online ablations as complementary evidence.
Response to Question 1: Same-State Compression and Recall Interventions
Question 1: Can you validate the predicted memory needs through controlled interventions that compare task performance with and without compression or recall at the same decision point?
Thank you very much for recognizing the systematic experimental analysis and for suggesting controlled interventions at the same decision point. Your suggestion provided a clear, actionable direction for moving from label prediction to actual memory operations and their downstream effects. We greatly value this guidance and added same-state paired compression and recall experiments accordingly.
We conducted the interventions on the fixed WorkBuddyBench Pilot40 list: 10 tasks each from Code, Office, Security, and Web, totaling 40 independent tasks and 120 terminal trajectories.
Each task first reaches a predetermined memory decision point. We save the environment, conversation history, and external-memory snapshot, then create three branches from that same snapshot:
| Condition | Operation at the designated decision point |
|---|---|
| A: Retain history | Skip the designated compression and retain the original history |
| B: Compression only | Compress, but skip the designated recall injection |
| C: Compression + recall | Compress and perform the designated recall injection |
Only the designated memory operations differ; subsequent behavior follows the original agent policy. A means retaining history at the designated point, not guaranteeing that the entire subsequent trajectory never compresses.
| Metric | A: Retain history | B: Compression only | C: Compression + recall |
|---|---|---|---|
| Independent tasks | 40 | 40 | 40 |
| Mean Reward (%) | 70.07 | 69.90 | 74.58 |
| Reward SD (pp) | 10.37 | 9.60 | 10.22 |
| API tokens/task | 1,062,969 | 820,433 | 905,749 |
| 9B input tokens/task | 398,415 | 407,521 | 422,296 |
| 8B input tokens/task | 0 | 179,579 | 280,097 |
| 9B CUDA s/task | 33.89 | 36.64 | 35.72 |
| 8B CUDA s/task | 0 | 17.60 | 28.24 |
| E2E s/task | 910.43 | 640.19 | 699.73 |
B still embeds and stores summaries; it skips only the designated query injection, so its 8B cost is nonzero.
| Paired comparison | Reward difference (pp) | 95% CI | Tasks improved |
|---|---|---|---|
| B-A: Compression | -0.175 | [-1.14, +0.79] | 19/40 |
| C-B: Recall | +4.680 | [+3.25, +6.11] | 33/40 |
| C-A: Combined | +4.505 | [+2.80, +6.21] | 32/40 |
Compression alone yields a small mean Reward difference with a CI including zero, while reducing API tokens by 22.82% and E2E latency by 29.68% relative to retaining history. This means that no clear average quality difference was observed; it does not establish lossless compression or non-inferiority.
Adding recall after compression increases mean Reward by 4.68 percentage points, indicating a positive effect of historical-information recovery on this task set. Average improvement does not exclude harm to individual tasks or establish that every probe-predicted memory operation is correct or optimal.
These paired interventions provide complementary evidence about operational effectiveness. Their conclusions remain limited by the task set, acting model, environment configuration, and randomness. We will retain these limitations in the revision and identify broader task coverage and repeated runs as future work.
Once again, we sincerely thank you for recognizing the research value, experimental analysis, and system results. Your positive assessment is an important encouragement, for which we are deeply grateful. Your specific suggestions on evaluation scope and operational effectiveness have also helped us identify and address shortcomings, making the evidence and conclusions clearer and better supported.
We truly appreciate the valuable time and effort you have devoted to understanding, evaluating, and improving this work. Thank you for recognizing its contributions while identifying directions for further improvement. Please accept our sincere gratitude and our wishes for continued success, good health, and a fulfilling, enjoyable, and intellectually stimulating academic exchange at this year's ICLR.
Response to Weakness 1: Evaluation Beyond WorkBuddyBench
Dear Reviewer,
First, please accept our sincere thanks for the valuable time and effort you have devoted to reviewing our work with such care and rigor. We are especially grateful for your positive assessment of the research question and experimental analysis: your recognition of the connection between preaction hidden states and agent memory control, and of the systematic layerwise analysis, controls, baseline comparisons, and cross-backbone evaluation, is greatly encouraging. We also appreciate your recognition of PaMER+'s quality-token trade-off on the full WorkBuddyBench.
Your comments on evaluation scope and the effectiveness of memory operations helped us identify evidence that needed strengthening and improve our additional experiments and conclusions. We are deeply grateful for both your encouragement and your efforts to improve this work. Below, we address the two weaknesses you identified, followed by your question.
Weakness 1: The downstream evaluation is limited to WorkBuddyBench.
Thank you very much for recognizing the existing baseline and cross-model evaluations, and for encouraging us to examine other task environments. Your positive assessment encouraged us, while your suggestion clarified how to further assess applicability. We therefore added three benchmarks beyond WorkBuddyBench.
We extended evaluation to τ-bench, SWE-bench Verified, and Terminal-Bench 2.0, covering interactive tool use, software repair, and terminal tasks, respectively.
Within each benchmark, the base agent, PaMER, and PaMER+ use the same acting model and inference parameters; they differ only in context management. Exact model versions, inference parameters, and runtime configurations are provided in the revised experimental setup.
The table includes τ-bench overall and domain results, together with the SWE-bench Verified and Terminal-Bench 2.0 base-agent results already reported in our aggregate cost table.
SWE-bench Verified uses a fixed 100-task subset shared by all methods. τ-bench includes 165 tasks (50 Airline and 115 Retail); Terminal-Bench 2.0 includes 89. Each method runs once per task, and all tasks are included in performance and E2E statistics. PaMER+ retains the original implementation.
The results vary across environments:
These results extend the evidence beyond WorkBuddyBench and characterize the quality-cost trade-offs observed across the additional task environments.
Thank you again for helping us broaden the evaluation. The revision will report both benefits and costs and clearly specify each benchmark's task scope and settings.