- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ARSM is a training-free context-compression framework for LLM agents. It has two parts. First, interaction histories are reorganized into Hypothesis–Action–Result micro-chains, and each model output carries state-update metadata alongside the next action. Second, a five-layer dynamic state machine (global checklist, pitfall memory, loop alerts, milestones, active reasoning path) evolves through atomic operations gated by one of four routing signals, ONGOING, MILESTONE, FAILURE, and SUCCESS, while a compression-control parameter ρ jointly scales raw-trajectory retention and structured-state capacity. The executor stays frozen and the mechanism is implemented entirely through prompts and configuration. Experiments cover WebShop, a 16-objective multi-hop QA setting, and SWE-Bench Lite across six backbones: WebShop reward improves while total tokens drop by about two thirds, QA rises from 0.567 to 2.01, and SWE-Bench Lite token consumption falls by up to 82%. The paper also reports routing distributions, component ablations, solved-set overlap across ρ, and a trajectory-level evaluation.
Critical strengths
- The training-free, self-contained mechanism delivers real efficiency gains with a genuine operating knob. WebShop reward improves consistently across three model families while total tokens drop by about two thirds; ρ yields an explicit performance-cost frontier (Figures 5 and 7) that lets users pick a point by task and budget. Code and YAML configurations ship with the supplementary material, supporting reproducibility.
- The empirical analysis goes beyond headline numbers. The routing-signal distribution (Table 5) shows the state machine does not degenerate into an append-only log; the component ablation (Table 6) separates the contributions of HAR abstraction and state evolution; solved-set overlap across ρ (Table 8) shows a stable core with complementary coverage; the trajectory-level evaluation (Table 7) quantifies state fidelity and sufficiency; and the case study illustrates how information migrates between the two memory surfaces. This level of mechanistic characterization is uncommon in scaffolding papers.
Critical weaknesses
- Baseline health and statistical rigor are insufficient. No seeds or variance are reported for accuracy metrics, and several baselines appear to run far outside their design regime: Search-R1 scores 0.009 against 0.567 for the raw backbone, MEM1-Webshop reaches 0.153 reward against 0.286 for the raw backbone, and ContextBudget is evaluated without RL. These numbers are far below the results reported in the original papers and inflate ARSM's relative advantage. Meanwhile, on SWE-Bench Lite, ARSM trails SWE-Pruner by about 11 points at a similar token budget on the 9B model (29.7% vs. 40.7%) and Mini SWE-Agent by 5.2 points on the 27B model, so the abstract's claim of maintaining task performance while reducing token consumption is too strong for this benchmark; the body's more honest framing is a tunable trade-off.
- The formalism is disconnected from the actual mechanism. Equations 3 to 6 describe state updates through a transition-operator matrix with element-wise broadcasting and a one-hot routing vector, but what actually executes is an LLM emitting status tags and a controller applying hardcoded per-layer rules; no property of this state machine is stated or proved, the loop-detection similarity indicator 1_sim and the penalty increment γ are never defined, and Appendix A truncates in the middle of the prune operator. The mathematics adds nothing verifiable and forces the reader to rebuild the algorithm from appendix examples.
Questions for authors
- Please provide the implementation of 1_sim (similarity metric, threshold, and over what representations) and the value of , and complete the truncated operator definitions in Appendix A.
- Were the Search-R1, MEM1, and ContextBudget numbers re-run under your protocol or taken from their papers? Search-R1's 0.009 and MEM1-Webshop's 0.153 reward are far below their published levels; please either validate these baselines within their design regime or explain why they collapse here.
- Are the accuracy numbers single runs? Please report variance over at least three seeds for WebShop reward and SWE-Bench Pass@1, and add significance for the -sensitivity claims.
- On SWE-Bench Lite with the 9B model, ARSM trails SWE-Pruner by about 11 points at a comparable token budget. What is the source of this gap, and how should users choose between the two?
2 comments
Thank you for the detailed rebuttal, which addresses several concerns, although some issues regarding baseline fairness, formalization, and statistical robustness remain unresolved.
Thank you for the response. For the weakness and questions mentioned in the review:
W2 and Q1. State transitions and loop detection. Figure 3 and §3.2 show how the controller parses the status tag and applies the corresponding branch’s updates. The transition matrix summarizes those four branches. In Eq. (10), the binary indicator
1_simis 1 when a repeat failure is detected and 0 otherwise. Each detected repeat increments the loop counter by γ = 1. Appendix D gives an example of an inspection warning after three reads of the same file or path set without new evidence. The pruning operator on page 13 removes the last η entries from the active reasoning path in Eq. (11). The check operator on page 14 returns WARN when a loop counter reaches its threshold in Eq. (12).W1 and Q2. Baseline provenance. The Search-R1 result in Table 3 (summed EM = 0.009, summed F1 = 0.011) comes from MEM1’s Table 1 [1], which reports a performance collapse for the original Search-R1 at 16 objectives. We reran the MEM1-style truncation baseline on the same 16-objective QA test set, using the same E5 retrieval endpoint (top-k = 3), 20-turn cap, and evaluator as ARSM. In this baseline, each call used the original 16-objective prompt, a truncated window of preceding responses, and the latest retrieved evidence.
The 30B ContextBudget result comes from the published no-RL ablation in Table 1 and serves as a training-free reference [2]. We will place its 0.170 in the summed F1 column and label 8K as a configured context limit.
For MEM1-WebShop, we retrained MEM1 on the same backbone as ARSM, used GRPO, LoRA (rank = 8, alpha = 16) and a maximum of 1,024 generated tokens per call. Both evaluations used Qwen2.5-7B-Instruct, temperature zero, an 8,192-token context limit, and up to 20 environment interaction steps. Most MEM1 trajectories ended without a purchase, which helps explain the low reward in this resource-constrained reproduction. We will clarify the training and evaluation settings in the revised manuscript.
W1 and Q3. Seeds, variance, and ρ sensitivity. The supplementary YAML lists the seed configurations. Table 2 reports dispersion in peak token usage for WebShop. For the 9B SWE-bench comparisons below, we ran each issue five times with distinct seeds and computed Pass@1 and Pass@5 as defined in Appendix C.2.
Across five seeds, ARSM at ρ = 0.5 achieved a mean Pass@1 close to Mini SWE-Agent’s and showed less run-to-run variation than SWE-Pruner.
In the Qwen3.5-27B ρ-sensitivity analysis, Cochran’s Q test showed that Pass@1 differed across settings (Q(10) = 52.68, p = 8.55 × 10⁻⁸). In follow-up exact McNemar tests against ρ = 0.5, Pass@1 was significantly lower at ρ = 0.7 and ρ = 0.9 after Holm correction (adjusted p = 0.0031 and 0.0159, respectively).
In this analysis, increasing ρ from 0.5 to 0.7 or 0.9 reduced Pass@1 by 8.7 and 8.0 percentage points, respectively. Stronger compression can therefore lower task-solving accuracy at these settings.
W1 and Q4. The 9B gap and method choice. On SWE-bench Lite, SWE-Pruner achieves 40.7% Pass@1 at 2.29M reported tokens, compared with 29.7% at 2.40M for ARSM at ρ = 0.1. As discussed in Appendix C.3, one possible contributor is how the methods divide the work. SWE-Pruner uses a separately trained 0.6B model for line-level pruning, while ARSM’s backbone handles both repair and structured memory updates. The 9B backbone sometimes omits required memory fields and needs extra turns to recover. We observed this behavior to a lesser extent on 27B. At the reported 9B settings, SWE-Pruner is preferable when Pass@1 is the priority and the auxiliary model can be deployed. ARSM offers a single-backbone option with adjustable token usage.
[1] MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents
[2] ContextBudget: Budget-Aware Context Management for Long-Horizon Search Agents