- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how small language models can complete long-horizon agent tasks without changing the model itself. The authors identify two problems in recursive agents, namely planning that repeats without advancing execution, and context growth that buries or deletes evidence needed later. They propose REx, which has two parts. First, progressive planning with recursive execution. Each planning round emits either END or a variable length batch of subtasks, each labeled for direct execution by a local ReAct loop or for recursive decomposition. Planning resumes only when the batch finishes or a subtask fails. Second, subtree compression. When a decomposed subtask completes, its execution subtree in the shared global context is replaced by a compact memory, conditioned on the current goal, the parent goal, and the next planned sibling. On Claw-Eval, GAIA, and Agent-Diff with three Gemma-4 backbones, REx beats the best of four baselines in all nine settings, with gains of up to 19.88 points over ReAct. Ablations support the flexible batch length and the goal-conditioned compression, and a depth sweep shows that recursion helps but the gain is not monotonic in depth.
Critical strengths
- A well motivated design that separates task-level control from action-level execution. The batch mechanism directly addresses the failure mode observed in ReCAP traces, where small models keep replanning without producing new actions.
- Consistent gains across three benchmarks and three model scales, supported by careful ablations. The compression ablation is especially informative, since compression without goal conditioning performs worse than no compression at all, showing the benefit comes from retaining what unfinished work needs rather than from summarization alone.
- Honest efficiency reporting. The paper measures input tokens, output tokens, and model calls, and notes that REx uses fewer input tokens than ReAct but more output tokens and more calls.
Critical weaknesses
- Each setting appears to be a single temperature-0 run with no variance reported. On benchmarks of 161 to 224 tasks, some margins are only a few points, so it is hard to tell how stable the wins are.
- The compression component is only compared against REx variants. A comparison with existing context management methods such as HiAgent or AgentFold under the same backbone would make this contribution clearer.
Questions for authors
- How were the compression thresholds chosen for each benchmark, and how sensitive is the performance to this value?
- All experiments use the Gemma-4 family. Have you tried small models from another family to check that the gains are not tied to Gemma?
- The planner freely chooses the batch length and the EXECUTE or DECOMPOSE labels. How often do these outputs fail to parse and trigger the correction prompt, and does performance depend on that retry?
6 comments
Official Comment by Authors
Weakness 2: The compression component is only compared against REx variants. A comparison with existing context management methods such as HiAgent or AgentFold under the same backbone would make this contribution clearer.
Response: We thank the reviewer for this valuable suggestion. We agree that comparing REx with existing context-management methods would strengthen the evaluation. We therefore added HiAgent, a closely related method supporting subgoal-based summarization and historical retrieval, as a new baseline. REx outperforms HiAgent in all nine benchmark–backbone settings, with improvements ranging from 3.64 to 27.95 percentage points.
(1) New Experiment: Comparison with HiAgent. We implemented HiAgent following its original subgoal-memory mechanism, including memory updates and historical retrieval through update_subgoal and retrieve_subgoal_history. We evaluated both methods on Claw-Eval, GAIA, and Agent-Diff using the same three Gemma-4 backbones within each setting.
| Backbone | Benchmark | HiAgent Pass@1 | REx Pass@1 |
|---|---|---|---|
| E4B | Claw-Eval | 16.15% | 29.81% |
| E4B | GAIA | 12.12% | 15.76% |
| E4B | Agent-Diff | 9.38% | 16.07% |
| 12B | Claw-Eval | 21.74% | 49.69% |
| 12B | GAIA | 21.21% | 33.93% |
| 12B | Agent-Diff | 32.14% | 39.29% |
| 26B-A4B | Claw-Eval | 29.81% | 55.90% |
| 26B-A4B | GAIA | 30.30% | 40.00% |
| 26B-A4B | Agent-Diff | 47.32% | 60.27% |
(2) Results and Discussion. REx achieves higher Pass@1 than HiAgent across all nine settings, with particularly large improvements on Claw-Eval (+27.95 pp with 12B and +26.09 pp with 26B-A4B). Unlike HiAgent’s subgoal-based summarization, REx compresses completed recursive subtrees using current, parent, and next-sibling goals to preserve information needed by unfinished tasks. These results demonstrate REx’s end-to-end advantage over a closely related context-management method. We will include HiAgent in the main comparison table in the revised paper.
Official Comment by Authors
Question 1: How were the compression thresholds chosen for each benchmark, and how sensitive is the performance to this value?
Response: We thank the reviewer for this important question. We selected benchmark-specific compression thresholds based on the characteristics of execution traces and conducted additional sensitivity experiments across all three benchmarks. The default thresholds achieve the highest observed Pass@1 in all three settings, while alternative thresholds also yield comparable performance in several cases.
(1) Compression Threshold Selection. We use 3,072 tokens for Claw-Eval and 6,144 tokens for GAIA and Agent-Diff. Claw-Eval primarily involves structured workflows with relatively compact execution records, whereas GAIA often produces lengthy web observations, such as DOM and accessibility trees, and Agent-Diff involves structured records from multi-service API interactions. We therefore use larger thresholds for GAIA and Agent-Diff to avoid compressing useful execution information too early. Importantly, the threshold determines when compression is triggered, rather than the target length of the compressed memory: a completed subtree is compressed at a recursive return only when its current size exceeds the threshold.
(2) New Experiment: Threshold Sensitivity. We evaluated four thresholds on all three benchmarks using Gemma-4-26B-A4B, keeping all other settings unchanged.
| Compression Threshold | Claw-Eval Pass@1 | GAIA Pass@1 | Agent-Diff Pass@1 |
|---|---|---|---|
| 1,536 | 51.55% | 39.39% | 59.82% |
| 3,072 | 55.90% (default) | 36.97% | 55.80% |
| 6,144 | 52.80% | 40.00% (default) | 60.27% (default) |
| 8,192 | 49.69% | 34.55% | 57.59% |
Results. The results show that compression-threshold sensitivity is benchmark-dependent and non-monotonic. On Claw-Eval, the default 3,072-token threshold achieves the highest Pass@1 (55.90%). On GAIA and Agent-Diff, the 6,144-token default also performs best, while the smaller 1,536-token threshold achieves similar results (39.39% vs. 40.00% on GAIA and 59.82% vs. 60.27% on Agent-Diff). These results indicate that REx does not require a single precise threshold to achieve competitive performance, although overly frequent or delayed compression can reduce accuracy.
Official Comment by Authors
Response: We thank the reviewer for raising this important concern. To assess the reliability of our results, we conducted additional task-level paired statistical analyses across all nine benchmark–backbone settings. REx achieves higher observed Pass@1 than ReAct in all nine settings, with six showing 95% confidence intervals entirely above zero. These results provide additional statistical evidence while distinguishing clear improvements from smaller, inconclusive differences.
(1) Paired Statistical Analysis. We computed 95% paired bootstrap confidence intervals for the success-rate differences between REx and ReAct, using 10,000 task-level resamples per setting. Each resample preserves the paired outcomes of both methods on the same tasks. The resulting intervals quantify task-resampling uncertainty conditional on the observed runs.
(2) New Results. The results across all nine settings are summarized below.
Results. Six of the nine 95% confidence intervals lie entirely above zero, covering all three benchmarks and all three model scales. The strongest evidence comes from Claw-Eval, where REx improves Pass@1 by 16.78–19.88 percentage points. For smaller gains, such as Agent-Diff/26B (+3.13 pp), the confidence interval includes zero ([−2.24, 8.48] pp), indicating that the observed difference is not statistically conclusive. These analyses strengthen the evidence for REx’s effectiveness while providing a more precise characterization of result uncertainty.