- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes AdaCoM, an external context manager for frozen long-horizon language agents. Instead of requiring the task agent to summarize its own trajectory or use a predefined memory operation, AdaCoM allows a separate LLM to delete, rewrite, merge, or preserve arbitrary context messages. The manager is initialized through supervised demonstrations and then optimized with end-to-end reinforcement learning using final task outcomes and rule-based process rewards, while the underlying agent remains unchanged. Experiments show consistent improvements across several agent backbones over vanilla ReAct, self-management methods, and a trained fixed-summarization manager. The paper further analyzes cross-agent transfer and reports that managers transfer better between agents with similar vanilla performance. Based on the learned context lengths, the authors propose a Fidelity–Reliability Trade-off: stronger agents tend to benefit from retaining more raw context, whereas weaker agents benefit from more aggressive compression.
Critical strengths
-
Practical and flexible formulation. Separating context management from the frozen task agent makes the method applicable to closed-source agents, while the general edit space is more expressive than a fixed summarization operation.
-
Strong and relatively comprehensive experiments. AdaCoM is evaluated across multiple agents and two tool-using task families, with useful comparisons against ReAct, self-management, fixed summarization, untrained and SFT-only managers, and additional memory baselines.
-
Promising transfer and efficiency results. The manager improves most agent–manager combinations, and the reported wall-clock measurements suggest that shorter contexts and fewer redundant actions can offset manager inference latency.
Critical weaknesses
-
The novelty positioning is incomplete. This paper should explain more precisely what is new beyond external learned compression and flexible context operations.
-
The Fidelity–Reliability Trade-off is supported mainly by correlation. The central evidence is the ordering of vanilla accuracy and learned context length across four source agents. Vanilla benchmark accuracy may not capture context-handling capability, and the observed relationship may be confounded by model-specific reasoning styles, APIs, native context windows, or optimization differences.
-
Some training and evaluation signals limit the generality of the conclusions. BrowseComp-Plus training uses privileged gold/key-document process rewards, while MCP-Bench-Wiki is synthetically generated and evaluated primarily with proprietary LLMs. Moreover, the reported API cost excludes the local manager’s computation, so the total resource advantage is not yet fully quantified.
Questions for authors
-
How does AdaCoM differ technically and empirically from ACON and LongSeeker, and can these methods be discussed or compared more explicitly?
-
Does AdaCoM retain its gains on BrowseComp-Plus when trained without gold-document and key-document process rewards?
-
Is vanilla ReAct accuracy genuinely the best proxy for an agent’s ability to use long contexts? Have alternatives such as controlled long-context retrieval accuracy or sensitivity to distractors been considered?
-
How much of the cross-agent transfer pattern is quantitatively explained by source–target performance similarity? Can the authors report a correlation or regression analysis rather than relying mainly on visual inspection?
-
For the efficiency comparison, can the authors report total computation and cost including manager inference, rather than only task-agent API cost? Does the result remain favorable when the Qwen3-4B manager’s GPU cost is included?
5 comments
Response to Reviewer yfJ2 (Part 2/3)
W2, Q3, Q4: Why we use vanilla ReAct accuracy to measure capability in the Fidelity–Reliability trade-off, and evidence beyond correlation
The FR trade-off concerns how context management adapts to agents' ability to exploit raw contexts. To clarify, “exploiting raw contexts” goes beyond retrieval: the agent must use its accumulated search history to track task constraints, weigh incomplete or conflicting evidence, and decide what to search next. Stronger agents can do so reliably with longer raw histories, while weaker agents benefit from shorter, distilled contexts. AdaCoM reduces corresponding failure modes (Sec. 4.3), such as settling on answers that meet only some requirements or giving up while viable search directions remain. We therefore use vanilla ReAct accuracy as a task-matched measure of agentic capability for deep search, capturing overall task performance rather than retrieval alone.
Alternative measures(Q3)
We agree that external benchmarks offer a useful independent assessment of this capability and consider two types.
(1) Controlled long-context retrieval and distractor sensitivity: NoLiMa (ICML 2025) is a non-agentic needle-in-a-haystack benchmark. A sentence naming a person is inserted into a novel; a subsequent question shares no keywords with it, requiring retrieval through semantic association rather than keyword matching.
| Agent | BCP ReAct(%,↑) | Effectivelength(↑) | Drop with distractors(points,↓) |
|---|---|---|---|
| GLM-4.5-Air | 32.56 | 2K | 23.1 |
| GPT-OSS-20B | 28.89 | 1K | 11.6 |
| Kimi-K2 | 18.56 | 4K | 7.6 |
| DeepSeek-V3 | 17.78 | 4K | 29.5 |
| Seed-1.6-Flash | 16.44 | 1K | 0.0 |
Table R3: Following NoLiMa, effective length is the longest context retaining at least 85% of the mean accuracy at 250–1K tokens. Distractor drop is the accuracy loss in percentage points after adding distractors at 4K.
Neither metric consistently tracks BrowseComp-Plus (BCP) performance. Although GLM outperforms Kimi on BCP, it has a shorter effective length (2K vs. 4K) and suffers a larger accuracy drop from distractors (23.1 vs. 7.6 points).
Thus, the learned context strategies do not simply follow retrieval ability. Long-context benchmarks also rank these agents differently: AA-LCR places Kimi (53.0) above GLM (46.7) and DeepSeek (40.7), unlike NoLiMa.
(2) Long-horizon agentic benchmarks (multi-step search, coding, and tool use):
| Agent | BCP ReAct | BrowseComp (from GLM-4.5 technical report) | Terminal-Bench Hard (from Artificial Analysis) | MCP-Universe |
|---|---|---|---|---|
| GLM | 32.56 | 21.3 | 20.5 | 19.5 |
| Qwen3-Max | 27.78 | – | 20.5 | 18.2 |
| Kimi | 18.56 | 7.9 | 15.9 | 19.1 |
| DeepSeek | 17.78 | 1.5 | 6.8 | 14.3 |
Table R4: Performance on long-horizon agentic benchmarks.
These rankings broadly agree with BCP ReAct, except that Kimi-K2 slightly outperforms Qwen3-Max on MCP-Universe. This supports using long-horizon agentic benchmarks to assess agent capability on BCP.
Potential confounders of the FR trade-off (W2)
- Reasoning style: Style can explain exceptions, e.g., Gemini imitating the Qwen-trained manager's report-style notes and stopping its search (Appendix C.6). Yet agents from the same family can prefer different managers, in line with task performance: GPT-OSS-20B does best with the GLM-trained manager, GPT-4o-mini with the Kimi-trained manager.
- API: All agents use the same prompt, tools, retriever, step limit, and decoding settings.
- Native context window: GLM and DeepSeek both have 128K windows, yet their learned lengths span the four source agents' extremes: 7.0K and 1.9K.
- Optimization: All managers start from the same SFT checkpoint and share RL hyperparameters and rewards. Even without manager training, per-step self-summarization (SumAgent) improves Kimi's accuracy but lowers GLM's.
Together, these observations suggest that the FR trade-off is not primarily driven by these confounders.
Quantitative analysis(Q4)
Across all 28 cross-agent pairs, we regress transfer gain (target mean@3 with the transferred manager minus target vanilla ReAct accuracy) on capability gap (absolute source–target difference in ReAct accuracy). Targets differ in how much they can improve, so we use target fixed effects to compare managers on the same target [R1].
Results: For a given target, a 1-percentage-point increase in the source–target capability gap is associated with a 0.30 pp lower transfer gain. The gap explains 43% of variation across managers for the same target (p = 0.002). Choosing the closest source by ReAct accuracy yields an average regret of 0.92 points, versus 2.61 for random selection; regret is the gap to the best transferred manager for each target.
We will revise the paper as follows:
- Clarify that "exploiting raw contexts" refers to agentic capability for long-horizon search, including reasoning and decisions across steps.
- Add the external benchmark discussion and quantitative transfer analysis to the appendix.
[R1] Taskonomy: Disentangling Task Transfer Learning
Response to Reviewer yfJ2 (Part 3/3)
W3, Q2, Q5: Training signals, MCP-Bench-Wiki evaluation quality, and total inference cost
Dependence on privileged process rewards (Q2)
AdaCoM retains gains without privileged document-level rewards. In Appendix Table 7, we remove all process rewards (), including gold- and key-document rewards, and train the manager using only the final-answer reward. Mean@3 decreases from 26.19% to 24.38% on DeepSeek-V3 and from 36.22% to 33.33% on Qwen3-Max, still exceeding ReAct by 6.60 and 5.55 percentage points, respectively.
MCP-Bench-Wiki task quality and evidence of substantive gains (W3)
We agree that synthetic tasks and LLM-based evaluation require explicit quality checks. Building on MCP-Bench's construction framework, we apply additional task validation and inspect paired trajectories to check whether higher scores reflect better task completion and grounding.
Task and evaluation quality. Our pipeline uses five task generators and screens each candidate with three models other than its generator, requiring at least two solvability scores ≥7/10. Surviving tasks undergo end-to-end dry runs against the live Wikipedia MCP server and full-log quality filtering. Held-out tasks are generated separately with training-topic blacklists, deduplicated, and screened for execution quality, sufficient tool use, and substantive answers (Appendix B). These checks target dependence on a single generator, unsolvable tasks, execution failures, and task overlap.
MCP-Bench compares its LLM judge's assessments with those of three independent human annotators, reporting mean agreement of 1.43 on a 0–2 scale after prompt shuffling and score averaging (Appendix A.9, Table 12). We therefore adopt this evaluation framework, using Claude Opus 4.6 as the judge.
Checking the gains. To examine whether the score gains correspond to substantive improvements in research, we also inspect paired DeepSeek trajectories. The analysis indicates that AdaCoM reduces premature completion and improves cross-source verification by preserving evidence and unfinished research requirements. These observations are consistent with the aggregate gains on DeepSeek in task fulfillment (42.1 → 49.1) and information grounding (48.0 → 64.4) in Appendix C.1, Table 8, supporting substantive improvements in research quality. We will add representative case studies to the appendix.
Total inference cost including the manager (Q5)
In Table 4, we evaluated task-agent API costs using 50 BrowseComp-Plus questions with three runs per question. We now include an estimate of the manager's inference cost from the same runs. As our evaluation workload underutilizes the A100 GPUs hosting the Qwen3-4B manager, we use Aliyun[1] and Nscale[2] API tariffs as reference estimates of inference cost, avoiding charges for idle GPU time.
The manager averages 140,936 input / 5,350 output tokens per task with Qwen3-Max, and 262,477 input / 11,258 output tokens with GLM-4.5-Air. We use Alibaba Cloud's Qwen3-4B (Instruct) LoRA rates[1] for China (¥0.30/1.20 per million input/output tokens, converted at ¥7.00 per US dollar) and Nscale's historical Qwen3-4B-Instruct-2507 rates[2] ($0.01/0.03). The resulting costs are:
| Task agent | Manager pricing | ReAct ($) | AdaCoM agent ($) | AdaCoM manager ($, est.) | Net savings |
|---|---|---|---|---|---|
| Qwen3-Max | Aliyun[1] | 0.1700 | 0.0990 | 0.0070 | 37.7% |
| Qwen3-Max | Nscale[2] | 0.1700 | 0.0990 | 0.0016 | 40.8% |
| GLM-4.5-Air | Aliyun[1] | 0.0400 | 0.0240 | 0.0132 | 7.1% |
| GLM-4.5-Air | Nscale[2] | 0.0400 | 0.0240 | 0.0030 | 32.6% |
Table R5: Estimated total inference cost per task.
Costs are in USD per task, with savings calculated before rounding. Including the manager, estimated net savings remain 37.7–40.8% for Qwen3-Max and 7.1–32.6% for GLM-4.5-Air under these two reference API tariffs. We will update Table 4 and Section 4.5 with manager cost.
[1] Alibaba Cloud Model Studio. Token-based LoRA deployment pricing.
[2] Nscale. Historical Qwen3-4B-Instruct-2507 pricing, archived October 2025.
Response to Reviewer yfJ2 (Part 1/3)
We thank you for your constructive suggestions and for recognizing AdaCoM's practical, flexible formulation, comprehensive experiments, and promising transfer and efficiency results.
Our novelty lies in the question we raise and answer: Do different agents need different context management strategies, and if so, how do they differ? To answer it, AdaCoM learns from each frozen agent's task feedback when to edit, which messages to change, and what to preserve. Keeping the agent frozen makes the method applicable to agents that cannot be trained. The flexible edit space allows different strategies to emerge: across 8 agents from 7 model families, we find that suitable strategies differ across agents and vary with capability (Sections 4.3–4.4), and that managers transfer best between agents of similar capability (Section 5).
Technical differences
Only AdaCoM combines flexible context operations with a trained external manager, without fixing when to edit or imposing an editing template. ACON and LongSeeker do not study strategy differences across agents, and each covers only one or two model families.
Empirical studies
Table R1: Mean@3 (%) on BrowseComp-Plus, 150 questions × 3 runs.
(1) ACON. We use Qwen3-4B, the backbone of AdaCoM's manager, as the compressor. As in ACON, its guideline is optimized once with GPT-5 (on 200 training questions with DeepSeek-V3) and reused for GLM-4.5-Air, and the compression threshold is 8K tokens (4K and 16K were no better). ACON falls below ReAct on both agents. On GLM-4.5-Air, it compresses 4.8 times per trajectory on average, and in 30 incorrect rollouts the gold answer appeared in a tool result but was summarized away before the agent answered. Thus, with the same 4B model, prompt optimization alone is not enough to adapt it to each agent, whereas AdaCoM improves both agents after RL.
Table R2: Mean@1 (%) on BrowseComp-Plus, 50 questions × 1 run.
(2) LongSeeker. LongSeeker trains the agent itself (only its inference code is public), so it cannot be directly compared with AdaCoM. We therefore test its context management paradigm: keeping LongSeeker's interaction format, we compare frozen agents with and without its context operations.
Thus, when flexible operations are handed to a frozen agent, their effect depends on the agent. AdaCoM instead delegates editing to an external manager, which lets us discover each agent's strategy.
We will revise the paper as follows: