acceptodds
Under review as a conference paper at ICLR 2027

Efficient On-Device Agents via Adaptive Context Management

Abstract

On-device AI agents offer the potential for personalized, low-latency assistance, but their deployment is fundamentally constrained by limited memory capacity. Context in agentic settings worsens this problem due to large static tool schemas and a growing interaction history that continually expands the persistent KV cache. To maintain on-device feasibility, agents must operate near the minimum task-sufficient context, while preserving task performance. We introduce two complementary mechanisms: (1) a learned intra-session memory architecture that distills trajectories into an append-only Context State Object (CSO), preserving current state and relevant details from previous steps that may have future utility, while still supporting KV-cache reuse, and (2) a just-in-time schema-passing mechanism that loads full tool definitions only upon tool selection. We instantiate this framework by adapting 3B-parameter SLMs and profiling them on smartphone hardware. In our evaluations, CSO-based memory outperforms KV-cache compression and trained summarization while retaining performance near or even above full-history execution. Deployed on-device, our framework reduces initial tool context by over 6x and interaction-context growth by 10-25x, substantially reducing initial

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Context management, On-device ML, Agents, Key–value (KV) cache, Token efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_avNtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a learned State-Tracker that appends compact updates to a persistent context state and an Executor that acts from that state, together with compact tool schemas and just-in-time schema loading. It evaluates tool-use precision, recall, and F1 under two simulators, an external ToolAlpaca setting, and another backbone, and reports context, KV-memory, and initial latency measurements on a Samsung Galaxy S25. The intended contribution is a practical capability-memory trade-off for small on-device agents.

Critical strengths
  1. The approach separates growing interaction history from static tool-schema overhead, making the two mechanisms independently usable. Comparison with trained summarization and KV-compression baselines, evaluation under a second simulator, and external-distribution results are useful additions beyond a single synthetic benchmark.

  2. Actual device measurements strengthen the systems motivation. The paper also reports the State-Tracker's 1.31-second post-turn cost and the precision/recall trade-off from JIT loading rather than portraying compression as uniformly lossless. Appendix A.12 usefully clarifies the separate adapter-specific caches.

Critical weaknesses
  1. A central result is inconsistent with the stated metric definition. In Table 1, Combined on mixed tasks has F1=0.94, P=0.92, and R=0.92. Appendix A.5 specifies per-trajectory F1=2PR/(P+R), followed by macro-averaging. For every trajectory F1 <= (P+R)/2, so the same inequality must hold for those macro averages. The reported triple violates it by more than rounding can explain. Independently averaging F1, P, and R does not resolve this inequality. The evaluation code or table needs correction before the headline performance comparisons can be trusted.

  2. The principal tool-set metrics do not establish preservation of executable task behavior: matching tool identities does not check argument values, execution order, repeated calls, or the final state. This matters directly for a memory method whose key promise is retaining information required later. The supplemental quality judgments and external results show meaningful trade-offs: Table 7's Combined quality is below the full-history baseline on cloud and mixed tasks, while ToolAlpaca F1 drops from 0.69 to 0.65 and recall from 0.90 to 0.58. These results merit an argument-/outcome-aware evaluation before describing task capability as broadly maintained.

Questions for authors
  1. Please provide the exact aggregation code or per-trajectory P/R/F1 values and explain/correct Table 1's mixed-task Combined entry. Are any other reported metrics or significance tests affected?

  2. How do the methods compare on argument-correct tool calls and verified final task outcomes, especially cases requiring exact values from distant turns? Do the memory savings persist at a matched success level?

  3. For the device results, what are complete-episode latency and peak memory when both adapter caches, State-Tracker updates, and JIT selection passes are included? This would clarify the practical benefit beyond initial TTFT and individual context sizes.

Reviewer_N1cSOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes a CSO-based memory framework with dual LoRAs for efficient on-device tool agents. The append-only CSO compactly stores critical task states and enables reusable KV cache, while JIT schema loading reduces tool context overhead. Evaluations show the proposed method preserves the F1 score of tool invocation, reducing interaction context by 10-25x and tool context by over 6x, lowering on-device latency and memory costs.

Critical strengths
  1. This paper proposes a context compression framework tailored for GUI Agent tool invocation. It features a concise design and remarkable performance gains, possessing promising practical engineering value.

  2. The paper is well-structured and easy to comprehend with excellent readability.

Critical weaknesses

1.This paper targets minimum task-sufficient context, yet its design is empirically driven and lacks theoretical guarantees for achieving truly minimal context.

  1. The goal of minimum task-sufficient context requires minimizing context while preserving task completion ability. However, the paper only evaluates tool-call F1-score, precision and recall, and does not report how context compression affects downstream task success rates.

  2. The baseline methods adopted are mostly conventional approaches, while recent state-of-the-art KV cache/Context compression techniques are missing from the comparison.

  3. This work is dominated by engineering-oriented designs. Its methodological novelty and research contribution are insufficient to meet the publication standard of ICLR.

Questions for authors
  1. How can the minimum task-sufficient context of a specific on-device model be quantitatively measured? Could the authors supplement experiments to observe task performance variations under different context compression ratios?

  2. How does this compression approach verify that it approximates or achieves the minimum task-sufficient context?

3.Why is the task success rate not reported? Can tool-calling accuracy directly reflect the overall task success rate?

Reviewer_9BnFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies persistent on-device tool agents whose deployment cost is dominated by KV-cache growth rather than by model weights. It asks how little context is still task-sufficient, and splits that context into static tool schemas and dynamic interaction history. A State-Tracker LoRA turns each completed turn into an append-only Context State Object (CSO) update trained from a teacher that sees the full trajectory and is instructed to keep goals, exact values, completed steps, constraints, and unresolved errors. An Executor LoRA then acts from the CSO rather than the raw history, with separate persistent KV caches so the committed prefix stays reusable. Complementarily, schemas are deterministically minified, and just-in-time (JIT) loading shows names plus short descriptions at selection time and injects the full compact schema only after a tool is chosen.

Critical strengths
  1. The deployment constraint is the right one, and it is measured on device. Fitting a 3B is treated as already solved. The remaining cost is KV cache: 10K–20K tokens are about 500 MB–1 GB in this Q4 deployment, and JIT cuts the profiled initial tool context from 3,246 to 347 tokens. Interaction growth and schema cost are reported separately, which matches the two mechanisms.

  2. The memory design follows from a real inference constraint. Append-only updates leave the committed CSO prefix unchanged, so its KV cache remains valid; current user, tool, and assistant tokens stay ephemeral and are rewound. Separating LoRA_Mem and LoRA_Exec on one base model is a practical way to keep two behaviors without a second full model.

  3. Trained summarization uses the same base model, data, teacher, and training protocol, and differs in regenerating the full compressed state instead of emitting delta_t. HeadKV-R2, H2O-style eviction, StreamingLLM, and training-free recursive summarization cover the token-deletion axis. Two simulators, a second backbone, and ToolAlpaca prevent the paper from resting on a single Gemini-generated loop.

  4. Combined shifts toward higher precision and lower recall, and the discussion correctly treats Memory-Efficient as the capability point and Combined as the aggressive deployment point. ToolAlpaca recall falling from 0.90 (full history) to 0.72 (Memory-Efficient) is disclosed in the main text.

Critical weaknesses
  1. Tool-set F1 does not test the claimed memory. A required tool name counts as a hit even when the argument, identifier, or error recovery is wrong (Sec. 4.3). That misses the exact values the CSO is built to preserve. The Gemini judge, which does look at arguments, disagrees: Combined’s cloud-delegation score is 2.68, against 4.11 for Memory-Efficient and 3.79 for full history (Table 7), while cloud F1 stays at 0.99. On ToolAlpaca, Memory-Efficient recall falls from 0.90 to 0.72 and F1 does not beat full history.

  2. The append-only claim is weakly isolated. Against the matched summarizer, the Qwen3.5 gain is about +0.05 to +0.07 F1. Appendix A.14 is a composite, not a causal comparison. Prefix reuse is also unmeasured on device: the State-Tracker costs 1.31 s per turn, but the prefill saved versus a rewrite is only inferred from immutability and Figure 3.

  3. Labels see the future, and the main eval stays in-distribution. The teacher writes delta_t with the full trajectory, so the target uses information the deployed tracker does not have. The 406 tasks share the training scenario framework and differ only by 19 held-out tools. The full-history Multi-Tool F1 drop from 0.83 (Gemini) to 0.51 (Qwen3.5) is left unexplained.

  4. The short horizon hides the cost of never revising state. Over 25–30 turns the CSO only grows, so goal changes, resolved errors, and replaced identifiers are untested. JIT also has no recovery once the first-stage selection is wrong, beyond the reported precision–recall shift.

Questions for authors
  1. On the Qwen3.5 and ToolAlpaca trajectories, what fraction of required arguments are exactly correct, and what fraction of gold identifiers, constraints, and unresolved errors appear verbatim in the final CSO?

  2. Which metric and which direction enter the Wilcoxon tests, and do Memory-Efficient and Combined differ significantly from the matched trained summarizer on the same paired tasks?

  3. If the teacher is denied the future trajectory and sees only the current CSO, the new turn, and the goal, how much do the F1 and exact-argument numbers move?

  4. For Combined’s cloud-delegation judge score of 2.68, are the failures missed tool selections, bad arguments after a correct selection, or delegation quality?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.