- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a learned State-Tracker that appends compact updates to a persistent context state and an Executor that acts from that state, together with compact tool schemas and just-in-time schema loading. It evaluates tool-use precision, recall, and F1 under two simulators, an external ToolAlpaca setting, and another backbone, and reports context, KV-memory, and initial latency measurements on a Samsung Galaxy S25. The intended contribution is a practical capability-memory trade-off for small on-device agents.
Critical strengths
-
The approach separates growing interaction history from static tool-schema overhead, making the two mechanisms independently usable. Comparison with trained summarization and KV-compression baselines, evaluation under a second simulator, and external-distribution results are useful additions beyond a single synthetic benchmark.
-
Actual device measurements strengthen the systems motivation. The paper also reports the State-Tracker's 1.31-second post-turn cost and the precision/recall trade-off from JIT loading rather than portraying compression as uniformly lossless. Appendix A.12 usefully clarifies the separate adapter-specific caches.
Critical weaknesses
-
A central result is inconsistent with the stated metric definition. In Table 1, Combined on mixed tasks has F1=0.94, P=0.92, and R=0.92. Appendix A.5 specifies per-trajectory F1=2PR/(P+R), followed by macro-averaging. For every trajectory F1 <= (P+R)/2, so the same inequality must hold for those macro averages. The reported triple violates it by more than rounding can explain. Independently averaging F1, P, and R does not resolve this inequality. The evaluation code or table needs correction before the headline performance comparisons can be trusted.
-
The principal tool-set metrics do not establish preservation of executable task behavior: matching tool identities does not check argument values, execution order, repeated calls, or the final state. This matters directly for a memory method whose key promise is retaining information required later. The supplemental quality judgments and external results show meaningful trade-offs: Table 7's Combined quality is below the full-history baseline on cloud and mixed tasks, while ToolAlpaca F1 drops from 0.69 to 0.65 and recall from 0.90 to 0.58. These results merit an argument-/outcome-aware evaluation before describing task capability as broadly maintained.
Questions for authors
-
Please provide the exact aggregation code or per-trajectory P/R/F1 values and explain/correct Table 1's mixed-task Combined entry. Are any other reported metrics or significance tests affected?
-
How do the methods compare on argument-correct tool calls and verified final task outcomes, especially cases requiring exact values from distant turns? Do the memory savings persist at a matched success level?
-
For the device results, what are complete-episode latency and peak memory when both adapter caches, State-Tracker updates, and JIT selection passes are included? This would clarify the practical benefit beyond initial TTFT and individual context sizes.