- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to predict the next tool an LLM agent will call, so that the system can prepare this tool early and hide its latency. The authors build a global tool graph with tool nodes and semantic state nodes, and the transition edges store counts from training trajectories. The session text is encoded by BM25, the past tool sequence is encoded by a GRU, and both vectors are added into every layer of a two-layer R-GCN. A node head, an edge head with transition statistics, and a Markov prior are summed to rank the candidates, and the top tools receive tiered prefetch. On two τ-bench retail sets and APIGen-MT, the method reports 65% to 85% top-1 accuracy and 1.79x to 2.09x simulated speedup, and it beats rerun k-gram, smoothed Markov, and LLM speculator baselines. Ablations on ToolBench and τ-bench study the role of the transition features and of each input source.
Critical strengths
Strength 1. The paper targets a real gap in agent serving. Count-based predictors are fast but ignore the conversation, while LLM speculators read the conversation but cost seconds per step. The proposed model ranks all candidates with context at 18 ms and with no LLM calls, which makes tiered prefetch possible, and this is the main reason I see value in the work.
Strength 2. The evaluation is more careful than is common in this area. The authors rerun the baselines on the same test split, give the LLM competitors full history, and also compare methods inside the same Markov candidate set in Table 3. This in-set test shows a gain of 20 to 25 points over counting, which supports the claim that context helps beyond the previous tool.
Critical weaknesses
Weakness 1. The speedup claims rest on a simulator with fixed savings of 60% and 20% and a fixed 50 ms penalty, not on measured tool latencies. Under these rules, the speedup is mostly a direct function of top-1 accuracy, so it adds little new evidence about real benefit. In addition, the model predicts only tool names and not arguments, so it is unclear what heavy prefetch can actually do in practice compared with PASTE or Speculative Actions, which run full calls.
Weakness 2. Several design choices seem to be tuned on the test data. Section 3.4 says the best epoch is chosen by AUC on the evaluation split, which appears to be the shared test set, and Table 6 shows that early stopping alone adds 5.4 points. The window k, the learning rate, the truncation length, and the vocabulary repair also have large effects, and with only three seeds and a standard deviation of 10.17 on APIGen, I think the reported margins may be optimistic.
Questions for authors
Is the evaluation split used for epoch selection and early stopping the same as the test split in Table 2? If it is, what are the results when a separate validation split is used for selection?
What exact operations does heavy prefetch perform when only the tool name is known and the arguments are not? (Suggestion) How do the speedups change when measured tool latencies from a real serving system are used?
BUTTONInstruct is listed in Table 1, and ToolBench appears only in the ablations. Why are their main top-1 accuracy and speedup results not reported in Table 2?