- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies tool exposure as a decision that can both improve and reduce an LLM agent's accuracy. They proposes AgentRT which decides whether to expose tools using inexpensive prompt features, directly executes supported requests through a typed parser, and constrains the remaining tool interactions. Experiments report accuracy and energy improvements on a mixture of mathematical reasoning and synthetic computation tasks. Further evaluations examine paraphrase robustness, parser coverage, and differences across models.
Critical strengths
- The paper makes a useful empirical observation that exposing tools can damage accuracy, while withholding them can increase computation when the model struggles to answer directly. Measuring both correctness and energy makes this more informative than an analysis based only on tool-call counts. The rescued/broken decomposition also gives a clear way to quantify the accuracy headroom of admission for a fixed pair of policies.
- The evaluation explicitly examines limitations rather than reporting only favourable aggregate results. The parser ablation, paraphrase experiments, external coverage measurements, and newer-model results show that the benefits depend on both workload structure and model behaviour. In particular, acknowledging the admission gate's poor transfer helps separate the system design from the effectiveness of its current predictor.
Critical weaknesses
- The headline improvement is largely attributable to specialised symbolic execution, and the comparison does not isolate what AgentRT adds beyond this. A crucial missing comparison is the same parser followed by ordinary no-tool generation for unsupported requests, or the same parser attached to the strongest tool baseline. The held-out paraphrases provide useful evidence within the supported task families.
- Serving-state awareness is central to the motivation, but its benefit is not demonstrated. The main experiments use zero serving pressure, and the additional pressure settings do not establish behaviour under an actual capacity bottleneck. Moreover, raising an exposure threshold under load needs justification because the paper shows that exposure often reduces serving cost. A score threshold without a request-specific cost estimate does not establish that the resulting policy improves the accuracy-latency trade-off under pressure.
Questions for authors
- How does AgentRT compare with the same frozen parser followed by either no-tool generation or the strongest existing tool policy?
- Can you demonstrate the serving-state mechanism under measured load, reporting throughput, tail latency, and accuracy? How should the gate treat requests for which exposing a tool reduces cost?