- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes UTR, which compiles API documentation into an output embedding for tool selection and a compact memory for argument generation. Two-stage training enables new tools to be registered without further parameter updates. UTR achieves 74.77% complete-call EM on STQ, 71.51% macro NDCG@5 on ToolBench with disjoint training APIs, and 80.55% Hit@3 on new-tool queries after adding 5K APIs without retraining.
Critical strengths
-
UTR provides a coherent interface connecting document-based selection and invocation. Shared compilers let new APIs enter the output space and supply calling information without tool-specific optimization, addressing evolving tool inventories.
-
The experiments probe both transfer and the role of memory. Disjoint-API and library-expansion tests evaluate new-tool registration, while controlled caller comparisons and memory interventions examine whether compressed tool information supports argument generation.
Critical weaknesses
-
The practical value and design of compact memory need stronger justification. In the ToolBench control, memory reduces input length but yields lower gold-tool argument EM than full documents. Matched latency and total-cost comparisons are not reported, and the advantage of two compilers over a single shared compiler is not established.
-
Reliable unseen-tool agent execution remains insufficiently validated. The large-scale end-to-end development study uses full documents and mixed API exposure, leaving compact-memory transfer untested in that setting. A compact-memory STQ reproduction invokes tools on all 32 synthetic requests answerable without tools, even when explicitly prompted to answer directly (Appendix C.4), indicating unreliable abstention in this diagnostic.
Questions for authors
-
What motivates the two-compiler design? Is there evidence comparing it with a single shared compiler? For the existing memory/full-document comparisons, are matched latency and total costs, including training and registration, available?
-
Are compact-memory end-to-end results available under strict API holdout? Given Appendix C.4, what training or calibration enables the model to answer directly when no tool is needed?
3 comments
Response to Reviewer J7kG (Part 2/2)
W2: Reliable unseen-tool agent execution remains insufficiently validated. The large-scale end-to-end development study uses full documents and mixed API exposure, leaving compact-memory transfer untested in that setting. A compact-memory STQ reproduction invokes tools on all 32 synthetic requests answerable without tools, even when explicitly prompted to answer directly (Appendix C.4), indicating unreliable abstention in this diagnostic.
Q2: Are compact-memory end-to-end results available under strict API holdout? Given Appendix C.4, what training or calibration enables the model to answer directly when no tool is needed?
1) Compact memory end to end under a strict split
The paper's Table 4 uses full documents, where UTR solves 37.58% of tasks versus 17.25% for ToolRetriever. We now generate every argument from compact memory on the same 765 tasks, and also report the 240 tasks in which no API of any relevant provider was ever a positive training target, a stricter split than single-API holdout. Solve rate with 47,323 registered APIs:
| Configuration | All 765 tasks | Provider never a positive target (240) | Hardest group, G3-instruction |
|---|---|---|---|
| Arguments from compact memory | 30.00 | 31.04 | 26.2 |
| Full-document calling (Table 4 setup, re-run) | 34.84 | 36.04 | 26.2 |
| Full-document calling, ToolRetriever's candidates | 24.97 | 27.08 | — |
- No degradation under strict exposure: both versions score higher on the 240 tasks than on all 765, and tie at 26.2 on G3-instruction, the hardest group, where each task combines several tools from one collection.
- UTR's selection gain also holds on the strict split: with the caller fixed, UTR's candidates lead ToolRetriever's by +9.87 overall and +8.96 on the 240 tasks.
The full-document row re-runs the Table 4 setup on the same hardware as the memory version; it gives 34.84 against 37.58 in the paper, a difference that is not significant per task.
2) Answering directly when no tool is needed
The paper's benchmarks require a tool for every request, so UTR learns which tool to call and how. To decide whether to call, we add a pre-call check, as in standard function calling: the backbone sees the request with UTR's top-1 tool and either calls it or answers directly. It needs no extra training, direct-answer data or threshold, and leaves UTR unchanged. We test it on a mixed set of 1,066 STQ tool queries and 1,950 direct-answer requests in 8 kinds (arithmetic, sorting, reversal, copying, parity, upper-casing, clarification, "no tool is available"):
| Registry | False tool calls | Missed tool calls | Correct direct answers | Accuracy |
|---|---|---|---|---|
| 837 APIs | 0 / 1,950 | 9.57% | 83.1 | 77.2 |
| 1,836 APIs | 0 / 1,950 | 9.85% | 83.1 | 76.4 |
- Zero false calls: no direct-answer request triggers a tool, in any of the 8 kinds, and this still holds when the registry doubles.
- The check is what enables abstention: mixed-set accuracy rises from 25.0 without it to 77.2.
- Registration is unaffected: the check sits in front of UTR, so new tools are still registered by compiling their documents.
Thank you once more for your useful and precise questions. We will add the shared-compiler ablation, the cost table, the exposure split and the calibration study to the paper, and welcome any further discussion.
Thank you for the detailed response. Most of my concerns have been addressed. I will keep my score.
Response to Reviewer J7kG (Part 1/2)
We sincerely thank you for your support. We greatly appreciate your description of UTR as "a coherent interface connecting document-based selection and invocation" and your recognition of our transfer and memory experiments. Your questions guided us to measure the full cost of compact memory, test a shared compiler, and evaluate memory end to end under a strict exposure split.
1) What memory carries
On 997 unseen ToolBench calls, only 16 memory slots reach 42.1 argument EM, 80% of the 52.4 from full documents, with half the input. The information is tool-specific: another tool's memory gives 0.3, and the memory of the tool with the most similar parameter names is still 25.5 lower. UTR's caller accepts either memory or documents, so both can be used as needed.
2) Matched latency and total cost
A matched pair trained with the same recipe and measured on the same hardware:
Training cost is the same, registration is a one-off 73 ms, and each step reads half the input or less; the saving grows with steps and candidates, and prefill is 14.1× faster at 100 candidates.
3) Two compilers vs. a shared compiler
The two representations sit at opposite ends of the decoder. Selection uses one vector in the output layer, scored with the LM head to single out the correct tool among many; calling uses several slots in the input embedding space to carry parameter names, types and required fields. Separate compilers let each be optimized for its own use.
In the shared variant, the output row is pooled from the memory slots and projected, so the selection loss also trains the memory compiler. Both start from the same stage-1 checkpoint, where the shared output rows match the original ones at cosine ≥ 0.99999, so any difference comes from sharing alone. Both follow the matched setting of App. C.2.2 with 1,000 stage-2 steps and seed 17. STQ, 1,066 queries, 837 unseen APIs:
Separate compilers are preferable for three reasons: