- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents an empirical comparison of Direct Tool Exposure (DTE) and Agent-as-a-Tool (AaT) from the service provider's perspective. Through controlled experiments involving five LLMs and 250 synthetic document-management tasks, the authors investigate the effects of delegation, model capability, and tool scalability. The results suggest that AaT primarily benefits from capability transfer through stronger inner agents and can reduce token consumption as the number of exposed tools increases.
Critical strengths
S1. Systematic experimental design.
The paper conducts extensive experiments across multiple models, task complexities, and tool configurations. The same-model comparisons provide a useful approach to separating architectural effects from model capability differences.
S2. Practical relevance.
The study addresses a relevant design choice for tool-based agent systems. Its analysis of capability transfer and token efficiency provides useful insights for service providers considering different tool exposure strategies.
Critical weaknesses
W1. Limited novelty of the main findings.
The main findings suggest that weaker clients benefit from stronger inner agents and that AaT reduces tool-schema overhead. These results are largely intuitive, and the study provides limited insights beyond these expected observations.
W2. Insufficiently controlled architectural comparison.
Although same-model comparisons control for model capability, DTE and AaT still differ in inference overhead and context management. Without matching computational budgets, the effect of delegation itself cannot be fully isolated.
W3. Limited generalizability to real-world settings.
All experiments are conducted in a single synthetic document-management environment, and AaT is restricted to a single delegation call. It remains unclear whether the findings generalize to realistic multi-service workflows involving iterative interaction and replanning.
Questions for authors
Please see the critical weaknesses above.
3 comments
Thank you for the detailed response. While some of my concerns have been addressed, I still think that the paper's main findings, particularly regarding capability transfer and token efficiency, are largely consistent with intuitive expectations, and thus offer relatively limited new insights. Additionally, the experiments are restricted to a single synthetic environment and a single-delegation setting, leaving the generalizability of the conclusions to more complex real-world workflows uncertain.
Nevertheless, I acknowledge the empirical value of this work. The experimental design is systematic, covering different model capabilities, task complexities, and tool configurations. The same-model comparisons and statistical analyses also provide meaningful support for the main conclusions. The authors' response further clarifies the rationale behind the experimental setup.
Overall, considering these points, I am willing to raise my rating from 2 to 3.
Reply to Reviewer 269K
We thank the reviewer again for recognizing the value of this work.
For a better clarification of the new insights of this paper, we would recommend the reviewer to check our latest reply to Reviewer 7PKA as well, and see whether the response can answer some of the remaining concerns regarding novelty and insights.
Again, thank you for your valuable feedback!
Response to Reviewer 269K
We thank the reviewer for recognizing the systematic design and its practical relevance.
W1. Limited novelty; findings are intuitive.
The finding that weaker clients benefit from a stronger inner agent is intuitive only if one assumes that the natural-language delegation boundary is nearly lossless. That assumption is exactly what the same-model controls highlighted in S1 test, and no prior work had measured it. With Opus as inner agent, the weakest client matches DTE(Opus) within ±5pp although its own hard-task success is 11.0%, and five clients with DTE baselines spanning 11–96% fall within 2.0pp of each other. The same controls also show where the assumption weakens (a −6.0pp, model-specific estimate for Sonnet), which an intuition-level argument cannot reveal.
W2. Computational budgets are not matched.
Both architectures share the same per-tier primitive tool-call budget, and it never binds.
The residual difference is about three additional LLM calls and 18–21% more tokens for AaT, two of which are the client's delegation and summary. That difference is inherent to the boundary being measured, and it favors AaT. Same-model AaT still does not outperform DTE, so our "no intrinsic architectural advantage" conclusion is conservative with respect to compute.
Context handling is identical: both use the same agent loop without compaction, and the inner agent's context is the DTE context with the instruction replaced by the client's request. Capability transfer also does not rely on extra compute: AaT(Qwen→Opus) matches DTE(Opus) accuracy at slightly lower token cost (64K vs. 67K). We will add this compute breakdown to the appendix.
W3. Single synthetic domain; single delegation call.
Both are deliberate identification choices rather than oversights.
Agent.as_tool(), MCP tools and Browser Use are invoked. Iterative replanning across services is a client-side orchestration question, which we deliberately separate (§4.2).We therefore state our conclusions conditionally on service-local requests, and we regard replication in other domains as complementary future work rather than a gap in the present design. We will also soften the introduction's "without loss of generality" footnote to state the decomposition into atomic requests as a scope assumption.