- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a taxonomy-driven framework for designing and evaluating LLM agents for natural-language map queries. The authors introduce a faceted taxonomy covering context binding, grounding, entity retrieval, derivation, and network analysis. The taxonomy is validated through LLM and human-expert annotations on three existing datasets.
Based on the taxonomy, the authors construct a set of 12 tools with reduced functional overlap and develop a pipeline for generating evaluation queries with explicit capability coverage. A relatively simple agent is then used to compose these tools into executable query plans. Experiments on MapQA, MapEval-API, and a synthesized dataset show improvements over existing map-query agents. The authors also investigate how overlapping tool interfaces affect planners of different sizes.
Critical strengths
-
The paper addresses an interesting limitation of existing map-query agents, where tool inventories are often designed around the capabilities required by particular benchmarks. The taxonomy validation using existing datasets and human annotation also provides some evidence for its consistency.
-
The authors evaluate the proposed agent with Qwen, Ministral, and Gemma models at different scales.
Critical weaknesses
-
The paper argues that the proposed taxonomy provides a more systematic foundation for tool design and evaluation. However, the proposed agent differs from existing baselines in several aspects, including tool coverage, tool interfaces, backend implementation, and planning strategy. As a result, it is difficult to determine how much of the performance improvement comes specifically from taxonomy-driven tool design.
-
The experiment changes the complexity of the retrieval interface. The performance degradation may not be caused by functional overlap alone.
-
The effect is not consistent across all model scales, with some larger models performing similarly or slightly better under the merged configuration.
-
Limited evaluation on stronger LLM planners.
Questions for authors
-
Could the authors provide a controlled ablation that separates the contribution of taxonomy-driven tool design from the effects of tool coverage and the agent's planning strategy? For example, how would the same planner perform with an alternative tool inventory providing equivalent functionality?
-
In the functional-overlap ablation, how can the authors distinguish the effect of overlapping functionality from the increased complexity of the tool schema? Have other types of functional overlap been considered?
-
The taxonomy is validated using annotations from existing datasets, with human evaluation conducted on a subset. Could the authors clarify how the human-annotated subset was selected and whether any missing capabilities were identified during annotation?
-
The evaluation is limited to open-weight models. Is there a particular reason for excluding closed-source frontier models? Given that larger models appear less sensitive to tool overlap in Table 6, would the proposed taxonomy-driven tool design provide similar benefits when using stronger proprietary planners?
5 comments
Thank you for the detailed response and additional experiments. The revised ablation with TI0, TI1, and TI2 is helpful, and I appreciate the clarification of the human annotation process and the inclusion of a stronger model.
My main concern is still about the contribution of the taxonomy itself. The new experiments show that different ways of organizing retrieval, ranking, and aggregation tools can affect agent performance. However, all three inventories are designed within the proposed framework. It is still difficult to determine whether the improvements come from the taxonomy-driven design or simply from having a better-separated tool interface. A comparison with an alternative tool inventory of equivalent functionality would be more convincing.
I also think the interpretation of functional overlap remains somewhat unclear. TI1 and TI2 control the retrieval schema, but differ in the number of available tools, while TI0 and TI2 have the same tool count but different retrieval interfaces. The results help distinguish some factors, but do not fully isolate the effect of overlap.
The stronger-model evaluation is useful, although I would have liked to see the same tool-inventory ablation with the stronger planner to understand whether the reported benefits persist.
Overall, the rebuttal addresses several of my questions and improves the experimental analysis. However, I think the evidence is still insufficient to support the paper's main claim about taxonomy-driven tool design. I will maintain my original rating.
Thank you for your prompt response. We reply paragraph by paragraph.
My main concern is still about the contribution of the taxonomy itself. .... However, all three inventories are designed within the proposed framework. It is still difficult to determine whether the improvements come from the taxonomy-driven design or simply from having a better-separated tool interface. A comparison with an alternative tool inventory of equivalent functionality would be more convincing.
The problem we address remains unresolved in the existing studies examined in Table 1. Their agents and benchmarks lack important capabilities for supporting or evaluating map queries. For such an unsolved problem, our contribution is a systematic way to derive an inventory covering the capabilities defined by the taxonomy and to construct evaluation data with corresponding coverage. That is to say,
- taxonomy-driven design is our method, and a better-separated tool interface is one of the benefits that our method produces.
- Asking whether the improvement comes from taxonomy-driven design or from better-organized tools is like asking whether a recipe helped or whether the ingredients were simply combined better. Combining them better is what the recipe provides.
You ask for “an alternative tool inventory of equivalent functionality”. But Table 1 shows that the existing inventories examined do not provide that coverage. Constructing such “an alternative” requires designing another inventory, establishing its functional equivalence, and controlling aspects like “tool interfaces, backend implementation, and planning strategy” (quoted from your first comment). Equivalent functionality cannot simply be assumed. Verifying it requires an explicit account of the capabilities each inventory supports. Our taxonomy provides that reference. Matching tool counts or passing the same benchmark would not establish equivalent coverage, since the benchmarks themselves have gaps, also shown in Table 1. This amounts to developing and validating another solution to the same unresolved problem, potentially enough work for another paper.
To put it another way, even if developing such an alternative is feasible,
- You are asking us to solve the problem twice and show that one solution beats the other to establish a contribution. Table 1 already shows that none of the examined methods provide equivalent capability coverage.
We already addressed the request for controlled comparisons: TI0, TI1, and TI2 provide the same capabilities with the same backend, planner, and evaluation settings. Their shared foundation makes these controls possible. These experiments test whether the tool organization derived by our method is effective.
I also think the interpretation of functional overlap remains somewhat unclear. TI1 and TI2 control the retrieval schema, but differ in the number of available tools, while TI0 and TI2 have the same tool count but different retrieval interfaces. The results help distinguish some factors, but do not fully isolate the effect of overlap.
On overlap, the claim is also updated in the revised paper, not just the experiments. Section 6.3 explicitly states that TI0 versus TI1 tests merging operations, TI0 versus TI2 tests the expanded retrieval interface at a fixed tool count, and TI1 versus TI2 tests adding standalone tools under an identical retrieval schema. The differences in tool count and retrieval interface raised in your comment are already stated in the paper and reflected in its interpretation.
The revised conclusion is: “Together, these comparisons support the taxonomy-derived separation of operations into dedicated tools.” Saying that the comparisons do not fully isolate overlap does not challenge this updated claim. It repeats the objection to the earlier interpretation that we have already revised. If the interpretation remains “somewhat unclear”, please identify the specific statement in the revised paper that is unclear or draws a conclusion the comparisons do not support.
The stronger-model evaluation is useful, although I would have liked to see the same tool-inventory ablation with the stronger planner to understand whether the reported benefits persist.
The benefit for smaller planners is one of the paper’s main claims, and the experimental results support it across all three tested LLM families. Even if a stronger planner were less sensitive to these choices, that would not diminish their demonstrated value for smaller planners at all.
Given the contributions and improvements you have acknowledged, requesting another independently developed solution, repeating an objection to a claim we have already revised, and asking whether the benefit extends to a stronger planner do not justify maintaining your original rating. We ask you to reconsider the overall assessment to reflect the revised paper and evidence.
Thank you for the constructive review. We redesigned the ablation, added GPT-6.1 Sol, and clarified the human annotation subset (Section 6.3, Tables 3–6, Appendix A.2.4). The paper is also updated accordingly and uploaded.
W1-W2 / Q1-Q2. Attribution to taxonomy-driven design
We compare three tool inventories with the same capability coverage (Section 6.3). For each backbone, 600 queries, map backend and snapshots, planning and selection, sampling, inference budgets, and scoring are fixed. Tool schemas and examples are adapted to each inventory:
retrievecan rank or aggregate entities from earlier tool calls, replacing the standalonerankandaggregatetools.rankandaggregatetools.The pairwise comparisons are:
retrieve, with the tool count changing from 12 to 10.rankandaggregate. This evaluates the expanded retrieval interface and its resulting overlap at a fixed tool count.retrieveargument schema, keeping its schema complexity fixed. TI2 adds standalonerankandaggregate, increasing the tool count from 10 to 12. This tests whether offering standalone tools changes answer accuracy.Table 6. Answer accuracy (%). Rank/Agg. combines the disjoint subsets whose reference plans require ranking or aggregation. Parentheses give percentage-point changes from each model's TI0.
Section 4.1 derives TI0 by grouping taxonomy leaves with shared input/output signatures. Overall, TI0 > TI2 > TI1 on five of six backbones. TI0's lead over TI1 supports separate operations despite more tools. Its advantage over TI2 favors retrieval focused on entity selection at the same tool count. Among these five, TI2 recovers part of TI1's loss under the same retrieval schema. We evaluate ranking and aggregation as the overlap cases. Appendix G shows Qwen 7B preserving the garage filter with TI0 but dropping it and adding unrequested grouping with TI1/TI2.
W3. Differences across model scales
TI0 provides a larger benefit for the smaller model in each tested family. Ministral 8B and 14B have similar TI0 accuracy, yet lose 8.00 and 3.00 points under TI1. This supports the value of the taxonomy-derived inventory especially for smaller planners (Section 6.3).
Q3. Human subset selection and missing capabilities
Appendix A.2.4 specifies 120 queries, with 40 sampled uniformly at random from each dataset. A domain expert who did not design the taxonomy used the same definitions and examples as the LLM annotators. The protocol assessed annotation consistency and did not include an open-ended audit of missing capabilities.
W4 / Q4. Stronger planners
Tables 3-5 add GPT-6.1 Sol on all three datasets. It achieves 99.67%/99.00% accuracy on BY/IL, compared with 54.40% macro accuracy across all nine MapQA types. Traces identify location-resolution failures on 362 of 948 MapQA queries. These results reinforce the motivation for our synthesized benchmark: supplied grounding reduces location-resolution uncertainty, and replaying reference plans verifies that gold answers are reproducible on the backend. These controls make comparisons more directly reflect agent planning and tool use.
Thank you again for helping strengthen the study. We would appreciate your reassessment.