acceptodds
Under review as a conference paper at ICLR 2027

Taxonomy-Driven Map Query Agent Design and Evaluation

Abstract

Large language models (LLMs) equipped with tools can resolve map queries expressed in natural language. However, existing systems often assemble their tools and evaluation around target benchmarks, inheriting their capability gaps and lacking an independent reference for assessing coverage. We address this limitation with a faceted taxonomy of capabilities for map queries over 2D vector map data, validated through LLM and human-expert annotation on three datasets. From this taxonomy, we systematically construct a twelve-tool inventory with minimized functional overlap and synthesize evaluation data that covers every capability it defines. We build a simple agent to compose these tools into executable query plans. Experiments across three LLM families show that our agent improves on baseline results for two external benchmarks while maintaining relatively stable accuracy across planner scales. The ablation shows that overlapping tool interfaces reduce accuracy more for smaller planners, supporting the taxonomy's role in the systematic construction of tool inventories with minimal functional overlap.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
natural-language map queries, geospatial question answering, large language models, tool-using agents, faceted taxonomy, synthetic data generation, capability coverage, agent evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AV4aOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a taxonomy-driven framework for designing and evaluating LLM agents for natural-language map queries. The authors introduce a faceted taxonomy covering context binding, grounding, entity retrieval, derivation, and network analysis. The taxonomy is validated through LLM and human-expert annotations on three existing datasets.

Based on the taxonomy, the authors construct a set of 12 tools with reduced functional overlap and develop a pipeline for generating evaluation queries with explicit capability coverage. A relatively simple agent is then used to compose these tools into executable query plans. Experiments on MapQA, MapEval-API, and a synthesized dataset show improvements over existing map-query agents. The authors also investigate how overlapping tool interfaces affect planners of different sizes.

Critical strengths
  1. The paper addresses an interesting limitation of existing map-query agents, where tool inventories are often designed around the capabilities required by particular benchmarks. The taxonomy validation using existing datasets and human annotation also provides some evidence for its consistency.

  2. The authors evaluate the proposed agent with Qwen, Ministral, and Gemma models at different scales.

Critical weaknesses
  1. The paper argues that the proposed taxonomy provides a more systematic foundation for tool design and evaluation. However, the proposed agent differs from existing baselines in several aspects, including tool coverage, tool interfaces, backend implementation, and planning strategy. As a result, it is difficult to determine how much of the performance improvement comes specifically from taxonomy-driven tool design.

  2. The experiment changes the complexity of the retrieval interface. The performance degradation may not be caused by functional overlap alone.

  3. The effect is not consistent across all model scales, with some larger models performing similarly or slightly better under the merged configuration.

  4. Limited evaluation on stronger LLM planners.

Questions for authors
  1. Could the authors provide a controlled ablation that separates the contribution of taxonomy-driven tool design from the effects of tool coverage and the agent's planning strategy? For example, how would the same planner perform with an alternative tool inventory providing equivalent functionality?

  2. In the functional-overlap ablation, how can the authors distinguish the effect of overlapping functionality from the increased complexity of the tool schema? Have other types of functional overlap been considered?

  3. The taxonomy is validated using annotations from existing datasets, with human evaluation conducted on a subset. Could the authors clarify how the human-annotated subset was selected and whether any missing capabilities were identified during annotation?

  4. The evaluation is limited to open-weight models. Is there a particular reason for excluding closed-source frontier models? Given that larger models appear less sensitive to tool overlap in Table 6, would the proposed taxonomy-driven tool design provide similar benefits when using stronger proprietary planners?

5 comments

Thank you for the constructive review. We redesigned the ablation, added GPT-6.1 Sol, and clarified the human annotation subset (Section 6.3, Tables 3–6, Appendix A.2.4). The paper is also updated accordingly and uploaded.

W1-W2 / Q1-Q2. Attribution to taxonomy-driven design

We compare three tool inventories with the same capability coverage (Section 6.3). For each backbone, 600 queries, map backend and snapshots, planning and selection, sampling, inference budgets, and scoring are fixed. Tool schemas and examples are adapted to each inventory:

  • TI0: the original 12-tool inventory, with retrieval, ranking, and aggregation separate.
  • TI1: a merged 10-tool inventory. retrieve can rank or aggregate entities from earlier tool calls, replacing the standalone rank and aggregate tools.
  • TI2: an overlapping 12-tool inventory. It combines TI1's retrieval interface with the standalone rank and aggregate tools.

The pairwise comparisons are:

  • TI0 vs. TI1: Both provide the same operations without duplicate ranking or aggregation interfaces. The comparison assesses the effect of merging these operations into retrieve, with the tool count changing from 12 to 10.
  • TI0 vs. TI2: Both expose the same 12 tool names, including standalone rank and aggregate. This evaluates the expanded retrieval interface and its resulting overlap at a fixed tool count.
  • TI1 vs. TI2: Both use an identical expanded retrieve argument schema, keeping its schema complexity fixed. TI2 adds standalone rank and aggregate, increasing the tool count from 10 to 12. This tests whether offering standalone tools changes answer accuracy.

Table 6. Answer accuracy (%). Rank/Agg. combines the disjoint subsets whose reference plans require ranking or aggregation. Parentheses give percentage-point changes from each model's TI0.

BackboneTIAll(600)Rank/Agg.(262)Rank(181)Agg.(81)
Qwen 2.5 72BTI084.3381.6881.7781.48
Qwen 2.5 72BTI182.17(-2.17)77.86(-3.82)79.56(-2.21)74.07(-7.41)
Qwen 2.5 72BTI283.00(-1.33)79.01(-2.67)77.90(-3.87)81.48(+0.00)
Qwen 2.5 7BTI041.0046.9553.0433.33
Qwen 2.5 7BTI134.17(-6.83)30.15(-16.79)39.23(-13.81)9.88(-23.46)
Qwen 2.5 7BTI236.33(-4.67)37.40(-9.54)46.96(-6.08)16.05(-17.28)
Ministral 3 14BTI088.8387.0283.4395.06
Ministral 3 14BTI185.83(-3.00)76.72(-10.31)79.01(-4.42)71.60(-23.46)
Ministral 3 14BTI288.17(-0.67)83.59(-3.44)82.32(-1.10)86.42(-8.64)
Ministral 3 8BTI086.0079.7783.9870.37
Ministral 3 8BTI178.00(-8.00)61.07(-18.70)79.01(-4.97)20.99(-49.38)
Ministral 3 8BTI281.67(-4.33)69.47(-10.31)82.32(-1.66)40.74(-29.63)
Gemma 4 31BTI084.0089.6991.1686.42
Gemma 4 31BTI185.50(+1.50)93.13(+3.44)90.06(-1.10)100.00(+13.58)
Gemma 4 31BTI285.17(+1.17)93.13(+3.44)90.06(-1.10)100.00(+13.58)
Gemma 4 E4BTI068.5069.8571.8265.43
Gemma 4 E4BTI160.67(-7.83)51.53(-18.32)62.98(-8.84)25.93(-39.51)
Gemma 4 E4BTI263.00(-5.50)55.73(-14.12)64.64(-7.18)35.80(-29.63)

Section 4.1 derives TI0 by grouping taxonomy leaves with shared input/output signatures. Overall, TI0 > TI2 > TI1 on five of six backbones. TI0's lead over TI1 supports separate operations despite more tools. Its advantage over TI2 favors retrieval focused on entity selection at the same tool count. Among these five, TI2 recovers part of TI1's loss under the same retrieval schema. We evaluate ranking and aggregation as the overlap cases. Appendix G shows Qwen 7B preserving the garage filter with TI0 but dropping it and adding unrequested grouping with TI1/TI2.

W3. Differences across model scales

TI0 provides a larger benefit for the smaller model in each tested family. Ministral 8B and 14B have similar TI0 accuracy, yet lose 8.00 and 3.00 points under TI1. This supports the value of the taxonomy-derived inventory especially for smaller planners (Section 6.3).

Q3. Human subset selection and missing capabilities

Appendix A.2.4 specifies 120 queries, with 40 sampled uniformly at random from each dataset. A domain expert who did not design the taxonomy used the same definitions and examples as the LLM annotators. The protocol assessed annotation consistency and did not include an open-ended audit of missing capabilities.

W4 / Q4. Stronger planners

Tables 3-5 add GPT-6.1 Sol on all three datasets. It achieves 99.67%/99.00% accuracy on BY/IL, compared with 54.40% macro accuracy across all nine MapQA types. Traces identify location-resolution failures on 362 of 948 MapQA queries. These results reinforce the motivation for our synthesized benchmark: supplied grounding reduces location-resolution uncertainty, and replaying reference plans verifies that gold answers are reproducible on the backend. These controls make comparisons more directly reflect agent planning and tool use.

Thank you again for helping strengthen the study. We would appreciate your reassessment.

Thank you for the detailed response and additional experiments. The revised ablation with TI0, TI1, and TI2 is helpful, and I appreciate the clarification of the human annotation process and the inclusion of a stronger model.

My main concern is still about the contribution of the taxonomy itself. The new experiments show that different ways of organizing retrieval, ranking, and aggregation tools can affect agent performance. However, all three inventories are designed within the proposed framework. It is still difficult to determine whether the improvements come from the taxonomy-driven design or simply from having a better-separated tool interface. A comparison with an alternative tool inventory of equivalent functionality would be more convincing.

I also think the interpretation of functional overlap remains somewhat unclear. TI1 and TI2 control the retrieval schema, but differ in the number of available tools, while TI0 and TI2 have the same tool count but different retrieval interfaces. The results help distinguish some factors, but do not fully isolate the effect of overlap.

The stronger-model evaluation is useful, although I would have liked to see the same tool-inventory ablation with the stronger planner to understand whether the reported benefits persist.

Overall, the rebuttal addresses several of my questions and improves the experimental analysis. However, I think the evidence is still insufficient to support the paper's main claim about taxonomy-driven tool design. I will maintain my original rating.

Thank you for your prompt response. We reply paragraph by paragraph.

My main concern is still about the contribution of the taxonomy itself. .... However, all three inventories are designed within the proposed framework. It is still difficult to determine whether the improvements come from the taxonomy-driven design or simply from having a better-separated tool interface. A comparison with an alternative tool inventory of equivalent functionality would be more convincing.

The problem we address remains unresolved in the existing studies examined in Table 1. Their agents and benchmarks lack important capabilities for supporting or evaluating map queries. For such an unsolved problem, our contribution is a systematic way to derive an inventory covering the capabilities defined by the taxonomy and to construct evaluation data with corresponding coverage. That is to say,

  • taxonomy-driven design is our method, and a better-separated tool interface is one of the benefits that our method produces.
  • Asking whether the improvement comes from taxonomy-driven design or from better-organized tools is like asking whether a recipe helped or whether the ingredients were simply combined better. Combining them better is what the recipe provides.

You ask for “an alternative tool inventory of equivalent functionality”. But Table 1 shows that the existing inventories examined do not provide that coverage. Constructing such “an alternative” requires designing another inventory, establishing its functional equivalence, and controlling aspects like “tool interfaces, backend implementation, and planning strategy” (quoted from your first comment). Equivalent functionality cannot simply be assumed. Verifying it requires an explicit account of the capabilities each inventory supports. Our taxonomy provides that reference. Matching tool counts or passing the same benchmark would not establish equivalent coverage, since the benchmarks themselves have gaps, also shown in Table 1. This amounts to developing and validating another solution to the same unresolved problem, potentially enough work for another paper.

To put it another way, even if developing such an alternative is feasible,

  • You are asking us to solve the problem twice and show that one solution beats the other to establish a contribution. Table 1 already shows that none of the examined methods provide equivalent capability coverage.

We already addressed the request for controlled comparisons: TI0, TI1, and TI2 provide the same capabilities with the same backend, planner, and evaluation settings. Their shared foundation makes these controls possible. These experiments test whether the tool organization derived by our method is effective.


I also think the interpretation of functional overlap remains somewhat unclear. TI1 and TI2 control the retrieval schema, but differ in the number of available tools, while TI0 and TI2 have the same tool count but different retrieval interfaces. The results help distinguish some factors, but do not fully isolate the effect of overlap.

On overlap, the claim is also updated in the revised paper, not just the experiments. Section 6.3 explicitly states that TI0 versus TI1 tests merging operations, TI0 versus TI2 tests the expanded retrieval interface at a fixed tool count, and TI1 versus TI2 tests adding standalone tools under an identical retrieval schema. The differences in tool count and retrieval interface raised in your comment are already stated in the paper and reflected in its interpretation.

The revised conclusion is: “Together, these comparisons support the taxonomy-derived separation of operations into dedicated tools.” Saying that the comparisons do not fully isolate overlap does not challenge this updated claim. It repeats the objection to the earlier interpretation that we have already revised. If the interpretation remains “somewhat unclear”, please identify the specific statement in the revised paper that is unclear or draws a conclusion the comparisons do not support.


The stronger-model evaluation is useful, although I would have liked to see the same tool-inventory ablation with the stronger planner to understand whether the reported benefits persist.

The benefit for smaller planners is one of the paper’s main claims, and the experimental results support it across all three tested LLM families. Even if a stronger planner were less sensitive to these choices, that would not diminish their demonstrated value for smaller planners at all.


Given the contributions and improvements you have acknowledged, requesting another independently developed solution, repeating an objection to a claim we have already revised, and asking whether the benefit extends to a stronger planner do not justify maintaining your original rating. We ask you to reconsider the overall assessment to reflect the revised paper and evidence.

Reviewer_NZN8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to design and evaluate agents for natural-language queries over two-dimensional vector maps. It introduces a faceted capability taxonomy covering context binding, grounding, entity retrieval, derivation, and network analysis. The taxonomy guides the construction of a tool inventory and typed query-plan DAGs. These plans are instantiated against real map data, verbalized into natural-language queries, and executed to obtain reference answers. The resulting agent generates, validates, executes, and revises candidate plans. The paper evaluates annotation agreement and agent performance on MapQA, MapEval-API, and a synthetic dataset, reporting improvements over MapAgent on MapEval-API.

Critical strengths
  1. A meaningful problem formulation with practical value. The paper identifies how tool inventories and evaluation benchmarks can inherit each other’s capability blind spots. Introducing a shared capability taxonomy provides a useful reference for auditing both systems and datasets, making capability omissions visible beyond aggregate accuracy.

  2. A clear and traceable connection between tools and evaluation data. Capability labels, typed tool signatures, executable query plans, natural-language queries, and reference answers form a coherent and traceable pipeline. Fixed OSM snapshots and plan re-execution support reproducibility, while diverse DAG structures provide a useful foundation for evaluating compositional tool use.

  3. Useful empirical evidence and transparent reporting. Experiments across model families and scales provide evidence for the proposed design, including improvements over MapAgent on MapEval-API. The authors disclose limitations in released baseline implementations, report results under both stricter and original scoring procedures, and appropriately acknowledge that the illustrative ablation trace does not independently establish a causal effect of functional overlap.

Critical weaknesses

Major Weaknesses

  1. Annotation consistency does not establish the sufficiency of the taxonomy as a design principle. Section 3.2 reports annotations by two LLMs over 4,543 queries and by one expert over 120 queries, with a leaf-level Krippendorff’s alpha of 0.764. This supports consistent application of the supplied definitions. However, all annotators use the same taxonomy, boundary rules, and examples, so agreement cannot rule out shared omissions or establish that the chosen granularity and tool grouping are preferable to reasonable alternatives. An independent, open-ended audit of uncovered or difficult-to-classify requirements would strengthen this claim. Comparisons with alternative capability partitions or manually designed tool inventories, under matched backends, planners, and inference budgets, would also help distinguish the contributions of expanded coverage, interface design, and this particular taxonomy.

  2. Candidate selection and revision may favor nonempty answers, while important inference details remain unspecified. Section 5 prioritizes nonempty results and nonzero scalars and permits a revised plan to replace the original when an empty or zero result becomes nonempty or nonzero. However, Table 14 reports 41 valid empty or zero-valued answers. A revision could obtain a nonempty result by relaxing a distance threshold, removing an attribute constraint, or changing the reference region; the manuscript does not establish that revisions preserve the original requirements. The authors should report accuracy before and after selection and revision on these cases, constraint changes, and incorrect replacement rates, together with an ablation of the nonempty preference. Reproducibility also requires the inference-time candidate count, sampling settings, full prompts, structure-grouping and tie-breaking rules, and a clear algorithm or flowchart for generation, execution, and revision. Figures 1 and 3 primarily describe tool derivation and data synthesis rather than this inference procedure.

  3. Coverage of model types and analysis across benchmark dimensions remain insufficient. Tables 3–5 already evaluate nine configurations from Qwen2.5, Ministral 3, and Gemma 4, providing evidence across families and scales. The analysis nevertheless mainly examines aggregate performance when replacing the backbone of the same agent. Representative reasoning-oriented and strong proprietary planners, evaluated with matched backend evidence and tool access and explicit inference budgets, would broaden the assessment; the model-selection rationale and scope of the conclusions should also be stated. More importantly, Tables 14–16 characterize the dataset’s capability and structural distribution without systematically showing how individual models perform across capability facets and key leaves, DAG topology and size or dependency depth, capability combinations, regions, and valid empty versus nonempty answers. The gap between high tool-multiset F1 and lower answer accuracy in Table 5 also calls for separate analysis of tool selection, parameter binding, dependency composition, and execution failures. Reporting performance, sample counts, and uncertainty for these subsets would clarify which abilities the benchmark distinguishes and whether model rankings change with task complexity.

  4. The semantic reliability and representativeness of the synthetic evaluation require independent validation. Table 13 uses the same GPT-5.6 Sol model for verbalization and semantic review. Separate sessions and withholding reference answers are useful safeguards, but correlated semantic omissions remain possible. Re-execution verifies consistency between the serialized plan and its answer, rather than independently establishing equivalence between the natural-language question and the plan. A stratified, blinded human audit should check constraints, quantifiers, operand order, and dependencies. In addition, Table 15 contains 403 queries exercising N4, while C1, C2, G1, G2, G4, R2, R4, and D4 each occur in only six queries. A few occurrences per leaf do not permit stable estimates for rare capabilities, and aggregate accuracy provides limited evidence about them. The authors should increase rare-capability coverage or limit the corresponding claims, explain sampling quotas and aggregation weights, and provide balanced or macro-averaged results when sample sizes permit. External benchmarks mitigate circular validation, but do not resolve these semantic-quality and representativeness concerns.

Minor Weaknesses

  1. The original accuracies in the main results and ablation need reconciliation. Each region contains 300 queries. The Qwen-7B accuracies in Table 5, 62.33% and 66.67%, imply approximately 64.50% over the combined dataset, whereas the Original row in Table 6 reports 62.33% over all 600 queries. For Qwen-72B, the corresponding combined accuracy is approximately 76.83%, compared with 77.00% in Table 6. Please clarify whether these are independent runs or different samples and report the resulting variability; otherwise, the statistics should be checked.

  2. The declared tool signature and the execution example appear inconsistent. Table 2 declares that optimize_stops returns ordered coordinates, and Table 12 also specifies a coords-to-coords operation. In Table 20, however, its result is passed directly to measure to compute route length. Table 11 does not declare the necessary conversion from coords to geom or route. Please explain whether the tool additionally constructs a route or whether an implicit step is omitted, and align the contract with the example.

  3. The generation rule for reverse-geocoding examples is missing. Appendix C.1 describes Table 12 as the complete production-rule inventory, but no rule invokes rev_geocode, even though Table 16 reports three queries using this tool in each region. Please provide the relevant answer goal, production rule, or additional insertion mechanism so that full tool coverage can be traced to the generation procedure.

  4. Capability-coverage and scoring conventions need more concrete explanation. Table 1 lists G2 as missing from GeoQuestions1089, whereas Table 10 reports 5.6% empirical utilization; related differences also occur for some MapEval leaves. The appendix distinguishes documented question-type coverage from query-level annotations, but concrete examples are needed to explain how these differences affect the wording of capability omissions. Appendix E.3 should also specify exact-match normalization for option identifiers, answer text, capitalization, whitespace, and numerical units.

Questions for authors
  1. What evidence beyond annotation agreement supports the chosen taxonomy boundaries and granularity? Can you provide an independent audit of uncovered requirements or a controlled comparison with a reasonable alternative tool partition?

  2. How does selection and revision preserve every constraint in the original query? What are the before-and-after results and incorrect replacement rates on the 41 valid empty or zero-valued cases? Please also specify candidate counts and the complete selection and revision procedure.

  3. Can you provide per-model results by capability, DAG topology and complexity, and region, together with errors separated into tool selection, parameter binding, dependency composition, and execution? Do the aggregate trends across model scales hold within these subsets?

  4. What independent evidence establishes question-plan semantic equivalence after verbalization? Please report any blinded human audit or other independent checks, and clarify how limited samples for rare capabilities affect the claims of comprehensive evaluation.

  5. What explains the discrepancies between the Original accuracies in Tables 5 and 6? Also, does optimize_stops return only ordered coordinates or a constructed route, and what conversion makes the Table 20 plan valid under the declared types?

3 comments

Response (1/2)

Thank you for the detailed review. We redesigned the ablation, improved synthesis coverage and checks, clarified candidate selection, and aligned tool inputs and outputs. The revised manuscript has been uploaded.

Major W1 / Q1. Taxonomy and tool design

Section 3.1 defines capabilities in terms of map operations. Section 4.1 groups leaves with shared input/output signatures into tools. Annotation agreement assesses consistent use of the definitions. We test the tool grouping by comparing alternative inventories with the same capabilities.

Following your suggestion, we redesigned the ablation across six backbones (Section 6.3). Within each backbone, capability coverage, 600 questions, backend and snapshots, planning, selection, sampling, inference budgets, and scoring are fixed. Tool definitions and examples use the corresponding inventory:

  • TI0: Original 12 tools, with retrieval, ranking, and aggregation separate.
  • TI1: 10 tools. Expanded retrieve ranks or aggregates entities from earlier calls, replacing standalone rank and aggregate.
  • TI2: 12 tools. TI1's retrieval schema plus standalone rank and aggregate.

Each pair keeps a specific aspect of tool organization fixed:

  • TI0-TI1: Both avoid duplicate ranking/aggregation interfaces. This compares separate and merged designs, with tool count changing from 12 to 10.
  • TI0-TI2: Both expose the same 12 tool names. This tests expanded retrieval and its resulting overlap at fixed tool count.
  • TI1-TI2: Both use an identical retrieval schema. Adding standalone tools (10->12) tests their effect on accuracy without changing retrieval schema complexity.

Table 6. Answer accuracy (%). Subsets follow reference plans. Rank/Agg. is the disjoint union of ranking and aggregation queries. Parentheses give percentage-point changes from each backbone's TI0.

BackboneTIAll(600)Rank/Agg.(262)Rank(181)Agg.(81)
Qwen 2.5 72BTI084.3381.6881.7781.48
Qwen 2.5 72BTI182.17(-2.17)77.86(-3.82)79.56(-2.21)74.07(-7.41)
Qwen 2.5 72BTI283.00(-1.33)79.01(-2.67)77.90(-3.87)81.48(+0.00)
Qwen 2.5 7BTI041.0046.9553.0433.33
Qwen 2.5 7BTI134.17(-6.83)30.15(-16.79)39.23(-13.81)9.88(-23.46)
Qwen 2.5 7BTI236.33(-4.67)37.40(-9.54)46.96(-6.08)16.05(-17.28)
Ministral 3 14BTI088.8387.0283.4395.06
Ministral 3 14BTI185.83(-3.00)76.72(-10.31)79.01(-4.42)71.60(-23.46)
Ministral 3 14BTI288.17(-0.67)83.59(-3.44)82.32(-1.10)86.42(-8.64)
Ministral 3 8BTI086.0079.7783.9870.37
Ministral 3 8BTI178.00(-8.00)61.07(-18.70)79.01(-4.97)20.99(-49.38)
Ministral 3 8BTI281.67(-4.33)69.47(-10.31)82.32(-1.66)40.74(-29.63)
Gemma 4 31BTI084.0089.6991.1686.42
Gemma 4 31BTI185.50(+1.50)93.13(+3.44)90.06(-1.10)100.00(+13.58)
Gemma 4 31BTI285.17(+1.17)93.13(+3.44)90.06(-1.10)100.00(+13.58)
Gemma 4 E4BTI068.5069.8571.8265.43
Gemma 4 E4BTI160.67(-7.83)51.53(-18.32)62.98(-8.84)25.93(-39.51)
Gemma 4 E4BTI263.00(-5.50)55.73(-14.12)64.64(-7.18)35.80(-29.63)

Overall accuracy follows TI0 > TI2 > TI1 in general. The comparisons favor separate ranking/aggregation tools and retrieval without these additional functions. TI0's benefit is larger for the smaller backbone in every tested family, supporting the taxonomy-derived inventory particularly for smaller planners.

Major W2 / Q2. Selection and repair

We removed the nonempty/nonzero preference and reran all experiments. Successful empty/zero answers no longer trigger revision. An execution error triggers one repair attempt: the LLM receives the error and is told to fix it without changing filters merely to obtain a nonempty answer. The revised plan is accepted only if execution succeeds. Selection and repair are described in Section 5, settings in Table 19, and prompts in supplementary materials (zip) prompts/planner_requests.json.

For the original 41 cases requested in Q2, revision lost one correct answer per model: Qwen 7B fell from 29 to 28 correct and 72B from 37 to 36. Selection also missed an available correct candidate on three tasks per model.

On the revised 60 empty/zero tasks, no repair changed a correct answer to an incorrect one across all ten backbones. Full before/after and selection counts are in supplementary analysis/more_results.md.

Continued in reply 2/2.

Response (2/2)

Major W3 / Q3. Stronger planners and breakdowns

Tables 3–5 add GPT-6.1 Sol, with 99.67%/99.00% accuracy on BY/IL. Supplementary analysis/more_results.md reports all ten backbones by capability, plan topology, number of calls, dependency depth, region, empty/zero versus nonempty answers, and capability pairs, with sample counts and confidence intervals.

The main findings are:

  • Giving each of the 22 leaf subsets equal weight, Qwen 72B scores 82.04% and 7B 36.83%.
  • Qwen 72B exceeds 7B on all 22 leaves and four topologies. Gemma 31B leads E4B on 21 leaves but trails on G2. The Ministral models exchange ranks across leaves.
  • Qwen 7B has 72 invalid-argument and 14 invalid-reference failures. Of its 354 incorrect outcomes, 96 have perfect tool F1. The supplement separately reports tool/dependency differences and execution failures. Appendix G shows an omitted filter.

Major W4 / Q4. Synthesis reliability and coverage

We checked four accepted and one rejected pilot sample per region and agreed with all ten LLM judgments. This was an author check, not a blinded audit.

The revised pipeline adds an LLM reader that describes the task without seeing the reference plan, followed by comparison against that plan (Section 4.2 and Appendix C). Verbalization and review use GPT-6.1 Terra in separate sessions. Replay reproduces all 600 stored answers. Supplementary data/ and synthesis/ contain the dataset and review records.

Sampling now prioritizes leaves underrepresented among accepted questions. Every leaf occurs at least 40 times per region, N4 falls from 403 to 126 queries, and the largest-to-smallest leaf count ratio falls from 67.17 to 3.78 (Tables 14-16). Table 13 specifies sampling targets. The averages in our Q3 response give each leaf equal weight.

Minor W1 / Q5. Table consistency

After rerunning the experiments, Tables 5–6 use the same TI0 predictions. Qwen 7B's regional counts give (122+124)/600 = 41.00%, and 72B's give (256+250)/600 = 84.33%. The numbers agree for all six shared backbones.

Minor W2 / Q5. Tool contracts

optimize_stops previously returned a route despite declaring coordinates. It now returns ordered coordinates. Route length therefore requires optimize_stops -> route -> measure (Appendix B). All 600 reference plans pass the input/output compatibility checks. Appendix G presents a new ablation trace.

Minor W3. Reverse geocoding

Table 12 adds reverse-address: coord -> text via rev_geocode, with the address answer goal in Appendix C.1. Tables 15-16 report 41 BY and 42 IL examples.

Minor W4. G2 and scoring

Table 1 uses documented question types, whereas Table 10 uses LLM labels. The annotation logs shows a representative example that Qwen labels a request for JFK Airport's coordinates as G2 (“reverse geocoding to get coordinates”). But actually it requires to resolve a named entity to coordinates (G3), not converting coordinates to an textual address (G2). Therefore, G2 remains missing in GeoQuestions1089. We updated Appendix A.2.5 accordingly. The example is in supplementary taxonomy/g2_example.json.

Appendix F.3 specifies exact option-identifier matching for our agent. MapAgent's option text is matched after ignoring letter case and trimming outer whitespace. Internal whitespace and numerical units are not normalized.

Thank you again for helping strengthen the study!

Dear Reviewer NZN8,

As the discussion period draws to a close, we would like to briefly follow up on our responses. If you have had a chance to review them, we would appreciate hearing whether they address your concerns or whether any points would benefit from further clarification.

Thank you for your time and consideration.

Best regards, The Authors

Reviewer_siqxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a capability taxonomy for natural-language queries over 2D vector maps. It uses the taxonomy to derive twelve tools and construct a 600-query dataset with executable plans and verified answers. An LLM agent composes these tools to answer map queries. Experiments evaluate the agent on two external benchmarks and the synthesized dataset, and study how overlapping tool interfaces affect planners of different sizes.

Critical strengths
  1. The writing is clear.
  2. The taxonomy provides a useful framework for connecting capability coverage, tool design, and evaluation.
  3. The agent consistently improves over MapAgent across nine backbone configurations on MapEval-API.
Critical weaknesses
  1. The overlap ablation changes two things at once: it adds ranking and aggregation arguments to the retrieval tool while keeping the standalone tools. The performance drop may therefore come from the more complex retrieval interface rather than overlapping tools. This limits the evidence for the claimed benefit of minimizing functional overlap.
  2. The agent prefers nonempty or nonzero results and accepts revisions that turn empty or zero answers into nonempty or nonzero ones. However, 41 synthesized tasks have valid empty or zero answers, and no semantic check is described to ensure that these revisions preserve the query requirements. This creates a correctness risk that is not separately evaluated.
Questions for authors
  1. Does the performance drop remain when the expanded retrieval interface is retained but the standalone ranking and aggregation tools are removed?
  2. On the 41 tasks with valid empty or zero answers, how often does selection or revision change a correct answer into an incorrect one? Is there any check that the revised plan preserves the original query constraints?
4 comments

Thank you for your insightful review. We redesigned the ablation, removed the nonempty preference, reran all experiments, and clarified selection and repair (Sections 5, 6.3). The paper is also updated accordingly and uploaded.

W1 / Q1. Expanded retrieval versus functional overlap

Following your suggestion, we compare three inventories with the same capability coverage across six backbones (Section 6.3). For each backbone, all inventories share: 600 questions, map backend and snapshots, planning and selection, sampling, inference budgets, and scoring. Tool schemas and examples are adapted to each inventory:

  • TI0: the original 12-tool inventory, with retrieval, ranking, and aggregation separate.
  • TI1: a merged 10-tool inventory. retrieve can rank or aggregate entities from earlier tool calls, replacing the standalone rank and aggregate tools.
  • TI2: an overlapping 12-tool inventory. It combines TI1's retrieval interface with the standalone rank and aggregate tools.

Each pair keeps a specific aspect fixed:

  • TI0 vs. TI1: Both provide the same operations without duplicate ranking or aggregation interfaces. This tests merging these operations into retrieve, with tool count changing from 12 to 10.
  • TI0 vs. TI2: Both expose the same 12 tool names, including standalone rank and aggregate. This evaluates the expanded retrieval interface and its resulting overlap at a fixed tool count.
  • TI1 vs. TI2: Both use an identical expanded retrieve argument schema, keeping its schema complexity fixed. TI2 adds standalone rank and aggregate (10→12 tools), testing their effect on answer accuracy.

Table 6. Answer accuracy (%). Rank/Agg. combines disjoint ranking and aggregation subsets identified from reference plans. Parentheses give percentage-point differences from each model's TI0.

BackboneTIAll(600)Rank/Agg.(262)Rank(181)Agg.(81)
Qwen 2.5 72BTI084.3381.6881.7781.48
Qwen 2.5 72BTI182.17(-2.17)77.86(-3.82)79.56(-2.21)74.07(-7.41)
Qwen 2.5 72BTI283.00(-1.33)79.01(-2.67)77.90(-3.87)81.48(+0.00)
Qwen 2.5 7BTI041.0046.9553.0433.33
Qwen 2.5 7BTI134.17(-6.83)30.15(-16.79)39.23(-13.81)9.88(-23.46)
Qwen 2.5 7BTI236.33(-4.67)37.40(-9.54)46.96(-6.08)16.05(-17.28)
Ministral 3 14BTI088.8387.0283.4395.06
Ministral 3 14BTI185.83(-3.00)76.72(-10.31)79.01(-4.42)71.60(-23.46)
Ministral 3 14BTI288.17(-0.67)83.59(-3.44)82.32(-1.10)86.42(-8.64)
Ministral 3 8BTI086.0079.7783.9870.37
Ministral 3 8BTI178.00(-8.00)61.07(-18.70)79.01(-4.97)20.99(-49.38)
Ministral 3 8BTI281.67(-4.33)69.47(-10.31)82.32(-1.66)40.74(-29.63)
Gemma 4 31BTI084.0089.6991.1686.42
Gemma 4 31BTI185.50(+1.50)93.13(+3.44)90.06(-1.10)100.00(+13.58)
Gemma 4 31BTI285.17(+1.17)93.13(+3.44)90.06(-1.10)100.00(+13.58)
Gemma 4 E4BTI068.5069.8571.8265.43
Gemma 4 E4BTI160.67(-7.83)51.53(-18.32)62.98(-8.84)25.93(-39.51)
Gemma 4 E4BTI263.00(-5.50)55.73(-14.12)64.64(-7.18)35.80(-29.63)

The drop persists without standalone tools: overall accuracy mostly follows TI0 > TI2 > TI1, including all three smaller models. TI0's advantage over TI1 favors separate operations despite more tools. At the same tool count, TI0's lead over TI2 favors retrieval focused on entity selection. TI2 recovers part of TI1's loss under the same retrieval schema.

TI0's advantage is larger for the smaller backbone in each tested family, supporting the taxonomy-derived organization particularly for smaller planners.

Appendix G presents an example, where Qwen 7B preserves the garage filter with TI0 but omits it and adds unrequested grouping with TI1/TI2.

W2 / Q2. Valid empty answers and harmful revisions

We removed the preference for nonempty/nonzero answers and reran all experiments.

The original audit covered 41 valid empty/zero tasks with Qwen 7B and 72B. Selection missed a correct available candidate on three tasks per model. Revision caused one harmful replacement per model: 29->28 correct for 7B and 37->36 for 72B, with no beneficial changes. Four 7B cases without a selection event count as incorrect before revision.

Section 5 treats successful empty/zero answers as valid. Repair is triggered only by execution failure and accepted only when it resolves that failure. The prompt requires preserving query constraints.

On the revised 60 empty/zero tasks (Appendix C.4, Table 14), saved traces show no correct-to-incorrect repair changes across all ten backbones. After repair, Qwen 7B improves from 35 to 36 correct, while 72B remains at 52/60. For both Qwen models, voting misses a correct candidate on four tasks. Supplementary material (zip) provides analysis/more_results.md, providing all-backbone counts, with per-task audits in analysis/empty_answer_audit.csv and saved traces in traces/.

Thank you again for your time and effort. We hope all concerns are addressed.

Dear Reviewer siqx,

As the discussion period draws to a close, we would like to briefly follow up on our responses. If you have had a chance to review them, we would appreciate hearing whether they address your concerns or whether any points would benefit from further clarification.

Thank you for your time and consideration.

Best regards, The Authors

Thank you for the additional experiments and revisions. The new ablation and the changes to empty-answer handling largely address my concerns. I will maintain my positive rating.

Thank you for your acknowledgement of our work. We sincerely appreciate your time and effort throughout the discussion.

Comments on the paper

1 comment

We thank the reviewers for their time and feedback. In this comment, we summarize the problem this paper addresses, our contributions, and additional evidence provided during rebuttal.

Map-query agents and their benchmarks can share the same capability gaps. Table 1 identifies concrete omissions in the existing systems and datasets examined. Consequently, strong benchmark performance alone cannot establish coverage of the capabilities that map queries may require.

Our paper addresses this problem through a common analytical model that connects capability specification, tool derivation, and evaluation-data construction. Our contributions are detailed as follows.

  1. A common capability reference and a systematic tool-derivation method. We introduce a faceted taxonomy for queries over 2D vector maps, examine its capability coverage through LLM and human-expert annotation, and use it to identify capability gaps for existing agents and benchmarks (Table 1). Grouping executable capabilities by their I/O signatures yields a twelve-tool inventory (Table 2). This provides an explicit procedure connecting capability requirements to tool interfaces.

  2. A method for generating reproducible evaluation data. The taxonomy guides the construction of typed query plans, which are instantiated against fixed map snapshots, executed to obtain reference answers, and verbalized into natural-language queries. Reference answers are verified by replay, while semantic checks assess the correspondence between the question and plan. We demonstrate the method with 600 queries across two regions and 96 tool-DAG structures. Every taxonomy leaf and tool occurs at least 40 times per region, and replay reproduces all 600 stored answers (Section 4.2, Appendix C).

  3. Effective agents with smaller planners. Our simple plan-and-execute agent outperforms the compared systems even with substantially smaller planners across three LLM families. Our Qwen 2.5 14B configuration exceeds Spatial-Agent’s reported 72B result on its six MapQA types. On MapEval-API under exact-match scoring, our Qwen 2.5 7B configuration achieves 50.00%, versus MapAgent’s 37.33% using 72B. Our agent also leads across all ten tested backbones, demonstrating the practical effectiveness of the resulting design (Section 6.2).

    Accuracy is relatively stable across planner scales on the external benchmarks, while the synthesized dataset shows more pronounced scale-dependent differences. This motivates the controlled ablation of tool organization across planner scales.

  4. Controlled evidence of substantial benefits for smaller planners. The ablation compares the original 12-tool (TI0), merged (TI1), and overlapping (TI2) inventories with the same capabilities, backend, planner, questions, inference settings, and scoring.

    ComparisonHeld fixedChange tested
    TI0 vs. TI1No duplicate ranking/aggregation interfacesMerge ranking/aggregation into retrieval, reducing tools from 12 to 10.
    TI0 vs. TI2Same 12 tool namesExpand retrieval to include ranking/aggregation, testing focused versus expanded retrieval.
    TI1 vs. TI2Identical expanded retrieval schemaAdd dedicated ranking/aggregation tools, increasing tools from 10 to 12.

    We can observe in general TI0 > TI2 > TI1. In every tested family, TI0 benefits the smaller planner more than its larger counterpart. The smaller planners gain 6.83–8.00 percentage points over TI1 and 4.33–5.50 over TI2, showing that our design provides substantial benefits to smaller models (Section 6.3).

  5. Community contributions toward trustworthy evaluation. GPT-6.1 Sol’s traces reveal location-resolution failures on 362 of 948 MapQA queries in our evaluated setup, showing why grounding and backend support must be examined when interpreting planning performance. We also demonstrate that MapAgent’s released evaluator credits incorrect answers, including opposite directions. Our agent leads under both exact-match scoring and the original evaluator, with the latter comparison covering nine open-weight backbones.

    These findings, together with documented baseline reproduction limitations, give subsequent studies concrete checks for answerability, scoring correctness, and reproducibility, helping the community avoid misleading comparisons (Section 6.2, Appendix F).


During rebuttal, we redesigned the ablation, strengthened synthesis coverage and semantic checks, removed the nonempty/nonzero preference, added GPT-6.1 Sol, and reran all experiments. The paper is revised accordingly. The supplement provides the dataset, prompts, selected traces, and results by capability and plan structure in analysis/more_results.md.

We appreciate the reviewers' suggestions, which have strengthened the study. As discussion approaches its close, we would welcome feedback on the revised manuscript, particularly any remaining concerns or questions we can address.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.