acceptodds
Under review as a conference paper at ICLR 2027

GeoToolAct: Evaluating Tool Execution in Remote Sensing Agents

Abstract

Remote sensing agents rely on multi-step tool calls to perform complex analytical tasks, yet reliable execution remains challenging. To analyze execution failures, we conceptually divide the model’s tool-call generation process into two stages: Tool Plan, which selects tools and their invocation order, and Tool Act, which constructs arguments after target-tool selection and before execution. Using this distinction, our trajectory analysis shows that Tool Act errors are less frequent than Tool Plan errors but more strongly associated with end-to-end task failure. This finding motivates examining argument construction alongside tool planning. However, existing remote sensing and geospatial agent benchmarks report argument accuracy but offer limited support for comparing performance across argument-construction requirements. To enable this comparison, we analyze argument-construction errors and identify three capability dimensions: adherence to the current tool schema (Schema-Prior), locating and faithfully reproducing required historical values (Contextual Value), and determining currently valid values after resource updates (State-Update). We build GeoToolAct around these dimensions by reconstructing observed failures into diagnostic tasks with fixed target tools and controlled non-target requirements. The benchmark covers 144 core tasks and 44 target tools, with four execution-history variants per task up to 120K tokens, preserving core tasks and reference answers. Evaluating 19 models on GeoToolAct, we find that the highest Overall Direct Accuracy is only 78.53% despite explicit target-tool specification, indicating substantial room to improve argument-construction reliability. Models also exhibit substantially different performance profiles, and sensitivity to longer execution histories varies across models and capabilities. Retries with evaluator-generated argument feedback yield uneven gains and incur additional generation costs. These results highlight the need for dedicated evaluation and targeted improvement of argument-construction capabilities to support reliable tool use in remote sensing.

Primary area:
datasets and benchmarks
Keywords:
Remote Sensing, Large Language Models, Tool-Using Agents, Tool Calling, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CFUiOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper separates tool-call errors of remote sensing agents into Tool Plan (choosing the tool) and Tool Act (filling in the arguments once the tool is fixed), and builds a diagnostic benchmark for the second. Failures collected from runs on ThinkGeo, Earth-Bench and GeoAgentBench are rewritten into 144 tasks with a fixed target tool, grouped into three requirement types (Schema-Prior, Contextual Value, State-Update), and each task is padded with unrelated call history up to 120K tokens. Nineteen models are tested with three runs each. The best overall accuracy is 78.53%, models differ a lot across the three types, longer history hurts in a model-dependent way, and retrying with evaluator feedback helps unevenly.

Critical strengths
  1. Isolating argument construction from tool selection is a sensible way to localise failures that existing remote sensing agent benchmarks only report inside end-to-end results. The design is checked instead of assumed. The paired experiment in Appendix B shows that supplying the correct tool leaves argument accuracy unchanged (92.0% vs 91.8%), and each task is evaluated at four history lengths so the length effect is measured within the same task.

  2. The results tell us something concrete about where current models fail. Tracking the currently valid resource is the weakest point (37.6% on average over the general models, against 64.6% for schema adherence), strong models have different profiles, the effect of longer history is not monotonic, and retrying with feedback helps schema errors much more than state errors. The evaluation behind this is careful, with 19 recent models, three runs each and a two-annotator audit of 120 samples.

Critical weaknesses
  1. The benchmark is narrow. Table 7 shows that 64 of the 144 tasks fall in a category with a single tool name, and Appendix F states that all 60 Contextual Value instances use the same ten-input contract of build_crop_stress_assessment. The Contextual Value score is therefore a score on one tool, and State-Update has only 30 tasks. The authors acknowledge that the three task families differ in tools and templates, but this limits how far the per-type profiles in Section 5.2 can be generalised.

  2. The link to real agent performance is not established. The motivating analysis uses one model on one benchmark (Qwen3.5-27B on the 248 Earth-Bench tasks) with LLM-judged labels, and Appendix H itself notes that the difference between the two risk ratios is not tested. The tasks are mined from failures and then rebuilt under controlled conditions, which is fine for a diagnostic benchmark, but there is no experiment showing that a GeoToolAct score predicts end-to-end success on the source benchmarks. The three requirement types have also been studied for general function calling (ComplexFuncBench, LongFuncEval, HammerBench, none cited), and since the tasks are text only it is not clear what is specific to remote sensing beyond the tool names.

Minor points:

  • Names are not consistent. The three types are Schema-Prior / Contextual Value / State-Update in the text, Schema / Context / State in Table 2 and Schema / Value / State in Fig. 3(b). Fig. 1(a1) plots GeoBenchX, while Section 4.1 lists GeoAgentBench as the third source.
  • Sections 5.3 and 5.4 rely on the Fig. 3 heatmap only. Please add a table with the numbers.
  • "ReAct@4" is a single call retried with evaluator feedback, which is not a ReAct loop. The name is misleading.
Questions for authors
  1. How do the results change if build_crop_stress_assessment is excluded, or if Contextual Value is reported per tool? Can the Contextual Value family be extended to more tools?
  2. Do GeoToolAct scores correlate with end-to-end success of the same models on the source benchmarks? A rank correlation over a few models would already help.
  3. The tasks are mined from failures of Qwen3.5-27B and DeepSeek-v4-flash. Does this bias the comparison against these two model families?
  4. Was the LLM judge checked against human labels on any subset?
  5. Appendix B needed manual review to accept 76 of 98 unmatched outputs. Was a sample of the calls marked wrong in the main evaluation checked for false negatives?

As a suggestion that does not affect my rating, the four remote sensing models in Table 2 are small vision-language models used far outside their intended setting, and their near-zero scores tell us little.

3 comments
AuthorsOpenReview ↗

Response to Reviewer CFUi (1/2)

Thank you for the thoughtful review, and for noting that our design is checked rather than assumed and that the results show concrete failure points across models. Several of the new analyses below are shared with the other reviewers; we summarize them here and point to the full tables.

Q1 / W1 (Contextual Value beyond one tool; scale). We built 80 new Contextual Value tasks on four additional interfaces (NDVI composition, thresholding, trend analysis, spatial statistics; 20 each) with the same controls and 80K expansion, and ran four Table 2 models three times (full table in our reply to dkn5, Q1). Mean accuracy on the new tasks is 81.3 (DeepSeek-v4-flash), 61.3 (Kimi-K2.7-Code), 42.5 (GLM-5.2) and 19.6 (Kimi-K2.6), vs. 85.6 / 64.4 / 34.4 / 49.4 on build_crop_stress_assessment at 80K. Absolute levels vary by interface, but the model ordering is identical on all four new interfaces, and Contextual Value keeps its place in the profiles of DeepSeek-v4-flash, Kimi-K2.7-Code (strongest family) and GLM-5.2 (weakest). Excluding build_crop_stress_assessment altogether and recomputing Overall from the Schema-Prior and State-Update columns of Table 2 leaves the ranking of the 15 general models largely unchanged (Spearman ρ = 0.95); Gemini-3-Flash stays first, and the largest moves are GPT-5 (5th→tied 2nd with GLM-5.2, from its strong State-Update) and DeepSeek-v4-flash (2nd→4th, whose strength is Contextual Value). For the small State-Update family, redrawing the distractor histories of 30 instances per family three times leaves rankings and profiles unchanged (reply to VMcA, Q4). We will revise Sec. 5.2 to state that orderings and profiles, rather than absolute Contextual Value levels, are what we expect to generalize, and will extend Contextual Value to more interfaces and add target tools to Schema-Prior and State-Update in the released benchmark.

Q2 / W2 (link to end-to-end success). We ran four Table 2 models end-to-end under each benchmark's official protocol:

ModelGeoToolActEarth-Bench Acc.GeoBenchX Match
DeepSeek-v4-flash77.1470.9750.50
GLM-5.267.0769.3550.99
Kimi-K2.7-Code58.9175.0049.50
Kimi-K2.654.1761.6946.53

Of the six model pairs, GeoToolAct orders 5 consistently on GeoBenchX and 4 on Earth-Bench. Full agreement is not expected, since E2E success also depends on Tool Plan and execution, which GeoToolAct holds fixed; GeoToolAct localizes argument-construction failures and complements rather than replaces E2E evaluation.

Q3 (mining from two model families). Mining determines which decision points enter the pool, not the tasks models are evaluated on: each failure point is rebuilt with a fixed target tool, controlled non-target requirements and new histories (Sec. 4.3–4.4), so no model is scored on reproducing a mining model's mistake. Table 2 shows no penalty: DeepSeek-v4-flash ranks second overall (77.14) and first on Contextual Value, and Qwen3.5-Plus ranks third (69.16), above GLM-5.2 and GPT-5. In App. I, the failures of all 19 models fall into the error forms derived from the mining pool, with only 11 Schema-Prior calls in a residual category. We used two mining models because each must run full trajectories on all three sources with every step judged, and no finite set of mining models covers every family; the pipeline is model-agnostic and we plan to extend mining to further families (details in our reply to dkn5, Q2).

Q4 / W2 (LLM judge vs. human labels). Two researchers who work on GIS agents but were not involved in this project independently re-annotated the Earth-Bench trajectories under the protocol of App. A. Tool Act prevalence / RR is 41.6% / 1.76 [1.27, 2.45] and 40.8% / 1.74 [1.26, 2.42] for the two annotators, vs. 44.0% / 1.85 for the LLM judge; Tool Plan RR stays non-significant (1.09 and 1.10, CIs include 1). Trajectory-level κ is 0.88 (Act) and 0.89 (Plan). We will add this to App. H and, since the two RRs were not tested against each other, replace "more strongly associated" in the abstract and Fig. 1 (full table in our reply to VMcA, W1). We agree that the motivating analysis covers one model on one benchmark; we will present it as a motivating observation, and the cross-model link to end-to-end success is addressed by Q2.

AuthorsOpenReview ↗

Response to Reviewer CFUi (2/2)

Q5 (false negatives in the main evaluation). The review rate in App. B reflects its setting rather than the scorer: App. B scores natural decision points against the reference calls published by the source benchmarks, and the 76 accepted outputs are mainly free-text arguments paraphrased with the same meaning, optional arguments the task leaves unconstrained, and cases where the published reference conflicts with the task (e.g., the task asks for LST above 300 K while the reference uses 310). GeoToolAct removes these sources of ambiguity by construction: arguments are structured values traceable to the schema, task or history, otherwise-free choices such as output paths are fixed in the task, and gold calls pass deterministic verification and a two-annotator audit (Sec. 4.5; App. D). The scorer compares parsed calls, fills declared defaults and accepts equivalent representations (App. G).

To check directly, two annotators independently reviewed 90 randomly sampled first-attempt calls marked wrong (30 per family, stratified by length, fixed seed), with model names hidden. A call counted as a false negative if, passed to the tool as written, it would produce the same result as the gold call. Neither annotator judged any of the 90 a false negative (0/90; 95% Wilson upper bound 4.1%, 11.4% per family), and no gold defect was found. The rejected calls had wrong values, stale or wrong resources, missing or extra arguments, or argument types that violate the declared schema. Empty or truncated outputs, which contain no complete call, were excluded. We will add the audit and its per-call table to the appendix.

W2 (related work and remote-sensing content). We will cite and discuss ComplexFuncBench, LongFuncEval and HammerBench. We do not claim tool-conditioned argument accuracy as new; our contribution is the requirement-controlled construction that turns it into diagnostic profiles, with the same gold call across four history lengths. Most target tools are remote-sensing operators whose arguments carry physical roles (e.g., NIR/Red and NIR/SWIR band lists, a representative thermal-band index, albedo vs. day/night temperature inputs for thermal inertia); a value in the wrong role keeps the call type-valid but computes a different quantity. The benchmark is text-based by design: imagery is processed by the tools and referenced through paths and outputs (details in our reply to VMcA, Q2).

Presentation. We will revise the following points.

  • Names. Schema-Prior, Contextual Value and State-Update will be used consistently across text, tables and figures.
  • Fig. 1(a1) vs. Sec. 4.1. The two use different benchmarks because they need different properties. Fig. 1(a1) only needs a task-level E2E outcome and a tool-call count per task, spread over enough call counts to fit a confidence band, which GeoBenchX provides. Construction (Sec. 4.3) instead needs a verified correct trajectory for each failure point, i.e., step-level references: ThinkGeo and Earth-Bench publish step-level reference calls, whereas GeoBenchX does not, and some of its tasks end in a rejection tool, leaving no final result to confirm the preceding calls. We will clarify this in the caption of Fig. 1.
  • Fig. 3. We will add the numbers behind the heatmap as an appendix table.
  • ReAct@4. We will rename it Feedback@4 and describe it as a single call retried with evaluator feedback for up to four attempts.
  • Remote-sensing specialized models. We will move them to the appendix and state why we include them: they are the domain-specialized models a practitioner would naturally consider as an agent backbone, trained mainly for perception-style tasks (VQA, grounding, captioning). Their near-zero accuracy (at most 5.86% Overall) shows that current remote-sensing specialization does not by itself provide the argument construction tool execution requires. Since they are also small, we present this as an observation about current specialized models, not a controlled comparison.
  • Tool counts. We will consistently report 44 tool interfaces (43 tools).
Reviewer_CFUiOpenReview ↗

Response

Thanks for the detailed response and the additional experiments. The false-negative audit, the human re-annotation of the Earth-Bench analysis and the explanation of how the mined failure points are rebuilt answer my Q3–Q5, and the presentation changes are fine.

I still have two concerns. On Contextual Value, the new interfaces help, but the numbers in the response also suggest that the original single-tool score is not reliable on its own. Kimi-K2.6 is ahead of GLM-5.2 on build_crop_stress_assessment (49.4 vs 34.4) but well behind it on the four new interfaces (19.6 vs 42.5). I think the 80 new tasks should go into Table 2 and the per-type profiles in Section 5.2, rather than staying in the rebuttal.

On the link to end-to-end success, four models is a small sample. Kimi-K2.7-Code is third on GeoToolAct but first on Earth-Bench, and on GeoBenchX the top three models are within 1.5 points, so the pairwise agreement there does not say much. I agree with the response that GeoToolAct is best read as a diagnostic for argument construction rather than a predictor of end-to-end success, and the paper should be written that way. I will keep my score at 3.

Reviewer_VMcAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces GeoToolAct, a benchmark of how well remote-sensing agents construct tool arguments once the correct tool is given ("Tool Act"), evaluated separately from tool selection ("Tool Plan"). The motivation is an Earth-Bench analysis in which Tool Act errors are rarer than tool-selection errors but more strongly associated with task failure. The authors define three capabilities: adhering to the tool's schema, recovering values from the execution history, and tracking updates to resource states. They build 144 diagnostic instances from the failures of two models on three existing benchmarks and pad each one with unrelated history to four context lengths, for 576 samples in total. Across 19 models, the best overall accuracy is 78.53%, and models show distinct capability profiles. Sensitivity to longer histories also varies across models, and feedback-based retries help unevenly.

Critical strengths

S1. The split of argument construction into three requirements (schema adherence, value binding from history, state tracking) is clear and maps to recognizable failure forms. The per-model capability profiles are informative, e.g., GPT-5 is strong on State-Update but weaker on Contextual Value.

S2. Benchmark construction and reporting are careful. All samples pass deterministic verification, a human audit uses two annotators plus adjudication, run-to-run variance is reported, and the statistics are presented transparently.

Critical weaknesses

W1. The motivation and the claimed gap are weak given prior work.

  • Evaluating argument correctness conditional on the correct tool is not new for remote-sensing agents. CangLing-KnowFlow[1] adopts the step-wise protocol of GTA and reports ArgAcc, defined as argument correctness given that the correct tool is selected, together with per-LLM parameter accuracy. This is essentially the Tool Act setting, yet neither work is cited.
  • The stated gap, that existing argument metrics are embedded in end-to-end execution and cannot isolate argument construction, therefore does not hold. The paper's own App. B points the same way: argument accuracy under oracle tool fixation (91.8%) matches argument accuracy conditional on a correct free selection (92.0%). The existing conditional metric thus already measures the same quantity, and the remaining novelty is mainly the split into three error types.
  • The motivating evidence is also weak. It relies on one model, one benchmark and LLM-judge labels that were not validated against humans. The confidence intervals of the two risk ratios overlap ([1.37, 2.51] vs. [0.90, 1.65]).

W2. The benchmark contains no remote-sensing-specific content, and its difficulty comes mainly from a small, padded set of instances.

  • The three capabilities are domain-agnostic, and the tasks are text-only JSON argument filling with no imagery.
  • Argument requirements that are specific to remote sensing are not modeled: coordinate reference systems and projections, sensor band order, spatial resolution and resampling, temporal windows and cloud filtering, NoData values, and raster/vector conventions. "Remote sensing" appears only in tool names and file paths, so the benchmark is hard to distinguish from general function-calling benchmarks such as BFCL or ToolSandbox.
  • The benchmark is also small and concentrated. All 60 Contextual Value instances use the same ten-input contract of a single tool, and State-Update has only 30 instances.
  • At Base length, nearly all general models are close to ceiling on Contextual Value (Fig. 3a). The headline 78.53% therefore depends largely on the padding with unrelated history.

Reference:

[1] Z. Chen, H. Wang, J. Yao, J. Zhang, P. Ghamisi, J. Zhou, P. M. Atkinson, B. Zhang. CangLing-KnowFlow: A Unified Knowledge-and-Flow-fused Agent for Comprehensive Remote Sensing Applications. arXiv:2512.15231, 2025.

Questions for authors
  • How does Tool Act differ from the tool-conditioned argument accuracy (ArgAcc) used in GTA and CangLing-KnowFlow? What does GeoToolAct diagnose that those metrics cannot?
  • Which parts of the benchmark are specific to remote sensing? Could instances be added that test remote-sensing-specific requirements, such as CRS consistency, band order or NoData handling?
  • Do GeoToolAct scores correlate with the same models' end-to-end success on the source benchmarks?
  • Are the 40K/80K/120K histories nested or sampled independently? How much do the results vary when the distractors are resampled?
2 comments
AuthorsOpenReview ↗

Response to Reviewer VMcA (1/2)

We thank the reviewer for the careful review, for recognizing our construction and reporting, and for the concrete questions, which prompted the new analyses below.

Q1 (vs. GTA and CangLing-KnowFlow). We will cite and discuss GTA and CangLing-KnowFlow, together with T-Eval, API-Bank, BFCL, ComplexFuncBench, LongFuncEval and HammerBench, and revise our intro sentence on existing RS benchmarks. We agree that tool-conditioned argument accuracy is established and do not claim it, or providing the target tool, as a contribution. Conditional ArgAcc reports whether an argument is wrong, not which requirement failed. GeoToolAct turns recurring RS failure patterns (schema violations, wrong values taken from earlier outputs, outdated versions of updated artifacts) into controlled instances in which one requirement dominates (Table 1), with length variants that share the same gold call. App. B is consistent with this: the quantity is the same (91.8 vs. 92.0), so fixing the tool does not inflate it, but on natural steps it is a near-ceiling aggregate. Under controlled construction it yields profiles that differ across models: GPT-5 is strong on State-Update but weaker on Contextual Value, while GLM-4.7-Flash scores 99% on Contextual Value but 3% on State-Update at Base. We read these as diagnostic profiles, not independent mechanisms (Sec. 3.2).

Q2 (RS content). Most target tools are RS operators whose arguments carry physical roles. The 54 Schema-Prior instances cover 33 tools, including spectral indices (NDVI, NDWI, NDSI, NDBI, NDTI), fire radiative power, single-channel LST, multi-frequency brightness temperature, temperature–emissivity separation, apparent thermal inertia and bi-temporal change detection. Arguments include separate NIR/Red and NIR/SWIR lists, a representative thermal-band index, joint band/threshold/direction conditions, and albedo vs. day/night temperature inputs; a value in the wrong role stays type-valid but computes a different quantity. Contextual Value and State-Update bind RS products (thermal anomaly, evapotranspiration, crop mask, LST, NBR) and their updated versions. CRS consistency, band order and NoData handling are not included because none of the failure points collected by our pipeline (Sec. 4.1–4.2) involved these requirements. They can be added with the same construction: in our reply to Reviewer dkn5 (Q1), we build 80 Contextual Value tasks on four new tool interfaces under the same controls, and we plan to add such requirements in the same way. The benchmark is text-based by design: imagery is processed by the tools and referenced through paths and outputs. Unlike BFCL or ToolSandbox, each instance keeps the same task and gold call across four history lengths and isolates one requirement, so GeoToolAct shows which requirement fails and how failures grow with history, which an aggregate function-calling score does not.

Q3 (E2E success). We ran four Table 2 models end-to-end under each benchmark's official protocol:

ModelGeoToolActEarth-Bench Acc.GeoBenchX Match
DeepSeek-v4-flash77.1470.9750.50
GLM-5.267.0769.3550.99
Kimi-K2.7-Code58.9175.0049.50
Kimi-K2.654.1761.6946.53

Of the six model pairs, GeoToolAct orders 5 consistently on GeoBenchX and 4 on Earth-Bench. Full agreement is not expected, since E2E success also depends on Tool Plan and execution, which GeoToolAct holds fixed. GeoToolAct localizes argument-construction failures; it complements rather than replaces E2E evaluation.

AuthorsOpenReview ↗

Response to Reviewer VMcA (2/2)

Q4 (nesting and resampling). Thank you; this variance was not quantified originally. Schema-Prior histories are nested (40K ⊂ 80K ⊂ 120K); Contextual Value and State-Update histories are sampled separately per length and overlap partially. In all three families, the task, target tool, target evidence and gold call are identical across lengths; only the added history differs. We took 30 instances per family and redrew their distractor histories three times from the benchmark's pool under the same rules (mean ± std over resamples, one run each, averaged over 40K/80K/120K):

ModelSchema-PriorContextual ValueState-Update
DeepSeek-v4-flash72.2 ± 4.485.2 ± 2.856.3 ± 3.9
GLM-5.283.3 ± 2.242.6 ± 3.945.2 ± 5.0
Kimi-K2.7-Code53.3 ± 5.657.0 ± 3.930.7 ± 8.3
Kimi-K2.645.2 ± 2.843.7 ± 0.620.0 ± 10.0

In every resample, the within-family ordering matches Table 2, except Kimi-K2.6 vs. GLM-5.2 on Contextual Value (within variation), and each model's profile is unchanged. Absolute State-Update accuracy of the two weakest models is sample-sensitive (up to ±10), so rankings and profiles are the diagnostic output. The released benchmark will include the resampled instances. Our reply to dkn5 (Q3) further separates length, distractor content and evidence position at a fixed 80K.

W1 (motivating analysis). Two researchers who work on GIS agents but were not involved in this project independently re-annotated the Earth-Bench trajectories under the protocol of App. A, excluding runs with no labelable decision due to infrastructure errors (empty agent input, API rate limiting).

Prevalence / RR [95% CI]Annotator 1Annotator 2Paper (LLM judge)
Tool Act41.6% / 1.76 [1.27, 2.45]40.8% / 1.74 [1.26, 2.42]44.0% / 1.85 [1.37, 2.51]
Tool Plan53.6% / 1.09 [0.78, 1.51]53.2% / 1.10 [0.79, 1.54]54.4% / 1.22 [0.90, 1.65]

Trajectory-level κ is 0.88 (Act) and 0.89 (Plan). Both annotators reproduce Table 9: Tool Act errors are significantly associated with failure, Tool Plan errors are not. Counting the excluded runs as error-free gives Act RR 1.41 (lower bound ≥ 1.05). We will add this to App. H and, since the two RRs were not tested against each other, replace "more strongly associated" in the abstract and Fig. 1. The analysis covers one agent on one benchmark and only motivates evaluating Tool Act; GeoToolAct does not depend on it.

W2 (history and scale). "Unrelated" was unclear: we meant answer-preserving history (same intent, target tool, evidence and gold call, and no new valid answer; Sec. 4.4, App. D) and will rename it. It is not neutral filler, since the target evidence must be found among it. The decline with history is part of what we measure: high Base accuracy shows the items are solvable, and longer variants test whether this holds as history grows. We will report each length separately rather than only the 78.53% aggregate. On scale, we agree with the two points raised: all 60 Contextual Value instances share one interface, and State-Update has only 30 instances. For the first, we built 80 Contextual Value tasks on four new interfaces (reply to dkn5, Q1); absolute levels vary, but the model ordering is identical on all four. For the second, Q4 shows that rankings and profiles are stable when histories are redrawn, and we plan to add target tools to Schema-Prior and State-Update.

Reviewer_dkn5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GeoToolAct, a diagnostic benchmark for evaluating argument construction in remote sensing agents. It distinguishes Tool Plan from Tool Act. Constructed from execution failures of Qwen3.5-27B and DeepSeek-v4-flash, the benchmark contains 144 core tasks covering 44 target tools. Experiments on 19 models reveal substantial argument-construction errors, varying long-context robustness, and uneven gains from feedback-based correction.

Critical strengths

The research question is well defined and practically relevant. The paper focuses on reliable argument construction in remote sensing and geospatial agents, addressing execution errors that can occur even when the correct tool has been selected. The paper separates Tool Plan from Tool Act and evaluates three argument-construction dimensions. This reveals model weaknesses that end-to-end success rates alone may miss. The benchmark incorporates substantive and transparent quality control, combining automated verification of all 576 samples with an independent human audit of 120 samples. The authors distinguish first-pass from post-correction results and disclose the correction of two State-Update samples, strengthening confidence in benchmark quality.

Critical weaknesses

Insufficient validation of the capability decomposition. State-Update also requires contextual retrieval, making it difficult to distinguish retrieval failures from state-tracking failures. Differences in tools, argument structures, and information presentation further complicate cross-dimension comparisons. Limited diversity of failure sources and templates. Error mining uses only two models across three benchmarks, potentially favoring their particular failure patterns. Moreover, all 60 Contextual Value core instances share one tool interface. Inserting unrelated execution records increases context length while also introducing distractors and shifting evidence positions. The results therefore do not isolate the effect of length itself. Separate controls for length, distractor content, and evidence position are needed.

Questions for authors

All 60 Contextual Value core instances use the same tool interface. This seems narrow for drawing conclusions about contextual-value binding more broadly. Results on a few additional interfaces would help show whether the findings depend on this particular schema. The benchmark is built from failures of only two models. I would like to see some discussion of which errors this selection may miss, and whether models from other families expose failure patterns that the benchmark does not cover. Adding unrelated execution records changes both context length and the amount of distracting information, while also moving the relevant evidence. The current results show sensitivity to these combined changes, but do not establish which factor matters most. Further analysis separating these effects would help clarify the findings.

2 comments
AuthorsOpenReview ↗

Response to Reviewer dkn5 (1/2)

We appreciate the reviewer finding the research question practically relevant and our quality control transparent. We ran all three suggested analyses; the results are below.

Q1 / W2 (Contextual Value beyond one interface). To test whether the Contextual Value results depend on build_crop_stress_assessment, we built 80 new Contextual Value tasks on four additional interfaces (NDVI composition, thresholding, trend analysis, spatial statistics; 20 each), following the Contextual Value controls (Table 1) and the 80K expansion (Sec. 4.4). Four Table 2 models, paper decoding settings (16,384 output tokens), three runs:

ModelValue 80K (Table 2)New tasksNDVIThresh.TrendSpatial
DeepSeek-v4-flash85.681.3 ± 1.358.373.396.796.7
Kimi-K2.7-Code64.461.3 ± 5.448.341.781.773.3
GLM-5.234.442.5 ± 8.740.035.053.341.7
Kimi-K2.649.419.6 ± 2.98.315.026.728.3

Absolute levels vary by interface, but the model ordering is identical on all four, and the top two models match Table 2. Contextual Value also keeps its place in the profiles of DeepSeek-v4-flash and Kimi-K2.7-Code (their strongest family) and GLM-5.2 (its weakest). We will add these results to the appendix, revise Sec. 5.2 to state that orderings and profiles, rather than the absolute Contextual Value level, are what we expect to generalize, and extend Contextual Value to more interfaces in the released benchmark.

Q2 / W2 (two mining models). Mining determines which decision points enter the pool, not the tasks models are evaluated on: each failure point is rebuilt with a fixed target tool, controlled non-target requirements and new histories (Sec. 4.3–4.4), so no model is scored on reproducing a mining model's mistake. Table 2 shows no penalty for the mining families: DeepSeek-v4-flash ranks second overall (77.14) and first on Contextual Value (84.44), and the Qwen3.5 models range from 69.16 (Qwen3.5-Plus, third overall, above GLM-5.2 and GPT-5) to 34.95 (Qwen3.5-9B). The failure patterns mined from these two models already cover the requirement types we observe across models: in App. I, the failures of all 19 models, most of them from families never used for mining, fall into the error forms derived from the mining pool (schema violations, wrong historical values, outdated or mixed versions), with only 11 Schema-Prior calls in a residual category of other argument-value errors. We mined with two models because each mining model must run full agent trajectories on all three source benchmarks with every step judged, which is costly, and because no finite set of mining models covers every family. With any choice, decision points that all mining models handle correctly do not enter the pool, so failures specific to other families at those points are not measured. The pipeline itself is model-agnostic, and we plan to extend mining to further model families.

AuthorsOpenReview ↗

Response to Reviewer dkn5 (2/2)

Q3 / W3 (length vs. distractor content vs. evidence position). From the paper's 80K samples we built three controls, each isolating one factor while keeping the target evidence, final prompt and total token count fixed (within 1%; 5% for general-domain):

  • Neutral (length alone): every distractor block is replaced in place by token-matched neutral file-listing records, so length and evidence positions are unchanged but no tool-call content competes with the target.
  • General-domain (remote-sensing content): distractors are replaced by real general-domain tool-use episodes (ToolACE), keeping realistic tool-call records without remote-sensing content. This applies only to Schema-Prior and State-Update: Contextual Value tasks locate evidence by its position among historical tool calls ("the 7th historical tool call"), and ToolACE episodes interleave natural-language assistant turns, which changes what is counted rather than the content.
  • Gathered (evidence position): all original distractors are kept, but the target calls are moved into one contiguous block. This applies only to Contextual Value: Schema-Prior values are given in the task, so there is no historical evidence to move, and in State-Update the order of the update chain is the requirement itself.
ModelSchema 80K / Neutral / GeneralValue 80K / Neutral / GatheredState 80K / Neutral / General
DeepSeek-v4-flash67.9 / 75.3 / 70.485.6 / 99.4 / 98.364.4 / 100.0 / 42.2
Kimi-K2.7-Code50.6 / 86.4 / 77.564.4 / 99.4 / 100.027.8 / 96.7 / 46.7
Kimi-K2.643.2 / 78.4 / 74.749.4 / 98.9 / 98.926.7 / 82.2 / 5.6
GLM-5.275.9 / 87.7 / 87.734.4 / 71.1 / 82.852.2 / 74.4 / 24.4

Neutral filler raises accuracy in every cell, so length alone explains little of the 80K drop. General-domain history lowers State-Update for all four models (5.6–46.7 vs. 74.4–100.0 with neutral filler), so realistic tool-call records, not remote-sensing content specifically, drive the difficulty. For Schema-Prior, general-domain history stays within 9 points of neutral filler, as expected: its values are given in the task, so none has to be found in the history. Gathering the Contextual Value targets restores accuracy while keeping every distractor, so dispersion of the evidence matters more than the presence of distractors. We will add these controls and revise Sec. 5.3: the length variants measure robustness to a growing realistic execution history, not to length itself.

W1 (State-Update vs. retrieval). On the 30 State-Update cores at 80K, we added a random call identifier to every tool result and rewrote only the final instruction to name the producing call for each input ("the path written as new_path in the call whose result has call_id ca938"). This removes version determination and leaves retrieval; gold answers and the output budget are unchanged. Three runs:

ModelState 80K (Table 2)Producing call specified
DeepSeek-v4-flash64.477.8 ± 3.1
Kimi-K2.7-Code27.876.7 ± 2.7
Kimi-K2.626.720.0 ± 9.4
GLM-5.252.237.8 ± 11.0

Accuracy rises for DeepSeek-v4-flash (+13.4) and sharply for Kimi-K2.7-Code (+48.9), so a large part of their State-Update errors are errors in determining the currently valid version. For Kimi-K2.6 and GLM-5.2, most remaining errors are earlier versions from the same rename chain or reconstructed paths absent from the history, i.e., failures to locate and copy one path among near-identical versions; these are also the two weaker of the four on Contextual Value (Q1 table). The families therefore overlap for some models, as App. I notes, and we do not claim independence. Since the families also differ in tools and templates, absolute scores are not calibrated across families; we read profiles as relative patterns of which requirement dominates a model's errors. We will add this control to the appendix and state this reading in Sec. 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.