- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper separates tool-call errors of remote sensing agents into Tool Plan (choosing the tool) and Tool Act (filling in the arguments once the tool is fixed), and builds a diagnostic benchmark for the second. Failures collected from runs on ThinkGeo, Earth-Bench and GeoAgentBench are rewritten into 144 tasks with a fixed target tool, grouped into three requirement types (Schema-Prior, Contextual Value, State-Update), and each task is padded with unrelated call history up to 120K tokens. Nineteen models are tested with three runs each. The best overall accuracy is 78.53%, models differ a lot across the three types, longer history hurts in a model-dependent way, and retrying with evaluator feedback helps unevenly.
Critical strengths
-
Isolating argument construction from tool selection is a sensible way to localise failures that existing remote sensing agent benchmarks only report inside end-to-end results. The design is checked instead of assumed. The paired experiment in Appendix B shows that supplying the correct tool leaves argument accuracy unchanged (92.0% vs 91.8%), and each task is evaluated at four history lengths so the length effect is measured within the same task.
-
The results tell us something concrete about where current models fail. Tracking the currently valid resource is the weakest point (37.6% on average over the general models, against 64.6% for schema adherence), strong models have different profiles, the effect of longer history is not monotonic, and retrying with feedback helps schema errors much more than state errors. The evaluation behind this is careful, with 19 recent models, three runs each and a two-annotator audit of 120 samples.
Critical weaknesses
-
The benchmark is narrow. Table 7 shows that 64 of the 144 tasks fall in a category with a single tool name, and Appendix F states that all 60 Contextual Value instances use the same ten-input contract of
build_crop_stress_assessment. The Contextual Value score is therefore a score on one tool, and State-Update has only 30 tasks. The authors acknowledge that the three task families differ in tools and templates, but this limits how far the per-type profiles in Section 5.2 can be generalised. -
The link to real agent performance is not established. The motivating analysis uses one model on one benchmark (Qwen3.5-27B on the 248 Earth-Bench tasks) with LLM-judged labels, and Appendix H itself notes that the difference between the two risk ratios is not tested. The tasks are mined from failures and then rebuilt under controlled conditions, which is fine for a diagnostic benchmark, but there is no experiment showing that a GeoToolAct score predicts end-to-end success on the source benchmarks. The three requirement types have also been studied for general function calling (ComplexFuncBench, LongFuncEval, HammerBench, none cited), and since the tasks are text only it is not clear what is specific to remote sensing beyond the tool names.
Minor points:
- Names are not consistent. The three types are Schema-Prior / Contextual Value / State-Update in the text, Schema / Context / State in Table 2 and Schema / Value / State in Fig. 3(b). Fig. 1(a1) plots GeoBenchX, while Section 4.1 lists GeoAgentBench as the third source.
- Sections 5.3 and 5.4 rely on the Fig. 3 heatmap only. Please add a table with the numbers.
- "ReAct@4" is a single call retried with evaluator feedback, which is not a ReAct loop. The name is misleading.
Questions for authors
- How do the results change if
build_crop_stress_assessmentis excluded, or if Contextual Value is reported per tool? Can the Contextual Value family be extended to more tools? - Do GeoToolAct scores correlate with end-to-end success of the same models on the source benchmarks? A rank correlation over a few models would already help.
- The tasks are mined from failures of Qwen3.5-27B and DeepSeek-v4-flash. Does this bias the comparison against these two model families?
- Was the LLM judge checked against human labels on any subset?
- Appendix B needed manual review to accept 76 of 98 unmatched outputs. Was a sample of the calls marked wrong in the main evaluation checked for false negatives?
As a suggestion that does not affect my rating, the four remote sensing models in Table 2 are small vision-language models used far outside their intended setting, and their near-zero scores tell us little.
3 comments
Response to Reviewer CFUi (2/2)
Q5 (false negatives in the main evaluation). The review rate in App. B reflects its setting rather than the scorer: App. B scores natural decision points against the reference calls published by the source benchmarks, and the 76 accepted outputs are mainly free-text arguments paraphrased with the same meaning, optional arguments the task leaves unconstrained, and cases where the published reference conflicts with the task (e.g., the task asks for LST above 300 K while the reference uses 310). GeoToolAct removes these sources of ambiguity by construction: arguments are structured values traceable to the schema, task or history, otherwise-free choices such as output paths are fixed in the task, and gold calls pass deterministic verification and a two-annotator audit (Sec. 4.5; App. D). The scorer compares parsed calls, fills declared defaults and accepts equivalent representations (App. G).
To check directly, two annotators independently reviewed 90 randomly sampled first-attempt calls marked wrong (30 per family, stratified by length, fixed seed), with model names hidden. A call counted as a false negative if, passed to the tool as written, it would produce the same result as the gold call. Neither annotator judged any of the 90 a false negative (0/90; 95% Wilson upper bound 4.1%, 11.4% per family), and no gold defect was found. The rejected calls had wrong values, stale or wrong resources, missing or extra arguments, or argument types that violate the declared schema. Empty or truncated outputs, which contain no complete call, were excluded. We will add the audit and its per-call table to the appendix.
W2 (related work and remote-sensing content). We will cite and discuss ComplexFuncBench, LongFuncEval and HammerBench. We do not claim tool-conditioned argument accuracy as new; our contribution is the requirement-controlled construction that turns it into diagnostic profiles, with the same gold call across four history lengths. Most target tools are remote-sensing operators whose arguments carry physical roles (e.g., NIR/Red and NIR/SWIR band lists, a representative thermal-band index, albedo vs. day/night temperature inputs for thermal inertia); a value in the wrong role keeps the call type-valid but computes a different quantity. The benchmark is text-based by design: imagery is processed by the tools and referenced through paths and outputs (details in our reply to VMcA, Q2).
Presentation. We will revise the following points.
- Names. Schema-Prior, Contextual Value and State-Update will be used consistently across text, tables and figures.
- Fig. 1(a1) vs. Sec. 4.1. The two use different benchmarks because they need different properties. Fig. 1(a1) only needs a task-level E2E outcome and a tool-call count per task, spread over enough call counts to fit a confidence band, which GeoBenchX provides. Construction (Sec. 4.3) instead needs a verified correct trajectory for each failure point, i.e., step-level references: ThinkGeo and Earth-Bench publish step-level reference calls, whereas GeoBenchX does not, and some of its tasks end in a rejection tool, leaving no final result to confirm the preceding calls. We will clarify this in the caption of Fig. 1.
- Fig. 3. We will add the numbers behind the heatmap as an appendix table.
- ReAct@4. We will rename it Feedback@4 and describe it as a single call retried with evaluator feedback for up to four attempts.
- Remote-sensing specialized models. We will move them to the appendix and state why we include them: they are the domain-specialized models a practitioner would naturally consider as an agent backbone, trained mainly for perception-style tasks (VQA, grounding, captioning). Their near-zero accuracy (at most 5.86% Overall) shows that current remote-sensing specialization does not by itself provide the argument construction tool execution requires. Since they are also small, we present this as an observation about current specialized models, not a controlled comparison.
- Tool counts. We will consistently report 44 tool interfaces (43 tools).
Response
Thanks for the detailed response and the additional experiments. The false-negative audit, the human re-annotation of the Earth-Bench analysis and the explanation of how the mined failure points are rebuilt answer my Q3–Q5, and the presentation changes are fine.
I still have two concerns. On Contextual Value, the new interfaces help, but the numbers in the response also suggest that the original single-tool score is not reliable on its own. Kimi-K2.6 is ahead of GLM-5.2 on build_crop_stress_assessment (49.4 vs 34.4) but well behind it on the four new interfaces (19.6 vs 42.5). I think the 80 new tasks should go into Table 2 and the per-type profiles in Section 5.2, rather than staying in the rebuttal.
On the link to end-to-end success, four models is a small sample. Kimi-K2.7-Code is third on GeoToolAct but first on Earth-Bench, and on GeoBenchX the top three models are within 1.5 points, so the pairwise agreement there does not say much. I agree with the response that GeoToolAct is best read as a diagnostic for argument construction rather than a predictor of end-to-end success, and the paper should be written that way. I will keep my score at 3.
Response to Reviewer CFUi (1/2)
Thank you for the thoughtful review, and for noting that our design is checked rather than assumed and that the results show concrete failure points across models. Several of the new analyses below are shared with the other reviewers; we summarize them here and point to the full tables.
Q1 / W1 (Contextual Value beyond one tool; scale). We built 80 new Contextual Value tasks on four additional interfaces (NDVI composition, thresholding, trend analysis, spatial statistics; 20 each) with the same controls and 80K expansion, and ran four Table 2 models three times (full table in our reply to dkn5, Q1). Mean accuracy on the new tasks is 81.3 (DeepSeek-v4-flash), 61.3 (Kimi-K2.7-Code), 42.5 (GLM-5.2) and 19.6 (Kimi-K2.6), vs. 85.6 / 64.4 / 34.4 / 49.4 on
build_crop_stress_assessmentat 80K. Absolute levels vary by interface, but the model ordering is identical on all four new interfaces, and Contextual Value keeps its place in the profiles of DeepSeek-v4-flash, Kimi-K2.7-Code (strongest family) and GLM-5.2 (weakest). Excludingbuild_crop_stress_assessmentaltogether and recomputing Overall from the Schema-Prior and State-Update columns of Table 2 leaves the ranking of the 15 general models largely unchanged (Spearman ρ = 0.95); Gemini-3-Flash stays first, and the largest moves are GPT-5 (5th→tied 2nd with GLM-5.2, from its strong State-Update) and DeepSeek-v4-flash (2nd→4th, whose strength is Contextual Value). For the small State-Update family, redrawing the distractor histories of 30 instances per family three times leaves rankings and profiles unchanged (reply to VMcA, Q4). We will revise Sec. 5.2 to state that orderings and profiles, rather than absolute Contextual Value levels, are what we expect to generalize, and will extend Contextual Value to more interfaces and add target tools to Schema-Prior and State-Update in the released benchmark.Q2 / W2 (link to end-to-end success). We ran four Table 2 models end-to-end under each benchmark's official protocol:
Of the six model pairs, GeoToolAct orders 5 consistently on GeoBenchX and 4 on Earth-Bench. Full agreement is not expected, since E2E success also depends on Tool Plan and execution, which GeoToolAct holds fixed; GeoToolAct localizes argument-construction failures and complements rather than replaces E2E evaluation.
Q3 (mining from two model families). Mining determines which decision points enter the pool, not the tasks models are evaluated on: each failure point is rebuilt with a fixed target tool, controlled non-target requirements and new histories (Sec. 4.3–4.4), so no model is scored on reproducing a mining model's mistake. Table 2 shows no penalty: DeepSeek-v4-flash ranks second overall (77.14) and first on Contextual Value, and Qwen3.5-Plus ranks third (69.16), above GLM-5.2 and GPT-5. In App. I, the failures of all 19 models fall into the error forms derived from the mining pool, with only 11 Schema-Prior calls in a residual category. We used two mining models because each must run full trajectories on all three sources with every step judged, and no finite set of mining models covers every family; the pipeline is model-agnostic and we plan to extend mining to further families (details in our reply to dkn5, Q2).
Q4 / W2 (LLM judge vs. human labels). Two researchers who work on GIS agents but were not involved in this project independently re-annotated the Earth-Bench trajectories under the protocol of App. A. Tool Act prevalence / RR is 41.6% / 1.76 [1.27, 2.45] and 40.8% / 1.74 [1.26, 2.42] for the two annotators, vs. 44.0% / 1.85 for the LLM judge; Tool Plan RR stays non-significant (1.09 and 1.10, CIs include 1). Trajectory-level κ is 0.88 (Act) and 0.89 (Plan). We will add this to App. H and, since the two RRs were not tested against each other, replace "more strongly associated" in the abstract and Fig. 1 (full table in our reply to VMcA, W1). We agree that the motivating analysis covers one model on one benchmark; we will present it as a motivating observation, and the cross-model link to end-to-end success is addressed by Q2.