- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
GeoVR studies reinforcement learning for Google Earth Engine code generation. It constructs a hierarchical reward from execution correctness, output type, output schema, and required analytical operations. Schema checks use execution artifacts such as bands, coordinate reference systems, and spatiotemporal properties, while method checks use the program AST. The method combines these rewards with GRPO and execution-feedback repair. Evaluation on AutoGEEval++ includes comparisons against binary-reward RLVR and ablations of the reward components.
Critical strengths
-
The paper identifies a meaningful domain problem: successful execution alone does not establish that a geospatial program satisfies the requested analytical constraints. Turning observable execution artifacts into supervision is practically useful.
-
The component ablations show that schema checks, method checks, and hierarchical gating each contribute to benchmark performance. This supports the value of the domain-specific verifier rather than relying only on comparisons with general-purpose models.
Critical weaknesses
-
HSVR looks largely like a domain-specific test suite turned into an RL reward. Checking types, metadata, and analytical constraints is useful, but the paper does not clearly explain what new learning principle this introduces.
-
The hierarchy is a fairly straightforward way to enforce prerequisites, and the main comparison changes both the reward and the repair mechanism. This makes it difficult to tell how much of the contribution comes from a new reward design rather than careful engineering for GEE.
Questions for authors
-
What is the precise distinction between HSVR and a conventional domain-specific test suite whose assertions check output type, schema, metadata, and analytical operations? Which part of the contribution changes the RL methodology, rather than expanding the set of properties verified by execution-based tests?
-
Beyond a conventional gated or constrained reward, what new principle does the Execution → Type → {Schema, Method} hierarchy introduce? Please explain what the gating ablation establishes about this principle, as opposed to the usefulness of enforcing prerequisites in this particular verifier.
-
Which components of GeoVR constitute a reusable method beyond GEE, and which require task- or platform-specific specification? Please clarify how required operations and schema constraints are derived without simply enforcing the reference implementation, and what existing evidence supports accepting functionally equivalent alternative workflows.
3 comments
Follow Up on Our November 16 Response
Dear Reviewer Uruc,
As the discussion period is approaching its deadline, we would like to kindly follow up on our response posted on November 16.
In response to your concerns, we clarified how HSVR differs from conventional test suites and explained its methodological contribution. We also elaborated on the rationale behind hierarchical gating and provided the corresponding ablation results. Additionally, we clarified which components of GeoVR are reusable across geospatial platforms and how our task specifications accommodate functionally equivalent implementations. We further added controlled reward and repair ablations to distinguish their respective contributions.
These clarifications and results are included in our revised manuscript. We would appreciate it if you could revisit our response. If you have any remaining questions or concerns, please let us know before the discussion closes, if possible. We would be happy to provide further clarification.
Thank you very much for your time and consideration.
Follow-up on Our Response
Dear Reviewer Uruc,
We apologize for following up once more as the discussion period is drawing to a close.
We understand that you may be very busy, and we truly appreciate the time and effort you have devoted to reviewing our paper.
We have made our best effort to address your concerns through detailed explanations and additional controlled experiments in our November 16 response. We would be very grateful if you could kindly take another look at our response when you have a moment.
If you find that our clarifications and additional evidence help resolve your concerns, we would greatly appreciate it if you might consider revisiting your overall assessment. We would also be happy to address any remaining questions.
Thank you again for your patience, thoughtful feedback, and valuable time.
Clarified HSVR vs. test suites and the gating principle, added reward/repair ablation
We thank the reviewer.
Weakness 1 and Question 1. HSVR differs from a domain test suite in three ways. (1) What is verified. A test suite asserts output values for fixed inputs. A GEE output is a geographic object, and the same area value can come from a wrong period, a wrong projection, or a table missing the region field, each making the result unusable. HSVR extracts signatures by object type: bands, CRS, resolution, and spatiotemporal extent for images, and fields, geometry type, and extent for vectors. (2) How checks are combined. Test assertions are independent and at most summed. HSVR applies Schema and Method only after the primary result and output type pass, so a wrong-type program cannot earn reward from coincidentally correct bands or CRS. (3) How constraints are set. Tests derived from a reference assert all its properties and force reproduction of one implementation. HSVR activates only properties the instruction requires and can verify, and marks the rest
unconstrained(p. 5, L217–220). If no output projection is requested, UTM and WGS84 receive the same reward, as donormalizedDifferenceand a band expression for NDWI.The new learning principle is to build rewards from geospatial constraints in execution artifacts, ordered by their dependencies. This structure sets the ranking within each GRPO group. Binary reward saturates once most candidates pass, and within-group advantages collapse. In Fig. E2, Binary-RLVR's maximum advantage keeps decreasing while GeoVR keeps a broader positive range (p. 24, L1286–1289). With close Execution Rewards, 0.910 vs. 0.945, Theme pass@1 differs by 10.35 points. Figs. G1–G3 show executable programs with correct primary results that HSVR separates and binary reward cannot. Figures: G1 · G2 · G3 · E2
Weakness 2. Table 3 crosses reward type with repair (p. 6, L308–310).
The two K=0 runs share the GEE environment, tasks, GRPO configuration, and execution count. Only the reward differs, and HSVR leads by 2.74, 7.62, and 5.75 pass@1 points and 2.97 GeoCodeEval points. HSVR at K=0 also beats Binary at K=2. Repair adds 0.99, 2.28, and 4.60 points, so both contribute, the reward more. The checks are defined per object type rather than per task, and the gain carries over to Llama-3.2-3B, where GeoVR exceeds Binary-RLVR by 4.59 points on Theme and 3.96 on GeoCodeEval. The hierarchy's order follows the geospatial workflow, and removing it with identical checks costs 8.05 Theme points (Question 2).
Question 2. Reward organization should follow the dependencies among task constraints. In a geospatial workflow, the output type decides which operations, such as spatial joins, zonal statistics, and export, are possible, and bands, CRS, extent, and fields are attributes of an object of the correct type. HSVR w/o Gating keeps identical checks and training conditions (p. 8, L418–420) and removes only these dependencies, which amounts to a test-suite-style sum. It loses 1.51, 3.81, and 8.05 pass@1 points on Atomic, Combined, and Theme (Table 4), so the drop grows with the number of constraints. Output Type Mismatch rises from 2.33% to 3.26% and Parameter Error from 1.83% to 2.37% (Table 5): without gating, wrong-type programs earn partial reward, and the policy errs more on prerequisites. The same dependencies exist in ArcPy and PyQGIS.
Question 3. Reusable components are the task-specification format, gated scoring, and GRPO training. The format covers output type, dataset, extent, time range, structural properties, operations, prohibited patterns, and
unconstrainedmarks. Platform-specific components are signature extraction, operation rules, and error classification; for example, ArcPy reads the spatial reference and cell size from a raster object. Each task's specification is built from the instruction and the reference program's signature and AST. The reference supplies candidate properties, and only those the instruction requires are constrained. For a July 2022 NDWI median composite, region, period, NDWI, and median compositing are constrained, while the reference's cloud threshold, projection, and export scale are not. Automatic consistency checks pass 96.8% of specifications; failures are corrected or removed. Manual inspection of 200 random tasks found 189 (94.5%) consistent and not excluding reasonable alternatives (p. 27, L1439–1442).