- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces UrbanSurface-Agent, a physics-gated scientific agent for improving urban land surface parameterization. The framework combines LLM-based mechanism reasoning, deterministic parameter search, SUEWS simulations, and a Physics Judge that evaluates numerical improvement, process consistency, and temporal robustness. Failed and successful interventions are stored in scientific memory to inform subsequent proposals.
The authors evaluate the framework across 20 Urban-PLUMBER sites, examining 1,410 parameter candidates. The framework identifies admissible interventions at 16 sites, with 14 meeting the practical success criteria on a held-out future period. The results suggest that process-wise evaluation can prevent parameter corrections that improve aggregate error at the expense of individual physical processes.
Critical strengths
-
Physically motivated agent framework. I like the separation between LLM-based hypothesis generation, numerical evaluation, and independent acceptance. This prevents the LLM from directly modifying model outputs or determining whether its own proposed intervention is valid. The integration with an existing physical simulator also makes the framework relatively interpretable.
-
Systematic evaluation of parameter interventions. The experiments demonstrate why aggregate improvement alone can be misleading. Among 1,410 candidates, 350 improve the aggregate objective but fail process consistency, while another 89 fail temporal consistency. The explicit treatment of failed interventions and abstention is particularly useful for scientific model development.
Critical weaknesses
-
The Physics Judge does not directly verify physical consistency. The proposed Physics Judge primarily evaluates changes in prediction error across three outputs, together with temporal consistency and parameter validity. However, satisfying these conditions does not necessarily imply that the parameterization is physically correct. In particular, there is no independent evaluation of energy or water balance closure, and compensating errors may still exist within the accepted parameterizations. I would like to see stronger physical constraints beyond non-degradation of the selected outputs.
-
The improvement in predictive performance is relatively marginal. Despite the additional complexity, the full framework improves test nRMSE from 0.6556 to 0.6532, corresponding to an all-site gain of only 0.356% (Table 2). The practical success criterion also permits up to 0.1% degradation in an individual output, and only 5 of the 16 frozen interventions achieve exact non-degradation on the held-out period. It is unclear whether these improvements are sufficiently substantial relative to observational uncertainty and the additional computational cost.
-
The comparison against baseline optimization methods is not entirely controlled. The full framework uses approximately 95 SUEWS evaluations per site, compared with 25 for sensitivity-based search and approximately 50 for the other baselines. Sensitivity-based search already achieves a 0.247% all-site gain using substantially fewer evaluations. Although the appendix includes a matched maximum-budget memory ablation, the main performance comparison remains confounded by different computational budgets. A stronger comparison under identical evaluation budgets would help establish the efficiency of the proposed framework.
-
The contribution of LLM reasoning and scientific memory remains unclear. The paper introduces mechanism hypotheses and corresponding outcomes, but provides limited quantitative validation of whether the proposed physical explanations are actually correct. Most results focus on whether interventions satisfy predefined acceptance criteria. The memory ablation in Appendix H.5 shows an improvement in discovery rate from 40% to 50% at 100 calls, but also identifies negative transfer at several sites. I would like to see whether similar improvements could be obtained using deterministic, failure-informed search without an LLM, and whether the generated hypotheses provide meaningful scientific insight beyond parameter optimization.
-
Limited evidence of generalization beyond the evaluated sites. The evaluation uses a chronological split at the same 20 sites, meaning the results primarily demonstrate temporal transfer after site-specific adaptation. This does not establish whether the accumulated scientific memory can generalize to unseen urban environments. Moreover, the parameter search is restricted to changes within ±5% of the baseline and at most two parameters, limiting the range of physical corrections that can be explored. Evaluation on entirely held-out cities would strengthen the claim that the framework discovers transferable scientific knowledge.
Questions for authors
See above.