acceptodds
Withdrawn from ICLR 2027

UrbanSurface-Agent: Physics-Gated Scientific Reasoning for Urban Surface Parameterization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Urban land-surface numerical models are important tools for simulating urban energy exchanges and local climate. However, improvements in model performance do not necessarily imply scientifically valid parameterization, because compensating errors among coupled processes can make inappropriate parameter changes appear beneficial. To address this issue, we introduce UrbanSurface-Agent, a physics-gated scientific agent that formalizes parameter discovery as an experience-guided proposal–verification process. Model–observation discrepancies are used to formulate mechanism-guided parameter interventions with expected process responses, which are evaluated by SUEWS and independently adjudicated by a Physics Judge using semantic, process-wise, and temporal criteria. Experimental outcomes, including successful, failed, and boundary-case interventions, are retained in scientific memory to guide subsequent proposals. We evaluate UrbanSurface-Agent on 20 Urban-PLUMBER flux-tower sites under a sealed temporal protocol. Across 1,410 adaptation candidates, 350 improve the aggregate objective but fail process consistency and 89 further fail temporal consistency, demonstrating that numerical improvement alone is insufficient for reliable parameter discovery. The framework freezes interventions at 16 sites, with 14 satisfying practical success criteria on the sealed future period. These results demonstrate that reliable scientific-agent parameter discovery requires more than numerical optimization: proposed corrections must be tested against the physical processes they are intended to represent, and transferable scientific knowledge is better represented by context-dependent parameter–process evidence than by universal numerical corrections.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Scientific Agents, Physics-Guided Machine Learning, Simulator-Based Reasoning, Parameter Discovery, Urban Surface Parameterization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Sp3vOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces UrbanSurface-Agent, a physics-gated scientific agent for improving urban land surface parameterization. The framework combines LLM-based mechanism reasoning, deterministic parameter search, SUEWS simulations, and a Physics Judge that evaluates numerical improvement, process consistency, and temporal robustness. Failed and successful interventions are stored in scientific memory to inform subsequent proposals.

The authors evaluate the framework across 20 Urban-PLUMBER sites, examining 1,410 parameter candidates. The framework identifies admissible interventions at 16 sites, with 14 meeting the practical success criteria on a held-out future period. The results suggest that process-wise evaluation can prevent parameter corrections that improve aggregate error at the expense of individual physical processes.

Critical strengths
  • Physically motivated agent framework. I like the separation between LLM-based hypothesis generation, numerical evaluation, and independent acceptance. This prevents the LLM from directly modifying model outputs or determining whether its own proposed intervention is valid. The integration with an existing physical simulator also makes the framework relatively interpretable.

  • Systematic evaluation of parameter interventions. The experiments demonstrate why aggregate improvement alone can be misleading. Among 1,410 candidates, 350 improve the aggregate objective but fail process consistency, while another 89 fail temporal consistency. The explicit treatment of failed interventions and abstention is particularly useful for scientific model development.

Critical weaknesses
  • The Physics Judge does not directly verify physical consistency. The proposed Physics Judge primarily evaluates changes in prediction error across three outputs, together with temporal consistency and parameter validity. However, satisfying these conditions does not necessarily imply that the parameterization is physically correct. In particular, there is no independent evaluation of energy or water balance closure, and compensating errors may still exist within the accepted parameterizations. I would like to see stronger physical constraints beyond non-degradation of the selected outputs.

  • The improvement in predictive performance is relatively marginal. Despite the additional complexity, the full framework improves test nRMSE from 0.6556 to 0.6532, corresponding to an all-site gain of only 0.356% (Table 2). The practical success criterion also permits up to 0.1% degradation in an individual output, and only 5 of the 16 frozen interventions achieve exact non-degradation on the held-out period. It is unclear whether these improvements are sufficiently substantial relative to observational uncertainty and the additional computational cost.

  • The comparison against baseline optimization methods is not entirely controlled. The full framework uses approximately 95 SUEWS evaluations per site, compared with 25 for sensitivity-based search and approximately 50 for the other baselines. Sensitivity-based search already achieves a 0.247% all-site gain using substantially fewer evaluations. Although the appendix includes a matched maximum-budget memory ablation, the main performance comparison remains confounded by different computational budgets. A stronger comparison under identical evaluation budgets would help establish the efficiency of the proposed framework.

  • The contribution of LLM reasoning and scientific memory remains unclear. The paper introduces mechanism hypotheses and corresponding outcomes, but provides limited quantitative validation of whether the proposed physical explanations are actually correct. Most results focus on whether interventions satisfy predefined acceptance criteria. The memory ablation in Appendix H.5 shows an improvement in discovery rate from 40% to 50% at 100 calls, but also identifies negative transfer at several sites. I would like to see whether similar improvements could be obtained using deterministic, failure-informed search without an LLM, and whether the generated hypotheses provide meaningful scientific insight beyond parameter optimization.

  • Limited evidence of generalization beyond the evaluated sites. The evaluation uses a chronological split at the same 20 sites, meaning the results primarily demonstrate temporal transfer after site-specific adaptation. This does not establish whether the accumulated scientific memory can generalize to unseen urban environments. Moreover, the parameter search is restricted to changes within ±5% of the baseline and at most two parameters, limiting the range of physical corrections that can be explored. Evaluation on entirely held-out cities would strengthen the claim that the framework discovers transferable scientific knowledge.

Questions for authors

See above.

Reviewer_c4VUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces UrbanSurface-Agent for calibrating parameters in the SUEWS urban land-surface model. The framework combines LLM proposals, deterministic candidate generation, simulator evaluation, and experience memory. It screens interventions through checks on parameter validity, errors across outputs, and performance across temporal blocks, while retaining successful and failed trials to guide subsequent search. Evaluation covers 20 Urban-PLUMBER sites with chronological adaptation and held-out test periods, comparisons against conventional optimization methods and a memory-free LLM, and component ablations. The full system accepts updates at 16 sites, with 14 meeting predefined practical success criteria on the test period and a mean error reduction of 0.356% across all sites.

Critical strengths

The paper separates LLM proposals from acceptance under fixed rules, checking aggregate error, degradation in individual outputs, and performance across temporal blocks. This helps reject interventions whose average improvement masks localized degradation and provides a clear, verifiable basis for accepting parameter updates.

Critical weaknesses
  1. The methodological novelty and contribution to scientific reasoning beyond conventional constrained parameter calibration remain unclear. The method primarily integrates an existing simulator, bounded search, experience retrieval, and rule-based acceptance. The Physics Judge checks parameter validity and error behavior but does not independently validate the proposed physical explanations. The experiments also do not adequately demonstrate the added value of LLM-based mechanistic reasoning.

  2. The evaluation does not sufficiently isolate the agent’s contribution. The full system adds memory, more search rounds, and a larger simulation budget, while the memory-free LLM does not outperform sensitivity search. The existing memory ablation needs two further controls: sensitivity search with the same budget, and a method retaining memory while removing the LLM. Moreover, the full system uses approximately 3.8 times as many simulation calls, yet increases the mean error reduction from 0.247% to 0.356%. These results do not sufficiently justify the additional complexity and computational cost.

Questions for authors
  1. Please compare the full system with sensitivity search + Judge under the same total simulation budget, including sensitivity screening, and ensure that both methods can search the same parameter space. Report discovery rates during adaptation and test-period errors and success rates after freezing parameters, to distinguish methodological gains from those due to additional computation.

  2. Please add a control that retains memory but removes the LLM. Keep experience retrieval, candidate ranking, numerical search, and the Judge, while replacing LLM proposals with sensitivity rules or a deterministic strategy. This would test whether LLM reasoning adds value beyond experience reuse and numerical search.

  3. Mechanistic expectations are currently defined primarily in terms of improvements in output errors. Please explain how this supports judgments about physical mechanisms. Report how many proposals include testable expectations recorded before simulation and how well these expectations match the observed responses. As additional validation of the scientific reasoning claim, I suggest controlled experiments with known parameter perturbations to test whether the system can distinguish mechanistically justified corrections from compensating errors.

  4. Please justify the practical success criteria: why require at least a 0.05% aggregate error reduction while allowing up to 0.1% degradation in an individual output? Relate these thresholds to observational uncertainty and application needs. Also report absolute error changes for each output and the sensitivity of success rates to threshold selection.

  5. The main full-system results come from a single 20-site trajectory, whereas some baselines use five seeds. Please compare the full system and major baselines under a consistent replication protocol, reporting paired site-level gains and confidence intervals to assess whether the small average improvements are stable.

  6. Please report total runtime, hardware and parallelization settings, and a breakdown of time spent on simulation, LLM calls, and memory processing under the same computing environment. Include actual LLM call counts, token usage, and costs. Separate memory construction costs from subsequent use and explain how construction costs are amortized across reuse.

Reviewer_4QozOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies parameter correction for urban land-surface models through a discovery workflow combining LLM-generated hypotheses, SUEWS simulations, and a fixed Physics Judge. It retains successful and failed trials and searches during an adaptation period before freezing parameters for evaluation on a future interval. Of 20 sites, 16 receive frozen interventions, of which 14 satisfy the authors’ practical future-period succes

Critical strengths

Separation of proposal, verification, and future evaluation. The parameter ontology, independent acceptance rules, and frozen evaluation define the roles of the LLM and simulator and specify when the baseline should be retained. Full analysis includes failures and additional ablations. Result analysis part reports site-level outcomes, failure categories, and negative transfer, and includes a frozen cross-site memory ablation with the same maximum budget, clarifying the scope of the main results.

Critical weaknesses

The matched-budget memory result needs numerical uncertainty. Table 10 reports a 10-percentage-point discovery gain at that horizon, while the text acknowledges different realized call counts and negative transfer at two sites. It states that a site-clustered bootstrap confirms a positive average effect, but gives no numerical interval. Please report that interval and its resampling details so the strength of this evidence can be assessed. Practical future-period gains need clearer uncertainty and sensitivity reporting. The paper clearly distinguishes five exactly non-degrading transfers from nine additional grade-A transfers under its 0.1% practical margin, and explains the operational purpose of that margin. What remains unclear is how the 0.356% all-site gain and the 14 practical-success count vary with sampling uncertainty and the chosen 0.05%/0.1% thresholds.

Questions for authors

See as above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.