acceptodds
Under review as a conference paper at ICLR 2027

SciExam for ENSO: Can AI Agents Build Climate Models?

Abstract

Language-model agents are increasingly asked to carry out open-ended scientific research, yet their results are usually graded against a known answer, a rubric, or a language-model reviewer, none of which can tell whether a new scientific model is valid. The AI Science Exam for El Nino–Southern Oscillation (SciExam for ENSO) is a benchmark in which agents build low-order stochastic models of ENSO, the dominant mode of interannual climate variability, from real observations. Within a six-hour budget, agents process the observations, write their own diagnostics, which are then frozen, and develop a model using only these diagnostics as feedback. Hidden graders then test whether the model reproduces ENSO's statistics, recovers unobserved variables, and forecasts held-out years, and score a published model in the same way. Across twelve agent systems, six produce models that score higher than the published model, mainly through better reconstruction and forecasting. The simplified forms of the stronger models are each compatible with one of the two competing explanations of ENSO's warm–cold asymmetry, an open debate that the task never mentions. Controlled runs of the top system under varied information suggest that its scores do not come from recalling the dated observational record and that the information it receives shapes how it builds its model. SciExam for ENSO can thus evaluate agent research where no answer is known, and the results suggest that agents can already build competitive models whose structures bear on questions that scientists still debate.

Primary area:
datasets and benchmarks
Keywords:
AI for Science, Autonomous Agents, Long-Horizon Evaluation, Scientific Benchmarking, Dynamical Systems, Conceptual Climate Modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BygbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SciExam for ENSO evaluates whether AI agents can build scientific models when no unique correct equation is available. Agents construct low-dimensional stochastic models of El Niño–Southern Oscillation (ENSO) from real observations. Within a shared six-hour budget, they process the data, develop diagnostic scripts that are subsequently frozen, and use those diagnostics to guide model development. Hidden graders evaluate long-term statistics, reconstruction of unobserved variables, and forecasting on held-out years.

Each of twelve agent systems is run once. Six final models exceed a published reference model under the same composite evaluation. Post-hoc simplification connects stronger models to two established explanations of ENSO warm–cold asymmetry. Additional runs examine how available information affects one system. These results demonstrate competitive model instances, while reliable success rates and the underlying ENSO mechanism remain unresolved.

Critical strengths
  1. The benchmark makes open-ended scientific modeling directly testable.

    Agents submit executable dynamical models evaluated against observations. Long simulations, variable reconstruction, and held-out forecasting test complementary aspects of the same equations and parameters. Data-processing checks, frozen diagnostics, and hidden scores provide a concrete workflow for sustained autonomous modeling. This design evaluates the behavior of scientific outputs beyond task completion. (§3; Appendices A and C)

  2. The experiments produce competitive models and examine the structures behind their performance.

    Six final models outperform the published reference, mainly through reconstruction and forecasting. The strongest model also approaches the reference in statistical fidelity. (§4; Figure 3b; Table 3) The reported weight-sensitivity analysis retains the same six systems above the reference under equal weighting. (Appendix C, p. 16) Structural simplification further identifies reduced models compatible with established ENSO explanations. (§5.1; Figure 5; Table 3; Appendix G) Together, these results provide useful model instances and interpretable evidence beyond aggregate scores.

Critical weaknesses
  1. The scores can underrepresent the loss of an important physical feature, while the additional selection criterion is not fully quantified.

    Appendix G reports that further simplifying GPT-6 Astra and the two DeepSeek models to linear drift with additive noise preserves at least 95% of the composite score and 90% of each component score, despite producing nearly symmetric temperature distributions and reduced event diversity. Multiplicative noise refers to random forcing whose strength depends on the system state. The authors appropriately reject these reductions through an additional temperature-distribution check.

    Although the authors compare simulated and observed TET_E distributions, they do not specify a quantitative asymmetry measure or an explicit threshold for rejecting a reduction as “nearly symmetric.” Nor do they systematically report asymmetry measures for retained and rejected reductions. It therefore remains unclear how sensitively the scores reflect this important feature and how robust the two-family classification is to the additional criterion. This limits the interpretation of both the composite scores and the structural analysis. (Appendix G, pp. 22–23)

  2. Single runs limit conclusions about repeatability and the effects of information access.

    Each agent system is run once, and each information condition is tested once on one system. The authors explicitly frame these comparisons as model instances and a case study, which is appropriate. Nevertheless, the experiments cannot estimate how reliably systems exceed the reference or separate information-condition effects from variation between agent runs.

    The competitive result with anonymized years is useful evidence, but it does not by itself exclude other contributions from pretrained knowledge of the observations. These limitations do not invalidate the submitted models, but they constrain conclusions about reliable modeling ability and the effects of information access. (§5.2; Appendices B.1 and F)

Questions for authors
  1. Asymmetry criterion: How was a temperature distribution judged to be “nearly symmetric” in Appendix G? Please specify the measure and threshold, and report the corresponding values for the original, retained, and rejected reductions of the three models discussed above. Does the two-family classification remain unchanged under an explicit quantitative criterion?
  2. Repeatability: Have representative systems and information conditions been evaluated across repeated runs? If so, please report the variation in final scores, the frequency of exceeding the reference, and whether the modeling-strategy changes described in §5.2 recur. This would help distinguish repeatable effects from differences between individual runs.
4 comments
AuthorsOpenReview ↗

Response to Reviewer Bygb

We sincerely appreciate reviewer Bygb for the positive comments on our contributions, especially for recognizing that the benchmark makes open-ended scientific modeling directly testable and that the structural simplification gives interpretable evidence beyond aggregate scores.

[W1, Q1] Quantitative asymmetry measure, threshold and robustness of the two-family classification.

[…] they do not specify a quantitative asymmetry measure or an explicit threshold for rejecting a reduction as "nearly symmetric." […]

We thank the reviewer for this suggestion, which turns a qualitative judgement into an explicit criterion. The submission computed the asymmetry of each reduction but reported neither the measure nor the threshold. The revision states both and reports the values for every reduction (Appendix G.1).

Measure: the skewness of simulated TET_E in each of 30 free-run segments of 70 years, reported as mean ± sd (observed 0.83, 95% block-bootstrap interval [0.28, 1.20]). Criterion: a model keeps the asymmetry if the observed value lies within the central 95% of its segment skewnesses. A reduction is retained if it passes the score screen and this criterion.

  1. Every linear-drift, additive-noise reduction has skewness 0.01 to 0.06, and none of its segments is as skewed as the observations. All original models have 0.65 to 1.45.
ModelVersionTET_E skewnessMeets criterion
GPT-6 Astraoriginal0.92 ± 0.13yes
retained (nonlinear + additive)1.05 ± 0.27yes
rejected (linear + additive)0.04 ± 0.20no
DeepSeek V4.1 Flashoriginal1.45 ± 0.26no
retained (linear + multiplicative)1.07 ± 0.31yes
rejected on scores (nonlinear + additive)0.80 ± 0.20yes
rejected (linear + additive)0.06 ± 0.20no
DeepSeek V4 Prooriginal0.95 ± 0.47yes
retained (linear + multiplicative)0.71 ± 0.70yes
rejected (nonlinear + additive)0.19 ± 0.17no
rejected (linear + additive)0.01 ± 0.18no
rejected (linear + additive, no seasonal drift)0.04 ± 0.14no

Classification: Every retained reduction meets the criterion and every linear-drift, additive-noise reduction fails it, so the vote is unchanged (4 linear + multiplicative, 5 nonlinear + additive).

Sensitivity: The linear + additive reductions of GPT-6 Astra and both DeepSeek models pass the score screen, while their skewness falls from 0.92, 1.45 and 0.95 to 0.06 or below. The composite is therefore not a sufficient check of asymmetry, and the revised Section 5.1 reports skewness alongside the scores.

Seeds: Rescored with five further seeds, every retained reduction keeps at least 95.5% of its composite. The repeated-run simplification averages the official seed and four further seeds (Appendix D).

[W2, Q2] Repeated runs across systems and information conditions.

Have representative systems and information conditions been evaluated across repeated runs? […]

We thank the reviewer for this suggestion, which shows that the main findings recur across runs. All repeats have finished: three runs per system and for the anonymized-years condition (General Response, [Repeated Runs], Appendix D). GPT-6 Astra and Claude Fable 5.1 exceed the reference in every run, and the asymmetry of the two best agent systems recurs. Repeating the simplification (Appendix D), GPT-5.6-Sol, Opus 5 and Gemini 3.8 Flash reduce to nonlinear drift with additive noise in all three runs. Claude Fable 5.1 again replaces its initial recharge oscillator with a fitted dense operator in both offline repeats. Of the seven systems with more than one classified run, six never fall into the other family, although GPT-6 Astra, Claude Fable 5.1 and DeepSeek V4.1 Flash are each unassigned in one run. DeepSeek V4 Pro changes family in its one repeat above the analysis threshold, so its assignment describes a single run. The two web-access conditions remain single runs.

[W2] Contributions of pretrained knowledge beyond recall of the dated record.

[…] it does not by itself exclude other contributions from pretrained knowledge of the observations.

We thank the reviewer for this precise point, and we agree. The anonymized run excludes only recall of the dated record, and over three runs it scores lower than offline (0.407 against 0.489), so Section 5.2 no longer rules out a contribution from recall. Other contributions, such as familiarity with the ENSO literature, are not excluded either, as Section 5.2 says.

We sincerely appreciate the reviewer's feedback and are willing to address any further comments.

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer Bygb,

Thank you again for your positive and constructive review. We hope our response and the revised PDF (changes in blue) have addressed your questions. In particular, the paper now states the asymmetry measure, threshold and values, under which the two-family classification is unchanged, and reports three runs of every system that can still be run and of the anonymized-years condition.

As the discussion period ends on November 18 (AoE), please let us know if any question remains, and we will answer it before then. If you find that the revision strengthens the paper, we would be grateful if you would take it into account in your final assessment.

Best regards, The Authors

Thank you for your detailed responses and the revisions. I appreciate the additional experiments and clarifications regarding the asymmetry criterion and repeatability. I find your responses to my original concerns satisfactory.

I have a few additional comments regarding the scope and novelty of the benchmark:

  1. Limited Benchmark Scope: While I appreciate the depth of the ENSO modeling task, the benchmark currently contains only a single scientific problem. For a benchmark intended to evaluate scientific modeling capabilities, I find this scope somewhat limited.

  2. Novelty of the Evaluation Framework: I recognize the value of evaluating open-ended scientific models through executable, outcome-based verification. However, similar evaluation paradigms, in which agents produce executable artifacts assessed by task-specific verifiers, already exist in benchmarks such as KernelBench and Terminal-Bench-Science, as well as infrastructure such as Harbor. While the ENSO-specific scientific evaluation is valuable, I would appreciate further clarification on how the proposed evaluation framework differs methodologically from these existing approaches.

  3. Self-generated and Frozen Diagnostics: I consider this a potentially more distinctive methodological contribution. Having agents develop their own diagnostic criteria and freeze them before model development is an interesting design choice. However, its incremental value has not been established through controlled ablations, such as comparing self-generated versus predefined diagnostics, or frozen versus modifiable diagnostics. Such comparisons would help substantiate this aspect of the framework's novelty.

Overall, I view SciExam as a carefully designed scientific benchmark task that supports open-ended modeling and provides meaningful scientific evaluation. However, given its limited task diversity and the remaining questions about methodological novelty, I am inclined to maintain my current rating of 3.

AuthorsOpenReview ↗

Response to the additional comments on scope, novelty and frozen diagnostics

We thank the reviewer for reading our responses and for the further comments. We are glad that our responses resolved the original concerns, and we answer the three new points in turn.

  1. Benchmark scope: We chose to focus on a single problem for two reasons. First, depth in one problem can change practice in that field. ENSO is the dominant mode of interannual climate variability and one of the most active topics in climate science, and low-order conceptual models remain a central tool for studying and forecasting it. Researchers still build these models through hand calibration or conventional fitting. In SciExam, agents build them from raw observations in six hours, which is long for a benchmark run but short for a scientist, and the best of them exceed a published model under the same tests. For a field this active, this substantially changes how researchers can build and test such models. Second, we believe that in AI for science depth should come first. Benchmarks built for breadth suit coding, where many tasks have checkable answers, but scientific discovery usually comes from sustained work on one problem, and a shallow test cannot tell a valid model from a good fit. Depth allows tests beyond the score: repeated runs of every system, a published expert model and automated baselines under the same graders, held-out years, and analysis of the structure of the agent models. The protocol does not depend on ENSO, and our next step is to apply it with experts in other fields, so that the benchmark grows broader while keeping this depth.

  2. Novelty of the evaluation framework: SciExam indeed shares the infrastructure of KernelBench and Terminal-Bench-Science, in which agents produce executable artifacts that task-specific verifiers check, and we regard this as a strength. A good benchmark should measure different content within a common framework, so that researchers can run and compare it alongside existing ones on harnesses such as Harbor. SciExam differs in what it measures. In both of those benchmarks, the task authors fix a correct answer before the agent starts. SciExam has no such answer. Its graders test, against observations, whether a model behaves as a valid model of ENSO must, and compare it with a published expert model under the same tests. We have added these benchmarks to the related work with this distinction (Appendix K).

  3. Self-written and frozen diagnostics: We appreciate that the reviewer sees this design as a distinctive part of the framework. We arrived at it during development, where we tried the alternatives in turn. Without diagnostics, agents had no way to judge a revision, and over long runs they stalled, making small edits that did not improve the model while the cost kept rising. With diagnostics that the agent could keep revising, the same pattern returned, as agents spent much of the budget rewriting checks and files rather than the model (Appendix B). A single run could then cost several hundred US dollars, so a benchmark of twelve systems with repeated runs would cost thousands of dollars per round. Freezing the diagnostics before model development largely removed this pattern. Under the final protocol a run costs between 2 and 132 US dollars (Appendix J), and the frozen diagnostics carry information about model quality. When we reran each frozen suite on every saved checkpoint of the first run (Appendix B), six suites ranked checkpoints by their own verdicts, four of them selected the final checkpoint, and selecting by them put seven systems above the reference instead of six. As the Limitations paragraph states, we leave a controlled comparison with a variant that permits revisions to future versions.

We sincerely appreciate the reviewer's engagement throughout the discussion and are happy to answer any further questions.

Reviewer_M1ELOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SciExam for ENSO, in which agents have six hours to process observations, develop diagnostics that are subsequently frozen, and construct stochastic dynamical models. Hidden tests evaluate statistical properties, variable reconstruction, and forecasting on held-out years. In single runs of twelve systems, six final models exceed the composite score of a published reference model. Post-hoc structural simplification identifies reduced models compatible with two established explanations of ENSO asymmetry.

Critical strengths
  1. The benchmark turns open-ended scientific modeling into an executable and testable task. Agents must process real observations, design diagnostics, and construct stochastic dynamical models that undergo independent scientific evaluation. This provides a valuable setting for assessing modeling capabilities beyond reproducing a fixed reference answer.

  2. The analysis examines both model behavior and model structure. Long-run simulation statistics, variable reconstruction, and held-out forecasting provide complementary tests, while the equations and their structural simplification allow readers to inspect which mechanisms support performance. Together, these analyses provide scientific interpretability beyond fitting errors or system rankings.

Critical weaknesses
  1. A single ENSO task provides limited support for broader modeling capabilities and protocol transferability. ENSO is a well-studied system, and the task prescribes five state variables and supplies background literature. It is therefore difficult to distinguish general modeling ability from familiarity with established ENSO modeling paradigms. Year anonymization does not rule out the latter, and no additional physical systems validate the proposed protocol's transferability. These limitations constrain the scope of the conclusions that can be drawn from the benchmark.

  2. The reported performance advantage lacks sufficient replication and matched comparisons. Each system and information condition is evaluated only once, while development trajectories fluctuate and some systems show substantial best/final checkpoint gaps. These observations raise concerns about sensitivity to search trajectories and stopping time, but cannot substitute for estimates of run-to-run variability. Moreover, the main scientific-model comparison uses a published conceptual model that was not tuned to the composite objective; matched comparisons with applicable statistical modeling or equation-discovery methods, such as LIM, SINDy, and LLM-SR, are missing. Consequently, six systems exceeding this reference does not yet establish a robust, well-calibrated performance advantage over existing modeling approaches.

Questions for authors
  1. Is protocol transferability supported by evaluations beyond ENSO, or does it currently refer to the reusability of the execution structure? What scope of modeling capability do the authors consider the present results to establish?

  2. Are repeated-run results available to support differences between systems and information conditions? If not, how should these conclusions be interpreted given the unestimated run-to-run variability?

  3. Are matched results available for applicable statistical modeling or equation-discovery methods using the same training data, state variables, and evaluation conditions? Such comparisons would materially affect whether exceeding the published reference establishes additional value from general-purpose agents.

4 comments
AuthorsOpenReview ↗

Response to Reviewer M1EL

We sincerely appreciate reviewer M1EL for the positive comments on our contributions, especially for recognizing that the benchmark turns open-ended scientific modeling into an executable and testable task. We address each point as follows:

[W1, Q1] Scope of protocol transferability and of the established modeling capability.

Is protocol transferability supported by evaluations beyond ENSO, or does it currently refer to the reusability of the execution structure? […]

We thank the reviewer for pointing out this ambiguity, which helps us state our claims precisely. It refers to the execution structure. The data gate, the self-written, then frozen diagnostics, the hidden graders and the fixed budget do not depend on ENSO. We did not evaluate another system, so the results do not show that the agents' ability transfers. Section 3 now says so. The results support a narrower claim: on one climate system, agents build calibrated stochastic models from raw observations that beat published conceptual models, come within 0.03 of a strong empirical baseline, and at best reproduce ENSO's asymmetry.

[W1] General modeling ability versus familiarity with established ENSO paradigms.

It is therefore difficult to distinguish general modeling ability from familiarity with established ENSO modeling paradigms. […]

This question is central to interpreting the results. We agree that familiarity cannot be excluded, and a human ENSO scientist also starts from the literature. The benchmark tests whether that knowledge becomes a calibrated model that passes independent tests: the agents try a known approach, test it with their own diagnostics, and change it when it fails. Offline, Claude Fable 5.1 replaced a recharge oscillator that proved unstable with a fitted dense linear operator, and did so again in both repeats. The best resulting models combine the coupling and the asymmetry with statistics scores no automated baseline reaches ([W2, Q3]). With anonymized years the mean of three runs is lower than offline (0.407 against 0.489), so recall of the dated record may contribute, and other uses of pretrained knowledge are not excluded. Section 5.2 and the Limitations say so.

[W2, Q3] Matched comparisons with statistical modeling and equation-discovery methods.

Are matched results available for applicable statistical modeling or equation-discovery methods […]?

We thank the reviewer for this suggestion, which makes the comparison with established methods direct. We fitted conceptual and empirical baselines to the same 1980 to 2014 indices, with the same interface and graders (General Response, [Conceptual and Empirical Baselines], Appendix E). LLM-SR yields no stochastic forcing, so it cannot be scored as submitted. Against conceptual models the advantage holds: nine systems exceed the 5D model and six exceed XRO in at least one run. Against empirical models the composite alone does not show one, since the seasonal LIM (0.518) is level with the best agents, and we agree that exceeding the 5D reference is not enough on its own. The added value is a model right in both respects: the LIM is nearly symmetric, while the best agent models keep the coupling and the asymmetry in every run.

[W2] The reference model was not tuned to the composite objective.

[…] a published conceptual model that was not tuned to the composite objective

We appreciate this point, which clarifies what the reference comparison can show. The 5D model was calibrated by its authors against the full observational record, with ENSO's long-run statistics as the target, which is why its statistics score (0.727) is above every baseline. Statistics can be hand-tuned, because the role of each term of a conceptual model is understood, but no comparable tuning improves dynamical consistency or forecast skill. Tuning it to the composite would also need the hidden graders, which no agent had. Only one final model exceeds it in statistics, the third run of GPT-6 Astra (0.742), which also exceeds it on the other two dimensions (Appendix D).

[W2, Q2] Repeated runs and run-to-run variability.

Are repeated-run results available to support differences between systems and information conditions? […]

We thank the reviewer for this question, which makes the comparison between systems more reliable. See General Response, [Repeated Runs], and Appendix D. GPT-6 Astra and Claude Fable 5.1 exceed the reference in every run, and seven systems on their mean final score. Mean best and mean final rankings agree (Spearman 0.91).

We sincerely appreciate the reviewer's feedback and are willing to address any further comments.

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer M1EL,

Thank you again for your thoughtful review. We hope our response and the revised PDF (changes in blue) have addressed your concerns. In particular, the paper now includes matched comparisons with the seasonal LIM, VAR, CS-LIM, SINDy and XRO under the same graders, three runs of every system that can still be run, and three runs of the anonymized-years condition.

As the discussion period ends on November 18 (AoE), please let us know if any concern remains, and we will answer it before then. If our response has resolved your concerns, we would be grateful if you would consider updating your score.

Best regards, The Authors

Thank you for the repeated runs, matched baselines, and additional held-out reconstruction and forecast uncertainty analyses. These additions alleviate my concerns about experimental reliability. However, my main reservation remains the benchmark’s task coverage.

As clarified in the rebuttal, the evaluation is still limited to a single ENSO modeling task on one physical system, while transferability refers only to the execution structure. Increasing the number of evaluated systems, repeated runs, and scoring dimensions on the same task does not broaden the coverage of independent modeling problems. The current evidence therefore remains insufficient to establish whether the evaluation consistently provides meaningful distinctions between agent capabilities across different modeling challenges. I recognize the value of the ENSO task and its accompanying analyses, but remain unconvinced that its task coverage and representativeness sufficiently support the significance of the benchmark contribution for scientific modeling agents.

AuthorsOpenReview ↗

Response to the additional comments on task coverage

We thank the reviewer for confirming that the repeated runs, matched baselines and held-out analyses resolve the concerns on experimental reliability. The paper does not claim that the results transfer to other problems, and the Limitations paragraph states that "transfer to other systems is untested". We explain below why we believe the contribution is significant within this scope.

  1. As in our response to Reviewer Bygb, depth matters for two reasons. First, it can change practice in a large field. ENSO is the dominant mode of interannual climate variability and is studied by a large community, and its conceptual models span many types, from the recharge oscillator to stochastic, multiscale and nonlinear extensions such as XRO. Researchers still build them through hand calibration or conventional fitting, whereas in SciExam agents build them from raw observations, and the best of them exceed a published model under the same tests. Second, AI for science needs this depth. Scientific discovery usually comes from sustained work on one problem, and a shallow test cannot tell a valid model from a good fit.

  2. The benchmark distinguishes agents consistently. GPT-6 Astra and Claude Fable 5.1 exceed the reference in all three runs, while MiniMax-M3 and GLM-5.2 stay below it in all three (Appendix D, Table 6), and rankings by best and final checkpoint agree (Spearman 0.91). The graders also separate different abilities. The published model leads on statistics (0.727) but is weakest on dynamical consistency (0.323), and the seasonal LIM leads on dynamical consistency (0.707) but cannot reproduce the warm and cold asymmetry. Only the best agent models do well on both (Appendix E, Table 8).

  3. Single-problem benchmarks are established. Benchmarks built around one important problem have become standards in their fields. WeatherBench (Rasp et al., 2020) evaluates one task, forecasting two global fields three and five days ahead, and ClimateBench (Watson-Parris et al., 2022) evaluates one task, emulating how one Earth system model responds to emissions on one held-out scenario. Both became standard references for data-driven weather and climate modeling.

We sincerely appreciate the reviewer's engagement throughout the discussion and are happy to answer any further questions.

Reviewer_xRr6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark in which agents construct five-variable stochastic ENSO models from real observations within six hours. Agents process raw data, develop and freeze their own diagnostics, and then calibrate models without access to grader scores. Hidden graders assess long-run statistics, reconstruction of unobserved variables, and forecasting on held-out years. Twelve agent systems are evaluated once each; six final models exceed a published reference model in composite score. Post-hoc simplification relates the stronger models to two established explanations of ENSO asymmetry. Three additional runs of the highest-scoring system explore changes in access to dates, literature, and external data.

Critical strengths
  1. Evaluation of executable scientific outputs. The same submitted model must support long simulations, conditional reconstruction, and held-out forecasting. This provides more substantive evidence than evaluating the plausibility of a generated research report alone. Keeping scores and held-out observations inaccessible during development is also a valuable design choice.
  2. Detailed reporting and appropriately disclosed limitations. The paper documents the task, model interface, scoring formulas, scaffolds, costs, checkpoints, and representative equations. It explicitly acknowledges single-run evaluation, differences between systems and underlying models, and selection bias in post-hoc simplification. These disclosures make the results easier to interpret.
Critical weaknesses
  1. The evaluation is too narrow to establish a reliable comparative benchmark. Although twelve systems are included, they all solve one fixed five-variable ENSO modeling problem, and each system is run only once. The three information-access conditions are also each run once for one selected system. Multiple checkpoints from a trajectory do not constitute independent trials. The experiments demonstrate that some individual runs produce competitive artifacts, but do not establish expected system performance, success rates, or robustness across scientific problems. The authors acknowledge this distinction; nevertheless, the limited coverage remains a central constraint on the benchmark contribution. Repeated runs on representative systems and additional task instances would materially strengthen the evidence.
  2. Strong automated modeling baselines are missing. The main comparisons use one published conceptual model and persistence for forecasting. These do not isolate the value of the agentic workflow relative to standard automated identification and calibration. Seasonal linear stochastic models, VAR models, and sparse candidate-term fitting under the same data restrictions would be informative controls. This is particularly relevant because a leading model simplifies to dense linear coupling with state-dependent noise. Without such baselines, it is unclear how much of the result requires autonomous scientific reasoning rather than established fitting procedures.
  3. Reconstruction supports dependence modeling more directly than physical mechanism validity. The reconstruction test uses 1980-2014, the same period provided to agents for model development. Target variables are hidden from the smoother at evaluation, but their training records were available to the agent. Successful reconstruction therefore demonstrates learned conditional relationships on the development period, rather than independent validation of a physical mechanism. Reconstruction on held-out years and comparisons with simpler covariance-based models would clarify what this dimension establishes. Forecasting is genuinely held out, but its approximately ten-year record also calls for uncertainty estimates that account for overlapping forecasts and temporal dependence.
  4. The mechanism analysis involves additional human selection on the evaluation criteria. After the agent runs, the authors remove terms, refit coefficients to simulations of the original models, and retain reductions according to the same graders and an asymmetry requirement. This is useful exploratory analysis, but the resulting simplifications are not solely agent-produced discoveries, and improved scores after selection are not independent validation. Compatibility with two established model families does not distinguish their physical explanations. A quantitative asymmetry criterion and evaluation of selected reductions on independent periods or seeds would strengthen the analysis.
  5. The diagnostic-freezing and submission policies need direct validation. The paper motivates frozen diagnostics as protection against evaluation drift, but does not isolate their benefit against allowing diagnostic revisions. Freezing can also prevent agents from correcting inadequate tests. In addition, agents are told their best saved submission counts, whereas the primary comparison uses the final checkpoint. This mismatch can alter exploration behavior and final-model quality. A consistent policy in which agents select a final submission using their own diagnostics would provide a cleaner assessment.
Questions for authors
  1. Can you repeat representative systems, including the strongest system and the information-access conditions, and report score distributions and success rates? How sensitive are the conclusions to stochastic run variation?
  2. How do automated seasonal linear stochastic models, VAR models, and sparse equation-fitting baselines perform with the same observations and evaluation interface?
  3. Do the reconstruction gains persist on years excluded from model development? Can you report uncertainty for forecast improvements using temporal blocks or event-based resampling?
  4. Does freezing diagnostics improve final scientific quality relative to permitting revisions? How do results change when the agent must explicitly select its final model under a consistent submission rule?
  5. How is preservation of ENSO asymmetry quantified during simplification, and do the selected reductions retain their advantages under independent evaluation seeds or periods?
3 comments
AuthorsOpenReview ↗

Response to Reviewer xRr6 (1/2)

We sincerely appreciate reviewer xRr6 for the positive comments on our contributions, especially for recognizing the value of evaluating executable scientific outputs with scores and held-out observations kept from the agent, and the detailed reporting of the task and its limitations. We address each point as follows:

[W1, Q1] Repeated runs on representative systems and score distributions.

The evaluation is too narrow to establish a reliable comparative benchmark. […]

We thank the reviewer for this suggestion, which makes the comparison between systems more reliable. Every system that can still be run under its original conditions now has three six-hour runs, and so does the anonymized-years condition (General Response, [Repeated Runs], and new Appendix D). The conclusions of the submission hold: six systems exceeded the reference in the first run and seven exceed it on their mean final score, with GPT-6 Astra and Claude Fable 5.1 above it in every run. The structural families also recur across runs (our reply to Bygb, [W2, Q2]), and two of three anonymized-years runs exceed the reference. The literature and external-data conditions remain single runs, presented as a case study.

[W2, Q2] Automated seasonal linear, VAR and sparse equation-fitting baselines.

Strong automated modeling baselines are missing. […]

We thank the reviewer for this suggestion, which places the agents against much stronger baselines. We added all three kinds: seasonal linear stochastic models (seasonal LIM and CS-LIM), a seasonal VAR(1), and sparse candidate-term fitting (SINDy), all fitted to the 1980 to 2014 indices the agents had and scored by the same graders (General Response, [Conceptual and Empirical Baselines], and Appendix E). The seasonal LIM is the strongest, level with GPT-6 Astra on the composite (0.518 against a mean of 0.506), but being linear and Gaussian it is nearly symmetric (skewness 0.03 against 0.83 observed), and CS-LIM with multiplicative noise does not close this gap. The two best agent systems keep the asymmetry in every run. SINDy is weaker (0.399).

[W2] How much of the result requires agentic reasoning rather than standard fitting.

[…] a leading model simplifies to dense linear coupling with state-dependent noise. […]

This question goes to the core of what the benchmark measures, and we thank the reviewer for raising it. To answer it directly, we added the established fitting procedures for dense linear coupling, the seasonal LIM and VAR(1) ([W2, Q2]). They forecast and reconstruct as well as the best agent, but they are bound to a structure fixed in advance. A linear Gaussian model is nearly symmetric (TET_E skewness 0.03, against 0.83 observed) and rarely produces extreme El Nino events, and the automated baselines that reach the asymmetry (calibrated CS-LIM, SINDy, nonlinear XRO) score at most 0.49 on statistics, against 0.53 to 0.74 for the two best agent systems in every run.

The agents are not bound to a fixed structure. The task fixes the variables and the interface but not the functional form (Appendix A), and the agents choose and revise the form during a run. All six first-run models that exceed the reference start, like the baselines, from a regression of the data, then add their own nonlinear drift and state-dependent noise and revise them against their own diagnostics. In every run of the nine systems that exceed the reference at least once, the TET_E skewness is 0.53 to 1.45, against 0.03 for the seasonal LIM. Arriving at a model that is both well coupled and asymmetric within one six-hour run takes many modeling decisions, and this is what the established fitting procedures do not supply.

AuthorsOpenReview ↗

Response to Reviewer xRr6 (2/2)

[W3, Q3] Reconstruction on held-out years and uncertainty of forecast improvements.

Do the reconstruction gains persist on years excluded from model development? […]

We thank the reviewer for this suggestion, which tests whether the results hold beyond the years the agents saw. We reran the reconstruction on 2015 to 2024 for the final models of 31 of the 32 main runs, frozen as submitted, with anomalies taken about the period mean (table in Appendix F). Held-out scores stay at their in-sample level: over 31 runs the median change is -0.02 and the rank correlation 0.72. Of the 29 runs that exceed the reference in-sample, 25 also exceed it on the held-out years, 20 by more than the year-to-year sampling variability. The seasonal LIM scores 0.628 on the held-out years, level with the best agents.

The forecast score compares the root-mean-square error of the 50-member ensemble-mean forecast with that of persistence at leads of 1 to 12 months (Appendix C). Because consecutive forecasts overlap and the record holds few large events, the revision adds block resampling of the 108 initializations and the removal of each major event in turn (second table in Appendix F). With 24-month blocks, 27 of 31 runs beat persistence with 90% intervals above zero, and 29 still do with any single major event removed. The reference's own gain (0.18) is not significant, and the gain over the reference is positive in 25 runs but significant in only 11. The data therefore support that most agent models beat persistence and that the top systems beat the reference on average, but not a ranking among the top systems, and the revision says so.

We agree with the reviewer's reading: as the seasonal LIM shows, this dimension measures a stable dependence between variables, necessary but not sufficient for a mechanism.

[W4, Q5] Quantitative asymmetry criterion and independent seeds.

How is preservation of ENSO asymmetry quantified during simplification […]?

We thank the reviewer for this question, which makes the mechanism analysis quantitative. We agree that the reductions are post-hoc, and Section 5.1 now calls the analysis exploratory (Section 5.1, Appendix G.1). The reductions are partly our selection and show which family a model is compatible with, not which explanation is correct, and they were not re-evaluated on independent periods.

  1. The revision reports the measure and its threshold, and the classification is unchanged under this criterion (details in our reply to Bygb, [W1, Q1]).
  2. Rescored with five further seeds, every retained reduction keeps at least 95.5% of its composite. Repeated on all three runs, with scores averaged over the official seed and four further seeds, the family assignments recur (Appendix D).
  3. Leaving the least certain assignment (DeepSeek V4 Pro) unassigned changes the vote from 4 against 5 to 3 against 5, and no model changes family.

[W5, Q4] Validation of the diagnostic-freezing and submission policies.

A consistent policy in which agents select a final submission using their own diagnostics would provide a cleaner assessment. […]

We appreciate the reviewer raising both policies, which will shape the design of future runs.

Freezing: An agent that can edit its tests can loosen a failing check instead of fixing the model. We agree that the benefit is not isolated. A comparison with freezing disabled needs a second set of six-hour runs, which we will add for the camera-ready version, and the Limitations paragraph states this.

Submission rule: We agree that a rule in which the agent selects its own final model is cleaner, and we will use it in future runs. The best checkpoint can only be found with the hidden graders, so the main comparison uses the final checkpoint, which can only understate each run (in the first run, eight systems exceed the reference by best checkpoints and six by final). Over three runs, the rankings by mean final and mean best score have a Spearman correlation of 0.91. In the first run, letting each agent's frozen diagnostics choose among its checkpoints (Appendix B), four of the six suites that judge model quality choose the final one. DeepSeek V4.1 Flash chooses one scoring 0.451 (final 0.351) and Kimi K3 one scoring 0.470 (final 0.479), so seven systems instead of six exceed the reference. Other suites give no verdict, so future runs will ask for one.

We sincerely appreciate the reviewer's constructive feedback and are willing to address any additional feedback.

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer xRr6,

Thank you again for your detailed review. We hope our response and the revised PDF (changes in blue) have addressed your concerns. In particular, every system that can still be run now has three runs, the automated baselines you suggested are scored with the same graders, held-out reconstruction and forecast intervals are reported, and the asymmetry criterion is quantitative.

As the discussion period ends on November 18 (AoE), please let us know if any concern remains, and we will answer it before then. If our response has resolved your concerns, we would be grateful if you would consider updating your score.

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2)

We thank the reviewers for the careful and constructive comments spanning the repeatability and baselines of the benchmark (xRr6), the scope of its conclusions and matched comparisons (M1EL), and the quantification of asymmetry in the mechanism analysis (Bygb).

→ Strength: We appreciate that the reviewers see the benchmark as a direct, executable test of scientific models rather than of generated reports (xRr6, M1EL, Bygb), value the analysis of model structure beyond aggregate scores (M1EL, Bygb), and find the reporting transparent (xRr6).

→ Weakness: The reviews share four requests: repeated runs, held-out tests of reconstruction and forecasts, stronger automated baselines, and a quantitative asymmetry criterion. We address each with new experiments below. The revised PDF is uploaded, with all changes in blue.

[Repeated Runs]: We thank the reviewers for asking for repeated runs, which make the comparison between systems more reliable. Every system that can still be run under its original conditions was run twice more, which gives three independent six-hour runs per system, scored with the released graders (new Appendix D). The legacy models GPT-5.5 and Qwen3-Max are no longer served as before and keep their single runs. The anonymized-years run is repeated in the same way, and the literature and external-data conditions remain single runs. The table below (full version in Appendix D) shows the following insights:

  1. The two best agent systems are stable. GPT-6 Astra and Claude Fable 5.1 exceed the reference (0.378) with their final model in every run, and Fable varies least. MiniMax-M3, GLM-5.2 and Qwen3-Max stay below it in every run.
  2. The conclusions of the submission hold. Six systems exceeded the reference in the first run, and seven exceed it on their mean final score (GPT-5.5 from its single run). Four of them exceed it in two of three runs. The revision ranks the systems by their mean score with its spread, and Kimi K3, second in the first run, is ninth on its mean.
  3. The ranking does not depend on using the final checkpoint. The rankings by mean final and mean best score have a Spearman correlation of 0.91, and the best checkpoint is 0.04 higher on average.
  4. The asymmetry recurs. Every run of the nine systems that exceed the reference at least once keeps a TET_E skewness of 0.53 to 1.45 (observed 0.83, seasonal LIM 0.03). Fable's statistics score is 0.651 to 0.715 in every run, above every automated baseline, and the third run of GPT-6 Astra exceeds even the 5D model (0.742 against 0.727).
  5. With anonymized years, Fable scores 0.463, 0.334 and 0.425 (mean 0.407, against 0.489 offline). Two of three runs exceed the reference, but the lower mean means that recall of the dated record may contribute, and the revision says so.
SystemFinal mean ± sdBest mean ± sd
GPT-6 Astra0.506 ± 0.0710.525 ± 0.072
Claude Fable 5.10.489 ± 0.0230.500 ± 0.030
GPT-5.6-Sol0.404 ± 0.0440.436 ± 0.054
GPT-5.5 (single run)0.4030.404
DeepSeek V4.1 Flash0.396 ± 0.0420.450 ± 0.021
Claude Opus 50.395 ± 0.0420.425 ± 0.072
Gemini 3.8 Flash0.379 ± 0.0090.396 ± 0.004
DeepSeek V4 Pro0.362 ± 0.0750.394 ± 0.071
Kimi K30.348 ± 0.1180.424 ± 0.072
MiniMax-M30.248 ± 0.0490.311 ± 0.063
GLM-5.20.230 ± 0.0770.297 ± 0.067
Qwen3-Max (single run)0.0540.120
Fable 5.1, anonymized years0.407 ± 0.0660.439 ± 0.051

[Held-Out Reconstruction and Forecast Uncertainty]: Following xRr6, we reran the reconstruction on 2015 to 2024, which no agent saw, for the final models of 31 of the 32 main runs, and added intervals for the forecast gains (Appendix F). Held-out scores stay at their in-sample level (median change -0.02 over 31 runs), and 27 of 31 runs beat persistence with 90% intervals above zero. As xRr6 notes, this dimension measures a stable dependence between variables, necessary but not sufficient for a mechanism (details in our reply to xRr6, [W3, Q3]).

AuthorsOpenReview ↗

General Response (2/2)

[Conceptual and Empirical Baselines]: Our claim concerns conceptual models, low-dimensional stochastic models whose terms represent physical mechanisms. Besides the published 5D model, we now include the conceptual model XRO [1] and, following xRr6 and M1EL, empirical models: the stationary and seasonal linear inverse model (LIM) [2], a seasonal VAR(1), a LIM with correlated additive and multiplicative noise (CS-LIM) [3], and SINDy [4]. Their setup is the same as for the agents: they are fitted to the same 1980 to 2014 indices, submitted through the same interface and scored by the same graders, and none was tuned against a grader score (Appendix E).

The table below (full version in Appendix E) shows the following insights:

  1. Against conceptual models the advantage holds. Nine systems exceed the 5D model and six exceed XRO in at least one run, so knowing the recharge-oscillator paradigm is not enough to reach the top scores.
  2. A purely empirical model scores slightly higher on the composite. The seasonal LIM (0.518) is level with GPT-6 Astra (mean final 0.506, mean best 0.525). The stationary LIM scores 0.382, close to the 5D model (0.378), so this strength comes from the seasonal operator.
  3. The 5D model reproduces ENSO's statistics but not the coupling between variables. The seasonal LIM captures the coupling but, being linear and Gaussian, is nearly symmetric (eastern Pacific skewness 0.03 against the observed 0.83), and the automated baselines that reach the asymmetry (calibrated CS-LIM, SINDy, nonlinear XRO) score at most 0.49 on statistics, against 0.53 to 0.74 for the two best agent systems in every run. The best agent models have both, in every run ([Repeated Runs]).
ModelKindCompositeStatisticsDyn. consistencyPrediction
GPT-6 Astra, mean bestagent0.5250.7050.6860.270
Seasonal LIMempirical0.5180.5830.7070.328
Seasonal VAR(1)empirical0.5080.5500.6940.337
GPT-6 Astra, mean finalagent0.5060.6530.6790.266
Claude Fable 5.1, mean bestagent0.5000.6990.5980.276
Claude Fable 5.1, mean finalagent0.4890.6750.5970.270
XRO, nonlinearconceptual0.4360.3820.6880.287
SINDy, state-dependent noiseempirical0.3990.3390.6250.274
Stationary LIMempirical0.3820.2870.6240.271
5D referenceconceptual0.3780.7270.3230.157

[Asymmetry Criterion and Independent Seeds]: Following Bygb and xRr6, the revision states the asymmetry criterion (Appendix G.1): a model keeps the asymmetry if the observed TET_E skewness of 0.83 lies within the central 95% of its 70-year segment skewnesses. Every retained reduction meets it and every linear, additive reduction (skewness 0.01 to 0.06) fails it, so the two-family classification is unchanged, and the results hold when rescored with five further seeds (details in our reply to Bygb, [W1, Q1]).

[Paper Revision Updates]: The revised PDF marks all changes in blue.

  • Appendix D (new): three runs per system, final and best rankings, asymmetry and simplification across runs (xRr6, M1EL, Bygb).
  • Appendix E (new): automated baselines with scores, asymmetry and a structural comparison with the agent models (xRr6, M1EL).
  • Appendix F (new): held-out reconstruction and forecast uncertainty (xRr6).
  • Appendix G.1 (new): asymmetry criterion, skewness of every reduction, independent seeds (Bygb, xRr6).
  • Sections 4 and 5: agents placed against the baselines, what dynamical consistency measures, the mechanism analysis presented as exploratory (xRr6, M1EL, Bygb).
  • Abstract and Sections 3 and 6: following M1EL and Bygb, transferability narrowed to the execution structure, the anonymized-years run said to exclude recall of the dated record only, new Limitations paragraph.

We sincerely appreciate the reviewers' constructive feedback and are willing to address any additional feedback.

References

  1. Zhao, S., et al. "Explainable El Niño predictability from climate mode interactions." Nature, 2024.
  2. Penland, C., and Sardeshmukh, P. D. "The optimal growth of tropical sea surface temperature anomalies." Journal of Climate, 1995.
  3. Martinez-Villalobos, C., et al. "Calculating state-dependent noise in a linear inverse model framework." Journal of the Atmospheric Sciences, 2018.
  4. Brunton, S. L., Proctor, J. L., and Kutz, J. N. "Discovering governing equations from data by sparse identification of nonlinear dynamical systems." PNAS, 2016.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.