- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SciExam for ENSO evaluates whether AI agents can build scientific models when no unique correct equation is available. Agents construct low-dimensional stochastic models of El Niño–Southern Oscillation (ENSO) from real observations. Within a shared six-hour budget, they process the data, develop diagnostic scripts that are subsequently frozen, and use those diagnostics to guide model development. Hidden graders evaluate long-term statistics, reconstruction of unobserved variables, and forecasting on held-out years.
Each of twelve agent systems is run once. Six final models exceed a published reference model under the same composite evaluation. Post-hoc simplification connects stronger models to two established explanations of ENSO warm–cold asymmetry. Additional runs examine how available information affects one system. These results demonstrate competitive model instances, while reliable success rates and the underlying ENSO mechanism remain unresolved.
Critical strengths
-
The benchmark makes open-ended scientific modeling directly testable.
Agents submit executable dynamical models evaluated against observations. Long simulations, variable reconstruction, and held-out forecasting test complementary aspects of the same equations and parameters. Data-processing checks, frozen diagnostics, and hidden scores provide a concrete workflow for sustained autonomous modeling. This design evaluates the behavior of scientific outputs beyond task completion. (§3; Appendices A and C)
-
The experiments produce competitive models and examine the structures behind their performance.
Six final models outperform the published reference, mainly through reconstruction and forecasting. The strongest model also approaches the reference in statistical fidelity. (§4; Figure 3b; Table 3) The reported weight-sensitivity analysis retains the same six systems above the reference under equal weighting. (Appendix C, p. 16) Structural simplification further identifies reduced models compatible with established ENSO explanations. (§5.1; Figure 5; Table 3; Appendix G) Together, these results provide useful model instances and interpretable evidence beyond aggregate scores.
Critical weaknesses
-
The scores can underrepresent the loss of an important physical feature, while the additional selection criterion is not fully quantified.
Appendix G reports that further simplifying GPT-6 Astra and the two DeepSeek models to linear drift with additive noise preserves at least 95% of the composite score and 90% of each component score, despite producing nearly symmetric temperature distributions and reduced event diversity. Multiplicative noise refers to random forcing whose strength depends on the system state. The authors appropriately reject these reductions through an additional temperature-distribution check.
Although the authors compare simulated and observed distributions, they do not specify a quantitative asymmetry measure or an explicit threshold for rejecting a reduction as “nearly symmetric.” Nor do they systematically report asymmetry measures for retained and rejected reductions. It therefore remains unclear how sensitively the scores reflect this important feature and how robust the two-family classification is to the additional criterion. This limits the interpretation of both the composite scores and the structural analysis. (Appendix G, pp. 22–23)
-
Single runs limit conclusions about repeatability and the effects of information access.
Each agent system is run once, and each information condition is tested once on one system. The authors explicitly frame these comparisons as model instances and a case study, which is appropriate. Nevertheless, the experiments cannot estimate how reliably systems exceed the reference or separate information-condition effects from variation between agent runs.
The competitive result with anonymized years is useful evidence, but it does not by itself exclude other contributions from pretrained knowledge of the observations. These limitations do not invalidate the submitted models, but they constrain conclusions about reliable modeling ability and the effects of information access. (§5.2; Appendices B.1 and F)
Questions for authors
- Asymmetry criterion: How was a temperature distribution judged to be “nearly symmetric” in Appendix G? Please specify the measure and threshold, and report the corresponding values for the original, retained, and rejected reductions of the three models discussed above. Does the two-family classification remain unchanged under an explicit quantitative criterion?
- Repeatability: Have representative systems and information conditions been evaluated across repeated runs? If so, please report the variation in final scores, the frequency of exceeding the reference, and whether the modeling-strategy changes described in §5.2 recur. This would help distinguish repeatable effects from differences between individual runs.
4 comments
Looking forward to your feedback
Dear Reviewer Bygb,
Thank you again for your positive and constructive review. We hope our response and the revised PDF (changes in blue) have addressed your questions. In particular, the paper now states the asymmetry measure, threshold and values, under which the two-family classification is unchanged, and reports three runs of every system that can still be run and of the anonymized-years condition.
As the discussion period ends on November 18 (AoE), please let us know if any question remains, and we will answer it before then. If you find that the revision strengthens the paper, we would be grateful if you would take it into account in your final assessment.
Best regards, The Authors
Thank you for your detailed responses and the revisions. I appreciate the additional experiments and clarifications regarding the asymmetry criterion and repeatability. I find your responses to my original concerns satisfactory.
I have a few additional comments regarding the scope and novelty of the benchmark:
-
Limited Benchmark Scope: While I appreciate the depth of the ENSO modeling task, the benchmark currently contains only a single scientific problem. For a benchmark intended to evaluate scientific modeling capabilities, I find this scope somewhat limited.
-
Novelty of the Evaluation Framework: I recognize the value of evaluating open-ended scientific models through executable, outcome-based verification. However, similar evaluation paradigms, in which agents produce executable artifacts assessed by task-specific verifiers, already exist in benchmarks such as KernelBench and Terminal-Bench-Science, as well as infrastructure such as Harbor. While the ENSO-specific scientific evaluation is valuable, I would appreciate further clarification on how the proposed evaluation framework differs methodologically from these existing approaches.
-
Self-generated and Frozen Diagnostics: I consider this a potentially more distinctive methodological contribution. Having agents develop their own diagnostic criteria and freeze them before model development is an interesting design choice. However, its incremental value has not been established through controlled ablations, such as comparing self-generated versus predefined diagnostics, or frozen versus modifiable diagnostics. Such comparisons would help substantiate this aspect of the framework's novelty.
Overall, I view SciExam as a carefully designed scientific benchmark task that supports open-ended modeling and provides meaningful scientific evaluation. However, given its limited task diversity and the remaining questions about methodological novelty, I am inclined to maintain my current rating of 3.
Response to the additional comments on scope, novelty and frozen diagnostics
We thank the reviewer for reading our responses and for the further comments. We are glad that our responses resolved the original concerns, and we answer the three new points in turn.
-
Benchmark scope: We chose to focus on a single problem for two reasons. First, depth in one problem can change practice in that field. ENSO is the dominant mode of interannual climate variability and one of the most active topics in climate science, and low-order conceptual models remain a central tool for studying and forecasting it. Researchers still build these models through hand calibration or conventional fitting. In SciExam, agents build them from raw observations in six hours, which is long for a benchmark run but short for a scientist, and the best of them exceed a published model under the same tests. For a field this active, this substantially changes how researchers can build and test such models. Second, we believe that in AI for science depth should come first. Benchmarks built for breadth suit coding, where many tasks have checkable answers, but scientific discovery usually comes from sustained work on one problem, and a shallow test cannot tell a valid model from a good fit. Depth allows tests beyond the score: repeated runs of every system, a published expert model and automated baselines under the same graders, held-out years, and analysis of the structure of the agent models. The protocol does not depend on ENSO, and our next step is to apply it with experts in other fields, so that the benchmark grows broader while keeping this depth.
-
Novelty of the evaluation framework: SciExam indeed shares the infrastructure of KernelBench and Terminal-Bench-Science, in which agents produce executable artifacts that task-specific verifiers check, and we regard this as a strength. A good benchmark should measure different content within a common framework, so that researchers can run and compare it alongside existing ones on harnesses such as Harbor. SciExam differs in what it measures. In both of those benchmarks, the task authors fix a correct answer before the agent starts. SciExam has no such answer. Its graders test, against observations, whether a model behaves as a valid model of ENSO must, and compare it with a published expert model under the same tests. We have added these benchmarks to the related work with this distinction (Appendix K).
-
Self-written and frozen diagnostics: We appreciate that the reviewer sees this design as a distinctive part of the framework. We arrived at it during development, where we tried the alternatives in turn. Without diagnostics, agents had no way to judge a revision, and over long runs they stalled, making small edits that did not improve the model while the cost kept rising. With diagnostics that the agent could keep revising, the same pattern returned, as agents spent much of the budget rewriting checks and files rather than the model (Appendix B). A single run could then cost several hundred US dollars, so a benchmark of twelve systems with repeated runs would cost thousands of dollars per round. Freezing the diagnostics before model development largely removed this pattern. Under the final protocol a run costs between 2 and 132 US dollars (Appendix J), and the frozen diagnostics carry information about model quality. When we reran each frozen suite on every saved checkpoint of the first run (Appendix B), six suites ranked checkpoints by their own verdicts, four of them selected the final checkpoint, and selecting by them put seven systems above the reference instead of six. As the Limitations paragraph states, we leave a controlled comparison with a variant that permits revisions to future versions.
We sincerely appreciate the reviewer's engagement throughout the discussion and are happy to answer any further questions.
Response to Reviewer Bygb
We sincerely appreciate reviewer
Bygbfor the positive comments on our contributions, especially for recognizing that the benchmark makes open-ended scientific modeling directly testable and that the structural simplification gives interpretable evidence beyond aggregate scores.[W1, Q1] Quantitative asymmetry measure, threshold and robustness of the two-family classification.
We thank the reviewer for this suggestion, which turns a qualitative judgement into an explicit criterion. The submission computed the asymmetry of each reduction but reported neither the measure nor the threshold. The revision states both and reports the values for every reduction (Appendix G.1).
Measure: the skewness of simulated TE in each of 30 free-run segments of 70 years, reported as mean ± sd (observed 0.83, 95% block-bootstrap interval [0.28, 1.20]). Criterion: a model keeps the asymmetry if the observed value lies within the central 95% of its segment skewnesses. A reduction is retained if it passes the score screen and this criterion.
Classification: Every retained reduction meets the criterion and every linear-drift, additive-noise reduction fails it, so the vote is unchanged (4 linear + multiplicative, 5 nonlinear + additive).
Sensitivity: The linear + additive reductions of GPT-6 Astra and both DeepSeek models pass the score screen, while their skewness falls from 0.92, 1.45 and 0.95 to 0.06 or below. The composite is therefore not a sufficient check of asymmetry, and the revised Section 5.1 reports skewness alongside the scores.
Seeds: Rescored with five further seeds, every retained reduction keeps at least 95.5% of its composite. The repeated-run simplification averages the official seed and four further seeds (Appendix D).
[W2, Q2] Repeated runs across systems and information conditions.
We thank the reviewer for this suggestion, which shows that the main findings recur across runs. All repeats have finished: three runs per system and for the anonymized-years condition (General Response, [Repeated Runs], Appendix D). GPT-6 Astra and Claude Fable 5.1 exceed the reference in every run, and the asymmetry of the two best agent systems recurs. Repeating the simplification (Appendix D), GPT-5.6-Sol, Opus 5 and Gemini 3.8 Flash reduce to nonlinear drift with additive noise in all three runs. Claude Fable 5.1 again replaces its initial recharge oscillator with a fitted dense operator in both offline repeats. Of the seven systems with more than one classified run, six never fall into the other family, although GPT-6 Astra, Claude Fable 5.1 and DeepSeek V4.1 Flash are each unassigned in one run. DeepSeek V4 Pro changes family in its one repeat above the analysis threshold, so its assignment describes a single run. The two web-access conditions remain single runs.
[W2] Contributions of pretrained knowledge beyond recall of the dated record.
We thank the reviewer for this precise point, and we agree. The anonymized run excludes only recall of the dated record, and over three runs it scores lower than offline (0.407 against 0.489), so Section 5.2 no longer rules out a contribution from recall. Other contributions, such as familiarity with the ENSO literature, are not excluded either, as Section 5.2 says.
We sincerely appreciate the reviewer's feedback and are willing to address any further comments.