acceptodds
Under review as a conference paper at ICLR 2027

HazardCast: Benchmarking LLM Agents for Forecast Construction Across Five Natural Hazards

Abstract

Natural-hazard forecasting requires turning heterogeneous environmental observations into prediction procedures that respect task-specific targets, horizons, units, and information constraints. Large language model (LLM) agents can inspect data, write code, fit models, and revise workflows, but evaluating them requires more than measuring final prediction error. We introduce HazardCast, a benchmark of 24 forecasting questions spanning tropical cyclones, wildfire, heat, drought, and floods. Each question provides prepared environmental data and an explicit prediction contract while leaving feature construction and forecasting methodology to the agent. Evaluation combines native forecast losses, performance relative to a fitted local model, and checks of numerical and forecast-time validity. Across a 24-question comparison, selected agent workflows frequently improve on local models, but gains are uneven. Higher configured reasoning effort does not yield a uniform advantage, paired training conditions show substantial sensitivity to population design, and run-level audits identify concrete output transformations, prediction restrictions, and delivery failures. HazardCast therefore evaluates not only whether an agent produces accurate forecasts, but whether the procedure producing them is scientifically valid.

Primary area:
datasets and benchmarks
Keywords:
Large language models, scientific agents, natural hazard forecasting, environmental data, benchmark design, forecast verification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bzNqOpenReview ↗
Rating
2
Confidence
2(−2)
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces HazardCast, a benchmark for evaluating the ability of large language model agents to construct forecasting workflows across 24 tasks covering tropical cyclones, wildfire, heat, drought, and floods. Each task provides prepared environmental data and specifies the prediction target, forecast horizon, permissible information, and output requirements, while allowing agents to perform feature construction, model fitting, and prediction generation. The evaluation protocol combines task-specific forecast errors, normalized scores relative to fitted local baselines, and checks of numerical validity and compliance with forecast-time information constraints. The authors compare multiple agent configurations and examine how reasoning effort, forecast horizon, and training-data distributions affect forecasting performance. The paper also audits saved code, predictions, and execution records to analyze failure mechanisms involving output transformations, restrictive prediction rules, and delivery and feedback failures.

Critical strengths

The benchmark addresses a valuable research problem in automated natural-hazard forecasting. HazardCast organizes 24 tasks across five hazard families into explicit task specifications covering diverse prediction targets, forecast horizons, and output formats. This provides a concrete experimental setting for evaluating agents’ ability to construct predictive workflows across heterogeneous scientific tasks.

Critical weaknesses

The experimental design is flawed, limiting the credibility of the results. Different agents have access to different tools and information, making it difficult to attribute performance differences directly to their capabilities. Moreover, some results rely on later-issued forecasts or information that became available after the forecast origin, and therefore do not reflect performance achievable at the intended prediction time. Averaging these results together makes it difficult to determine the actual gains under valid forecasting conditions. I recommend standardizing tool and information access across agents and separately reporting results that use only information available at the time of prediction.

Questions for authors
  1. After excluding results that use later-issued forecasts or information available only after the forecast origin, what are the mean score and the number of tasks exceeding the local baseline for each agent?
  2. What are the specific differences in tool access, accessible data, and validation feedback across agents?
  3. Were multiple independent runs conducted for each experimental condition? If so, could you report the variability and clarify how the retained workflow was selected?
4 comments
AuthorsOpenReview ↗

Response to Weakness 1, Question 1

Weakness 1: Thank you for pointing this out, we agree that the original comparison confounded model capability with runner and information access. We have completed a new 24-task comparison of Codex+Astra, dsh+DeepSeek Flash and Qwen Code+Qwen3.8-Flash, all at High effort. Within each task, the 72 selected completions pass matching checks for starter-file hashes, task prompt, evaluator, replay inputs and scored cases. Shared controls are 60 active development minutes plus 90 replay minutes, 20 aggregate validation calls, 8 GiB, two numerical threads per process, local file/code tools with network access disabled for agent code, and a nominal 2,000-token tool-output allowance.

The frozen predictor is replayed separately at each forecast origin, with only that origin's permitted inputs and history. It cannot borrow a shorter-lead forecast or outcome from a later evaluation row. The revised Astra/DeepSeek/Qwen means are 52.76/51.40/49.44, compared with 55.53/52.97/53.15 originally. Astra remains highest, but DeepSeek now exceeds Qwen. The new scores exceed the historical local reference on 19/24, 14/24 and 14/24 tasks. Authorized replacements and failed attempts are disclosed; these are selected-completion results, not first-attempt success rates, and the changes cannot be attributed to isolation alone.

We also narrow the interpretation. These are agent-system comparisons under common recorded benchmark controls: native prompts, tool schemas and context handling still differ; Astra's account output ceiling is unverified and its context targets are inferred. DeepSeek and Qwen use verified 128,000-output-token requests and corresponding context settings. Isolation prevents cross-case lookahead and preserves the integrity of the controlled experiment.

TaskAstra HighDeepSeek HighQwen High
C142.4545.0345.59
C251.6249.8252.13
C355.0153.2652.90
C+147.8754.1154.28
C+250.5250.7550.89
W161.2948.9718.68
W251.8251.5551.07
W352.1453.0151.49
W+150.4044.5049.67
W+250.6050.2750.23
W+349.7050.2250.08
H151.9449.0146.01
H351.6352.1051.59
H+165.9364.2166.05
H+253.0152.5052.41
D150.0050.0050.00
D250.1449.9549.81
D351.4450.9953.66
D+147.5049.8546.82
D+261.6765.0565.46
F153.8651.1349.94
F253.6748.9139.27
F350.9747.8736.88
F+161.1450.5151.73
mean, all 2452.7651.4049.44

Question 1: Across all 24 tasks, the completed new origin-isolated runs give the following results:

SystemMean scoreTasks above the local baseline
Astra52.7619/24
DeepSeek51.4014/24
Qwen49.4414/24

Astra has the highest mean and exceeds the baseline on the most tasks.

AuthorsOpenReview ↗

Response to Question 2, 3

Question 2: In the new 24-task comparison, the shared capabilities are local file reading/writing, search and code execution in a restricted task workspace. The systems use the same prepared training inputs, labels, task prompt and numerical environment within each task. Agent code cannot access the network, private validation/test files, or another evaluation case. Scientific-document retrieval and specialist forecast services from the historical Full-tools condition are not supplied in this comparison. At inference, the evaluator passes the same permitted origin-specific inputs to each frozen predictor.

Validation access is also shared: at most 20 calls during the 60-minute development period, returning aggregate losses for the submitted candidate on fixed validation cases. It does not return per-case labels or test scores. Failed validation calls consume the allowance. Test replay has a separate 90-minute ceiling. The common resource settings are 8 GiB and two numerical threads per process, with a nominal 2,000-token tool-output allowance.

Equal access does not imply identical feedback histories. Across the 24 selected completed runs per system, the retained response files and native logs show:

SystemValidation callsNumerical responsesCalls per runRuns receiving numerical feedback
Astra1821792–1524/24
DeepSeek1641591–1624/24
Qwen43431–524/24

Astra and DeepSeek each had three runs with at least one nonnumerical response, but every selected run eventually received a numerical loss. For example, the repaired F+1 runs used eight, three and three calls for Astra, DeepSeek and Qwen. Different candidates naturally receive different losses, and agents decide how many evaluations to request.

The remaining differences are in the native runners: Codex, dsh and Qwen Code expose different tool names/schemas, system prompts and context management. All request High effort. DeepSeek and Qwen forward requests with 128,000 output tokens and verified contexts of 258,400 tokens. Astra uses account login; its output ceiling is unverified and its historical context targets are inferred. The tool-output limit is nominal, with runner-specific truncation implementations. We therefore claim shared benchmark access and budgets, not identical inference behavior.

Question 3: Thank you for pointing this out. We evaluated three separately developed DeepSeek Flash High workflows on each of five tasks, one per hazard family: C3, W+3, H+1, D+1 and F+1. Within task, inputs, prompt, evaluator, scored cases and recorded controls matched, including 60 minutes for development and 90 for replay. Across the three completions, mean ± sample SD scores were 53.29 ± 0.71 (C3), 50.52 ± 0.61 (W+3), 65.17 ± 1.10 (H+1), 49.61 ± 0.43 (D+1) and 50.85 ± 1.84 (F+1). C3, W+3 and H+1 exceeded the local reference of 50 in all three runs; D+1 did so in none and F+1 in two. The selected runs therefore show consistent above/below-reference outcomes on four tasks. Score variation is small on C3, W+3 and D+1 (SD 0.43–0.71), larger on the selected H+1 subset (1.10), and greatest on F+1 (1.84), where the above-reference conclusion changes between episodes.

Thank you again for your thoughtful review. We have posted detailed responses and uploaded the revised manuscript. When you have an opportunity, we would appreciate your assessment of whether these address your concerns.

The main updates relevant to your review are:

  • Tool and information access: Sections 3.1 and 4.1, together with Appendix C.5, describe the shared task inputs, tools, validation allowance and development/replay budgets, while explicitly identifying remaining native-harness differences.
  • Later-information use and aggregate results: Sections 3.1–3.3 explain forecast-origin replay and distinguish execution controls from input-timing validity. Section 4.2 and Table 2 report the updated three-system comparison, with all 24 task results in Appendix C.1.
  • Run variability and selection: Section 4.3 (RQ2c), Figure 2(c) and Appendix C.4 report the repeated-run results; Appendix C.5 describes the selection and reporting conventions.

We sincerely appreciate your time in considering the revised evidence.

Thank you for the detailed response and additional experiments. However, forecast-origin isolation does not fully guarantee that all input features were available at prediction time. I therefore retain some concerns about the scientific validity and interpretation of the results. I will maintain my overall rating of 2 while lowering my confidence from 4 to 2, reflecting both the improvements and the remaining uncertainty.

Reviewer_p31qOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces HazardCast, a benchmark of 24 forecasting questions across five natural-hazard families for evaluating large language model (LLM) agents to build forecasting workflows. HazardCast aims to assess both the accuracy of the forecasts and the scientific validity of the procedures that produce them. The experimental results show that agents often outperform fitted local models, although gains are uneven and sensitive to training-population design.

Critical strengths

The paper is well-presented and easy to follow.

The paper evaluates the scientific validity of agent-built forecasting procedures along with forecast accuracy, which ensures the task specifications and limitations are followed.

The proposed benchmark covers diverse hazards and tasks, and provides very detailed failure case analysis. This gives insights on LLM Agent design for hazard forecasting.

Critical weaknesses

The biggest concern is that the main comparisons are not controlled. The workflows differ in runner, capabilities and information access, and some headline results break the benchmark’s own forecast-time constraints (i.e. using later-issued forecasts). Since scientific validity is the paper’s central claim, there seems to be too many confounds of what the benchmark actually measures.

The related work lacks closely related agentic benchmarks such as ZephyrusBench[1] and TimeSeriesGym[2]. This makes it unclear how HazardCast’s validity checks differ from existing evaluations of agent-built forecasting workflows.


[1] Varambally, Sumanth, et al. "Zephyrus: An agentic framework for weather science." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Cai, Yifu, et al. "Timeseriesgym: A scalable benchmark for (time series) machine learning engineering agents." arXiv preprint arXiv:2505.13291 (2025).

Questions for authors

Please refer to the strengths and weaknesses. Additionally, figure 1 in the paper looks too dense, is it possible to add more graphical demonstrations?

5 comments
AuthorsOpenReview ↗

Response to Weakness 2

Weakness 2: Thank you for pointing us to ZephyrusBench and TimeSeriesGym. We compared their tasks and scoring procedures in Appendix E (Table 15). ZephyrusBench evaluates an agent’s answer to a weather question. An agent can call the supplied Stormer forecaster and use its output to identify regions at risk. TimeSeriesGym evaluates predictor construction on selected tasks. Its LANL earthquake challenge asks the agent to build a predictor and scores its estimate of time to fault failure. Its published tasks do not score forecasts of a hazard’s geographic extent or observed impacts.

HazardCast asks the agent to design a reusable predictor on all 24 tasks across five hazard families. ZephyrusBench supplies a forecaster for the agent to query, while TimeSeriesGym requires predictor construction on a subset of its engineering tasks. We evaluate the agent-designed predictor against a fitted local baseline model, using the same cases, forecast leads, inputs, and outcome definitions for both. The baseline is refit separately for each task and is not given the agent’s architecture, feature transformations, or learned parameters; it represents what can be achieved with a conventional task-specific model under the same information constraints. ZephyrusBench compares the accuracy of answers from different agent configurations. TimeSeriesGym uses challenge-specific references, including the predictor’s own performance before tuning in its volatility challenge. This comparison isolates whether the agent’s modeling choices improve forecast skill rather than merely reflecting differences in data or evaluation.

We score both hazard footprints, such as newly inundated areas, and observed consequences, such as next-day heatstroke transports. ZephyrusBench’s extreme-weather task scores the affected locations but has no corresponding health-impact target. TimeSeriesGym’s earthquake task scores when a fault fails, with no spatial footprint or impact target. We report the agent model’s performance, the baseline’s performance, and their paired difference on each task, so “improvement” means that the agent model makes more accurate forecasts than its matched local baseline on identical cases and leads. In ZephyrusBench, a gain can mean more weather questions answered correctly. In TimeSeriesGym’s volatility challenge, it means a reduction in percentage error after hyperparameter tuning. We also retain the full modeling procedure for inspection, including the selected inputs, transformations, architecture, and training decisions. ZephyrusBench inspects tool-use traces, and selected TimeSeriesGym tasks require agents to save code and models before and after tuning.

BenchmarkForecastingMulti-hazard coverageAgent builds predictorHazard footprints and impacts
[ZephyrusBench]◐◐—◐
[TimeSeriesGym]◐◐◐—
[WeatherBench 2]✓——◐
[AtmosSci-Bench]————
[Climate-Agent-Bench-85]◐✓—◐
[MLE-bench]◐◐✓—
[ScienceAgentBench]—◐◐—
[PaperBench]——◐—
[AstaBench]——◐—
HazardCast (revised)✓✓✓✓
AuthorsOpenReview ↗

Response to Question1

Question 1:Thank you for this suggestion. We redesigned Figure 1 as a compact, graphical overview of three stages: data preparation, forecaster construction, and evaluation. Hazard and data-source icons, a feature matrix, an illustrative predictor tree, and a forecast-origin timeline replace text-heavy descriptions. Arrows show the validation-feedback loop and the transition from model development to a frozen predictor; the evaluation panel separates origin inputs from outcomes available only to the scorer. We also shortened labels, aligned the boxes and connectors, and reduced internal padding. These changes make the workflow and information flow easier to follow while retaining the five hazard families and all 24 tasks.

AuthorsOpenReview ↗

Response to Weakness 1

Weakness 1: Thank you for pointing this out, we agree that the original comparison confounded model capability with runner and information access. We have completed a new 24-task comparison of Codex+Astra, dsh+DeepSeek Flash and Qwen Code+Qwen3.8-Flash, all at High effort. Within each task, the 72 selected completions pass matching checks for starter-file hashes, task prompt, evaluator, replay inputs and scored cases. Shared controls are 60 active development minutes plus 90 replay minutes, 20 aggregate validation calls, 8 GiB, two numerical threads per process, local file/code tools with network access disabled for agent code, and a nominal 2,000-token tool-output allowance.

The frozen predictor is replayed separately at each forecast origin, with only that origin's permitted inputs and history. It cannot borrow a shorter-lead forecast or outcome from a later evaluation row. The revised Astra/DeepSeek/Qwen means are 52.76/51.40/49.44, compared with 55.53/52.97/53.15 originally. Astra remains highest, but DeepSeek now exceeds Qwen. The new scores exceed the historical local reference on 19/24, 14/24 and 14/24 tasks. Authorized replacements and failed attempts are disclosed; these are selected-completion results, not first-attempt success rates, and the changes cannot be attributed to isolation alone.

We also narrow the interpretation. These are agent-system comparisons under common recorded benchmark controls: native prompts, tool schemas and context handling still differ; Astra's account output ceiling is unverified and its context targets are inferred. DeepSeek and Qwen use verified 128,000-output-token requests and corresponding context settings. Isolation prevents cross-case lookahead and preserves the integrity of the controlled experiment.

TaskAstra HighDeepSeek HighQwen High
C142.4545.0345.59
C251.6249.8252.13
C355.0153.2652.90
C+147.8754.1154.28
C+250.5250.7550.89
W161.2948.9718.68
W251.8251.5551.07
W352.1453.0151.49
W+150.4044.5049.67
W+250.6050.2750.23
W+349.7050.2250.08
H151.9449.0146.01
H351.6352.1051.59
H+165.9364.2166.05
H+253.0152.5052.41
D150.0050.0050.00
D250.1449.9549.81
D351.4450.9953.66
D+147.5049.8546.82
D+261.6765.0565.46
F153.8651.1349.94
F253.6748.9139.27
F350.9747.8736.88
F+161.1450.5151.73
mean, all 2452.7651.4049.44
Reviewer_wgGXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HazardCast evaluates whether LLM agents can construct forecasting workflows across 24 tasks covering tropical cyclones, wildfire, heat and associated impacts, drought, and floods. Agents receive prepared data and task specifications, choose features and methods, and implement predictors. Evaluation combines native forecast losses, scores relative to fitted local models, and checks of numerical and forecast-time validity. Experiments compare three agent configurations, DeepSeek reasoning-effort settings, forecast horizons, and seven paired training conditions. Selected workflows often outperform local models, while the audit identifies harmful output mappings, restricted predictions, later-information use, and tool failures.

Critical strengths
  1. The tasks specify targets, leads, units, chronological splits and scoring rules while allowing agents to choose their methods. The distinction between forecast cases and scoring units is appropriate: many pixels or rows do not constitute independent events. This provides a useful framework for evaluating forecast construction across different hazards (Section 2; Appendices A, B and D).
  2. The audit directly tests some failure mechanisms. In W3, removing only final class weights, with the fitted model and 630 test cases unchanged, reduces Basic's error from 0.4413 to 0.3667 (16.9%). D3 High's single probability per year and F+1 High/Max's exact specialist fallback are also informative diagnoses. The authors distinguish these findings from hypotheses based on residual patterns (Section 4.5; Tables 4 and 12).
Critical weaknesses
  1. Headline averages include runs with known later-information use or unresolved timing. DeepSeek Basic C1/C2 uses later-issued forecasts, and W+1 Basic uses later-event values (Section 4.2; Tables 8 and 12). This also affects the effort comparison: H1 Low scores 72.79 but carries a timing qualification in Table 12 (p. 23). Excluding H1 for all three settings changes the Low/High/Max means from 52.67/51.99/52.06 to approximately 51.79/52.08/52.13, calculated from rounded Table 10 values. This sensitivity check does not establish Max's superiority; H3 High/Max and W+1 High also need resolution. Report valid, invalid and unresolved outcomes separately, then compare scores on common tasks with matched permitted access. Past-only or same-time context should be judged against the task policy.
  2. The PDF does not fully specify the fitted local models: algorithms, features, hyperparameters, model selection and final fitting. Table 5's simple references do not identify the denominators in Equation 1. I also found no link to a reproducibility package with task definitions, data preparation, scorers and baseline recipes. Separate supplementary materials could address this concern. RQ3 also needs training sample sizes, class frequencies and accompanying workflow changes; six comparisons use selected evaluation cohorts. Together with the unresolved validity results, these gaps motivate weak rejection. I would reconsider after receiving reproducible baseline specifications and summaries that separate validity statuses.
Questions for authors
  1. Which runs are valid, invalid or unresolved at the original forecast issue time? How do Tables 2 and 9 change on a common valid task set, particularly after resolving H1 Low, H3 High/Max and known later-information use?
  2. Where are the complete task contracts, feature-availability rules, data versions, scorers and fitted-local-model recipes? Do the supplementary materials include prompts, fitting permissions and the imported Qwen configurations?
  3. For each RQ3 pair, what changed between balanced and natural training: sample size, class frequencies, features, estimator, settings, budget or feedback? Which conditions were held fixed?
  4. Which tasks and configurations account for the 281 validation attempts without numerical responses across 16 DeepSeek runs (Table 4)? What fraction of each comparison group was affected?
  5. Robustness check: are independent episode repeats and uncertainty estimates available that respect event, map or temporal dependence? If not, a few repeats on tasks with verified timing could test the effort and training-population conclusions and affect my recommendation.
5 comments
AuthorsOpenReview ↗

reply to Weakness 1 and 2

Weakness 1: Thank you for identifying this problem. We reran all 24 tasks with paired inputs and common benchmark controls. During development, agents receive training data and aggregate validation feedback. Test replay freezes each predictor and supplies one forecast origin at a time, retaining only task-permitted history and same-time context. This blocks access to later evaluation rows.

For RQ1, Astra retains the highest mean: Astra/DeepSeek/Qwen score 52.76/51.40/49.44, versus 55.53/52.97/53.15 originally—decreases of 2.77/1.57/3.71 points. Thus, Astra's leading position is preserved, but DeepSeek overtakes Qwen and the original magnitude of gains is not preserved. The details are included in the paper.

TaskAstra HighDeepSeek HighQwen High
C142.4545.0345.59
C251.6249.8252.13
C355.0153.2652.90
C+147.8754.1154.28
C+250.5250.7550.89
W161.2948.9718.68
W251.8251.5551.07
W352.1453.0151.49
W+150.4044.5049.67
W+250.6050.2750.23
W+349.7050.2250.08
H151.9449.0146.01
H351.6352.1051.59
H+165.9364.2166.05
H+253.0152.5052.41
D150.0050.0050.00
D250.1449.9549.81
D351.4450.9953.66
D+147.5049.8546.82
D+261.6765.0565.46
F153.8651.1349.94
F253.6748.9139.27
F350.9747.8736.88
F+161.1450.5151.73
mean, all 2452.7651.4049.44
For RQ2a, nine isolated Basic-tools runs on H1, H3 and W+1 give Low/High/Max means of 49.59/50.11/49.74. High leads on H1 and H3; Low leads on W+1. The qualitative finding remains that more reasoning effort does not consistently improve performance, although High now has the largest mean.
TaskLowHighMax
------:---:---:
H148.2850.4649.28
H350.6751.2950.99
W+149.8148.5748.95
Three-task mean49.5950.1149.74

Weakness 2: Thank you for specifying the missing details. The repository now contains both September and November task releases, real prepared data, baseline-fitting recipes, native scorers and pinned dependencies. The recipes record the features, candidate estimators and hyperparameters, selection rules and final fitting procedure. Equation 1 uses each selected fitted model's loss on the same cases, weights and lead; Table 5's simple references are separate.

Fresh fits reproduced all 24 historical local baselines within tolerance, and all 24 native scorers passed. We also reproduced 96 historical DeepSeek main-table scores and seven natural-training scores. In addition to these full-suite checks, we tested the published download in an independent GitHub checkout: D2 verified baseline fitting from downloaded data, and C1 verified scoring of saved paper predictions. The README supplies executable commands and the repository records the verification results.

For RQ3, we compared balanced and natural training pools on seven tasks. Balanced pools sampled classes equally for W1, W3 and D+1, paired events with controls for W+2, F3 and F+1, and balanced five flow regimes for F1. Natural pools retained all eligible training examples. Both workflows were scored on the same cases within each task. Six tasks used selected evaluation cohorts; F1 used all eligible test gauge-days. We now report the sample sizes and class frequencies alongside these selection rules.

AuthorsOpenReview ↗

Response to Question 1,2,3,4

Question 1: The reruns use origin-isolated test replay: a frozen predictor receives only the permitted inputs for the current forecast origin, rather than the full evaluation batch. The replay checks found no cross-origin borrowing in the examined cases. This addresses the historical failure mode of using a later test row or its outcome to predict an earlier origin. We distinguish that execution result from the timing of information already embedded in the supplied data. Appendix C.1 records those source-level findings separately.

Change to Table 2. Across all 24 tasks, the Astra/DeepSeek/Qwen means change from 55.53/52.97/53.15 to 52.76/51.40/49.44, decreases of 2.77/1.57/3.71 points. Astra remains the highest-scoring system, while DeepSeek moves ahead of Qwen. Thus, the leading system is unchanged, but the ordering of the other two systems and the size of the gains change. We also report the common four-task observation-cutoff subset—C3, H3, H+2 and D+1—separately:

Change to Table 9. We replace the headline historical 24-task Full-tools effort summary with a separate, fully rerun three-task isolated Basic-tools study: nine runs covering Low, High and Max on H1, H3 and W+1. The original Low/High/Max means were 52.67/51.99/52.06; the new three-task means are 49.59/50.11/49.74. These aggregates differ in both task coverage and tool access, so their numerical differences are not estimates of the effect of isolation. Low had the highest historical mean; High has the highest new mean, leading on H1 and H3, while Low leads on W+1. The supported qualitative conclusion remains that increasing reasoning effort does not consistently improve performance. We do not substitute the new cells into the old Full-tools average.

Question 2: Thank you for pointing this out, We have included the relevant informations about contracts, availability rules, data versions, scorers and agent configurations in the supplementary materials.

Question 3: Thank you for asking how each pair was constructed. We selected the balanced training examples before the agent began. W1, W3 and D+1 used random undersampling within outcome classes. W+2 paired discovery days with comparable days without a discovery, F3 paired flood crossings with similar noncrossings at the same gauge, and F+1 matched dry controls by nearby water cover. F1 balanced five regimes of observed streamflow and kept discharge as the continuous target. The natural condition supplied every eligible training example from the same source population. Appendix C.4 gives the selection rule and counts for each task (Table 12) and the resulting predictors, settings and validation feedback (Table 13).

Within each pair we fixed the ordered input columns, DeepSeek Flash Max Basic configuration, nominal budget, and cases, leads, loss and local reference used for scoring. The agent developed a fresh predictor in each condition, so its features, estimator, calibration and feedback history could change. The prompts also differed in their description of the evaluation population, and W+2 differed in access to evaluation cases. The seven pairs therefore measure the workflows produced under each condition. The separate D+1 check holds the estimator fixed while training size and prevalence change together.

Question 4: Thank you for pointing this out. We traced the 281 requests to W+2, H+1, F3 and F+1. On each of these four tasks, none of the four historical DeepSeek configurations received a numerical validation loss. The counts were 65 requests for Max Basic, 64 for Low Full, 95 for High Full and 57 for Max Full. Thus 4 of 24 selected runs (16.7%) in each group received no numerical feedback despite making requests.

A broader definition counts any run with at least one unsuccessful request, including runs that also received losses. Those fractions were 6/24, 9/24, 8/24 and 11/24, in the same order. H3 Max Full belongs to this second group: two invalid artifact paths and ten numerical responses. The logs establish which feedback reached each agent. They do not isolate how much the missing feedback changed forecast accuracy.

AuthorsOpenReview ↗

Response to Question 5

Question 5: Thank you for pointing this out. We evaluated three separately developed DeepSeek Flash High workflows on each of five tasks, one per hazard family: C3, W+3, H+1, D+1 and F+1. Within task, inputs, prompt, evaluator, scored cases and recorded controls matched, including 60 minutes for development and 90 for replay. Across the three completions, mean ± sample SD scores were 53.29 ± 0.71 (C3), 50.52 ± 0.61 (W+3), 65.17 ± 1.10 (H+1), 49.61 ± 0.43 (D+1) and 50.85 ± 1.84 (F+1). C3, W+3 and H+1 exceeded the local reference of 50 in all three runs; D+1 did so in none and F+1 in two. The selected runs therefore show consistent above/below-reference outcomes on four tasks. Score variation is small on C3, W+3 and D+1 (SD 0.43–0.71), larger on the selected H+1 subset (1.10), and greatest on F+1 (1.84), where the above-reference conclusion changes between episodes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.