- Rating
- 2
- Confidence
- 2(−2)
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces HazardCast, a benchmark for evaluating the ability of large language model agents to construct forecasting workflows across 24 tasks covering tropical cyclones, wildfire, heat, drought, and floods. Each task provides prepared environmental data and specifies the prediction target, forecast horizon, permissible information, and output requirements, while allowing agents to perform feature construction, model fitting, and prediction generation. The evaluation protocol combines task-specific forecast errors, normalized scores relative to fitted local baselines, and checks of numerical validity and compliance with forecast-time information constraints. The authors compare multiple agent configurations and examine how reasoning effort, forecast horizon, and training-data distributions affect forecasting performance. The paper also audits saved code, predictions, and execution records to analyze failure mechanisms involving output transformations, restrictive prediction rules, and delivery and feedback failures.
Critical strengths
The benchmark addresses a valuable research problem in automated natural-hazard forecasting. HazardCast organizes 24 tasks across five hazard families into explicit task specifications covering diverse prediction targets, forecast horizons, and output formats. This provides a concrete experimental setting for evaluating agents’ ability to construct predictive workflows across heterogeneous scientific tasks.
Critical weaknesses
The experimental design is flawed, limiting the credibility of the results. Different agents have access to different tools and information, making it difficult to attribute performance differences directly to their capabilities. Moreover, some results rely on later-issued forecasts or information that became available after the forecast origin, and therefore do not reflect performance achievable at the intended prediction time. Averaging these results together makes it difficult to determine the actual gains under valid forecasting conditions. I recommend standardizing tool and information access across agents and separately reporting results that use only information available at the time of prediction.
Questions for authors
- After excluding results that use later-issued forecasts or information available only after the forecast origin, what are the mean score and the number of tasks exceeding the local baseline for each agent?
- What are the specific differences in tool access, accessible data, and validation feedback across agents?
- Were multiple independent runs conducted for each experimental condition? If so, could you report the variability and clarify how the retained workflow was selected?
4 comments
Response to Question 2, 3
Question 2: In the new 24-task comparison, the shared capabilities are local file reading/writing, search and code execution in a restricted task workspace. The systems use the same prepared training inputs, labels, task prompt and numerical environment within each task. Agent code cannot access the network, private validation/test files, or another evaluation case. Scientific-document retrieval and specialist forecast services from the historical Full-tools condition are not supplied in this comparison. At inference, the evaluator passes the same permitted origin-specific inputs to each frozen predictor.
Validation access is also shared: at most 20 calls during the 60-minute development period, returning aggregate losses for the submitted candidate on fixed validation cases. It does not return per-case labels or test scores. Failed validation calls consume the allowance. Test replay has a separate 90-minute ceiling. The common resource settings are 8 GiB and two numerical threads per process, with a nominal 2,000-token tool-output allowance.
Equal access does not imply identical feedback histories. Across the 24 selected completed runs per system, the retained response files and native logs show:
| System | Validation calls | Numerical responses | Calls per run | Runs receiving numerical feedback |
|---|---|---|---|---|
| Astra | 182 | 179 | 2–15 | 24/24 |
| DeepSeek | 164 | 159 | 1–16 | 24/24 |
| Qwen | 43 | 43 | 1–5 | 24/24 |
Astra and DeepSeek each had three runs with at least one nonnumerical response, but every selected run eventually received a numerical loss. For example, the repaired F+1 runs used eight, three and three calls for Astra, DeepSeek and Qwen. Different candidates naturally receive different losses, and agents decide how many evaluations to request.
The remaining differences are in the native runners: Codex, dsh and Qwen Code expose different tool names/schemas, system prompts and context management. All request High effort. DeepSeek and Qwen forward requests with 128,000 output tokens and verified contexts of 258,400 tokens. Astra uses account login; its output ceiling is unverified and its historical context targets are inferred. The tool-output limit is nominal, with runner-specific truncation implementations. We therefore claim shared benchmark access and budgets, not identical inference behavior.
Question 3: Thank you for pointing this out. We evaluated three separately developed DeepSeek Flash High workflows on each of five tasks, one per hazard family: C3, W+3, H+1, D+1 and F+1. Within task, inputs, prompt, evaluator, scored cases and recorded controls matched, including 60 minutes for development and 90 for replay. Across the three completions, mean ± sample SD scores were 53.29 ± 0.71 (C3), 50.52 ± 0.61 (W+3), 65.17 ± 1.10 (H+1), 49.61 ± 0.43 (D+1) and 50.85 ± 1.84 (F+1). C3, W+3 and H+1 exceeded the local reference of 50 in all three runs; D+1 did so in none and F+1 in two. The selected runs therefore show consistent above/below-reference outcomes on four tasks. Score variation is small on C3, W+3 and D+1 (SD 0.43–0.71), larger on the selected H+1 subset (1.10), and greatest on F+1 (1.84), where the above-reference conclusion changes between episodes.
Thank you again for your thoughtful review. We have posted detailed responses and uploaded the revised manuscript. When you have an opportunity, we would appreciate your assessment of whether these address your concerns.
The main updates relevant to your review are:
- Tool and information access: Sections 3.1 and 4.1, together with Appendix C.5, describe the shared task inputs, tools, validation allowance and development/replay budgets, while explicitly identifying remaining native-harness differences.
- Later-information use and aggregate results: Sections 3.1–3.3 explain forecast-origin replay and distinguish execution controls from input-timing validity. Section 4.2 and Table 2 report the updated three-system comparison, with all 24 task results in Appendix C.1.
- Run variability and selection: Section 4.3 (RQ2c), Figure 2(c) and Appendix C.4 report the repeated-run results; Appendix C.5 describes the selection and reporting conventions.
We sincerely appreciate your time in considering the revised evidence.
Thank you for the detailed response and additional experiments. However, forecast-origin isolation does not fully guarantee that all input features were available at prediction time. I therefore retain some concerns about the scientific validity and interpretation of the results. I will maintain my overall rating of 2 while lowering my confidence from 4 to 2, reflecting both the improvements and the remaining uncertainty.
Response to Weakness 1, Question 1
Weakness 1: Thank you for pointing this out, we agree that the original comparison confounded model capability with runner and information access. We have completed a new 24-task comparison of Codex+Astra, dsh+DeepSeek Flash and Qwen Code+Qwen3.8-Flash, all at High effort. Within each task, the 72 selected completions pass matching checks for starter-file hashes, task prompt, evaluator, replay inputs and scored cases. Shared controls are 60 active development minutes plus 90 replay minutes, 20 aggregate validation calls, 8 GiB, two numerical threads per process, local file/code tools with network access disabled for agent code, and a nominal 2,000-token tool-output allowance.
The frozen predictor is replayed separately at each forecast origin, with only that origin's permitted inputs and history. It cannot borrow a shorter-lead forecast or outcome from a later evaluation row. The revised Astra/DeepSeek/Qwen means are 52.76/51.40/49.44, compared with 55.53/52.97/53.15 originally. Astra remains highest, but DeepSeek now exceeds Qwen. The new scores exceed the historical local reference on 19/24, 14/24 and 14/24 tasks. Authorized replacements and failed attempts are disclosed; these are selected-completion results, not first-attempt success rates, and the changes cannot be attributed to isolation alone.
We also narrow the interpretation. These are agent-system comparisons under common recorded benchmark controls: native prompts, tool schemas and context handling still differ; Astra's account output ceiling is unverified and its context targets are inferred. DeepSeek and Qwen use verified 128,000-output-token requests and corresponding context settings. Isolation prevents cross-case lookahead and preserves the integrity of the controlled experiment.
Question 1: Across all 24 tasks, the completed new origin-isolated runs give the following results:
Astra has the highest mean and exceeds the baseline on the most tasks.