HazardCast: Benchmarking LLM Agents for Forecast Construction Across Five Natural Hazards

Natural-hazard forecasting requires turning heterogeneous environmental observations into prediction procedures that respect task-specific targets, horizons, units, and information constraints. Large language model (LLM) agents can inspect data, write code, fit models, and revise workflows, but evaluating them requires more than measuring final prediction error. We introduce HazardCast, a benchmark of 24 forecasting questions spanning tropical cyclones, wildfire, heat, drought, and floods. Each question provides prepared environmental data and an explicit prediction contract while leaving feature construction and forecasting methodology to the agent. Evaluation combines native forecast losses, performance relative to a fitted local model, and checks of numerical and forecast-time validity. Across a 24-question comparison, selected agent workflows frequently improve on local models, but gains are uneven. Higher configured reasoning effort does not yield a uniform advantage, paired training conditions show substantial sensitivity to population design, and run-level audits identify concrete output transformations, prediction restrictions, and delivery failures. HazardCast therefore evaluates not only whether an agent produces accurate forecasts, but whether the procedure producing them is scientifically valid.