SciDynBench-R: Benchmarking Error-Validity Disagreement in Scientific Dynamical Forecasting

Scientific dynamical forecasts are usually selected by aggregate point error, but scientific use often depends on regimes, events, oscillations, and constraints. We introduce SciDynBench-R, a benchmark that connects scientifically motivated distribution shifts with task-specific validity criteria across controlled ODEs, Palk inspired biological transport, and leakage safe reduced WeatherBench2 Z500 climate indices. Its reproducible protocols jointly evaluate point error and scientific validity and diagnose disagreement between their model rankings. Across 187 frozen baseline model/split means, SciDynBench-R finds 44 low error scientific failures: ODE forecasts in the wrong regime, biological forecasts with large calcium peak count errors, and climate forecasts selected by NRMSE that behave as event smoothers, missing high variability and amplification events despite low aggregate error. A climate ablation illustrates improvements in event fidelity at higher NRMSE among the evaluated configurations. The benchmark and frozen results provide a reference for examining when favorable point error coexists with failures of task-specific scientific properties.