- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces DamEQ-Bench, a physics-aware benchmark for gravity-dam seismic video generation built from large-scale shaking-table experiments. The benchmark contains 198 controlled seismic conditions (empty/impounded reservoir states) with ~1,300 multi-view UAV recordings and synchronized sensor measurements. The authors formulate the task as I2V/V2V generation conditioned on seismic prompts and evaluate 16 video generation models plus 2 static baselines using 6 visual-quality and 8 response-oriented metrics. They also investigate LoRA-based domain adaptation.
Critical strengths
Genuine physical experimental data at scale: The 198 shaking-table conditions with synchronized multi-view video and sensor data represent a substantial, costly, and reusable resource. This is far more valuable than synthetic or internet-scraped benchmarks and enables physically grounded evaluation that cannot be replicated without such infrastructure. Rigorous separation of visual and physical metrics: The explicit demonstration that visually plausible videos can be physically incorrect—and the provision of interpretable diagnostic metrics—advances the community's understanding of what video generation models actually learn. This framing is consequential for how future physics-aware generation work should be evaluated.
Critical weaknesses
Limited generalization scope: All data comes from a single 1:50 scaled specimen at one facility. The benchmark evaluates "within-system condition generalization" only, as the authors acknowledge. This limits claims about whether models learn generalizable seismic dynamics versus memorizing this specific structure's response. While the authors are transparent about this, it constrains the benchmark's broader impact. Reference model circularity risk: The ODE-Response and related metrics depend on a learned Neural ODE as "ground truth" for conditions where direct sensor comparison may be unavailable. If this model is imperfect, the benchmark may penalize physically correct generations or reward ones that match the ODE's biases. The paper does not report ODE validation error against held-out sensor data.
Questions for authors
ODE validation: What is the prediction error of the Neural ODE against held-out sensor measurements? How sensitive are the reported rankings to ODE accuracy? Evaluation subset size: How many samples were actually evaluated per model in Tables 2/H3/H4? Are the reported differences statistically significant (e.g., confidence intervals or standard errors)? LoRA training protocol: What was the training set size, and were evaluation conditions strictly held out? Could the observed improvements simply reflect overfitting to the experimental scene?
Response to Reviewer RYD9
Response to Reviewer RYD9
We sincerely thank the reviewer for recognizing the exceptional value of our large-scale experimental dataset and the importance of physically grounded video-generation evaluation. We have strengthened DamEQ-Bench through additional experimental validation, statistical significance analysis, ranking robustness tests, and a strictly partitioned LoRA evaluation protocol. Our key findings are summarized below.
DamEQ-Bench is built upon experiments conducted at the National Facility for Earthquake Engineering Simulation, featuring the world's largest shaking table in terms of platform dimensions and payload capacity (20 × 16 m; 1,350 t). We employed a 1:50 large-scale gravity-dam model designed according to physical similitude principles, enabling controlled investigation of seismic responses and complex dam–reservoir interactions.
The resulting dataset comprises 198 experimental seismic conditions, approximately 1,300 multi-view UAV recordings, and synchronized sensor measurements. This experimental infrastructure provides physically traceable observations that are difficult to obtain through synthetic data or conventional video collections.
By systematically varying seismic excitation and reservoir conditions, DamEQ-Bench establishes a rigorous testbed for condition-dependent physical generalization. Its experimental observations provide valuable foundations for evaluating learned structural dynamics and developing future physics-aware generative models.
We have incorporated additional quantitative validation and metric-ablation experiments to strengthen the reliability of our physical evaluation.
An independent single-degree-of-freedom response-validation experiment under El Centro excitations of 0.10g and 0.20g achieves Pearson correlations of 0.837 and 0.877, NRMSE values of 0.074 and 0.070, and peak-response errors of 0.1% and 5.3%, respectively. Predicted and measured dominant frequencies also show close agreement. These results provide quantitative evidence of response-reconstruction accuracy in the independent validation system.
We further conducted ranking robustness analyses to assess the influence of learned physical references. After removing ODE-Response, the resulting rankings retain strong agreement with the full evaluation (Spearman ρ = 0.988; Kendall τ = 0.933). Under the independently reported noODE evaluation of base generators, the leading models remain Wan2.1-I2V, LTX-Video-2B-distilled, and ExtDM.
Together, these analyses demonstrate that the benchmark's comparative findings are supported by multiple complementary physical diagnostics rather than a single learned-response metric.
Our benchmark encompasses 198 experimentally controlled seismic conditions, with multiple generated videos evaluated under each condition across different generative models. We further strengthen statistical reliability through 2,000 condition-level paired bootstrap resamples, accounting for dependencies among videos generated under the same seismic excitation.
The statistical analysis reports model-specific sample sizes and confidence intervals. For example, Wan2.1-I2V outperforms ExtDM by 0.026 in physical score, with a 95% confidence interval of [0.008, 0.045]. Additional metric-ablation analyses further demonstrate the robustness of the model rankings.
We established a condition-level training, validation, and testing protocol for the complete collection of 198 seismic conditions, with a proportional allocation of 82/45/71 conditions, respectively. The protocol is designed to separate seismic excitation records across training and evaluation subsets, enabling controlled assessment of generalization to unseen seismic conditions.
For Wan2.1-I2V, we selected the LoRA checkpoint at training step 2000, achieving a validation loss of 0.006674. The condition-level partitioning, checkpoint selection, and evaluation procedures provide a reproducible foundation for assessing domain adaptation and physical-response fidelity.
This protocol supports a systematic investigation of whether LoRA adaptation improves the generalization of video generation models across different seismic loading and reservoir conditions.