- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how temporal frame ordering affects video prediction and how conclusions about temporal-order sensitivity depend on the choice of reference comparisons. Experiments using SimVP, ConvLSTM, and PredRNN on Moving MNIST and KTH demonstrate that different reference-weighting strategies can produce different sensitivity estimates and, in certain cases, reverse the relative ordering of predictors.
Critical strengths
- The paper highlights an important issue in evaluating temporal understanding: conclusions drawn from frame-order perturbations may depend strongly on the construction of reference comparisons and the selected evaluation metrics.
- The authors provide extensive sensitivity analyses, including different weighting constraints, prediction horizons, and evaluation metrics.
Critical weaknesses
- The proposed framework primarily combines existing concepts.
- The experiments rely on a small number of pretrained model checkpoints and two relatively constrained datasets.
Questions for authors
- What is the principal methodological advancement beyond applying established transition-preserving permutations and constrained optimization to video prediction outputs?
- Since the three input-defined central weighting strategies generally produce consistent model rankings, how frequently would the demonstrated reversals occur under realistic evaluation protocols?
- Can the authors validate the findings on more diverse video prediction architectures and realistic video datasets with independent subjects and motion patterns?