acceptodds
Withdrawn from ICLR 2027

Beyond Frame Shuffling: Interpreting Temporal-Order Sensitivity in Video Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A video predictor's response to reordered frames is difficult to interpret: reordering can change local transitions and perturbation magnitude, while aggregate error scores can conceal changes in the predictions themselves. We study this ambiguity through controlled comparisons with explicitly preserved input properties. Our construction enumerates six histories with identical frame content, boundary frames, and final five observations, and groups them by their directed adjacent-frame-pair multisets. Within-group pairs preserve local transitions; between-group pairs provide references with the same frame content. We evaluate all pairs using prediction disagreement, pixelwise error change, and aggregate error change. Experiments comprise 896 forecasts on 32 Moving MNIST clips under unmasked and masked observations, using released SimVP and OpenSTL ConvLSTM predictors. For unmasked first-frame predictions, SimVP's transition-matched pairs exhibit 14.7% greater mean prediction disagreement than unmatched pairs. Post-hoc comparison against references with exactly equal input L1 distance reduces this gap to 6.3%, with a similar result under stricter per-pixel distance matching. The ConvLSTM predictor remains close to parity, and some masked comparisons reverse the ordering. Output-range decomposition and stepwise analysis further distinguish prediction changes from those retained by scalar errors. Together, the controls and decompositions turn order sensitivity into a precisely specified comparison: the input constraints define what is held fixed, and the readout defines which prediction changes remain visible.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video prediction, Temporal-order sensitivity, Transition-matched controls, Controlled evaluation, Error decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xs4pOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how temporal frame ordering affects video prediction and how conclusions about temporal-order sensitivity depend on the choice of reference comparisons. Experiments using SimVP, ConvLSTM, and PredRNN on Moving MNIST and KTH demonstrate that different reference-weighting strategies can produce different sensitivity estimates and, in certain cases, reverse the relative ordering of predictors.

Critical strengths
  1. The paper highlights an important issue in evaluating temporal understanding: conclusions drawn from frame-order perturbations may depend strongly on the construction of reference comparisons and the selected evaluation metrics.
  2. The authors provide extensive sensitivity analyses, including different weighting constraints, prediction horizons, and evaluation metrics.
Critical weaknesses
  1. The proposed framework primarily combines existing concepts.
  2. The experiments rely on a small number of pretrained model checkpoints and two relatively constrained datasets.
Questions for authors
  1. What is the principal methodological advancement beyond applying established transition-preserving permutations and constrained optimization to video prediction outputs?
  2. Since the three input-defined central weighting strategies generally produce consistent model rankings, how frequently would the demonstrated reversals occur under realistic evaluation protocols?
  3. Can the authors validate the findings on more diverse video prediction architectures and realistic video datasets with independent subjects and motion patterns?
Reviewer_BoMjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper examines how the interpretation of temporal order sensitivity in video prediction depends on input constraints, reference weights, and output measurements. The main constructions preserve frame content, boundary frames, and recent observations within each family, and group histories by their directed adjacent frame pair multisets. The analysis combines input distance matching, endpoint balance, and optimization over admissible reference weights to distinguish a contrast under one comparison rule from a direction that holds throughout a specified domain. Experiments using released OpenSTL SimVP+gSTA, ConvLSTM, and PredRNN predictors on Moving MNIST and KTH demonstrate reference dependent contrasts and selected model ordering reversals, with additional studies examining frozen weight transfer, content preserving permutations, and the relationship between prediction disagreement and excess forecast error.

Critical strengths
  • The paper carefully specifies what each comparison preserves. Input distance matching and endpoint balance address distinct aspects of the reference comparison, while shared weights permit a direct comparison between predictors under the same rule. Computing the admissible range makes the distinction between a selected point result and a direction supported throughout the declared domain precise.

  • The experiment with frozen reference weights provides meaningful confirmation of the main ordering reversal. In Section 4.4, the same two weight vectors reverse the first frame ordering of SimVP and ConvLSTM on new segments across all five subjects, including after model specific normalization. Although these are new segments of previously used videos, the result strengthens the evidence beyond the original optimization on saved outputs.

  • The distinction among prediction disagreement, pixelwise error change, and aggregate error change is informative. The separate response and forecast error analysis further explains why output changes do not directly determine accuracy changes. The paper also reports failed confirmation results and unsuccessful model selection outcomes, and explicitly distinguishes reference choice ranges from sampling uncertainty.

Critical weaknesses
  1. The significance of ambiguity over the full reference domain is less established than its mathematical existence. In Section 4.4, the three graph only weighting rules agree on all pooled model orderings, and Section 4.6 shows that agreement and local stability can coexist with a range that crosses zero. The paper acknowledges that its domain does not describe an observed distribution of evaluator preferences. Consequently, the evidence supports sensitivity within the declared domain, but leaves the relevance of that sensitivity to concrete evaluation choices insufficiently demonstrated.

  2. The connection between the principal matched/reference contrast and forecasting decisions remains limited. The prospective confirmation in Appendix S16 concerns native to perturbed response, which is distinct from the matched/reference contrast. In Appendix S13.4, the reported selection policies have higher MSE than their corresponding uniform random choice baselines on the same decided subsets, while the small gain obtained with a ConvLSTM fallback in Appendix S13.7 does not survive every subject deletion. These negative results do not invalidate the diagnostic, but they limit the demonstrated consequences of the matched/reference analysis for forecasting decisions.

  3. The benchmark evidence has a restricted scope. The principal Moving MNIST studies use 32 clips each, and the key KTH confirmation experiments use additional segments from videos of the same five subjects. The public artifact comparisons use fixed released checkpoints, while the content preserving extension admits only 15 of 30 candidate clips and replaces directed transition matching with a histogram of transition magnitudes. These choices are explicitly documented, but they limit the breadth of the empirical support and prevent the extension from being interpreted as confirmation under the original transition constraint.

  4. The abstract does not adequately represent the final empirical evidence. It emphasizes the initial 14.7% excess and the 6.3% contrast after distance matching, although the prospective Moving MNIST experiment does not confirm the positive direction. It also omits the KTH reference weighting results and frozen weight confirmation that form a substantial part of the main contribution. Revising the abstract to reflect these findings would make the scope and central message substantially clearer.

Questions for authors

• Clarification: Given the agreement among the three graph only rules and the reported local stability, how should an evaluator choose an admissible reference domain and weight floor for a particular forecasting question? Please explain what scientific considerations would justify requiring a direction to hold throughout that domain, beyond the mathematical feasibility of its weights.

• Validation request: Appendix S13.4 states that seven candidates produce an empty numerical support screen and that this is not an exact infeasibility proof. Can exact feasibility checks resolve these cases and establish whether the reported coverage of 15 out of 30 clips reflects the declared constraints or limitations of the numerical screen?

• Validation request: Using the available saved forecasts, how sensitive are the content preserving comparisons to the alternative histogram widths of 2/255 and 4/255 considered during development? Please distinguish changes in input eligibility from changes in model ordering on clips supported under the compared widths, since the extension uses a different transition descriptor from the original constructions.

Reviewer_ukfgOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This manuscript examines how video prediction models respond to temporally reordered input frames, focusing on separating order-sensitive responses from differences in frame content, local transitions, and perturbation magnitude. The authors construct alternative input histories with matched directed adjacent-frame-pair multisets and analyze prediction disagreement, pixelwise error changes, and aggregate error changes. They further examine whether response directions and cross-predictor response orderings depend on the weighting of eligible reference comparisons. Experiments use fixed SimVP, ConvLSTM, and PredRNN checkpoints on Moving MNIST and KTH, including prospective follow-ups and a content-preserving permutation extension. The results show that some response orderings can reverse under different admissible reference weights, and that a frozen pair of reversing weights reproduces a first-frame reversal on new segments. Ultimately, the paper advocates explicitly specifying reference rules and distinguishing reproducibility under a fixed rule from invariance to reference choice. Although the evaluation question is interesting, I find the presentation, practical significance, technical novelty, and empirical generality insufficiently convincing for acceptance.

Critical strengths
  1. Carefully specified input controls Family A enumerates six histories that share the same frame content, boundary frames, and final five observations, and groups them into three classes according to their directed adjacent-frame-pair multisets. Within-class comparisons therefore preserve position-independent additive transition summaries while varying their temporal arrangement. This construction makes the controlled properties more explicit than ordinary unconstrained frame shuffling.

  2. Follow-up experiments and transparent reporting of negative results The authors do not rely exclusively on optimized reversals in the original outputs. In Section 4.4, two shared reference-weight vectors are frozen before evaluating thirty additional segments, and the first-frame SimVP–ConvLSTM response ordering reverses in the pooled results and all five subjects. The paper also explicitly reports that the initial positive Moving MNIST direction is not confirmed by the prospective follow-up, and that the three graph-only central rules agree on all twelve pooled comparisons. These qualifications help establish the boundaries of the findings.

  3. Useful distinctions in the discussion The discussion offers useful insights into interpreting temporal-order sensitivity, particularly by distinguishing reproducibility under a fixed reference rule from invariance to reference choice. It also clarifies what the individual controls establish and separates changes in predictions from changes in task error. For example, the paper explicitly distinguishes the main matched–reference contrast from the native-to-perturbed response used in the task-consequence analysis. These distinctions help readers avoid interpreting the probe as a general measure of temporal ability.

Critical weaknesses
  1. Poor presentation and insufficient alignment between the framing and the final findings The manuscript is difficult to follow, and the presentation does not clearly prioritize its main contribution. The abstract foregrounds the initial 896 forecasts on 32 Moving MNIST clips and the positive contrasts of 14.7% and 6.3%, whereas the introduction reports that the prospective follow-up does not confirm this positive direction, with ratios of 0.9996 and 0.8978. The subsequent analysis focuses on reference-weight domains and response-ordering reversals, which are not adequately reflected in the abstract. This mismatch makes it difficult to identify the final claim and connect it to the relevant validation. A more direct explanation of the relationship between the initial observation, its non-confirmation, and the reference-domain analysis would substantially improve clarity.

  2. The practical significance of the findings is not sufficiently convincing The motivation for investigating these particular frame constructions and reference domains remains limited from a practical standpoint. The paper demonstrates that optimized admissible weights can produce different response conclusions, but it is less clear when this changes the conclusions of a realistic evaluation procedure. In Section 4.4, the three graph-only central rules disagree on only 1/60 subject comparisons and 0/12 pooled comparisons, and agree everywhere after model-specific normalization. This does not invalidate the domain-wide ambiguity, but it leaves the practical importance of that ambiguity insufficiently established. Although KTH provides real-video evidence, the paper does not convincingly demonstrate how the proposed analysis yields actionable diagnostic value in practical video prediction.

  3. The technical contribution appears limited The methodology primarily combines established tools: transition-preserving constructions, elementary error inequalities, convex-separation feasibility conditions, and linear-programming bounds. The paper itself acknowledges that the transition-multiset identity and convex-separation argument are not new theoretical results. I recognize that an evaluation paper need not introduce a new architecture or learning algorithm; nevertheless, the manuscript needs to establish more clearly why this particular combination provides a sufficiently significant methodological advance. At present, I find the contribution incremental relative to the complexity of the proposed comparison framework.

  4. The empirical evaluation remains narrow and conditional The evaluation extends beyond the initial 32-clip Moving MNIST study to additional Moving MNIST experiments and KTH, but it remains restricted to these two datasets and fixed SimVP, ConvLSTM, and PredRNN checkpoints. The supported comparison sets are small: the earlier KTH comparison admits 18/24 clips, the five-subject cohort admits 21/30 clips, and the content-preserving extension admits 15/30 candidates. These input-dependent restrictions limit the scope of the conclusions, even though the exclusions are reported transparently.

Questions for authors
  1. Could you evaluate the framework on more recent video predictors with substantially different architectures or generative mechanisms? Do the reference-sensitive response orderings persist under the same declared controls?

  2. Beyond the current KTH cohorts, can you demonstrate the validity and stability of the reference comparisons on a larger and more diverse real-world video dataset, while reporting the proportion of clips that satisfy the comparison requirements?

  3. What concrete evaluation or diagnostic decision should a practitioner change after applying this framework? In particular, how should the results be used when the graph-only central rules agree but the full admissible range allows a reversal?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.