acceptodds
Under review as a conference paper at ICLR 2027

Do Frame-Order Differences Between Video Pretraining Stages Survive Fine-Tuning?

Abstract

Linear probes on frozen features are a standard way to compare what different pretraining gives a video model. We ask whether the differences they reveal survive fine-tuning, for frame order, measured as the balanced-accuracy drop when a clip's frames are shuffled. Across the published pretraining stages of one architecture (VideoMAE), on two binary fall-detection datasets, one synthetic and one in the wild, frozen checkpoints differ markedly at the output, and supervised Kinetics training lowers the fall probe's order sensitivity even as it raises accuracy on the synthetic dataset. Fine-tuning collapses most of these differences. On the synthetic dataset the fraction of the gap between the self-supervised (MAE-only) and Kinetics checkpoints that fine-tuning closes is κ = 1.08 [0.84, 1.33], so the gap closes completely within error. It also closes under a standard fine-tuning schedule (κ = 0.85 [0.22, 1.36]), under re-drawn and tubelet-preserving shuffles and with nonlinear probes, and it does not survive on native-resolution input or with cross-fitted probes, where refitted probes even find the fine-tuned order reversed. On the in-the-wild dataset, where κ is uninformative, the spread between checkpoints shrinks by 69%. Between these two checkpoints, what closes is how the fall read-out uses frame order, not the linearly readable direction information the output carries: on a task defined by frame order their frozen outputs are not distinguishable, and both kept most of their direction information through fine-tuning for fall detection. In exploratory layer-wise probes, the frozen checkpoints differ mainly in the upper layers, and under the standard schedule they still differ at blocks 9 and 10 yet agree at the output. In TimeSformer, whose frozen Kinetics output carries no linearly readable direction information, the fall-task gap does not close under either recipe (κ = 0.08 and 0.43, both intervals containing 0), and most of the gap in direction information persists; the same holds in Hiera, where we predicted before fine-tuning that its fall-task gap would not close (κ = 0.03 [−0.54, 0.45]). On an order-defined task TimeSformer's gap closes (κ = 0.86 [0.69, 1.01]) and Hiera's, as the probe reads it, does not (κ = 0.25 [−0.24, 0.60]). Within VideoMAE, fall-task gaps closed even where differences in direction information remained. A frozen comparison of frame-order sensitivity on a task that does not require order thus describes, for VideoMAE, mostly how a probe reads the top of the network, and it cannot tell which differences fine-tuning will keep.

Primary area:
interpretability and explainable AI
Keywords:
video representation learning, probing, linear probes, frame order, self-supervised pretraining, fine-tuning, VideoMAE, TimeSformer, Hiera, pre-registered analysis

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_QMqFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether differences in frame-order sensitivity between pretrained video models persist after downstream end-to-end fine-tuning. The authors primarily compare VideoMAE checkpoints from different pretraining stages and evaluate their frame-order sensitivity on two fall-detection datasets, OF-Syn and OOPS. They observe that the differences between frozen checkpoints largely disappear after fine-tuning on OF-Syn, while the results on OOPS are less conclusive. Additional experiments involving an order-defined task, layer-wise probing, and other video architectures are conducted to investigate whether these changes reflect differences in temporal information availability or its task-specific utilization.

Critical strengths
  1. The paper focus on temporal representation learning and transfer, which are important topics in video understanding.

  2. The comparison between VideoMAE checkpoints before and after supervised Kinetics training provides a relatively controlled setting for investigating the effects of different pretraining stages.

  3. The authors conduct extensive experimental analyses, including multiple random seeds, different fine-tuning schedules, alternative shuffling strategies, and layer-wise probing.

Critical weaknesses
  1. Insufficient motivation and unclear significance of the research question.

Understanding temporal representation learning and transfer is important. However, the paper does not adequately explain why differences in frozen frame-order sensitivity should be expected to persist after end-to-end fine-tuning, which explicitly adapts the entire network to a new task. The authors also fail to establish that existing studies rely on this assumption or that its violation has meaningful practical consequences. Therefore, the significance of the specific research question remains unclear.

  1. A mismatch between the evaluation tasks and the research objective.

The main experiments rely on fall detection, which can potentially be solved using static visual cues rather than temporal ordering. This limitation is evident in the authors' own results: frozen VideoMAE checkpoints differ substantially in sensitivity on OF-Syn Fall (0.151 vs. 0.048), but show similar sensitivity on Up/Down (0.325 vs. 0.303). These task-dependent measurements do not directly characterize temporal representation capabilities, weakening the connection between the experiments and the paper's broader conclusions.

  1. Inconsistent findings and limited generalizability.

The main finding is primarily supported by VideoMAE on synthetic OF-Syn. On real-world OOPS, the reported κ = 0.70 [-0.41, 1.34] is inconclusive. Furthermore, TimeSformer and Hiera retain their sensitivity differences after fine-tuning on the same OF-Syn Fall task. Although the authors acknowledge these discrepancies, they do not establish what determines whether the differences persist, leaving the conclusions highly architecture- and task-dependent.

Questions for authors

see weakness

Reviewer_Cu9FOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper tests frame order sensitivity in pre-trained vs fine-tuned video models on fall-detection task on two benchmarks. While different pre-trained models may show different frame order sensitivity, if fine-tuned, this difference disappears.

Critical strengths
  • The question of what pre-trained frozen representations encode and what subset of that may or may not be instilled by fine-tuning is interesting.
Critical weaknesses

While the core question can be interesting for a scientific study, the paper is entirely and completely AI generated slop. This by itself is not a negative signal - but the paper has no structure, no meaningful research question and no insightful observations.

For instance, just reading the abstract: it does not state the motivation, what experiments were designed and why, or what observations were made. It feels like a complete dump of all the results.

The following is an actual paragraph from conclusion:

Four practices follow. Report a probed property across depth, as the drop from each model’s own maximum, not only at the output or as a peak location. Check a ranking between checkpoints in the regime in which it will be used: a frozen ranking licenses a choice for frozen use. Where the probing task does not need the property, also probe one that does (Section 4.2). And record the preprocessing and library version, and assert a checkpoint-file statistic at load: both move frozen values, preprocessing by checkpoint-specific amounts.

The authors have not even bothered to clean the LLM output to make it reasonable looking.

Questions for authors

None

Reviewer_nRBUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies if the standard linear probe is enough to evaluate and compare pretrianing video models. The authors propsoe 4 best practices for probing: across depth, match eval setup, use property-dependent tasks, and standardize the preprocessing. The key takeaway on current linear probing is frozen linear probing mainly reflects the top layers of the frozen network and cannot reliably predict how fine-tuned models wil utlize the temporal information.

Critical strengths

the authors have propsoed some actionable best practice.

the paper validate finding across different video archtectures and demonstrate the frozen differences will change across intermediate layers.

Critical weaknesses

W1. The paper lacks a strong justification on why comparing frozen-frame order sensitivity matters to the broader video modeling research. It is somewhat unclear to me how resolving this gap translate to real world problems like deployment or improving downstream model performance.

W2. Wonder what is the differences it will make on current paper conclusion if the author also consider parameter-efficient finetuning, along side full finetuning

W3. the empirical eval focus on binary fall detection, the task is too narrow.

W4. the authors admit in the end that they cannot explain the performance gap discrepencies btw videoMAE and TimeSformer/Hiera. The underlying arch mech remains an unresolved variable.

Questions for authors

please refer to weakness session above, i mainly interested in W1-W3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.