acceptodds
Under review as a conference paper at ICLR 2027

Image Robustness Does Not Imply Video Robustness: The Limits of Single-Frame Evaluation

Abstract

Single-image corruption benchmarks such as ImageNet-C are a cheap way to choose a robust backbone for a video system, but this shortcut works only if image and video corruptions rank the models the same way. We test how well a model's robustness to single-image corruptions actually predicts its robustness to video corruptions. We evaluate models on image and video corruptions under identical conditions and check whether the single-image robustness ranking predicts the video robustness ranking, using a rank-correlation threshold for reliable prediction. Our central finding is that image robustness transfers to video only under certain corruption conditions. Specifically, we observed that the single-image ranking transfers to video to the extent that a single frame shows the corruption. When a corruption realization is shared across a video clip, the single-image ranking is preserved almost perfectly across seven settings. By contrast, when a corruption disrupts the temporal structure of a clip, the single-image ranking does not reliably predict the video ranking in any setting. These temporal-structure corruptions include frame drop, repetition, and reordering, which damage the frame sequence even though every delivered frame stays clean. Corruptions that a single frame shows only in part give mixed results, for example when the corruption strength drifts over time or the camera shakes. Single-image robustness had essentially no rank correlation with video robustness to temporal-structure corruptions among 37 video-pretrained 3D backbones. The same pattern was observed in codec and transport pipelines: Single-image robustness predicts robustness to codec compression well, but poorly predicts robustness to packet loss or variable-frame-rate judder. Our theoretical analysis proves that two corruptions can look identical to a single-image benchmark yet reverse the ranking of two video models, implying that image-only testing cannot guarantee the right model choice for video. We recommend adding a separate targeted temporal evaluation whenever temporal structure can be disrupted, and selecting video models by single-image robustness only when a single frame shows the expected corruptions.

Primary area:
datasets and benchmarks
Keywords:
Video Understanding, Model Robustness, Distribution Shift, Action Recognition, Computer Vision, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XqGrOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper is on whether ranking a set of models based their performance under image degradation will align with their rank under video degradation. The result is somewhat expected: if the video corruption is mostly an image corruption present at similar levels in every frame, the ranking based on image performance will predict well video.

The paper also demonstrates that image-only degradation assessment will always have this limitation (cannot guarantee correct ranking for video-level degradations).

Critical strengths

As the main topic of the paper ("IMAGE ROBUSTNESS DOES NOT IMPLY VIDEO ROBUSTNESS") seems somewhat self evident, one of the strengths seems to be the attempt to perform theoretical analysis via an existence theorem that demonstrates "worst-case limits" that the rank reversal can happen for video-level (dynamic) degradation..

Critical weaknesses

The contributions are limited and somewhat expected (e.g., that one should ideally evaluate model performance on representative types of distortion, that is include video-specific degradations if expected to perform well on such tasks).

Even the largest strength ("theoretical analysis") feels somewhat inaccessible and hard to parse: It's hard to connect the mathematical abstractions to the practical examples.

The paper feels overly wordy for the magnitude of the contributions. Results are often forward linking to an even more verbose appendix. Some examples:

  • line 103 "Fig. 4 in App. F.1 illustrates why" -- which is on page 27 in the appendix.
  • line 274: " Table 3" which is on page 33
  • line 360: "with clean top-1 from 46.5% to 84.1% (Table 18)." Table 18 is on page 50.
  • line 468: (Fig. 6). On page 41.

The authors should exhibit more discretion when deciding what to include (and exclude) in order to make the paper clear, concise, and easy for a human to understand.

Questions for authors

None

6 comments

Thank you for the comments on novelty and accessibility. We clarify the empirical question and give a concrete explanation of the theorem.

1. What does the study establish beyond the intuition?

A frame's inability to show a timing fault does not tell us whether spatial robustness still identifies the most robust video models. Robustness to different faults could be shared across models. Our study tests that possibility and measures the cost of relying on it.

We isolate the effect of corruption type through a controlled comparison and additional ablations. For the same 37 Diving48 models, weights, clips, temporal heads, and input lengths stay fixed; only the corruption family changes. Spatial→coherent is ρ=0.992\rho = 0.992, while spatial→temporal-structure is ρ=0.053\rho = 0.053. Resampling whole architecture groups accounts for related models and preserves a positive paired difference: 0.939 [0.376, 1.306].

We also test whether the strong coherent result depends on spatial operators that produce identical outputs under independent and shared application. Removing these deterministic operators leaves only stochastic operators and preserves the separation (0.986/0.097). Thus the result persists when the two spatial tests use different random corruption patterns.

The distinction is large for models that look equivalent under spatial testing. An illustrative pair from the submission is:

ModelSpatial retentionTemporal-structure retentionPacket-loss retention, severity 5VFR retention, severity 5
VideoMAE-S, SSv2 initialization0.840.7240.2650.774
ViViT-B0.840.9200.4260.923

The spatial scores differ by less than 0.001, while temporal-structure retention differs by 0.197; the models rank last and first on that family, with the same ordering under both pipeline faults. The pair illustrates the largest temporal gap among spatial gaps ≤ 0.005: nearly equal spatial scores can conceal a large temporal difference.

An additional leave-one-corruption-out control asks whether poor prediction is simply due to testing an omitted fault. The spatial score predicts an omitted spatial operator at median ρ=0.832\rho = 0.832, but a temporal-structure operator at 0.118; other temporal operators predict an omitted temporal operator at 0.805. Prediction remains strong within spatial and temporal families, while the spatial→temporal gap persists.

The missing measurement has a measured benefit. Across all 435,897 five-model subsets of the original Diving48 pipeline results, mean retention regret falls from 0.02535 with spatial testing to 0.01132 with the fixed blend and 0.00244 with the probe. The probe's 90.4% reduction costs 20.5% as many corrupted-clip evaluations as the spatial sweep. Its correlation gain has architecture-bootstrap 95% CI [0.085, 0.887]; removing any one operator preserves 84.4–94.3% regret reduction.

These findings support one evaluation principle: measure spatial and temporal robustness separately, then use the information relevant to the target faults and selection objective. The study supplies the controlled evidence, information boundary, and measured selection benefit for applying it.

2. The theorem through a concrete example

Knowing how often each frame is corrupted does not tell us which frames are corrupted together. That missing information can reverse the model ranking.

Consider three frames: the ball is on the same side in frames 1 and 2, then either stays or switches sides in frame 3. For example, left → left → right has label “changed side.” Model A compares frames 1 and 3; model B compares 2 and 3. Both are 100% correct on clean clips. The starting side and stay/change label are sampled independently with equal probabilities, so one frame alone gives no label information.

Now mirror selected frames horizontally, swapping left and right:

Corruption processFrames mirrored in each random eventModel A accuracyModel B accuracy
Mirror frames 1 and 3 together40%: {1, 3}; 40%: {2}; 20%: none100%20%
Mirror frames 2 and 3 together40%: {1}; 40%: {2, 3}; 20%: none20%100%

Mirroring both frames used by a model preserves their same/different relationship; mirroring just one reverses it. In the first process, A is always correct, while B is correct only in the unchanged 20% of clips. The second process swaps their roles. Yet every frame is mirrored with probability 0.4 in both processes, for every clean clip. Thus an audit of individual-frame corruption distributions sees identical evidence even though the best model changes.

An image-only selector must therefore choose A with the same probability pp in both processes. Its regret is 0.8(1−p)0.8(1 - p) in the first and 0.8p0.8p in the second: the chance of choosing the worse model times the 0.8 accuracy gap. These regrets sum to 0.8, so at least one is 0.4 or larger. More single-frame samples cannot reveal which frames change together. This illustrates the submitted theorem's worst-case limit with clean accuracies matched.

Frame-level evidence is sufficient when loss adds up frame by frame. For a fixed label yy and finitely many possible values per frame, all ways of combining the same individual-frame distributions give the same expected loss if and only if

ℓ(z1,…,zT,y)=a0(y)+∑t=1Tat(zt,y).\ell(z_1,\ldots,z_T,y) = a_0(y) + \sum_{t=1}^{T} a_t(z_t,y).

Here ztz_t is frame tt, yy is the label, a0a_0 is a baseline, and ata_t is frame tt's contribution to the total loss ℓ\ell. We can average each frame's contribution separately and add the results. Those averages depend only on the individual-frame distributions, so which frames change together cannot affect the expected total loss. Errors in the ball example depend on the relationship between two frames, so this additive condition fails. Without it, identical frame-level distributions do not guarantee equal expected loss. The experiments measure when this missing temporal information changes rankings and how temporal testing improves model selection.

3. Making the main text self-contained

We have replaced the four distant references you identified with evidence at the point of use:

Reference in the submissionMain-text revision
Line 103 → Fig. 4, Appendix F.1Added corruption examples to Figure 1, showing what an individual frame reveals.
Line 274 → Table 3Placed per-family correlations and uncertainty beside the transfer comparison (Section 3; Tables 1–2).
Line 360 → Table 18Added a cited model inventory in Section 2 and Appendix G (Table 10), with protocols, pretraining, and clean-accuracy ranges.
Line 468 → Fig. 6Combined pipeline correlation, selection regret, and evaluation cost in Table 3.

We have also put the closest related-work comparison in the Introduction, defined literal T=1T = 1 and the multi-frame spatial reference side by side, and removed prose that repeats tables. The added literal single-frame experiment now appears beside the central claim (Section 3, Table 2). The supplementary code and manuscript evidence map provide executable checks and the location of each revision.

Thanks for the clarifications and for trying to fix up some of the issues in the main text.

With that and a second pass through the paper I think I can see a little bit clearer what the authors are trying to convey.

However, the result still seems expected. (e.g., Model A can perform better than Model B when benchmarked on class of distortions X, but then the performance can flip such that Model B > Model A when tested on some other class of distortions Y). If one cares about the performance of the model in an environment containing a mix of distortions X and Y you probably want to evaluate performance on both types of distortions (or something representative of them) when doing model selection. And if distortion Y is rare (lets say 1% chance of happening), then that should also factor into model selection.

So I'm still failing to see the significance of the main result in the paper. If one cares about performance of a model in an application where a new or different mix of distortions is possible, then one should probably evaluate using relevant distortions. For classification tasks where datasets already exist, simply evaluating on those distortions should straightforward (unlike say image / video quality assessment, where a new human study may need to be done in order to collect annotations).

Another reviewer had asked the question ("What is the main contribution of the paper? Is it a benchmark or a method?). There is likely a way to structure the narrative that better demonstrates the impact & conributuion of the results (e.g., an exhaustive benchmark result that previous methodologies have been evaluating models incorrectly and should be using a "temporal probe", or that the temporal aspect had not been considered enough is important due to reasons that the others would demonstrate, or that it isn't mandatory to exhaustively test many temporal distortions and that a particular combination of independent spatial and temporal probes are sufficient, etc.).

Reviewer_qFxsOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper analyzes how robustness to single-image corruptions relates to robustness to video corruptions. The paper's main finding is the fact that image robustness transfers to video only under certain corruption conditions. The experiments are evaluated on 37 video-pretrained backbones. The paper also proposes a temporal probe to improve prediction of video-robustness rankings.

Critical strengths
  1. The paper’s stated experiments scale is impressive. This base could help to create a valuable comprehensive benchmark.
  2. The paper presents theoretical contributions that strengthen their claims.
Critical weaknesses
  1. The paper does not provide code for the benchmark in the supplementary materials, especially the code for the experiments and proposed method. This is a serious reproducibility concern, especially paired with the significant size of the experiments and “datasets and benchmarks” as the primary contribution.

  2. The paper is difficult to read due to the text quality, heavy reliance on textual information, and constant references to other parts of the text. Right now, all the information needed to understand and verify the contributions is buried in the Appendix. The text needs a strong revision: the main results should be included in the main text, additional visual illustrations should be added, and the narrative should be improved to refer to the new content. The presence of inaccuracies in the references (described in Questions For Authors), structure of the paper (all main information in the very long appendix), and general text quality (long sentences, vague narrative) raises a serious concern about full text generation by an LLM. Additionally, the absence of the code also questions the validity of the results and experiments.

Questions for authors

Other main concerns:

  1. The paper contains inaccuracies in references. Specifically, “Abigail Z. Jacobs. Measurement and Fairness. In FAccT, pp. 375–385, 2021. ” entirely omits Wallach H. as the second author. “Du Tran, Heng Wang, Matt Feiszli, and Lorenzo Torresani. Video Classification With ChannelSeparated Convolutional Networks. In ICCV, pp. 5551–5560, 2019. ” places Matt Feiszli before Lorenzo Torresani.
  2. What is the main contribution of the paper? Is it a benchmark or a method?
  3. The results are rather predictable from the previous work, e.g [1].
  4. Table 16 states that the top 15 models achieves the correlation of 0.64 that is significantly more than random guessing that the paper claims.

[1] Gu K. et al. Using videos to evaluate image model robustness //arXiv preprint arXiv:1904.10076. – 2019.

Things to improve the paper that did not impact the score:

  1. The main text does not fully compare the previous works with the described research. All of the related work is contained in the Appendix, which does not help to understand the positioning of the contributions.
  2. What is image protocol or a multi-frame video protocol in Problem Formulation? ∆rel is introduced there but it does not appear again.
  3. The complete list of models used in the paper is never listed in the main text. The paper cites the models used in the main only in the related work in the Appendix. The complete list of models should include the references to the models. Additionally, UCF101 and HMDB51 datasets are never cited in the main text.

I believe the current text quality and absence of the code are blocking factors for acceptance. Since the paper requires a near-total rewrite, which is difficult to accomplish within the timeframe of a rebuttal, I lean towards rejection.

5 comments

Thank you for recognizing the experimental scale and theoretical contribution. We provide supplementary code and clarify the contribution, novelty, and evidence among the strongest models.

1. Supplementary code: experiment source and executable verification

We provide vrg_rebuttal_supplement.zip as supplementary material accompanying this rebuttal. The archive contains training/evaluation source, dataset and checkpoint wrappers, corruption and codec implementations, the fixed eight-condition probe and rank blend, complete numerical configurations, and 31,683 original condition-level outcomes. Its README maps the claims below to executable checks and recorded evidence.

From the extracted archive:

python reproduce.py --suite integrity
python -m pip install -r requirements-quick.txt
python reproduce.py --suite quick

The first command verifies the distributed files without additional dependencies. The CPU quick check reconstructs model scores, all 28 principal spatial→video correlations, and all 438,900 five-candidate subsets of the full and top-15 Diving48 pools, including ties and within-subset blend ranks. It also reconstructs the literal T=1T = 1 scores from the supplied correctness records. Datasets, checkpoints, and a GPU are unnecessary for these checks.

The full analysis commands rerun the added paired-bootstrap, grouped-model, and held-out analyses; CONFIGURATION.md specifies seeds, repetitions, severity grids, and codec/transport settings. Training and inference recipes support rerunning the benchmark with the public datasets and checkpoints.

Beyond reanalysis, a separate training replication of all 27 Diving48 Transformer-head models gives spatial-ranking agreement ρ=0.97\rho = 0.97 with the submitted runs. Spatial→coherent/structure correlations are 1.00/0.77, compared with 1.00/0.73 in the submission. The coherent-versus-temporal distinction therefore persists after retraining. These 27-model runs are reported separately from the 37-model video-pretrained experiment; the new weights support the added literal T=1T = 1 tests.

2. Main contribution: identifying what image robustness cannot tell us about video models

The primary contribution is a benchmark and scientific analysis establishing the limits of transferring robustness rankings from images to video. The temporal probe is the accompanying model-selection method. The central finding is that temporal robustness contains predictable, practically consequential information that spatial robustness can miss, even when spatial testing is highly predictive for other video corruptions.

The evidence establishes three connected results:

  1. A specific, reproducible separation. The same 37 Diving48 models give spatial→coherent ρ=0.992\rho = 0.992 and spatial→temporal-structure ρ=0.053\rho = 0.053, with each model's weights, clips, temporal head, and input length fixed. The contrast survives using only stochastic spatial operators (0.986/0.097), so it also holds when independent and shared corruption realizations differ. Coherent transfer remains 0.987–1.000 across seven settings.
  2. A measurable source of information beyond spatial robustness. Spatial robustness predicts an omitted spatial operator at median ρ=0.832\rho = 0.832, but a temporal-structure operator at 0.118; other temporal operators predict an omitted temporal operator at 0.805. The first two correlations differ by 0.714, with lineage-bootstrap 95% CI [0.387, 1.203]. Each held-out operator is excluded from its predictor. Holding out an entire spatial group still gives 0.831, and the spatial-within-family versus spatial-to-temporal gap has a positive interval in all four video-pretrained settings. These positive controls distinguish missing temporal information from a generally unreliable benchmark or an unpredictable target.
  3. A theoretical limit with a constructive remedy. Identical frame-level corruption distributions can reverse the optimal video-model choice, imposing a worst-case selection loss even with unlimited frame-level observations. The theory also characterizes when frame-level information is sufficient: on finite frame alphabets, for each fixed label, expected loss is invariant to every change of temporal coupling exactly when the loss is additive across frames. The probe measures temporal behavior directly; on the original Diving48 packet-loss/VFR results, blend/probe reduce mean five-candidate retention regret by 55.4%/90.4%, with the probe requiring 20.5% as many corrupted-clip evaluations as the spatial sweep at equal input length.

The benchmark, theorem, and probe establish the failure, its information limit, and a measured remedy. The selection gains do not depend on the ρ0=0.9\rho_0 = 0.9 threshold.

3. Why the result does not follow from Gu et al. or existing video-corruption benchmarks

Gu et al. (2019) show successful prediction of image-classifier behavior on neighboring video frames. Our result establishes limits on extending image-based proxies to models that integrate frames into a video decision.

Gu et al. (2019), Sections 2 and 4.3

  • Prior work: Translation/color robustness predicts conditional neighboring-frame accuracy across 24 image classifiers.
  • Additional result here: Coherent/structure ranking separation among the same video models, with a formal limit for selectors using only frame marginals.

Yi et al. (2021), Sections 3–4

  • Prior work: Spatial/temporal benchmarks, including compression, packet loss, and frame-rate conversion; architecture, accuracy, and capacity comparisons.
  • Additional result here: Tests whether spatial evaluation can substitute for temporal evaluation, separates visible damage from timing information, and quantifies model-choice errors.

Schiappa et al. (2023), Sections 5–6

  • Prior work: Architecture/pretraining effects and dataset-dependent sensitivity to spatial, temporal, and spatiotemporal perturbations.
  • Additional result here: Held-out-operator controls reveal predictable temporal information missed by spatial testing; aggregate agreement can conceal temporal misranking.

Zeng et al. (2024)

  • Prior work: Temporal-action-detection benchmarks diagnose localization vulnerability and improve robustness through training.
  • Additional result here: Selects among already-trained models using fixed-weight tests, with measured evaluation cost and selection benefit.

The distinction is what one frame reveals. Coherent corruption extends across time yet remains visible in one frame; timing faults can leave delivered frames intact. Temporal sensitivity alone does not establish whether shared architecture or pretraining makes spatial and temporal robustness co-vary. Our paired comparisons and held-out-operator controls test that possibility.

As a direct control, our Gu-inspired adaptation requires both sampled neighbors to be correct given a correct anchor. On the 27 replicated Diving48 candidates, it correlates with temporal-structure and codec-adjusted timing retention at 0.504/0.542. This uses our task and sampling, rather than replicating the protocol of Gu et al. (https://arxiv.org/abs/1904.10076).

Aggregate agreement can conceal temporal misranking. Averaging the four video families, excluding the spatial reference, gives spatial→aggregate ρ=0.876\rho = 0.876 on Diving48-3D and 0.929 on public SSv2, while temporal-structure correlations are 0.053/−0.474. Thus an apparently successful aggregate proxy can miss the temporal failure.

The remedy transfers beyond overlapping operators and evaluation clips. On Diving48-3D, removing any one probe operator preserves 84.4–94.3% regret reduction. Removing synthetic VFR and frame drop together retains pipeline ρ=0.937\rho = 0.937, versus spatial 0.341, without tuning the remaining battery or blend weight. On the separate 27-model replication, the clean-aware probe reduces accuracy regret relative to literal T=1T = 1 by 54.2% when selection and evaluation use disjoint clip halves.

We retain the established corrupted/clean accuracy normalization used by Yi et al. (https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/2a38a4a9316c49e5a833517c45d31070-Abstract-round2.html) and Schiappa et al. (https://openaccess.thecvf.com/content/CVPR2023/html/Schiappa_A_Large-Scale_Robustness_Analysis_of_Video_Action_Recognition_Models_CVPR_2023_paper.html), alongside absolute accuracy. The advance is the controlled transfer result, its information boundary, and the demonstrated value of the missing measurement.

4. Table 16 in the submission: the selection benefit persists among the strongest models

The submission’s Table 16 (Table 24 in the revision) strengthens the practical conclusion: restricting attention to the 15 strongest models by clean accuracy still leaves substantial value in temporal testing. The near-chance statement refers to the full 37-model pool, where pairwise concordance is 52.3%. That table explicitly reports the different restricted-pool result: ρ=0.639\rho = 0.639 and 73.3% concordance. A positive association and reliable selection of the best candidate are different properties, which we now evaluate together:

Diving48 candidate poolSpatial→structure ρ\rhoStructure pairs reversedMean pipeline-retention regret: spatial / blend / probe
All 37 models0.05347.7%0.02535 / 0.01132 / 0.00244
Top 15 by clean accuracy0.63926.7%0.02906 / 0.01345 / 0.00045

The last column evaluates actual packet-loss/VFR pipeline targets and averages all 435,897/3,003 five-model subsets, respectively, with blend ranks recomputed within each subset. Within the top 15, the probe reduces mean retention regret by 98.5% and raises the rate of selecting the highest-retention candidate from 26.4% to 87.3%. The benefit survives removing all 22 lower-clean-accuracy models and establishes the value of temporal testing among competitive candidates independently of a correlation threshold. We have placed the full-set scope of the near-chance statement and the top-15 selection evidence beside the first main-text claim, in Section 4.3, and in the updated Table 24 caption.

5. A self-contained main text and verifiable reference provenance

The code and revised manuscript support verification during the rebuttal. The revised main text brings together the question, controlled comparison, information limit, and selection benefit:

ConcernConcrete revision
Main evidence is difficult to locatePlaced per-family correlations and uncertainty beside the claim, followed by a pipeline selection-regret/cost table (Tables 1–3).
Heavy prose and distant appendix referencesAdded the corruption overview and worked theorem example; shortened sentences and removed repeated numerical prose.
Contribution and prior workStated the benchmark's scientific finding and the probe's methodological role; included the prior-work comparison in the Introduction.
Model and dataset referencesAdded a cited model inventory in Section 2 and Appendix G (Table 10); cited UCF101 and HMDB51 at first mention.
Protocol terminology and unused notationDefined the three protocols together, placed the T=1T = 1 evidence beside the claim (Table 2), and removed unused Δrel\Delta_{\mathrm{rel}}.

The author listings for these papers differ across the available publication records. We double-checked the references using DBLP and Google Scholar. Our current author strings and page ranges match the proceedings records we used: DBLP's Measurement and Fairness, FAccT 2021 and CSN, ICCV 2019. For CSN, they also match the official IEEE Computer Society proceedings record. Based on these checks, we consider the current entries correct for the cited proceedings records and have retained them. If a further correction is needed, please let us know and we will incorporate it. Thank you for carefully checking these references.

The submitted bibliography was assembled entry by entry from DBLP and Google Scholar; no bibliography entries were generated by AI. This collection process is distinct from the AI assistance disclosed in the submission. The authors are responsible for the text, proofs, code, and results, and the supplementary source, recorded outcomes, and separate training replication make the empirical claims directly checkable.

Reviewer_6Y6POpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether robustness to spatial corruptions is a reliable guide to robustness under video corruption. It compares clean-normalized retention rankings with model weights held fixed, using rank correlations, pairwise agreement, and bootstrap analyses. The static reference is a literal T = 1 score on appearance-dominated tasks. On motion-dependent tasks it instead evaluates multi-frame clips with spatial corruptions applied independently to each frame, retaining the video model's temporal head. Across seven model-discriminating settings, coherent spatial corruption largely preserves the static ranking, while temporal-structure corruption fails the authors' rho_0 = 0.9 criterion. The latter correlation is 0.05 for 37 Diving48 video-pretrained models and -0.47 for public SSv2 checkpoints. Codec and transport experiments extend the comparison to compression, packet loss, and variable-frame-rate judder. A binary two-frame construction establishes a worst-case limitation of marginal-only audits. Finally, an eight-condition temporal probe improves pipeline-rank prediction: the equal-weight blend raises the Diving48-3D correlation from 0.34 to 0.74.

Critical strengths
  1. The evidence spans tasks and model constructions rather than resting on one benchmark. The separation appears with temporal-head image encoders, video-pretrained models, public checkpoints, and DAVIS label propagation (Table 1; Appendices H-N). It also has a measurable selection cost: on Diving48-3D, temporal-structure retention ranges from about 0.724 to 0.920, static-to-structure concordance is 0.52, and the static-best model incurs retention regret of 0.059 (Section 4.2). These results make the model-selection concern concrete. This breadth is the strongest reason to accept the paper.
  2. The packet-loss and VFR experiments give the synthetic results practical support. The temporal-structure score predicts the raw pipeline ranking at 0.78, versus 0.34 for the static reference, with a paired model-bootstrap difference interval of [0.07, 0.80] (Appendix K). Matched compression-only controls help separate timing faults from codec damage. The marginal-only impossibility construction provides a useful explanation of why image information cannot guarantee the correct video ranking.
Critical weaknesses
  1. The headline claim is broader than the main empirical test. On Diving48 and SSv2, s_stat already uses multi-frame clips, the temporal head, and downstream fine-tuned weights (Section 2, lines 162-180). Appendix A states this clearly and motivates the substitution using the 0.98 agreement between single-frame and frame-independent video scores in the appearance regime. That agreement does not establish equivalence on motion-dependent tasks. The strongest experiments therefore test spatial-to-temporal corruption transfer within video models, not whether a pretrained single-image backbone ranking transfers to video. Theorem 1 establishes a worst-case limit but does not validate the observed coherent-transfer rule. The narrower finding remains valuable; the title, abstract, and selection guidance should reflect it unless a literal image-score comparison supports the broader interpretation.
  2. Reproduction settings and boundary inference still need clarification. Appendix G and Tables 17/18 provide detailed model inventories, and Listing 1 gives a temporal-mode implementation excerpt. The remaining gaps in the supplied PDF are numerical training/evaluation seeds, bootstrap repetition counts, corruption-realization counts, per-operator severity-grid values, and codec/transport settings such as bitrate or quality, GOP, packet-loss rates, and VFR timestamp distributions. No study-specific executable artifact is linked in the supplied PDF. These are specific obstacles to reconstructing the experiments, not a claim that the checkpoint inventory is absent. For inference, Appendix E discusses lineage dependence, and Table 12's caption and Appendix M.2 openly describe public-SSv2 motion as a boundary case: the one-sided rule rejects although the model-bootstrap interval is [0.37, 0.94]. Different one-sided tests and two-sided intervals need not give identical decisions, so this is not itself an error or an undisclosed inconsistency. What is missing is a sensitivity check for these weaker decisions using rank-appropriate inference that accounts for related architecture/pretraining lineages. This concern affects the borderline rejection counts more than the central near-zero-correlation findings.
Questions for authors
  1. Is there evidence that the literal single-frame ranking agrees with s_stat on the motion tasks, beyond the appearance-regime correlation of 0.98? If not, can the title, abstract, and practical guidance be narrowed to spatial-versus-temporal corruption transfer within video models?
  2. Can you supply the numerical seeds, bootstrap and corruption-realization counts, severity grids, and codec/transport configuration, including bitrate or quality, GOP, loss rates, and VFR timing? A compact configuration table or reproducible script would address most of this concern.
  3. Do the temporal-structure findings and the probe's correlation gains survive clustering by architecture/pretraining lineage, or keeping one representative per lineage? Please separate these strong results from boundary cases such as public-SSv2 motion, and compare the latter using an aligned one-sided confidence level.
  4. On the existing pipeline results, how much does the probe reduce actual model-selection regret compared with the static ranking? Please distinguish retention from absolute corrupted accuracy, since clean accuracy varies widely in Table 18. Appendix L also reports a mean compression-correlation decrease of 0.20 for the blend; when should a practitioner prefer the probe alone, the blend, or the static score? These questions concern the current selection claim, not a request for a new deployment study.
4 comments

Thank you for the specific questions. The added experiments directly confirm the single-frame result, the comparisons remain strong after grouping related models, and the probe improves actual model choices.

1. A single-frame test confirms the result on a motion task

Yes. On Diving48, evaluating the models one frame at a time gives nearly the same robustness ranking as our multi-frame spatial reference (ρ=0.940\rho = 0.940). That single-frame ranking predicts the same visual damage shared across frames much better than disrupted frame order or timing (0.943 versus 0.615). Each prediction sees only one frame.

We repeated training for all 27 image backbones with temporal Transformer heads, then evaluated the same weights on the same 1,500 clips in two ways. At T=1T = 1, each of eight frame positions is classified separately; we average correctness and divide by single-frame clean accuracy. At T=8T = 8, the model sees all eight frames together, with spatial corruption sampled independently per frame. Holding the trained model fixed makes this a direct test of what changes when temporal information is available.

Target predicted by literal T=1T = 1 robustnessSpearman ρ\rhoPaired-clip 95% CI
Multi-frame spatial reference0.940[0.848, 0.957]
Coherent corruption0.943[0.850, 0.957]
Temporal-structure corruption0.615[0.369, 0.697]
H.264/H.265 compression0.885[0.774, 0.915]
Packet loss + VFR, relative to compression-only controls0.348[0.079, 0.498]

A single-frame test ranks models well for shared visual damage, but less well when the frame sequence is disrupted. All rows compare the same 27 models. The coherent-minus-structure difference is 0.328, with paired-clip 95% CI [0.237, 0.521]. Resampling both clips and related model groups keeps the difference positive: [0.106, 0.741] for 11 architecture/pretraining groups and [0.147, 1.004] for six broader groups. Using only the center frame gives the same ordering: 0.861 for coherent versus 0.628 for temporal structure.

The same distinction matters for actual video delivery. To separate delivery faults from ordinary compression damage, we compare each packet-loss/VFR result with an otherwise identical encode/decode pass with no loss or timing change. We divide retention by that matched control and average the two ratios. Temporal-structure robustness predicts this adjusted target at 0.763, versus 0.348 for T=1T = 1: a paired gain of 0.415 [0.208, 0.621]. The gain also remains positive when clips and model groups are resampled together. Temporal testing therefore measures useful information beyond compression robustness.

We also test the complementary question of transfer from the initializing image checkpoints, using 36 classifiers, 15 ImageNet-C types, five severities, and 10,000 class-balanced images. Their ImageNet-C ranking predicts downstream spatial/temporal-structure rankings at 0.83/0.67 on Diving48, 0.74/0.76 on SSv2, and 0.00/0.16 on DAVIS. Together, the experiments distinguish transfer across tasks from transfer across corruption types. The fixed-weight T=1T = 1 comparison isolates the latter without a change of training or task.

These results directly support the paper's central claim: image robustness does not guarantee video robustness. The single-frame experiment demonstrates the distinction on a motion task, and the fixed-video comparisons establish it across architectures and tasks. We now distinguish initializing-checkpoint, single-frame, and multi-frame spatial evaluation side by side in Section 2, and make the single-frame versus multi-frame comparison explicit in the abstract. Experiments establish the observed coherent transfer; the theorem explains why frame-level information alone cannot guarantee the video ranking.

2. Supplementary code and complete numerical settings

The accompanying supplementary archive, vrg_rebuttal_supplement.zip, makes the results executable. It contains training/evaluation source, model wrappers, corruption and codec implementations, the fixed temporal probe, 31,683 original condition-level outcomes, and the added single-frame correctness records. Its README maps each response claim to the corresponding script and result files.

From the extracted archive, the dataset-free CPU check is:

python -m pip install -r requirements-quick.txt
python reproduce.py --suite quick

This reconstructs the 28 principal correlations, exact candidate-set selection losses, and the added single-frame point estimates. The full analysis commands recompute the paired-clip and grouped-model analyses from the supplied records. Training and inference recipes are included for rerunning the benchmark with the public datasets and checkpoints.

CONFIGURATION.md supplies the complete numerical recipe, including every operator's five-level severity grid. The main settings are:

ComponentReproduction setting
Training and evaluation seedsPyTorch/NumPy initialization and data-order seed 0; evaluation-subset seed 0. Original augmentation generators and the worker-seeded rerun recipe are documented separately. Three UCF101 backbones also have seed-1 repeats.
Corruption drawsOne realization per item, condition, and severity. Independent mode redraws per frame; coherent mode shares the same realization across the clip. Classification uses the cached clip index as seed; DAVIS/KITTI use 1000 × severity + item index.
Evaluation1,500 shared classification clips; 750 for K400 pipelines; 30 DAVIS sequences. Severities {1, 3, 5}, DAVIS {1, 5}, probe {3}.
Original statistical analyses2,000 model/clip bootstrap draws; 5,000 permutations; 4,000 pipeline-contrast draws; 5,000 probe/blend-contrast draws; seed 0.
Added analyses20,000 draws for grouped-model analyses and family contrasts; 5,000 literal paired-clip draws; 10,000 joint clip/group draws per grouping; 1,000 held-out partitions and 5,000 fixed-partition clip draws. Exact seeds are listed by analysis in CONFIGURATION.md.
Codec and transportH.264/H.265 bitrate grids, GOP settings, packet-deletion fractions, VFR duration distributions, and matched no-fault controls are specified in CONFIGURATION.md; PyAV 13.1.0 pins the transport implementation.

The archive separates replay of recorded outcomes from neural training/inference and preserves the original and rerun augmentation recipes. This lets reviewers both check the reported numbers and reuse the evaluation on additional models.

3. Related architectures and aligned inference

Yes. The difference between coherent and temporal-structure transfer remains positive in all seven settings after grouping related models. The Diving48 probe gain also remains positive. We resample whole architecture groups, and separately architecture/pretraining groups, recomputing ranks each time. Groups are defined by model identity.

Two questions make the inference easy to read:

  1. Does the spatial test predict coherent damage better than disrupted frame order or timing? Yes, in all seven settings under both groupings, with positive paired 95% intervals. This directly tests the paper's central pattern and does not use a 0.9 cutoff.
  2. Is temporal-structure agreement below the stricter 0.9 standard for substituting one evaluation for the other? Five settings satisfy this under both groupings. The table gives the one-sided 95% upper bound, U95U_{95}: if even this upper bound is below 0.9, the substitution standard is rejected.
Temporal-structure settingModelsObserved ρ\rhoArchitecture U95U_{95}Architecture/pretraining U95U_{95}
Diving48, Transformer head270.7280.9180.929
SSv2, Transformer head170.7180.8140.924
Diving48, video-pretrained370.0530.5420.512
SSv2, video-pretrained370.2230.5090.469
K400, public checkpoints270.4540.7000.700
SSv2, public checkpoints16−0.4740.7500.750
DAVIS, frozen backbones280.0040.4120.412

For Diving48-3D, the direct coherent-minus-structure difference is 0.939 [0.376, 1.306] under architecture resampling. The seven positive contrasts use pointwise 95% intervals. Thus the main finding is supported both by a direct comparison across all seven settings and by the strict substitution test in five settings.

Two ablations isolate possible explanations. First, we remove all deterministic spatial operators, whose independent and coherent outputs would coincide. The remaining stochastic operators still give 0.986 versus 0.097 on Diving48-3D, with positive paired contrasts in all seven settings under both groupings. Second, we remove architecture multiplicity by taking one model per group. Across all 19,008 compositions of eight architecture groups, spatial→structure correlation stays below 0.524 on both 37-model datasets. Across all 11,880 compositions of eleven architecture/pretraining groups, the maxima are 0.427/0.318 on Diving48/SSv2. The separation persists after controlling both operator overlap and related-model counts.

For the original Diving48 pipeline target, spatial/blend/probe correlations are 0.341/0.742/0.920. The paired architecture-bootstrap gains over spatial are 0.401 [0.115, 0.690] for the blend and 0.579 [0.085, 0.887] for the probe. The blend improves correlation in every one-representative composition under both groupings.

Aligned inference identifies the boundary cases without changing the main result. Public-SSv2 motion has ρ=0.765\rho = 0.765 and model/architecture U95=0.923/0.922U_{95} = 0.923/0.922, so its aligned bounds include 0.9. The two Transformer-head structure rows also include 0.9 under at least one grouping. We label these as boundary cases and show the original Fisher-z and aligned group-bootstrap decisions together in Appendix E.4 (Table 5 and the public-SSv2 discussion). The central evidence is the positive family contrast in all seven settings and the positive Diving48 pipeline gains, each tested directly with related models grouped together.

4. Actual model selection and a consistent practical rule

On the original 37-model Diving48 packet-loss/VFR target, the blend cuts mean retention regret by 55.4%, and the probe by 90.4%. We evaluate all 435,897 five-model subsets, average score ties uniformly, and recompute blend ranks inside each subset.

Selection ruleRetention regret ↓Accuracy regret: same choice, pp ↓Accuracy regret: clean-aware, pp ↓
Spatial reference0.025352.3920.760
Equal-rank blend0.011321.8640.278
Temporal probe0.002443.1360.035
Clean accuracy alone0.028870.6850.685

To choose the model that stays most accurate, combine its starting accuracy with how well it retains that accuracy. A model starting at 80% accuracy and retaining 90% is predicted to reach 72%; retaining 95% of a 50% starting accuracy gives only 47.5%. We therefore select for accuracy using clean video accuracy×predicted retention\text{clean video accuracy} \times \text{predicted retention}; the blend averages the ranks of these products. This clean-aware rule reduces accuracy regret by 63.4%/95.4% for blend/probe relative to clean×spatial. The table also shows the accuracy of the original retention-guided choices, making the two objectives directly comparable. Here pp means percentage points, and percentage reductions describe regret, not an increase in accuracy by that percentage.

These are exact subset means for the measured models. The paired architecture intervals above assess ranking gains; the following independent-clip control tests selection on new clips.

The benefit also appears when selection and evaluation use different clips. On the separate 27-model replication, we select on 750 clips and evaluate on the other 750, then swap the halves. Each of 1,000 partitions averages all 80,730 five-model subsets. The target is actual packet-loss/VFR accuracy.

Clean-aware selection scoreHeld-out accuracy regret, pp: mean ± SD
Literal T=1T = 10.261 ± 0.044
Multi-frame spatial reference0.227 ± 0.034
Temporal probe0.120 ± 0.042
Clean accuracy alone0.183 ± 0.076

The probe reduces held-out accuracy regret relative to the literal single-frame test by 54.2%, a paired reduction of 0.141 ± 0.040 pp. A separate within-half clip bootstrap on a fixed partition confirms a positive reduction: 0.135 pp [0.055, 0.408]. This directly answers whether temporal testing improves choices made using image-only evidence on new clips. The SDs describe the 1,000 data partitions; the fixed-partition interval is a separate uncertainty analysis. The spatial and clean-only rows provide descriptive controls, with their zero-spanning paired intervals included in the supplementary results.

Matching clean performance preserves the benefit. Among all 68 cross-architecture pairs within two clean-accuracy percentage points, clean×spatial versus clean×probe yields 1.195 versus 0.010 pp mean accuracy regret. Equal weighting of architecture pairs gives 1.180 versus 0.033 pp.

The practical rule is to test the damage expected in use and choose for the performance that matters. The results identify when each measurement earns its cost:

  • Compression: use the spatial score. It gives the lowest mean retention regret in all three evaluated codec settings. The reported 0.20 mean correlation decrease from blending is mainly driven by UCF101 (−0.50); the Diving48-Transformer/K400 decreases are 0.05–0.06. This identifies a setting where the spatial test already measures the relevant damage.
  • Disrupted frame delivery: use temporal testing to measure timing sensitivity. On Diving48-3D, the probe's mean-retention-regret benefit persists across all tested packet-loss/VFR mixtures and candidate counts 2–35.
  • Both sources of damage: combine the spatial and temporal rankings with the fixed 50:50 blend, then evaluate the shortlist on the intended pipeline. The weight is fixed independently of pipeline outcomes.

Screening gives a concrete benefit even when all 37 models are available. The pipeline-best retention model is ranked eighth by spatial testing, third by the blend, and second by the probe. With a budget of three subsequent pipeline evaluations, blend and probe screening both recover the best model; spatial screening leaves 0.027 retention regret. This is the measured value of bringing the right candidates into a small shortlist. Direct top-1 retention regrets are reported separately (0.027/0.027/0.029 for spatial/blend/probe), and all clean-aware top-1 rules select VideoMAE-H at 71.00% pipeline accuracy.

The supplementary comparisons also show why the recommendation depends on the target: in the smaller Transformer-head/K400 timing sets, the standalone probe does not lower mean retention regret and blend correlation-gain intervals span zero. The practical contribution is a measured rule for choosing the evaluation, supported by the large Diving48-3D decision gains, the independent single-frame comparison, and the shortlist savings above.

The probe uses the five temporal-structure and three motion-coupled operators at severity 3, with equal family weights. The battery and blend weight are fixed independently of pipeline outcomes. At equal video input length, its cost is 8/39 = 20.5% of the spatial sweep, or 20.5% extra when added to that sweep.

Comments on the paper

1 comment

We thank the reviewers for their assessments. The paper establishes an empirical and theoretical limit of image-based robustness evaluation: a spatial score can preserve video-model rankings almost perfectly for one corruption family while missing the information needed to rank the same models under temporal faults. Our benchmark measures this separation, our theory explains the missing information, and eight targeted temporal tests recover substantial model-selection value. The added single-frame experiment directly tests this claim on a motion-dependent task.

Three results connect the controlled comparison, theory, and practical benefit:

  • A reproducible separation. With weights, clips, temporal head, and input length fixed, spatial robustness predicts coherent-corruption robustness at Spearman ρ=0.987–1.000\rho = 0.987\text{–}1.000 across seven settings. On the 37-model Diving48 set, its correlation with temporal-structure robustness is 0.053. The separation survives architecture grouping and a stochastic-operator-only ablation.
  • Direct single-frame validation. On a separate 27-model Diving48 training replication, evaluating one frame at a time gives nearly the same model ranking as the multi-frame spatial reference (ρ=0.940\rho = 0.940). It predicts coherent damage much better than disrupted frame order or timing (0.943 versus 0.615). The difference remains positive when both clips and related model groups are resampled.
  • Measured selection value. On the original Diving48 packet-loss/VFR results, the fixed equal-rank blend and eight-condition temporal probe reduce mean five-candidate retention regret by 55.4% and 90.4%. At equal input length, the probe uses 20.5% as many corrupted-clip evaluations as the spatial sweep.

Retention is corrupted accuracy divided by clean accuracy; regret is performance lost relative to the best candidate. We report absolute corrupted accuracy separately. We provide vrg_rebuttal_supplement.zip as rebuttal supplementary material, containing experiment source, numerical configurations, recorded outcomes, and CPU reproduction commands. The revised manuscript now puts the protocols and main evidence together in the main text (Sections 2–4; Tables 1–3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.