- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper is on whether ranking a set of models based their performance under image degradation will align with their rank under video degradation. The result is somewhat expected: if the video corruption is mostly an image corruption present at similar levels in every frame, the ranking based on image performance will predict well video.
The paper also demonstrates that image-only degradation assessment will always have this limitation (cannot guarantee correct ranking for video-level degradations).
Critical strengths
As the main topic of the paper ("IMAGE ROBUSTNESS DOES NOT IMPLY VIDEO ROBUSTNESS") seems somewhat self evident, one of the strengths seems to be the attempt to perform theoretical analysis via an existence theorem that demonstrates "worst-case limits" that the rank reversal can happen for video-level (dynamic) degradation..
Critical weaknesses
The contributions are limited and somewhat expected (e.g., that one should ideally evaluate model performance on representative types of distortion, that is include video-specific degradations if expected to perform well on such tasks).
Even the largest strength ("theoretical analysis") feels somewhat inaccessible and hard to parse: It's hard to connect the mathematical abstractions to the practical examples.
The paper feels overly wordy for the magnitude of the contributions. Results are often forward linking to an even more verbose appendix. Some examples:
- line 103 "Fig. 4 in App. F.1 illustrates why" -- which is on page 27 in the appendix.
- line 274: " Table 3" which is on page 33
- line 360: "with clean top-1 from 46.5% to 84.1% (Table 18)." Table 18 is on page 50.
- line 468: (Fig. 6). On page 41.
The authors should exhibit more discretion when deciding what to include (and exclude) in order to make the paper clear, concise, and easy for a human to understand.
Questions for authors
None
6 comments
2. The theorem through a concrete example
Knowing how often each frame is corrupted does not tell us which frames are corrupted together. That missing information can reverse the model ranking.
Consider three frames: the ball is on the same side in frames 1 and 2, then either stays or switches sides in frame 3. For example, left → left → right has label “changed side.” Model A compares frames 1 and 3; model B compares 2 and 3. Both are 100% correct on clean clips. The starting side and stay/change label are sampled independently with equal probabilities, so one frame alone gives no label information.
Now mirror selected frames horizontally, swapping left and right:
| Corruption process | Frames mirrored in each random event | Model A accuracy | Model B accuracy |
|---|---|---|---|
| Mirror frames 1 and 3 together | 40%: {1, 3}; 40%: {2}; 20%: none | 100% | 20% |
| Mirror frames 2 and 3 together | 40%: {1}; 40%: {2, 3}; 20%: none | 20% | 100% |
Mirroring both frames used by a model preserves their same/different relationship; mirroring just one reverses it. In the first process, A is always correct, while B is correct only in the unchanged 20% of clips. The second process swaps their roles. Yet every frame is mirrored with probability 0.4 in both processes, for every clean clip. Thus an audit of individual-frame corruption distributions sees identical evidence even though the best model changes.
An image-only selector must therefore choose A with the same probability in both processes. Its regret is in the first and in the second: the chance of choosing the worse model times the 0.8 accuracy gap. These regrets sum to 0.8, so at least one is 0.4 or larger. More single-frame samples cannot reveal which frames change together. This illustrates the submitted theorem's worst-case limit with clean accuracies matched.
Frame-level evidence is sufficient when loss adds up frame by frame. For a fixed label and finitely many possible values per frame, all ways of combining the same individual-frame distributions give the same expected loss if and only if
Here is frame , is the label, is a baseline, and is frame 's contribution to the total loss . We can average each frame's contribution separately and add the results. Those averages depend only on the individual-frame distributions, so which frames change together cannot affect the expected total loss. Errors in the ball example depend on the relationship between two frames, so this additive condition fails. Without it, identical frame-level distributions do not guarantee equal expected loss. The experiments measure when this missing temporal information changes rankings and how temporal testing improves model selection.
3. Making the main text self-contained
We have replaced the four distant references you identified with evidence at the point of use:
| Reference in the submission | Main-text revision |
|---|---|
| Line 103 → Fig. 4, Appendix F.1 | Added corruption examples to Figure 1, showing what an individual frame reveals. |
| Line 274 → Table 3 | Placed per-family correlations and uncertainty beside the transfer comparison (Section 3; Tables 1–2). |
| Line 360 → Table 18 | Added a cited model inventory in Section 2 and Appendix G (Table 10), with protocols, pretraining, and clean-accuracy ranges. |
| Line 468 → Fig. 6 | Combined pipeline correlation, selection regret, and evaluation cost in Table 3. |
We have also put the closest related-work comparison in the Introduction, defined literal and the multi-frame spatial reference side by side, and removed prose that repeats tables. The added literal single-frame experiment now appears beside the central claim (Section 3, Table 2). The supplementary code and manuscript evidence map provide executable checks and the location of each revision.
Thanks for the clarifications and for trying to fix up some of the issues in the main text.
With that and a second pass through the paper I think I can see a little bit clearer what the authors are trying to convey.
However, the result still seems expected. (e.g., Model A can perform better than Model B when benchmarked on class of distortions X, but then the performance can flip such that Model B > Model A when tested on some other class of distortions Y). If one cares about the performance of the model in an environment containing a mix of distortions X and Y you probably want to evaluate performance on both types of distortions (or something representative of them) when doing model selection. And if distortion Y is rare (lets say 1% chance of happening), then that should also factor into model selection.
So I'm still failing to see the significance of the main result in the paper. If one cares about performance of a model in an application where a new or different mix of distortions is possible, then one should probably evaluate using relevant distortions. For classification tasks where datasets already exist, simply evaluating on those distortions should straightforward (unlike say image / video quality assessment, where a new human study may need to be done in order to collect annotations).
Another reviewer had asked the question ("What is the main contribution of the paper? Is it a benchmark or a method?). There is likely a way to structure the narrative that better demonstrates the impact & conributuion of the results (e.g., an exhaustive benchmark result that previous methodologies have been evaluating models incorrectly and should be using a "temporal probe", or that the temporal aspect had not been considered enough is important due to reasons that the others would demonstrate, or that it isn't mandatory to exhaustively test many temporal distortions and that a particular combination of independent spatial and temporal probes are sufficient, etc.).
Thank you for the comments on novelty and accessibility. We clarify the empirical question and give a concrete explanation of the theorem.
1. What does the study establish beyond the intuition?
A frame's inability to show a timing fault does not tell us whether spatial robustness still identifies the most robust video models. Robustness to different faults could be shared across models. Our study tests that possibility and measures the cost of relying on it.
We isolate the effect of corruption type through a controlled comparison and additional ablations. For the same 37 Diving48 models, weights, clips, temporal heads, and input lengths stay fixed; only the corruption family changes. Spatial→coherent is ρ=0.992, while spatial→temporal-structure is ρ=0.053. Resampling whole architecture groups accounts for related models and preserves a positive paired difference: 0.939 [0.376, 1.306].
We also test whether the strong coherent result depends on spatial operators that produce identical outputs under independent and shared application. Removing these deterministic operators leaves only stochastic operators and preserves the separation (0.986/0.097). Thus the result persists when the two spatial tests use different random corruption patterns.
The distinction is large for models that look equivalent under spatial testing. An illustrative pair from the submission is:
The spatial scores differ by less than 0.001, while temporal-structure retention differs by 0.197; the models rank last and first on that family, with the same ordering under both pipeline faults. The pair illustrates the largest temporal gap among spatial gaps ≤ 0.005: nearly equal spatial scores can conceal a large temporal difference.
An additional leave-one-corruption-out control asks whether poor prediction is simply due to testing an omitted fault. The spatial score predicts an omitted spatial operator at median ρ=0.832, but a temporal-structure operator at 0.118; other temporal operators predict an omitted temporal operator at 0.805. Prediction remains strong within spatial and temporal families, while the spatial→temporal gap persists.
The missing measurement has a measured benefit. Across all 435,897 five-model subsets of the original Diving48 pipeline results, mean retention regret falls from 0.02535 with spatial testing to 0.01132 with the fixed blend and 0.00244 with the probe. The probe's 90.4% reduction costs 20.5% as many corrupted-clip evaluations as the spatial sweep. Its correlation gain has architecture-bootstrap 95% CI [0.085, 0.887]; removing any one operator preserves 84.4–94.3% regret reduction.
These findings support one evaluation principle: measure spatial and temporal robustness separately, then use the information relevant to the target faults and selection objective. The study supplies the controlled evidence, information boundary, and measured selection benefit for applying it.