- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies what depth probes recover from pretrained video models. It shows that high scores on driving videos can largely reflect image position, rather than the detailed scene level geometry. The authors audit six choices: position and motion controls, diffusion noise level, network block, temporal window, and patch grid. Among ten checkpoints, controlled KITTI experiments show larger learned depth increments for predictive encoders than for generators. The exact generator count changes when the block is selected separately: Mochi reaches a small positive increment. Additional tests examine dense optical flow, nonlinear probes, VAE compression, and indoor videos. A second analysis also finds a shared subspace across several models that transfers ordinary depth probes well, while mostly encoding image layout under the tested controls.
Critical strengths
1: The paper distinguishes its findings from predictions that failed, including the locality screen and the held-out LTX forecast. This helps people to judge the actual evidence. The shared-subspace result is also useful: cross-model depth-probe transfer is near the self-transfer ceiling, yet the selected shared directions decode position particularly well and fail the residual-depth gate.
2: It makes six important choices explicit, including position controls, motion controls, noise level, layer selection, temporal window, and patch resolution. This framework can help researchers design clearer evaluations and explain why different studies reach different conclusions.
3: The paper does more than add a weak baseline. Position alone explains much of KITTI depth, and the authors then test whether the remaining advantage comes from motion summaries, readout choice, or the video VAE.
Critical weaknesses
1: Section 5.2 selects blocks and evaluates them using different splits of the same 200 scenes. Selection can therefore use labels from scenes that later enter evaluation; the paper acknowledges this and the below-nominal interval coverage.
2: Predicting the depth left after removing image position is different from measuring how much model features improve prediction beyond a position baseline. These measures use different reference scales, but the paper sometimes presents them together. This makes the main claim harder to interpret and needs a clearer explanation.
3: The author chooses different layers and evaluates them using different splits of the same scenes. Some scenes can therefore influence both decisions. Several proposed controls are also tested separately rather than together. A separate test set that combines the main controls would provide stronger support for the final conclusion.
Questions for authors
1: With block selection confined to an outer training set, does the dense-flow result still show a positive encoder-generator margin? Please include uncertainty and the two native-grid generators? 2: On the final KITTI protocol, how much of each learned increment remains when the input repeats the target frame? This would separate gains from video motion from gains available from static appearance cues?
2 comments
I appreciate the clarification, but I am still not fully convinced by the reported encoder-generator gap. The scene-disjoint test give unstable estimates, while the alternative still reuses scenes for layer selection and evaluation, so my concern about the realiability of the comparison remains, thus maintaining my original score.
Response to dutm
Thank you — W2 identified a real error that reached our abstract, and W1/W3 asked for a design we had run and failed to report. Both are fixed.
W1, W3: selection and scoring share the same 200 scenes; a separate test set would be stronger.
The scene-disjoint design is what we preregistered (PREREG H.1), and we ran it at n=200 — 100 scenes to select the block, 100 to score it. It fails its own abort gate, which asks that each model's committed block reproduce its published interval. The mechanism is in the floor, not the model:
V-JEPA2's trained Δ barely moves (+0.076 → +0.068); its untrained floor falls from −0.042 to −0.438, and the gap interval widens to [+0.034, +1.125]. Halving the training scenes costs the floor arm more than the trained arm, and only where the floor is negative — that is, for the encoders. The preregistered design therefore inflates precisely the class our headline favours, four- to sevenfold, and we report the seed-separated arm instead and state its weaker independence rather than take the larger number. This is now in §5.2 with the numbers in Appendix H; the four runs are in the released archive.
On controls tested separately: Table 12 applies five of the six simultaneously on 200 scenes — position and flow regressed out, generators read at a clean latent, a fixed grid, the depth target centred in the window. §5.2 adds the sixth. The one combination we do not yet have is the one you ask for in Q1.
W2: R²_resid and Δ have different reference scales and were presented together.
Correct, and it had reached the abstract, which wrote that a probe reading 0.848 "adds +0.122 over" a position baseline of 0.844. Those two numbers do not add: 0.848 is scored against the variance of depth, 0.122 against the 0.156 that position leaves. Changes:
The conclusion is unchanged in direction and slightly larger in size; what changed is that the ratio is now between two numbers that share a denominator.
Q1: with block selection confined to an outer training set, does the dense-flow result still show a positive margin, with uncertainty and the two native-grid generators?
We do not yet have that cross; we have each control separately, on 200 scenes:
Each control costs roughly a quarter to a third of the margin on its own, and neither closes it. We are running the two together, with the bootstrap interval and with Wan2.2 and LTX-Video at their native 8×8 grid. [CONTINGENT] We will post the result during the discussion period. We will not state the combined margin before measuring it.
Q2: how much of each increment survives when the input repeats the target frame?
On the motion-parallax stimuli this control is in the paper: freezing the middle frame drops the position-residualized readout from 0.517 to 0.076 for V-JEPA2 and from 0.948 to −0.009 for VideoMAE, against random-init controls of −0.025 and −0.017. What survives position control there is motion-derived, not static appearance. [CONTINGENT] We are running the same control on the final 200-scene KITTI protocol, preregistered with its design, reading rule and abort conditions before the first extraction (Section I of the released preregistration), and will post the result during the discussion period.