acceptodds
Under review as a conference paper at ICLR 2027

Most of the Geometry Credited to Video Foundation Models Is a Ground Plane: Six Probe Choices, and the Small Effect That Survives Them

Abstract

A video foundation model is credited with 3D structure when depth decodes from its features, and a naive probe on driving footage reads R² near 0.85. A quadratic in the patch's own row and column reaches R²=0.844 on the same data; regress it out, and the probe that read 0.848 adds +0.122 over it. Most of that number is a ground plane. Image position is one of six choices a depth probe makes, with motion, noise timestep, block, temporal window and patch grid; none is consistently reported, and in an exploratory audit on ten open checkpoints each moves the verdict, by 1.6× to about 50× or a change of sign. We give a configuration that closes all six and test what survives. On 200 real driving clips, every choice controlled but the block, which is fixed the usual way, four of four predictive encoders carry depth beyond position and flow and zero of six generators do; selecting the block on held-out splits as well keeps the encoder count and lets one generator cross at +0.0188, 2.5 times below the weakest encoder. The classes separate by magnitude, not by kind: an effect worth 0.05–0.12 of variance, an order of magnitude below the headline numbers it replaces. It survives a dense optical-flow control, which narrows the margin by a third, and a VAE control, after which encoders keep 70% of their advantage. Representational convergence, often read as corroboration, points the other way: the converging models that align patch for patch share a subspace that transfers a depth probe between them at ceiling, yet position-residualized depth decodes from it in 0/5 models, so what independently trained video models agree on is where things sit in the frame. Harness, per-experiment results, preregistration and decision ledger are released.

Primary area:
interpretability and explainable AI
Keywords:
video foundation models, representation probing, depth, 3D geometry, evaluation methodology, confounds, representational convergence, world models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dutmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what depth probes recover from pretrained video models. It shows that high scores on driving videos can largely reflect image position, rather than the detailed scene level geometry. The authors audit six choices: position and motion controls, diffusion noise level, network block, temporal window, and patch grid. Among ten checkpoints, controlled KITTI experiments show larger learned depth increments for predictive encoders than for generators. The exact generator count changes when the block is selected separately: Mochi reaches a small positive increment. Additional tests examine dense optical flow, nonlinear probes, VAE compression, and indoor videos. A second analysis also finds a shared subspace across several models that transfers ordinary depth probes well, while mostly encoding image layout under the tested controls.

Critical strengths

1: The paper distinguishes its findings from predictions that failed, including the locality screen and the held-out LTX forecast. This helps people to judge the actual evidence. The shared-subspace result is also useful: cross-model depth-probe transfer is near the self-transfer ceiling, yet the selected shared directions decode position particularly well and fail the residual-depth gate.

2: It makes six important choices explicit, including position controls, motion controls, noise level, layer selection, temporal window, and patch resolution. This framework can help researchers design clearer evaluations and explain why different studies reach different conclusions.

3: The paper does more than add a weak baseline. Position alone explains much of KITTI depth, and the authors then test whether the remaining advantage comes from motion summaries, readout choice, or the video VAE.

Critical weaknesses

1: Section 5.2 selects blocks and evaluates them using different splits of the same 200 scenes. Selection can therefore use labels from scenes that later enter evaluation; the paper acknowledges this and the below-nominal interval coverage.

2: Predicting the depth left after removing image position is different from measuring how much model features improve prediction beyond a position baseline. These measures use different reference scales, but the paper sometimes presents them together. This makes the main claim harder to interpret and needs a clearer explanation.

3: The author chooses different layers and evaluates them using different splits of the same scenes. Some scenes can therefore influence both decisions. Several proposed controls are also tested separately rather than together. A separate test set that combines the main controls would provide stronger support for the final conclusion.

Questions for authors

1: With block selection confined to an outer training set, does the dense-flow result still show a positive encoder-generator margin? Please include uncertainty and the two native-grid generators? 2: On the final KITTI protocol, how much of each learned increment remains when the input repeats the target frame? This would separate gains from video motion from gains available from static appearance cues?

2 comments
AuthorsOpenReview ↗

Response to dutm

Thank you — W2 identified a real error that reached our abstract, and W1/W3 asked for a design we had run and failed to report. Both are fixed.

W1, W3: selection and scoring share the same 200 scenes; a separate test set would be stronger.

The scene-disjoint design is what we preregistered (PREREG H.1), and we ran it at n=200 — 100 scenes to select the block, 100 to score it. It fails its own abort gate, which asks that each model's committed block reproduce its published interval. The mechanism is in the floor, not the model:

published gapscene-disjoint arminside published CI?
V-JEPA2+0.1187+0.5061no (4.3×)
V-JEPA2-ViT-H+0.0986+0.7050no (7.2×)
CogVideoX-2B+0.0110+0.0044yes
CogVideoX-5B+0.0039−0.0034yes

V-JEPA2's trained Δ barely moves (+0.076 → +0.068); its untrained floor falls from −0.042 to −0.438, and the gap interval widens to [+0.034, +1.125]. Halving the training scenes costs the floor arm more than the trained arm, and only where the floor is negative — that is, for the encoders. The preregistered design therefore inflates precisely the class our headline favours, four- to sevenfold, and we report the seed-separated arm instead and state its weaker independence rather than take the larger number. This is now in §5.2 with the numbers in Appendix H; the four runs are in the released archive.

On controls tested separately: Table 12 applies five of the six simultaneously on 200 scenes — position and flow regressed out, generators read at a clean latent, a fixed grid, the depth target centred in the window. §5.2 adds the sixth. The one combination we do not yet have is the one you ask for in Q1.

W2: R²_resid and Δ have different reference scales and were presented together.

Correct, and it had reached the abstract, which wrote that a probe reading 0.848 "adds +0.122 over" a position baseline of 0.844. Those two numbers do not add: 0.848 is scored against the variance of depth, 0.122 against the 0.156 that position leaves. Changes:

  • Table 1 row 1 now puts both cells on the variance of depth: naive 0.828 against +0.114 beyond position, a 7.3× swing (previously 0.848 against +0.122, 7×).
  • The abstract now says "of the variance that leaves, the probe that read 0.848 recovers 12%".
  • §4 states the denominator where R²_resid is introduced and says it is never differenced against Δ; Table 7 and the defaults table follow.

The conclusion is unchanged in direction and slightly larger in size; what changed is that the ratio is now between two numbers that share a denominator.

Q1: with block selection confined to an outer training set, does the dense-flow result still show a positive margin, with uncertainty and the two native-grid generators?

We do not yet have that cross; we have each control separately, on 200 scenes:

  • out-of-fold block, frame-difference proxy: weakest encoder +0.0475, Mochi-1 +0.0188 → margin +0.0287;
  • committed block, dense Farnebäck field: weakest encoder +0.0367, strongest generator +0.0132 → margin +0.0235;
  • committed block, proxy (published): margin +0.0344.

Each control costs roughly a quarter to a third of the margin on its own, and neither closes it. We are running the two together, with the bootstrap interval and with Wan2.2 and LTX-Video at their native 8×8 grid. [CONTINGENT] We will post the result during the discussion period. We will not state the combined margin before measuring it.

Q2: how much of each increment survives when the input repeats the target frame?

On the motion-parallax stimuli this control is in the paper: freezing the middle frame drops the position-residualized readout from 0.517 to 0.076 for V-JEPA2 and from 0.948 to −0.009 for VideoMAE, against random-init controls of −0.025 and −0.017. What survives position control there is motion-derived, not static appearance. [CONTINGENT] We are running the same control on the final 200-scene KITTI protocol, preregistered with its design, reading rule and abort conditions before the first extraction (Section I of the released preregistration), and will post the result during the discussion period.

I appreciate the clarification, but I am still not fully convinced by the reported encoder-generator gap. The scene-disjoint test give unstable estimates, while the alternative still reuses scenes for layer selection and evaluation, so my concern about the realiability of the comparison remains, thus maintaining my original score.

Reviewer_6R78OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies different ways to probe geometry in video foundation models. It predicts depth from frozen features and varies image position controls, motion controls, diffusion noise, layer choice, temporal windows, and patch resolution. On KITTI, image position explains much of the decoded depth, while the tested encoders retain more depth information beyond these controls than the generators. Additional synthetic and ScanNet experiments examine these findings. The paper also compares shared features across models and interprets their common information as image layout.

Critical strengths

Useful Controls: The position-only baseline shows why high depth-probe accuracy on driving scenes should be interpreted carefully. Comparing against untrained features is also appropriate. The nonlinear readout and ScanNet experiments strengthen the depth-probing results, and the authors openly report failed tests and changes from their original plan.

Critical weaknesses
  1. The Main Claim Is Too Broad: The paper shows that image position explains much of a depth score on KITTI. I am not convinced that this establishes its central claim that most geometry credited to video foundation models is a ground plane. Table 5 states that depth is the only target carried through the evaluation; pose, occupancy, and normals stopped at pilots, and the geometric-law test covers only reflection equivariance. The cited Probing the 3D Awareness of Visual Foundation Models also tests consistency across views, which this paper does not directly reassess. The ScanNet extension is useful but includes only one generator. The authors acknowledge these restrictions, yet the title and opening conclusion still extend the depth-probing finding to geometric representations generally. This mismatch concerns the paper's main scientific contribution, not just its wording.
Questions for authors
  1. Which completed tests support the broad conclusion about geometric representations, beyond depth recoverable under the evaluated probes?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 6R78: claim rescoped to depth, no result removed

Thank you for a precise objection. It is correct, and the revision acts on it.

Q1. Which completed tests support the broad conclusion about geometric representations, beyond depth recoverable under the evaluated probes?

None. Your reading of Table 5 is accurate: of the four targets the ladder preregistered only depth was carried through, pose, occupancy and normals stopped at pilots, ScanNet retests one generator, and we do not reassess the multi-view correspondence that el Banani et al. probe. The one completed test that is not about depth, L3's reflection lawfulness of the features, says nothing about a ground plane in either direction. Depth recoverable under the evaluated probes, the scope your question names, is the scope the evidence supports, and it is the scope the paper now claims.

Why rescoping changes the claim and leaves the evidence. You note that the mismatch concerns the main contribution and not only its wording. It sat in the main claim, and only there. Every confirmatory result in the paper, and each of the four contributions listed in §1, is a measurement of depth under the evaluated probes, so the rescoped paper loses no result and keeps no contribution that rests on an untested target. The revision states the claim at that scope:

  • Title, once the rules allow a change (see the note below): Most of the Depth Credited to Video Foundation Models Is a Ground Plane.

  • Conclusion, first sentence: "Most of the depth credited to video foundation models is a ground plane".

  • Limitations (§7) opens with the scope statement, instead of leaving it to be assembled from Table 5 and the appendices:

    Every confirmatory claim here is about depth. Of the four geometric targets the ladder preregistered only depth was carried through (Table 5), L3 tests one of its four residuals, and we do not reassess the multi-view correspondence el Banani et al. also probe: what these features carry about pose, occupancy or normals is untested here.

What the paper establishes at that scope. That monocular depth networks judge distance from vertical image position rather than apparent size is known (van Dijk & de Croon, ICCV 2019; now cited in §2). The probing studies that credit video foundation models with 3D understanding from shallow readouts (Huang et al., 2025), or define 3D awareness through them (el Banani et al., 2024), do not regress position out of the depth readout, and the paper measures what that omission costs:

  1. The naive number does not rank the models. Three of our checkpoints read R² = 0.782 to 0.848 naive on the same KITTI frames. Scored against the variance a quadratic in row and column leaves, they span +0.122 to −0.098, the last predicting depth worse than the ground plane alone (§1, Figure 1a).
  2. Position is one of six such choices. Motion control, noise timestep, block, temporal window and patch grid each move the verdict by 1.6× to about 50× or flip its sign on fixed models and fixed data, and we know of no published probe that controls more than one (Table 1).
  3. What survives is small, and it separates the classes by magnitude. With five of the six closed on 200 KITTI scenes, four of four predictive encoders keep depth beyond position and flow, worth 0.05–0.12 of variance, and none of the six generators clears its untrained floor. Selecting the block out of fold as well lets Mochi-1 cross at +0.0188, 2.5 times below the weakest encoder (§5.1–5.2).

On the title. The ICLR 2027 author guide does not permit title changes until the discussion period ends, so the OpenReview title and the revised PDF keep the submitted title for now. The title above is the one the paper will carry.

We would be grateful if you would reconsider your assessment in light of the rescoped claim.

I maintain my strong reject recommendation. Reading this apparent AI slop—the paper, the rebuttal, and ytFo’s reviews—has been a waste of my time. Please take your responsibilities seriously.

Reviewer_ytFoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper makes a valuable methodological contribution by showing how strongly depth probing results can depend on seemingly minor evaluation choices. The six-choice audit is clear, practically useful, and likely to improve how future work evaluates geometric representations in video models.

Critical strengths

The empirical analysis is unusually careful and transparent. The use of scene-disjoint splits, random-feature and random-initialization controls, permutation tests, bootstrap intervals, preregistration, and explicit reporting of failed predictions gives the conclusions substantial credibility.

I appreciate that the paper does not force an overly simple encoder-versus-generator narrative. When out-of-fold block selection changes the categorical result, the authors revise the claim accordingly and focus on effect magnitude rather than a binary distinction.

Critical weaknesses
  1. The title and some statements in the abstract are broader than the evidence currently supports. The main confirmatory analysis is based on a limited set of checkpoints and primarily on KITTI driving scenes, so the conclusion should be framed as a caution about depth probing under these protocols rather than a general claim that most video-model geometry is a ground plane.

  2. The encoder-generator comparison remains statistically fragile. It is based on only two encoder families and five generator families, and the reported family-level permutation result is at the smallest attainable significance level for this sample size. The paper should avoid presenting this as a robust class-level law.

  3. Position- and flow-residualized depth decoding is stronger evidence than naive probing, but it is still not equivalent to general 3D understanding. Remaining signal may reflect scene semantics or other visual regularities, and the causal generation-side intervention did not clearly separate geometric structure from appearance.

Questions for authors

All my concerns and questions have been present in weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer ytFo

Thank you. We agree with all three points, and each one changed the revision.

W1. The title and abstract are broader than the evidence.

Agreed. The revision reads as the caution you describe, a caution about depth probing under these protocols.

  • Title: "Geometry" becomes "Depth" at camera-ready, since the author guide permits no title change during the discussion. The Conclusion's first sentence already says depth.
  • Abstract: the convergence finding is scoped to the data it was measured on ("on driving footage what independently trained models agree on is where things sit in the frame"), and the class comparison to "these ten checkpoints".
  • Limitations (§7) now opens with what is untested: of the four preregistered geometric targets only depth was carried through, L3 tests one of four lawfulness residuals, multi-view correspondence is not reassessed, and ScanNet retests one generator. The confirmatory numbers are KITTI driving scenes, as it states.

W2. The encoder–generator comparison is statistically fragile.

Agreed, and the revision no longer leans on the permutation test. The paper already noted that p = 0.048 is the smallest value two families against five can produce. §5.1 now says what the separation rests on instead:

Two encoder families cannot support more than that, so the separation rests on the eight per-checkpoint intervals of Table 12 and is a description of the checkpoints measured, not a class-level law.

At the committed block those intervals exclude zero for all four encoders (the lowest bound is +0.023) and include it for all four generators. At the out-of-fold block Mochi-1 crosses at +0.0188, 2.5 times below the weakest encoder, which is why the abstract says the classes separate "by magnitude, not by kind" on these ten checkpoints.

W3. Residualized depth is not 3D understanding, the remainder may be semantics, and the generative test did not separate geometry from appearance.

Agreed on the first and the last. The claim is depth beyond position and flow, not 3D understanding, and the generative intervention is reported in §5.4 as a failed attempt rather than as support. On semantics and other visual regularities, the evidence differs by dataset.

  • Where the paper can rule it out. On the synthetic motion-parallax clips, shading and texture are drawn independently of depth, so no appearance or semantic cue predicts it. There V-JEPA2 and VideoMAE decode position-residualized depth at R²_resid = 0.517 and 0.948, freezing the middle frame drops it to 0.076 and −0.009, and random-initialized controls read −0.025 and −0.017. The remainder there is motion-derived. The revision states the independence in §4, where the reader can check it.

  • Where it cannot yet. On KITTI no control in the paper separates the remainder from single-frame cues, and Limitations now says so:

    Depth beyond position and flow is not 3D understanding either. On synthetic clips whose appearance is independent of depth it collapses when the middle frame is frozen (§4), but on KITTI no control here yet separates it from scene semantics or other single-frame cues.

    [CONTINGENT] We are running the same frozen-frame control on the 200-scene KITTI protocol, preregistered before the first extraction (Section I of the released preregistration), and will post how much of each model's increment survives a static input.

  • Two other regularities are bounded. A position basis as wide as the model's features absorbs six percent of V-JEPA2's increment, and a nonlinear readout (gradient boosting) leaves the encoder–generator split in place (§5.3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.