acceptodds
Under review as a conference paper at ICLR 2027

Spatial Mixing Predicts the Read-out Depth for Geometry in Vision Foundation Models

Abstract

Dense geometric tasks are read off a vision transformer's output block by convention, under the suspicion that depth trades spatial detail for abstraction. We find instead that the depth at which a linear probe best decodes a geometric attribute rises with the attribute's ground scale, so the best depth belongs to the attribute and not to the network alone. Over 32 backbones this ordering appears in every one whose coarse-scale representation is still improving at the output block, and it transfers to indoor photographs. Our account is that each block mixes a token with its neighbours over a length that grows with depth, so depth loses geometry below that length and keeps what is coarser. We measure that length without labels, as the radius over which a token responds to one occluded input patch. On overhead imagery, where every token has a known ground scale, dividing each target's scale by that radius brings the retention curves of nine matched backbones together for surface normals and aspect and not for band-passed height, and no rival length of the mixing does this. Blurring an intermediate block reproduces the output block's loss at a width the radius predicts. Narrowing late attention as far as semantic accuracy allows returns the lost surface geometry on three of six backbones and worsens it on none. Mixing is therefore part of the cause. An earlier block still holds the lost detail, and on the same imagery warping a tile by a known map names it without labels or training, recovering 80 to 88% of what a labelled sweep would gain over all 32 backbones.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
vision foundation model, remote sensing, effective receptive field

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kGxyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates which intermediate layer of a vision foundation model is most suitable for extracting geometric information. Its central hypothesis is that the optimal feature extraction depth depends on the spatial scale of the target, rather than solely on the network architecture or pretraining objective. The authors propose that spatial mixing increases with network depth, progressively weakening fine-scale geometric information while preserving or improving coarse-scale information.

Critical strengths

I see this as a substantial empirical study with a clear research question and several interesting observations. The experimental effort is impressive, and the authors are transparent about negative results and limitations. The most valuable contribution is the demonstration that optimal feature depth can vary systematically with target spatial scale, supported by extensive model comparisons and targeted interventions.

Critical weaknesses

My primary concern is that the paper does not establish a sufficiently clear boundary for when the proposed scale-depth relationship should hold.

Although the authors conduct extensive experiments across 32 vision backbones, the proposed ordering is observed in only 18 of them. The paper attempts to explain this variation by introducing a condition that coarse-scale representations must continue improving toward the output layer. However, this condition is not necessary, as nine models that violate it still exhibit the proposed ordering.

More importantly, the condition is determined through supervised layer-wise probing rather than through an independently measurable property of the network. Consequently, it is unclear how the proposed proposition could be applied to a previously unseen backbone without conducting experiments that the proposition is intended to explain or predict.

In general, the paper demonstrates a correlation between target scale and preferred feature depth in a subset of vision foundation models, but does not establish an independently actionable rule for determining when that correlation should hold.

Questions for authors
  1. What determines whether a backbone follows the proposed relationship? Can the authors identify an independently measurable criterion that predicts its applicability to previously unseen models, rather than categorizing models retrospectively based on layer-wise probing results?
  2. By different pretraining objectives, the feature abstraction capabilities, feature expressiveness are different for sure. How do the authors reconcile these differences with Equation (2), which assumes a target-independent abstraction factor?
1 comment

Thank you for the careful review, and for valuing the scale of the experiments and the transparent reporting of negative results. We have uploaded a revised version of the paper that reflects the points below.

W and Q1: A boundary for when the relation holds, and a criterion in advance

You note that the ordering appears in 18 of 32 backbones, that its premise is not necessary, and that the premise is read by supervised probing, so it is hard to apply in advance to an unseen backbone. We answer in three parts.

First, choosing a block does not need the premise: the warp rule of Section 8 uses neither labels nor the premise. With its evaluation fixed in advance and applied unchanged to all 32 backbones, it recovers 80-88% of the labelled sweep's gain (aspect). A new backbone needs only one unlabelled warp curve.

Second, the premise decides how far the relation reaches, not whether it exists. "18 of 32" counts the strictest form, in which the coarse target's best block must reach the output block. Proposition 1 actually rests on increasing differences: log⁡qℓcoarse−log⁡qℓfine\log q^{\mathrm{coarse}}_\ell-\log q^{\mathrm{fine}}_\ell should rise with depth, and under Eq. 2 the abstraction factor gℓg_\ell cancels exactly from this difference.

Probe noise breaks the rise at some pair of blocks even where the property holds, so we test its direction. For each of the 10 pairs of five band-passed height scales we take the Spearman correlation between depth and this difference; its median is positive in 27 of 32 backbones (27-28 at other floors). With no interaction the sign is a coin flip, so a sign test gives p=5.7×10−5p=5.7\times10^{-5}. Counting each training recipe once, it is 18 of 22 (p=0.0022p=0.0022). It is positive in all 9 that satisfy the premise and in 18 of the 23 that do not, including language-image alignment (4/6) and JEPA (2/2), where the argmax ordering never appears. So an interaction in the direction Proposition 1 requires is present in most backbones (C1 rejects the no-interaction model, p=0.0074p=0.0074), and the premise decides only whether its coarse end reaches the output block. Hence the premise is sufficient but not necessary.

Third, the premise is read in advance from one coarse-target probe, not a scale sweep, and does not depend on the label. Read from semantic segmentation, it agrees on 31 of 32 (p=0.0062p=0.0062). Recomputing the ordering on bands the premise does not read keeps the separation (p=0.0012p=0.0012). The three-block threshold was fixed before later backbones were added; the separation over 32 is exact p=0.0007p=0.0007, and p=0.0036p=0.0036 per training recipe. As you say, this probe still needs a label. We tried a label-free premise proxy and withdrew it because it failed on unseen backbones (Appendix D.3); a label-free applicability criterion remains an open problem this paper does not supply.

In short: the interaction appears in 27 of 32 backbones and 8 of 9 objectives; one coarse-target probe with a single label, semantic or height, decides whether it is the strict ordering; and the block to read needs no labels, even on an unseen backbone.


Q2: Objective-dependent abstraction and Eq. 2

You ask how a target-independent gℓg_\ell holds when objectives differ in abstraction. gℓg_\ell is a depth profile set separately per backbone, so objective differences are absorbed into gℓg_\ell (and the measured rℓr_\ell), and every test is within a backbone. That targets share gℓg_\ell within a backbone is not an empirical conclusion but a modelling assumption that isolates the interaction for testing. The direction test above checks the property Proposition 1 needs without this assumption, but cannot tell whether the interaction comes from mixing or from target-specific abstraction. We examine the source separately:

  1. Holding the block fixed and changing only mixing. Any blur degrades fine geometry, so what is tested is the width that matches the output block's loss, and that width follows the radius (p<0.04p<0.04). Narrowing late attention returns the normal on 3 of 6 backbones and worsens it on none.
  2. A photometric target built with the same functions is lost harder although it is not geometric (9/9, p=0.0039p=0.0039), while semantic boundaries survive. The loss follows scale, not the kind of target.
  3. The premise read from semantics and from height agrees on 31 of 32. This is agreement on whether the coarse target peaks near the output block, not on the whole profile.

Hence mixing is one cause of the fine-geometry loss, and a further share from target-specific abstraction is not excluded, as the paper concludes.


Thank you again. The revision reflects your comments: the direction test and its per-recipe count are in Section 5 and Appendix D.3, and Section 5 now separates "18 of 32" (the strict form) from "27 of 32" (the direction of the interaction). We would be grateful if you would reconsider your evaluation in this light.

Reviewer_Yt7ZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper turns the common suspicion that reading dense geometric tasks off a ViT's output block loses spatial detail into a measurement, a model, and a label-free rule. Core finding: the depth at which a linear probe best decodes a geometric attribute rises with the attribute's ground scale — fine geometry peaks at intermediate blocks, coarse geometry at the output block — and this holds across 32 backbones (9 of which meet the "still abstracting at the output block" premise), 9 pretraining objectives, and indoor photographs. The account: each block mixes a token with neighbours over a radius r_ℓ that grows with depth, so depth destroys geometry below the aggregation scale r_ℓ·τ and keeps coarser structure. The authors measure r_ℓ without labels as the radius over which a token responds to one occluded input patch; dividing each target's scale s by r_L collapses nine backbones' retention curves (surface normals, aspect; not band-passed height), and no rival mixing length (mean attention distance, gradient receptive field, autocorrelation) does. Causality is triangulated by (a) a blur of an intermediate block reproducing the output block's loss at the width √(r_L²−r_16²), and (b) narrowing late attention returning the surface normal on 3/6 backbones without worsening it. The repair: an earlier block still holds the lost detail, and an unlabeled warp names that block, recovering 80–88% of what a labeled sweep would gain over all 32 backbones.

Critical strengths
  1. Exceptional experimental rigor and causal care. The combination of observational ordering, label-free length measurement, permutation nulls, multiple-comparison correction, and three mutually reinforcing interventions (add mixing via blur, remove it via window, show no read-out of the output block recovers the loss) is well above the usual standard for analysis papers. The two opposite-direction controls separating scale from modality are especially elegant.
  2. Genuinely new, important insight with a practical payoff. That the best read-out depth moves with target scale (beyond the concurrent "intermediate blocks are better for geometry" observation), plus a measured, label-free aggregation scale and an unlabeled rule that recovers 80–88% of a labeled sweep over 32 backbones, is both conceptually novel and directly useful for foundation-model-based dense geometry.
Critical weaknesses
  1. The most actionable claims are scoped to overhead imagery. The radius calibration and the unlabeled block-choice rule are established where a token has a known ground scale (satellite/overhead). On photographs the calibration is explicitly "undecided rather than refuted" (too few backbones, and a perspective token covers many ground scales at once). So the practical, label-free selection contribution is validated mainly on overhead imagery, which bounds its generality to the natural-image settings where most dense-geometry decoders are deployed.
  2. Causal attribution is necessarily partial and not fully clean on every backbone. The window intervention returns the normal on only 3 of 6 backbones at the strongest admissible dose, and the blur equivalence acts on the read path rather than the network. The authors hedge correctly ("mixing is part of the cause"), and the three measurements agree, but the fraction of the fine-geometry loss attributable to late mixing — versus irreducible token-identity collapse, which they note fails in 4 backbones — is not quantified, so the causal share remains approximate.
Questions for authors
  1. The actionable rule is validated on overhead imagery; on photographs the calibration is undecided. Since a perspective token spans many ground scales, how do you expect the rule to transfer to natural images, and could a depth-aware per-token scale (e.g., from a monocular depth prior) make the label-free selection applicable there?
  2. The window intervention recovers the normal on only 3/6 backbones. Could you quantify how much of the remaining fine-geometry loss on the other 3 is not late-mixing (e.g., token-identity collapse, noted in 4 backbones)? This would bound the causal share of mixing.
  3. You differentiate from concurrent Zhou et al. (2026) / Sanghavi (2026) by the scale-movement of the peak. If those natural-image studies were probed at multiple scales, would their intermediate-block peak also move with target scale? I.e., is the scale-dependence incremental over their observation, or does it coexist in their results?
  4. The response radius uses channel whitening (shrinkage 0.2) and a single-patch occlusion. Is the calibration robust to the whitening shrinkage and to the occlusion protocol (single vs multiple/random patches)? Both are stated to be safe, but a brief sensitivity note would strengthen the narrowest claim.
1 comment

Thank you for the warm assessment and the concrete questions, and for valuing the rigour and practical use of the experiments. We have uploaded a revised version of the paper that reflects the points below.

W1 and Q1: Transfer to natural images

You note that the calibration and the unlabelled rule are established on overhead imagery, and ask whether a depth prior setting per-token scale would extend them to photographs. The phenomenon itself appears in photographs. On NYU Depth v2 the ordering holds in 4 of 6 backbones (median ρ=+0.707\rho=+0.707), and the output block reads the finest surface normal 5.3-7.6° worse than the best block. The calibration is undecided there because a token's ground size is not fixed in a perspective image: within one NYU frame metres per token vary by a median factor of 2.7, so s/(rLτ)s/(r_L\tau) has no single value (Appendix D.5). We think your depth prior is the missing link. Defining s/(rLτi)s/(r_L\tau_i) per token with τi=dip/f\tau_i=d_i p/f would make the scale-conditioned rule of Section 8 (Appendix F.4) usable token by token, and NYU's ground-truth depth would let the prior's and the rule's errors be evaluated separately.


W2 and Q2: The other three backbones and the share of mixing

Since the window returns the normal on 3 of 6 backbones, you ask us to quantify the part of the remaining loss that is not mixing. The window acts from block 16, so the loss it addresses is the loss after block 16, on the intervention's 71 tiles:

BackbonerLr_L (tokens)Normal lost from block 16 to outputWindow, strongest admissible dose
DINOv3-Web7.670.96°-0.148° (p=0.0035p=0.0035)
DINOv2-L7.490.85°-0.126° (p<0.0001p<0.0001)
CLIP-L6.800.36°-0.111° (p=0.011p=0.011)
ViT-L IN-21k7.630.17°-0.021° (n.s.)
DINOv3-Sat1.330.10°window wider than the response
MAE-L0.98none-0.045° (n.s.)

The other three lose at most 0.17° after block 16, far less than the three the window returns (0.36-0.96°), so what the window leaves unexplained there is at most 0.17°. CLIP-L and DINOv2-L return significantly although their token identity has collapsed, so that collapse does not prevent the return.

MAE-L, however, loses nothing after block 16, yet its normal improves at the weaker 41% dose (-0.062°, p=0.003p=0.003) and its matching accuracy rises 4.0-9.4 points past its own best block even under the hard mask. An improvement where there is no loss to return means the window has effects besides mixing, and Appendix E.3 reports it as a result against the account. We therefore read the window's return as an approximation of the mixing share, together with the blur. The blur adds only mixing after the forward pass, reproduces the output block's loss on all nine, and its width follows the added radius (p<0.04p<0.04).


Q3: Relation to concurrent work

You ask whether the concurrent studies' peaks would also move with target scale. The two results coexist; the intermediate-block peak they report is a single-scale slice. Probing band-passed targets at five scales on NYU, the best block moves with scale in 4 of 6 backbones: DINOv2, analysed by Zhou et al., reads ρ=+0.949\rho=+0.949, and the supervised ViT family of Sanghavi reads +0.894+0.894 at grid 32. A best block chosen at one scale is therefore not the best at another. Our work goes further by measuring the unit of that shift, rLτr_L\tau, without labels and by intervening on mixing directly.


Q4: Sensitivity to whitening and occlusion

You ask whether the calibration survives other shrinkages and occlusion protocols. We varied the shrinkage and the occlusion one at a time, on the collapse's own protocol (DFC2019, grid 64).

ChangeRank ρ\rho vs. paperNormal collapse (p)Aspect collapse (p)Band-passed height, control (p)
Shrinkage 0.5 or 1 (= paper)+1.00024.1% (0.0074)35.7% (0.031)-8.7% (0.76)
Shrinkage 0+0.98321.6% (0.021)41.1% (0.0063)-6.2% (0.91)
Sites redrawn (1)+0.95021.3% (0.015)41.1% (0.0063)-4.4% (0.57)
Sites redrawn (2)+0.93321.2% (0.018)41.1% (0.0063)-4.4% (0.58)
Four patches at once+0.88322.7% (0.0070)40.3% (0.0099)-2.3% (0.44)

The paper occludes 8 random sites per tile, one at a time (320 responses per backbone). The four-patch leg places sites at least 22 tokens apart and assigns each token's response to the nearest patch, which reads large radii smaller, yet the ranking and the collapse hold. The control never tightens, so selectivity holds.


Thank you again. The revision reflects these points: the loss after block 16 in Section 7 and Appendix E.3, the sensitivity analyses in Section 6 and Appendix D.1 (Table 7), and the depth-based route in Appendix D.5 and Section 9. We hope this clarifies the causal role of mixing and the basis of the radius.

Reviewer_GqvYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the optimal layer for extracting geometric information from visual foundation models varies according to the spatial scale of the target. The authors conduct layer-wise probing on backbones—controlling for architecture and ground coverage—and measure spatial mixing via the response radius resulting from the occlusion of a single input patch. They further validate their interpretations through feature blurring, constraints on attention in later layers, and the use of different decoders. Additionally, the paper proposes a method for selecting the optimal extraction layer using unlabeled matching curves derived from known image transformations; this approach recovers most of the performance that would otherwise be lost when relying on the final output layer for geometric tasks in remote sensing.

Critical strengths
  1. The study refines the observation that "intermediate layers are better suited for geometric tasks" into a testable relationship involving scale. In Section 5, the authors vary the scale within a consistent family of models while keeping feature and probe capacity constant. Among the 32 backbones examined, the 9 that satisfy the premise—showing consistent improvement in representations at coarser scales—also exhibit a clear, ordered relationship between scale and the optimal layer. The credibility of these empirical findings is further bolstered by geographically isolated data splits, bootstrapping over entire image patches, and corrections for multiple hypothesis testing.

  2. The chain of evidence goes beyond mere correlation. In Section 6, normalizing by the response radius reduces the dispersion of retention curves for surface normal and aspect estimation by 24% and 36%, respectively; the authors also compare this against radius-based permutations and other length-based metrics. Section 7 provides further validation by experimentally increasing or decreasing the degree of spatial mixing. In Section 8, the proposed unlabeled selection heuristic recovers 80–88% of the performance gains achieved through supervised layer selection for aspect estimation tasks, lending practical utility to the analysis. The authors also clearly delineate the scope regarding transfer to indoor settings and the effects of their interventions.

Critical weaknesses
  1. A clearer distinction should be made between the theoretical monotonic predictions and the empirical calibration results. Proposition 1 supports the idea that the optimal layer varies with scale. However, Figure 2—which attributes the transition point (where the target scale roughly equals the mixing length) to Eq. 2—makes a stronger claim than the proposition itself: even with fixed radius and inter-layer gain, the transition point can be shifted simply by varying the parameter c in A(x) = exp(-cx^2). Similarly, the relationship involving the squared difference of radii in Section 7 cannot be derived directly from the first-order mean distance defined in Section 3. While the correlation between curve convergence and blur width provides valid empirical support, these quantitative interpretations require additional assumptions regarding kernel shape and normalization—or should otherwise be explicitly framed as empirical regularities.

  2. The connection between the label-free layer-selection rule and the target scale remains unclear. In Section 8, a layer-wise curve is generated using a fixed transformation and a matching tolerance of 0.75 t; for a given image and backbone, the peak of this curve remains invariant regardless of the target scale. Yet, a key finding in Section 5 is that the optimal layer for a given backbone varies depending on the scale of the target. While the reported reconstruction results suggest that this rule serves as an effective proxy for fine-scale surface geometry in remote sensing, it does not yet explain how the rule selects distinct layers for targets of different scales. This limits the generalization from empirical observations to a universal selection criterion, though it does not negate the demonstrated improvements in reconstruction.

Questions for authors
  1. What assumptions—beyond those in Proposition 1—are required to justify the transition point in Figure 2 and the squared-radius-difference relationship in Section 7? Could you clarify the distinction between the theoretical ordering and the empirically derived calibration?

  2. For targets of different scales within the same image and backbone, does the method in Section 8 involve adjustments to image transformations, feature smoothing, or matching tolerances? If not, how should the range of target scales for which this rule is applicable be defined?

1 comment

Thank you for the careful review, and for recognising that we cast the scale-depth relation in a testable form and backed it with interventions. We have uploaded a revised version of the paper that reflects the points below.

W1 and Q1: What the crossover and the quadrature assume

You point out that Proposition 1 guarantees only an ordering, while Figure 2's crossover and Section 7's quadrature claim more.

Crossover. In Eq. 2 target and block meet only through s/(rLτ)s/(r_L\tau), so the crossover is proportional to rLτr_L\tau and, as you say, its constant is set by cc in AA. Figure 2 and test C4 test only this proportionality. The collapse statistic is a binned variance on the log⁡(s/rL)\log(s/r_L) axis, so it is unchanged when every radius is multiplied by one constant; recomputed at 0.5x, 1x and 3x it reads 24.1% (p=0.0074p=0.0074) each time. It uses neither cc nor β\beta. That the constant is near one (about 20 m for DINOv3-Web against rLτ=21.9r_L\tau=21.9 m on Section 6's reading) is a measurement and enters no test. Figure 2's caption now says so, and Section 6 states the invariance.

Quadrature. As you note, it does not follow from the first moment. What adds exactly when centred kernels are composed, whatever their shape, is the second moment. Since the mixing in Eq. 2 is centred on each token, we re-read the same occlusion responses on all nine backbones as the second moment about the occluded token (RMS radius); the first-moment values reproduce. With the quantity for which the theory is exact, the relation to the blur width becomes stronger:

Predictor of the blur widthAspect ρ\rho (exact pp)Normal ρ\rho (exact pp)
rL2−r162\sqrt{r_L^2-r_{16}^2}, first moment (paper)+0.636 (0.036)+0.650 (0.033)
rL2−r162\sqrt{r_L^2-r_{16}^2}, RMS (new)+0.669 (0.027)+0.667 (0.029)
rL/r16r_L/r_{16}, RMS (new)+0.695 (0.022)+0.700 (0.022)
rL/r16r_L/r_{16}, no shape assumption (paper)+0.753 (0.012)+0.750 (0.013)

The first-moment predictor was fixed from the model before the blur sweep finished, and test C4 keeps it (p=0.0359p=0.0359); the RMS reading is reported beside it, not substituted.

The one remaining assumption is Eq. 2's modelling assumption that adding a block adds one token-centred blur. Attention is not a convolution, so this is an approximation, and the blur experiment of Section 7 tests it directly on all nine backbones.

In short, theory gives the ordering (Proposition 1) and, under Eq. 2 with any AA, a crossover proportional to rLτr_L\tau and second moments adding in quadrature. The crossover's constant and the blur widths are measured, and the widths are the test of the quadrature.


W2 and Q2: How the unlabelled rule follows target scale

You ask whether the rule adjusts anything per scale, and over what range it applies, since a fixed warp at a 0.75-token tolerance gives one peak whatever the target scale. As submitted, it adjusts none of the three: a fixed warp, no smoothing and one tolerance. The revision lets scale enter in two ways.

Range. The rule picks one of two blocks per target, and scale decides which. As Section 8 and Appendix F.2 explain, the warp criterion is a localisation score and is meaningful only for targets finer than the aggregation scale rLτr_L\tau; coarser targets are read best at the output block (Section 5). Target scale is known before measurement, so this decision needs no labels. Implemented as stated and evaluated on 236 targets of 22 ViT-L backbones:

Block choiceShortfall vs. labelled sweep (%)
Output block always9.2
Warp block for every target, ignoring the range7.4
Within the range (warp block only for s<rLτs<r_L\tau)4.9
Same, with the tolerance matched to target scale (nearest swept tolerance)4.4

Respecting the range alone nearly halves the output block's shortfall.

Tolerance. Adjusting it, as you suggest, also follows scale. Varying it from 0.5 to 8 tokens moves the warp's peak deeper in 27 of 32 backbones (p=5.7×10−5p=5.7\times10^{-5}). For DINOv3-Web under a 3° rotation the peak is block 8 at 0.5 token, 10 at 1 and 12 at 2, and the output block from 3 tokens on, the same shape as its labelled ordering. With labels, the scale-wise three-block routing of Table 19 (0.82 M parameters) beats the 7.36 M hypercolumn on both targets, normal -12.8% and coarse height +94.6%.


Thank you again. The revision reflects these points: the proportionality and the measured constant in Figure 2's caption and Section 6; the RMS radius and its centring condition in Section 7 and Appendix E.1; the range rule as a new paragraph in Section 8, with the table above and the tolerance sweep in Appendix F.4 (Table 25). We believe this makes clearer that the calibration is a proportional prediction independent of the shape of AA, and that the unlabelled rule chooses its block by target scale.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.