- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates which intermediate layer of a vision foundation model is most suitable for extracting geometric information. Its central hypothesis is that the optimal feature extraction depth depends on the spatial scale of the target, rather than solely on the network architecture or pretraining objective. The authors propose that spatial mixing increases with network depth, progressively weakening fine-scale geometric information while preserving or improving coarse-scale information.
Critical strengths
I see this as a substantial empirical study with a clear research question and several interesting observations. The experimental effort is impressive, and the authors are transparent about negative results and limitations. The most valuable contribution is the demonstration that optimal feature depth can vary systematically with target spatial scale, supported by extensive model comparisons and targeted interventions.
Critical weaknesses
My primary concern is that the paper does not establish a sufficiently clear boundary for when the proposed scale-depth relationship should hold.
Although the authors conduct extensive experiments across 32 vision backbones, the proposed ordering is observed in only 18 of them. The paper attempts to explain this variation by introducing a condition that coarse-scale representations must continue improving toward the output layer. However, this condition is not necessary, as nine models that violate it still exhibit the proposed ordering.
More importantly, the condition is determined through supervised layer-wise probing rather than through an independently measurable property of the network. Consequently, it is unclear how the proposed proposition could be applied to a previously unseen backbone without conducting experiments that the proposition is intended to explain or predict.
In general, the paper demonstrates a correlation between target scale and preferred feature depth in a subset of vision foundation models, but does not establish an independently actionable rule for determining when that correlation should hold.
Questions for authors
- What determines whether a backbone follows the proposed relationship? Can the authors identify an independently measurable criterion that predicts its applicability to previously unseen models, rather than categorizing models retrospectively based on layer-wise probing results?
- By different pretraining objectives, the feature abstraction capabilities, feature expressiveness are different for sure. How do the authors reconcile these differences with Equation (2), which assumes a target-independent abstraction factor?
Thank you for the careful review, and for valuing the scale of the experiments and the transparent reporting of negative results. We have uploaded a revised version of the paper that reflects the points below.
W and Q1: A boundary for when the relation holds, and a criterion in advance
You note that the ordering appears in 18 of 32 backbones, that its premise is not necessary, and that the premise is read by supervised probing, so it is hard to apply in advance to an unseen backbone. We answer in three parts.
First, choosing a block does not need the premise: the warp rule of Section 8 uses neither labels nor the premise. With its evaluation fixed in advance and applied unchanged to all 32 backbones, it recovers 80-88% of the labelled sweep's gain (aspect). A new backbone needs only one unlabelled warp curve.
Second, the premise decides how far the relation reaches, not whether it exists. "18 of 32" counts the strictest form, in which the coarse target's best block must reach the output block. Proposition 1 actually rests on increasing differences: logqℓcoarse−logqℓfine should rise with depth, and under Eq. 2 the abstraction factor gℓ cancels exactly from this difference.
Probe noise breaks the rise at some pair of blocks even where the property holds, so we test its direction. For each of the 10 pairs of five band-passed height scales we take the Spearman correlation between depth and this difference; its median is positive in 27 of 32 backbones (27-28 at other floors). With no interaction the sign is a coin flip, so a sign test gives p=5.7×10−5. Counting each training recipe once, it is 18 of 22 (p=0.0022). It is positive in all 9 that satisfy the premise and in 18 of the 23 that do not, including language-image alignment (4/6) and JEPA (2/2), where the argmax ordering never appears. So an interaction in the direction Proposition 1 requires is present in most backbones (C1 rejects the no-interaction model, p=0.0074), and the premise decides only whether its coarse end reaches the output block. Hence the premise is sufficient but not necessary.
Third, the premise is read in advance from one coarse-target probe, not a scale sweep, and does not depend on the label. Read from semantic segmentation, it agrees on 31 of 32 (p=0.0062). Recomputing the ordering on bands the premise does not read keeps the separation (p=0.0012). The three-block threshold was fixed before later backbones were added; the separation over 32 is exact p=0.0007, and p=0.0036 per training recipe. As you say, this probe still needs a label. We tried a label-free premise proxy and withdrew it because it failed on unseen backbones (Appendix D.3); a label-free applicability criterion remains an open problem this paper does not supply.
In short: the interaction appears in 27 of 32 backbones and 8 of 9 objectives; one coarse-target probe with a single label, semantic or height, decides whether it is the strict ordering; and the block to read needs no labels, even on an unseen backbone.
Q2: Objective-dependent abstraction and Eq. 2
You ask how a target-independent gℓ holds when objectives differ in abstraction. gℓ is a depth profile set separately per backbone, so objective differences are absorbed into gℓ (and the measured rℓ), and every test is within a backbone. That targets share gℓ within a backbone is not an empirical conclusion but a modelling assumption that isolates the interaction for testing. The direction test above checks the property Proposition 1 needs without this assumption, but cannot tell whether the interaction comes from mixing or from target-specific abstraction. We examine the source separately:
Hence mixing is one cause of the fine-geometry loss, and a further share from target-specific abstraction is not excluded, as the paper concludes.
Thank you again. The revision reflects your comments: the direction test and its per-recipe count are in Section 5 and Appendix D.3, and Section 5 now separates "18 of 32" (the strict form) from "27 of 32" (the direction of the interaction). We would be grateful if you would reconsider your evaluation in this light.