acceptodds
Withdrawn from ICLR 2027

Below the Patch: Separating Information from Accessibility in Frozen Vision Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision transformers represent an image on a coarse patch grid, yet their frozen features are widely used for tasks requiring localization far below the patch stride. What, then, does “resolution” mean for a frozen representation? We argue that answering this question requires separating information from accessibility: a signal may be present in the input or representation while remaining inaccessible to a particular comparison and readout. We study this distinction using controlled sub-pixel displacement. Raw patches contain enough evidence to recover 2-D motion when local image structure is well conditioned ( up to ), even though a linear probe on the same pixels reports essentially zero. The ViT patch embedding is not the bottleneck either: on our grayscale input subspace it is injective, and a nonlinear decoder recovers displacement equally well from raw patch pairs and layer-0 token pairs (). The apparent loss emerges when the two observations are collapsed into an aligned difference, where pixel and token inputs fail alike. This bottleneck is structured rather than complete. The aligned difference retains the aperture-constrained component of motion in a linearly accessible form. Tracking this coordinate through depth reveals that frozen checkpoints differ sharply in how much fine-scale geometry they keep accessible: MAE largely preserves it, whereas a label-supervised ViT reduces late-layer linear accessibility to nearly zero. Intervening on the fitted direction strongly controls the downstream readout, and combining the accessible scalar with the corresponding image geometry supports sub-pixel refinement on naturalistic video. These results show that effective spatial resolution is not determined by patch stride alone. It depends on what information is present, how observations are compared, and what the readout can access.

Primary area:
interpretability and explainable AI
Keywords:
vision transformers, patch, representation probing, spatial precision

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_DkDgOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies sub-patch displacement in frozen vision transformers by separating information in the observations from accessibility under a chosen comparison and readout. Matched pixel and layer-0 token controls show that a nonlinear decoder can recover displacement from paired observations but performs poorly when they are collapsed into a difference. The authors then track linearly accessible normal flow through depth, compare seven checkpoints, manipulate a fitted access direction, and demonstrate aperture-aware refinement on MPI-Sintel. The contribution is an empirical analysis of representation accessibility rather than a new correspondence algorithm.

Critical strengths
  1. Rigorous experimental controls and measurement hygiene. The paper is unusually thorough in ruling out confounders before making claims about transformer internals. Sections 3.1–3.3 and Tables 16–18 cleanly separate local patch identifiability (via a classical Lucas–Kanade structure-tensor solver), layer-0 projection invertibility (W+W^+), and the pair-versus-difference bottleneck. The measurement hygiene in Appendix A (auditing and eliminating bilinear-warping spectral leaks) and the transparent self-correction in Appendix J (withdrawing an earlier isotropic bound on SAE reconstruction error) further strengthen the empirical credibility of the work.

  2. Well-bounded analysis across depth, causal interventions, and downstream correspondence. Tracking normal flow from layer 0 to layer 12 shows a striking divergence between MAE and supervised ViTs starting from nearly identical layer-0 accessibility (0.6570.657 vs. 0.6510.651). Importantly, the authors keep their claims well-calibrated: they separate observational public-checkpoint comparisons from controlled objective runs, and Table 7 clearly demonstrates that while a single ridge direction w^\hat{w} is load-bearing for a fixed readout, the 767-D orthogonal complement still retains 70–73% of the MLP-decodable signal after refitting. Both the positive gains on MPI-Sintel and the negative boundaries (degradation under non-rigid flow, 40 px translations, and SPair-71k) align with local aperture geometry.

Critical weaknesses
  1. Insufficient experimental specifications and unaligned numbers across several key tables (Appendices B & L, Tables 17–18). Several experiments that carry substantial weight in the paper lack essential reproducibility details or show unexplained numerical discrepancies:

    • In Appendix L, the controlled ViT-S/16 study on ImageNet-100 compares five training objectives ("strong" and "mild" reconstruction/supervision, plus invariance) alongside random initialization across only "seven runs" in total. The text does not specify the exact objective formulations, masking ratios, augmentations, optimizer schedules, or how 7 runs are distributed across 5 objectives (implying most objectives were evaluated with a single training seed).
    • In Appendix B and Tables 17–18, the MLP architecture, training/evaluation split sizes, and hyperparameter selection protocols for the pair-versus-difference decoders are omitted. Furthermore, Table 17 reports R2=0.944R^2 = 0.944 (median error 0.0960.096 px) for the raw-pixel structure-tensor solve after leaving out the most ambiguous λmin⁡\lambda_{\min} quartile (Q1), whereas in Table 18 the individual Q2, Q3, and Q4 structure-tensor scores are 0.961,0.976,0.961, 0.976, and 0.9820.982 (median errors 0.064,0.053,0.064, 0.053, and 0.0490.049 px). A pooled Q2–Q4 evaluation on the same data cannot yield 0.0960.096 px median error when every constituent quartile sits between 0.0490.049 and 0.0640.064 px. A smaller but noticeable baseline drift also appears for clean MAE layer-12 normal-flow R2R^2 across the paper (0.6310.631 in Table 1, 0.6100.610 in Table 11, 0.6290.629 in Table 7, 0.6480.648 in Table 8, 0.6460.646 in Table 9, and 0.6200.620 in Table 10).
  2. Understated direction-sampling uncertainty in the layer-6 intervention controls (Table 11 & Figure 4). In Table 11, the per-seed layer-12 flow R2R^2 for the MAE RMS-matched random control varies substantially across the three seeds (0.489,0.461,0.6090.489, 0.461, 0.609), and varies even more widely for the overlap-matched control (0.053,0.072,0.4510.053, 0.072, 0.451). However, the reported 95% bootstrap confidence intervals are extremely narrow ([0.513,0.527][0.513, 0.527] and [0.174,0.209][0.174, 0.209]). Although Appendix C states that seeds move the data, probe fit, and random control direction together so that the spread covers all sources, the bootstrap interval itself clearly resamples only image pairs conditional on the three fixed directions rather than capturing variance across randomly drawn directions. In addition, matching first-order overlap requires rescaling the random direction by 44.1×44.1\times in MAE, which drives ancillary orientation and contrast readouts to −1.516-1.516 and −0.815-0.815, pushing the perturbation well outside the local linear regime.

Questions for authors
  1. Appendix L specifications: Please provide the exact loss definitions (specifically how "strong" vs. "mild" reconstruction and supervision are parameterized), data augmentations, training epochs/schedules, and the allocation of the 7 runs across the 5 objectives and random initialization.

  2. Table 11 bootstrap & control directions: Can you confirm that the 95% CIs in Table 11 and Figure 4 are computed by resampling image pairs conditional on the 3 drawn control directions? Please clarify this distinction in the caption/text, and consider reporting the mean and standard deviation across a larger number of random directions (e.g., 20–50 random directions) so the control baseline reflects direction-level variance.

  3. Reconciling Tables 17 and 18 (and MLP details): What are the MLP architectures, training/validation split sizes, and preprocessing details for Tables 17–18? Why does the Q2–Q4 structure-tensor result in Table 17 (R2=0.944R^2 = 0.944, median error 0.0960.096 px) differ from the individual Q2–Q4 quartiles in Table 18 (R2∈[0.961,0.982]R^2 \in [0.961, 0.982], median error ∈[0.049,0.064]\in [0.049, 0.064] px)?

Reviewer_BRjROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies to what extent the ViT models preserve the pixel-wise or low-level details throughout different layers and which factors contribute to the loss. Study objectives include pre-training tasks, grid size, network depth, readout methods. In general, the paper conveys the message that a readout that fails doesn't mean the information is lost.

Critical strengths
  1. Comprehensive experiments across different backbones and datasets.
Critical weaknesses

Main concern: limited new knowledge.

  1. Lossless patchification is expected by construction. The standard ViT patch embedding is equivalent to a pixel unshuffle (space-to-depth) followed by a per-patch linear projection; for ViT-B/16 it maps 3×16×16 = 768 values to 768 channels. Sub-patch information is therefore preserved whenever this projection is injective, which the paper itself confirms (rank 256/256 on grayscale inputs, Sec. 3.2). This is a sanity check rather than a finding, yet it is presented as part of the first contribution ("We localise an apparent loss of sub-patch resolution to the comparison/readout rather than patchification", p. 3).

  2. The depth trend matches well-established understanding. It is long established that deeper layers trade low-level detail for task-relevant abstraction, and that how much is discarded depends on the training objective: objectives that reward invariance (label supervision, augmentation-invariant self-supervision) have no incentive to keep sub-pixel position, whereas pixel reconstruction must keep it. The paper acknowledges this in its related work ("Image-level objectives can discard equivariant or patch-local information that a reconstruction loss keeps", Sec. 2), and Park et al. (arXiv:2305.00729) report a similar contrast between masked image modeling and contrastive learning in ViTs. The headline results (MAE 0.657 → 0.631 vs. the label-supervised ViT 0.651 → 0.004, Table 1) and the controlled study showing that the objective changes the rate of loss (Sec. 4.3, App. L) therefore confirm an expected effect rather than reveal a new one.

  3. The findings are not turned into an improvement. No model, objective or method is changed based on the analysis. The only application, sub-pixel refinement on MPI-Sintel (Sec. 6), is described by the authors as "an accessibility result about the representation, not a method contribution" (p. 14). It still trails classical pixel-space Lucas–Kanade (PCK@1 0.358 vs. 0.398, Sec. 6.2), loses its advantage over feature upsampling at larger motions (translations up to 40 px: 0.114 vs. 0.108 for MAE, 0.073 vs. 0.108 for DINO) and gives no gain on SPair-71k ([email protected] 0.108 vs. 0.113 unrefined; Sec. 6.3).

Questions for authors

Please refer to the weakness section.

Reviewer_FGtXOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • This paper investigates the preservation and accessibility of sub-pixel information in frozen Vision Transformers (ViTs). The authors compare displacement reconstruction from pixel, token pairs and aligned differences, and examine layer-wise accessibility across multiple pretrained ViTs using normal flow as the target geometric quantity.
  • Through experiments across diverse backbones and image domains, they show that normal flow accessibility varies substantially across models and layers. They further demonstrate that the accessible information can support sub-pixel correspondence refinement. Based on these findings, the authors emphasize the importance of distinguishing information preserved in a representation from information accessible through a particular comparison and readout mechanism.
Critical strengths
  • This paper conducts extensive experiments across various pretrained ViT backbones, layers, and image domains. Furthermore, it seeks to validate its core claims from multiple perspectives, including pixel-to-token comparisons, controlled objective experiments, feature-direction interventions, and correspondence evaluations. The breadth of its experimental evaluation and its emphasis on systematic validation represent the paper’s primary strengths.
Critical weaknesses

1.Insufficient distinction from previously established properties

  • A fundamental limitation of this paper is its failure to distinguish between phenomena predictable by existing theory and genuinely new scientific discoveries regarding pre-trained ViTs.
  • In Section 3.3, the degradation in 2D displacement reconstruction following the pair-to-difference transformation can be largely explained by the loss of reference image gradient information under the assumption of brightness constancy. Similarly, the information preservation discussed in Section 3.2 stems from full-rank linear projections, while the concept of normal flow in Section 3.4 is already well-established within the geometry of optical flow. While these findings are valid, it remains unclear what they offer beyond what is already known.
  • Section 4 reports differences in accessibility across various backbones and layers, yet it fails to propose any generalizable mechanisms or predictive principles. Furthermore, Section 6 lacks a sufficient basis for the broader applicability or design implications of this analysis.
  • As an analytical study, this paper needs to clarify what significant new scientific knowledge it provides. Specifically, how it advances beyond existing geometric analyses and explorations of model representations.
  1. Lack of Motivation, Technical Clarity, and Interpretability
  • Even after a careful reading of the paper, it remains difficult to understand how the core concepts and experimental results translate into a scientific contribution. In particular, terms such as "direct-readout bottleneck," "aligned difference," and "load-bearing access direction" are introduced without sufficient intuitive explanation.
  • A major issue is the lack of a clear rationale for choosing "aligned difference readout." While Section 3.3 analyzes the limitations of this approach, the paper fails to justify why this specific configuration is essential for evaluating sub-pixel accessibility, especially when alternative comparison mechanisms such as correlation, matching, or cross-attention exist. Consequently, the broader implications of a failure in a given difference decoder remain unclear.
  • The problems with the presentation go beyond mere ambiguity in terminology or figures. For instance, Section F ("Additional Tables and Figures") contains only a heading with no actual content, jumping directly to Section G. Section H references Equation (3) which is not defined until Section J. While Section J includes phrases such as "as argued in a previous version of this work" and mentions a "withdrawn bound." Furthermore, Tables 5 and 6 repeat the same summary results, and data from various experimental runs are scattered across multiple tables without clear contextual explanation. It is also difficult to discern what additional insights beyond the quantitative results are provided by the qualitative visualizations in Figures 3 and 6.
  • These issues extend well beyond mere errors in the text. Taken as a whole, the manuscript is not sufficiently polished. Consequently, it becomes unnecessarily difficult for readers to understand the logical connections between the experimental evidence and the scientific conclusions of the study.
Questions for authors

Q1. Novelty and scientific significance

  • El Banani et al. [1] analyze the geometric capabilities and limitations of visual foundation models through single view geometry and multiview consistency. More recently, Zhou et al. [2] investigate how pretraining objectives influence geometric representations in ViTs through probing and subspace interventions. Given these prior studies, what specific findings in Sections 4 and 5 go beyond the previously established differences in geometric representations across pretrained ViTs and the effects of feature subspace interventions? In particular, does normal flow accessibility reveal a distinct scientific insight that could not be established through existing geometric probing approaches [3]?

References

[1] Mohamed El Banani et al. "Probing the 3D Awareness of Visual Foundation Models." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21795–21806, IEEE, 2024.

[2] Weichen Zhou et al. "Understanding Geometric Representations in Self-Supervised Vision Transformers via Subspace Intervention." European Conference on Computer Vision, pp. 357–374, Springer, 2026.

[3] John Hewitt and Percy Liang. "Designing and Interpreting Probes with Control Tasks." Proceedings of EMNLP-IJCNLP, pp. 2733–2743, 2019.

Q2. Relevance of the aligned difference bottleneck

  • The authors focus their analysis on the aligned difference readout and explicitly acknowledge that other comparison mechanisms, such as correlation, matching, and cross-attention, exist. However, it remains unclear why this particular readout is a sufficiently important or representative setting for investigating sub-pixel accessibility in frozen ViTs. Could the authors clarify which practical spatial prediction or correspondence methods rely on this comparison mechanism, and what broader conclusions can be drawn from the observed bottleneck beyond this specific readout?

Q3. Does normal flow accessibility predict independent spatial performance?

  • Section 4 measures normal flow accessibility across different backbones and layers, while Section 6 presents a correspondence refinement method that leverages this information. However, the relationship between the measured accessibility and actual spatial performance remains unclear. Can the authors provide a quantitative analysis of the relationship between normal flow accessibility and independently evaluated correspondence or localization performance? Specifically, under identical evaluation conditions, do representations with higher accessibility consistently yield better downstream performance or greater improvements over a baseline?
Reviewer_VaCkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether frozen ViTs really carry fine spatial detail below the patch level, which matters because people use them for precise localization. They test this by shifting images by known sub-pixel amounts and measuring what can be read back from the tokens. The answer is largely yes at the input: the patch embedding keeps the detail, and a nonlinear decoder can recover 2-D motion from a pair of early tokens. Deeper in the network, 'it depends' on the training objective. If trained for classification/captioning, it's likely there's loss but if trained for reconstruction there's more precision retained. Not a surprise.

Critical strengths

Strong technical methodology

Critical weaknesses

The main question i have is whether this is valuable. I'll be honest: most of the vision encoder that i work with or my team works with, if we need the very best results, we unfreeze them. If we want a classification task, we pick a classification-focused encoder like a SigLip and if we want dense predictions/segmentations/depth then we might favor a dense-pretrained encoder like Dino. Given that, knowing whether a frozen encoder retains spatial resolution into the network, and observing that different encoders behave differently, i would have said: yeah, that makes sense, kind of expected.

So to be honest, my feeling is to question the value. I don't want to declare that this is unvaluable because any time i see strong methodology, i'm happy. But i'd rather it be really useful too.

Questions for authors

none

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.