- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies sub-patch displacement in frozen vision transformers by separating information in the observations from accessibility under a chosen comparison and readout. Matched pixel and layer-0 token controls show that a nonlinear decoder can recover displacement from paired observations but performs poorly when they are collapsed into a difference. The authors then track linearly accessible normal flow through depth, compare seven checkpoints, manipulate a fitted access direction, and demonstrate aperture-aware refinement on MPI-Sintel. The contribution is an empirical analysis of representation accessibility rather than a new correspondence algorithm.
Critical strengths
-
Rigorous experimental controls and measurement hygiene. The paper is unusually thorough in ruling out confounders before making claims about transformer internals. Sections 3.1–3.3 and Tables 16–18 cleanly separate local patch identifiability (via a classical Lucas–Kanade structure-tensor solver), layer-0 projection invertibility (), and the pair-versus-difference bottleneck. The measurement hygiene in Appendix A (auditing and eliminating bilinear-warping spectral leaks) and the transparent self-correction in Appendix J (withdrawing an earlier isotropic bound on SAE reconstruction error) further strengthen the empirical credibility of the work.
-
Well-bounded analysis across depth, causal interventions, and downstream correspondence. Tracking normal flow from layer 0 to layer 12 shows a striking divergence between MAE and supervised ViTs starting from nearly identical layer-0 accessibility ( vs. ). Importantly, the authors keep their claims well-calibrated: they separate observational public-checkpoint comparisons from controlled objective runs, and Table 7 clearly demonstrates that while a single ridge direction is load-bearing for a fixed readout, the 767-D orthogonal complement still retains 70–73% of the MLP-decodable signal after refitting. Both the positive gains on MPI-Sintel and the negative boundaries (degradation under non-rigid flow, 40 px translations, and SPair-71k) align with local aperture geometry.
Critical weaknesses
-
Insufficient experimental specifications and unaligned numbers across several key tables (Appendices B & L, Tables 17–18). Several experiments that carry substantial weight in the paper lack essential reproducibility details or show unexplained numerical discrepancies:
- In Appendix L, the controlled ViT-S/16 study on ImageNet-100 compares five training objectives ("strong" and "mild" reconstruction/supervision, plus invariance) alongside random initialization across only "seven runs" in total. The text does not specify the exact objective formulations, masking ratios, augmentations, optimizer schedules, or how 7 runs are distributed across 5 objectives (implying most objectives were evaluated with a single training seed).
- In Appendix B and Tables 17–18, the MLP architecture, training/evaluation split sizes, and hyperparameter selection protocols for the pair-versus-difference decoders are omitted. Furthermore, Table 17 reports (median error px) for the raw-pixel structure-tensor solve after leaving out the most ambiguous quartile (Q1), whereas in Table 18 the individual Q2, Q3, and Q4 structure-tensor scores are and (median errors and px). A pooled Q2–Q4 evaluation on the same data cannot yield px median error when every constituent quartile sits between and px. A smaller but noticeable baseline drift also appears for clean MAE layer-12 normal-flow across the paper ( in Table 1, in Table 11, in Table 7, in Table 8, in Table 9, and in Table 10).
-
Understated direction-sampling uncertainty in the layer-6 intervention controls (Table 11 & Figure 4). In Table 11, the per-seed layer-12 flow for the MAE RMS-matched random control varies substantially across the three seeds (), and varies even more widely for the overlap-matched control (). However, the reported 95% bootstrap confidence intervals are extremely narrow ( and ). Although Appendix C states that seeds move the data, probe fit, and random control direction together so that the spread covers all sources, the bootstrap interval itself clearly resamples only image pairs conditional on the three fixed directions rather than capturing variance across randomly drawn directions. In addition, matching first-order overlap requires rescaling the random direction by in MAE, which drives ancillary orientation and contrast readouts to and , pushing the perturbation well outside the local linear regime.
Questions for authors
-
Appendix L specifications: Please provide the exact loss definitions (specifically how "strong" vs. "mild" reconstruction and supervision are parameterized), data augmentations, training epochs/schedules, and the allocation of the 7 runs across the 5 objectives and random initialization.
-
Table 11 bootstrap & control directions: Can you confirm that the 95% CIs in Table 11 and Figure 4 are computed by resampling image pairs conditional on the 3 drawn control directions? Please clarify this distinction in the caption/text, and consider reporting the mean and standard deviation across a larger number of random directions (e.g., 20–50 random directions) so the control baseline reflects direction-level variance.
-
Reconciling Tables 17 and 18 (and MLP details): What are the MLP architectures, training/validation split sizes, and preprocessing details for Tables 17–18? Why does the Q2–Q4 structure-tensor result in Table 17 (, median error px) differ from the individual Q2–Q4 quartiles in Table 18 (, median error px)?