acceptodds
Under review as a conference paper at ICLR 2027

Sub-Patch Semantics in DINOv3: Geometry, Recoverability, and Feature Upsampling

13 views

Abstract

Foundational models pretrained on large pool of unlabelled data has brought strong image representation across a wide range of domains. However, their semantically rich but low spatial resolution poses a challenge when working with pixel level dense prediction task such as segmentation or depth prediction. Existing work such as JAFAR, AnyUp try to recover this sub-patch information using the input image as a guide to upsample the ViT feature map. The upsamplers are trained to produce feature map are closer to the feature map as generated from a higher resolution image under the mean absolute error objective. However, a systematic study of what sub-patch information is already encoded within a patch token, how this information is geometrically organized, and to what extent it can be recovered directly from the coarser feature representation remains largely unexplored. In this work, we carry out a study of the geometric and semantic nature of the patch vectors. We find that patch embeddings are anisotropic, with distinct object representations occupying a shared, cone-like subspace where recovering the identity of objects in a mixed patch is challenging. We show that when different objects are present in the same patch, the resulting patch vector can't be explained as a linear decomposition of the mixed tokens. We show that the identity of the foreground object in such cases is still recoverable in a low-dimensional subspace. A Fisher-style criterion identifies such a subspace by selecting directions that separate foreground classes while remaining relatively stable across changes in background. Based on our observation, we propose an upsampler pre-trained on a labelled dataset that learns class prototypes and uses them to extract class-discriminative sub-patch information for coarse DINOv3 patch tokens without requiring an image-based guide that are used by models such as JAFAR or AnyUp. On semantic segmentation, our method improves linear-probe performance by approximately mIoU over the raw DINOv3 tokens, by mIoU over existing image-guided learned feature-upsampling baselines and outperforms transformer and DPT-based decoders.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Dinov3, ssl, self-supervised, segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DFFYOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates whether DINOv3 patch tokens retain semantic and spatial information below their native 16×16-pixel resolution. Using ADE20K compositions and synthetic shape–colour mixtures, it studies feature anisotropy, direction versus magnitude, within-patch location, foreground–background entanglement, and foreground retrieval through Fisher-style supervised projections. Based on these observations, it proposes an ADE20K-trained prototype-conditioned upsampler that consumes four layers of frozen DINOv3 features, predicts 16 child tokens per parent patch without RGB guidance at inference, and separates selected semantic directions from a complementary detail path. Evaluation covers synthetic retrieval, reconstruction of high-resolution DINOv3 features, ADE20K segmentation with linear and DPT-style heads, and transfer of the frozen upsampler to OpenEarthMap.

Critical strengths

Recovering dense features without running the ViT at substantially higher resolution is important for segmentation and other dense tasks

Critical weaknesses
  1. The main comparison confounds architecture with substantially stronger semantic supervision. The proposed upsampler is trained on ADE20K with high-resolution DINO targets, child semantic labels, class prototypes, routing cross-entropy, InfoNCE, a class-margin loss, an image-presence loss, and 5× weighting of minority children (pp. 4–5 and 16–17; Figure 1; Appendices C.1–C.3). The paper does not state that JAFAR or AnyUp was retrained with equivalent semantic supervision; Appendix C.4 says only that the downstream heads share a recipe. The cited JAFAR and AnyUp methods are task-agnostic and annotation-free in their original formulations.

  2. No ablation establishes that the representation findings cause the reported gains. The method combines four DINOv3 layers, 5×5 context, child-position queries, trainable Fisher-initialized directions, hard top-k selection, multiple prototypes, an image-level prior, semantic/detail paths, and seven losses (Section 3.3; Appendices C.1–C.3). No component ablation is reported, and W is optimized after initialization.

Questions for authors
  1. What exact data, supervision, checkpoints, losses, and tuning budgets were used for JAFAR and AnyUp? Can you provide same-supervision baselines and a reconstruction-only variant?

  2. Was the ADE20K validation set used both to select epoch 13 and to report Tables 2–3? What encoder and RGB-guidance resolution was used for every row?

  3. Which improvements remain after ablating Fisher initialization, top-K selection, prototypes, semantic losses, multilayer context, image-presence prediction, and the detail path?

  4. How are 128 Fisher directions obtained from the stated 64-dimensional search space, how is Q constructed relative to learned W, and how are hard selection and prototype assignment trained?

  5. Do the conclusions hold for natural mixed patches and unseen foreground identities, and against fitted affine mixtures and standard supervised linear classifiers?

1 comment
Reviewer_DFFYOpenReview ↗

Response to

I appreciate the author’s detailed response and additional experiments, which have addressed some of my concerns. However, having read the comments from other reviewers and the corresponding responses, I remain concerned about the paper’s contribution; therefore, I am maintaining my original score.

Reviewer_2g6yOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at the information that lies below the patch resolution in DINOv3 features. It uses feature-geometry analysis and controlled object insertions to study how well foreground identity and within-patch location can be recovered. The authors find that background subtraction and subspace removal can hide foreground details. However using a Fisher projection helps improve retrieval when dealing with new background combinations. They introduce a feature upsampler that combines -layer context selected semantic directions, class prototypes and a complementary detail branch. The upsampler is trained using ADE20K labels and high-resolution DINOv3 feature targets. It enhances feature reconstruction and segmentation performance on ADE20K and OpenEarthMap. Can be integrated into a DPT decoder.

Critical strengths
  1. The controlled foreground-recovery experiments provide evidence: Prototypes and directions are trained on background data. Tested on unseen background combinations. The incorrect-label and foreground/background specificity controls help show that successful retrieval is tied to foreground information, not fitting arbitrary labels (Section 5.3; Appendix B.5). The increase in block-6 Recall@1 from 0.200 to 0.927 in the 15-identity setting is a result. The distinction between removing background variation and keeping discriminative directions is informative.

  2. The full upsampler system shows improvements across evaluations: Centered feature cosine improves from 0.778 in JAFAR to 0.874. ADE20K segmentation improves by 1.71 mIoU over features. The DPT decoder gains 0.61 points (Tables 2–3). Transferring the trained upsampler to OpenEarthMap with a new segmentation head adds evidence of practical use.

Critical weaknesses
  1. The experiments do not isolate the value of the proposed readout: Feature reconstruction is already present in FeatUp’s network and LoftUp’s coordinate-based transformer. The key new idea here is the selected-direction prototype mechanism. . The proposed system includes four-layer context, ADE20K class supervision, prototype memory and multiple semantic losses all along with Fisher-based selection. Comparing to JAFAR and AnyUp, which use task- feature-reconstruction objectives does not account for these differences. Tables 2–3 show that the full supervised system helps. No ablation study shows that restricting each component to a small set of directions drives that improvement. This weakens the link between the representation analysis and the proposed architecture.

  2. Essential details of the read/write mechanism are missing: Section 3.3 points to Appendix C.2 for the gate write basis QQ and detail path. That section only explains prototype updates. Equation (4) is an orthogonal projector if the selected columns of QQ are orthonormal. The method for ensuring this is not explained. Appendix C.1 also does not clarify how the 64-dimensional Fisher search is adapted to initialize 128 directions. These omissions make it hard to verify the architecture’s idea of separating semantic and complementary feature content.

Questions for authors
  1. Can you provide a comparison between K=16K=16 and K=128K=128 keeping input layers, prototype memory, targets, losses and training budget constant? Results from feature reconstruction and segmentation would directly test the benefit of the restricted readout.

  2. How are QQ and the top-KK gate built and trained? What ensures the projection property in Equation (4). What search rank is used to initialize the 128-dimensional Fisher readout? Pseudocode with dimensions would help clarify this.

1 comment

I thank the authors for their response and the additional experiments.

My second concern is addressed about Q and the Fisher initialization. The ablations the authors provided are also helpful, specifically the improvement seen from using multiple layers. However, (K=16) and (K=128) perform quite similarly, and Fisher initialization offers very little advantage over PCA or random initialization.

And so although the results support the effectiveness of the full system, the contribution of prototype routing and semantic supervision are still unclear. I therefore keep my original rating. I appreciate the additional evidence and clarifications.

Reviewer_yutrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates sub-patch identity and location in frozen DINOv3 ViT-L/16 features using natural-image compositions and synthetic mixtures. It reports anisotropic geometry, failed simple background-removal operations, and improved held-out foreground retrieval through a supervised Fisher readout. It proposes a token-only upsampler using multi-layer context, class prototypes, and semantic/detail paths, trained with semantic labels and high-resolution feature targets. ADE20K linear-head mIoU increases from 54.99 to 56.70, DPT improves from 56.51 to 57.12, and the ADE20K-trained upsampler improves OpenEarthMap results with a newly trained head.

Critical strengths
  1. The foreground-recovery study includes meaningful generalization and label-capacity controls: Table 1 reports held-out Recall@1 of 1.000 at blocks 9/13 using 14 Fisher directions; Appendix B.5/Table 6 excludes held-out tokens from fitting and shows wrong-label retrieval near chance (0.069–0.084).
  2. The diagnostics distinguish several sources of recoverable information: Section 5.1 and Appendix B.1 test location, texture scrambling, and object–location binding, while Section 5.2 compares matched-background removal with unrelated-background and random controls.
  3. The complete upsampler shows useful reported feature and task improvements: Table 2 raises centred cosine from JAFAR’s 0.778 to 0.874, including 0.840 on minority cells, and Table 3 demonstrates improvements with both linear and DPT heads.
Critical weaknesses
  1. The comparisons do not isolate the proposed upsampling mechanism from additional supervision and inputs. The model uses four DINOv3 layers, labelled class prototypes, routing/classification losses, and minority weighting (Section 3.3; Table 7). Comparable baseline adaptation is not reported. Matching the downstream head recipe in Table 3 does not match feature-learning supervision, and absence of RGB guidance does not resolve this difference. Include a capacity/input/supervision-matched feature-only decoder and appropriately adapted baselines before attributing the gains to the proposed readout/prototype design.
  2. The link between the representation findings and the architecture lacks component ablations. Tables 2–3 report the full system without testing Fisher versus random/PCA initialization, prototype versus ordinary feature reconstruction, sparse versus dense semantic coordinates, or multi-layer versus final-layer inputs. Thus, the synthetic 15-identity Fisher result does not establish which proposed mechanisms cause the segmentation gains. Ablate these components under otherwise identical training, prioritizing Fisher initialization and prototype routing.
  3. The semantic-protection construction is insufficiently specified. Equations (3)–(4) block detail writes into selected semantic directions only if the selected columns of Q are orthonormal, or an appropriate alternative projector is used. Neither this constraint nor Q’s relationship to the learned Fisher readout W is specified. Section 3.3 points to Appendix C.2 for Q, gating, and detail-path construction, but that section only describes prototype updates. Provide the construction, maintained constraints, and top-K differentiation rule so that the claimed separation can be verified.
  4. Failed subtraction does not establish the broad non-additivity/upsampler-limitation claim. Sections 1 and 5.2 infer that mixtures cannot be linear combinations and image-guided weighted aggregation cannot reveal foregrounds. However, even a linear mixture af+(1−a)b need not make subtraction of b align with f. Appendix B.2 also reports improved subtraction retrieval at block 24 (0.676 versus 0.471). Fit appropriate mixture coefficients and test oracle neighbourhood linear/convex reconstructions, or restrict the conclusion to the particular removal operations examined.
Questions for authors

See the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.