acceptodds
Withdrawn from ICLR 2027

DiveUp: Learning Feature Upsampling from Diverse Vision Foundation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recently, feature upsampling has gained increasing attention owing to its effectiveness in adapting vision foundation models (VFMs) for pixel-level understanding tasks. While backbone-agnostic upsamplers offer lower training overhead and broader applicability, existing methods suffer from two fundamental limitations. First, relying solely on single-model intra-reconstruction forces the upsampler to overfit to the source model's inherent spatial misalignments and high-norm artifacts. Second, it remains unclear how to compose the training backbones for universal zero-shot generalization. To address these limitations, we propose DiveUp, a unified upsampling framework that breaks single-model dependency via two complementary mechanism: training on a diverse mix of VFMs, and supervision the upsampler with a cross-model relational guidance derived from a single spatially-aligned teacher. Concretely, Diveup formulates a local center-of-mass (COM) field as a universal relational representation that transfers the teacher's geometric structure to correct the student's spatial interpolation, without altering the backbone's native semantics. Furthermore, through systematic ablations across 14 diverse VFMs, we uncover two distinct principles governing zero-shot generaliztion: (1) at a fixed backbone budget, the representation diversity of the training mix, rather than the identity of any bacbbone, governs out-of-distribution geralizablitiy; and (2) explicit spatial alignment guidance is a separate regularization that is decisive for structurally noisy backbones. Extensive experiments on dense prediction demonstrate that DiveUp achieves state-of-the-art zero-shot performance across unseen architectures.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision Foundation Models, Self-supervised learning, Feature Upsampling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DAdtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DiveUp, a shared feature upsampler trained to reconstruct features from diverse vision foundation models while matching a local COM field from a spatially aligned teacher. It achieves competitive zero-shot performance on segmentation and depth estimation. An additional analysis relates the representational diversity of the training mixture to generalization.

Critical strengths
  • The method shows strong zero-shot generalization to unseen backbones, especially those with noisy spatial features.
  • It is a simple, model-agnostic way to transfer geometric structure without aligning heterogeneous feature spaces.
Critical weaknesses
  • The paper frames the COM guidance as a central contribution, but its practical impact in the full pipeline seems relatively minor. Its benefit is mainly visible in single-backbone settings or boundary-specific metrics, so its role as a core component need to be established more clearly.
  • The claim that mixture diversity drives generalization rests on a small number of mixtures. Since the effect of diversity is heavily confounded with the choice of model families, this conclusion lacks solid backing.
Questions for authors
  • Could the authors clarify the contribution of the COM loss in the final multi-VFM configuration, ideally with results over multiple seeds?
  • Is there additional evidence, such as mixtures that control for model family, supporting the relationship between mixture diversity and generalization?
Reviewer_y64pOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates training model-agnostic upsampler modules to adapt vision backbones to pixel-level understanding tasks. It argues that existing model-agnostic upsamplers are typically trained on features from a single model and hence they pick up that model's spatial misalignment and feature artifacts. It proposes a training algorithm that combines the reconstruction of pixel-level targets from features of diverse vision foundation models (VFMs) with an additional spatial supervision loss (COM loss). It demonstrates state-of-the-art zero-shot transfer across diverse visual backbones.

Critical strengths
  • The paper is written clearly, and it is mostly easy to follow (except Section 4.3).
  • It shows competitive performance on zero-shot transfer across diverse visual backbones.
Critical weaknesses

The paper does not clarify why achieving zero-shot transfer across diverse visual backbones is significant. The results in the tables show that the method mainly helps weaker visual backbones, while the paper fails to demonstrate significant improvements for stronger backbones. For example, bilinear performance for DINOv3-B and DINOv3-L is not reported, and the improvements for DINOv2-S are marginal. The paper targets a general setting in its narrative and experimental setup, without resource or multimodality constraints. In such a setting, there is no obvious reason to use a model-agnostic upsampler with a weak backbone (e.g., Swin-S) instead of simply using frozen features from a strong backbone (e.g., DINOv3-L, the strongest model used during training).

Other concerns.

  • As pointed out in the main weaknesses section, the method only matters if a practitioner cannot choose a single strong backbone like DINOv3-B or DINOv3-L and needs zero-shot transfer across diverse visual backbones. However, such a setting is not discussed or investigated in the current draft, and such an investigation would require a major update to the main narrative and experimental setup.
  • The COM loss has only a marginal effect in the general setting (around +0.2 as reported in Table S6 in Appendix S4.2). This result is left to the appendix, while the ablation study in the main text does not cover the main configuration. This weakens the soundness of the claim that the COM loss is useful and raises concerns about its significance as a contribution.
  • The experimental setup is not reproducible: the 14 test backbones are not named. Furthermore, their per-backbone scores are not reported for comparison.
  • Section 4.3 is hard to follow. It looks like it was largely generated by an LLM without proper human quality checks. I would suggest that the authors rewrite the entire section from scratch and also include the COM ablation for the deployed configuration in the main text.
Questions for authors
  • What is a plausible real-world scenario in which zero-shot transfer to diverse backbones is important, i.e., the proposed method will be more useful than the baselines or using frozen features of a strong backbone?
Reviewer_X3XkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DiveUp proposes a model agnostic training framework to do feature super-resolution. Most previous works focused on VFM specific upsampling with some recent works exploring VFM agnostic upsampling like NAF. DiveUp showcases their approach on NAF. During training they utilize multiple frozen VFMs by sampling across steps as well as adding geometrics guidance using Dinov3. Evaluation covers ablations, unseen backbones and few different benchmarks and setting comparing with other VFM specific and agnostic prior works. In all cases DiveUp shows strong performance.

Critical strengths
  1. The paper deals with an important problem where a lot of prior approaches have focused on VFM specific upsampling techniques like LiFT, LoftUp, Featup, UpliFT, Anyup etc. Having VFM agnostic strong upsamplers is a good direction and DiveUp presents one of the first strong results on how to do this.
  2. The approach is easy to follow and the different proposed parts like COM, VFM sampling and which VFMs to use, qualitative visualizations are all strong and make the approach sound.
Critical weaknesses
  1. The paper claims at a few places (abstract, Sec 2) that prior works use single model reconstruction based training but that claim is not true as both Anyup and NAF have experiments around mixture based trainings. That also leads to the main novelty being COM and the VFM mixture used here?
  2. Additionally, following up on previous point given COM might be the main contribution, yet COM adds only +0.20 mIoU in the deployed mixture (about +1.2 on boundary metrics, Table S6).
  3. The same SigLIP-B setting gives +4.78 (Table 3), +2.28 (Table S2), and absolute 79.61 in Table S3.
  4. In table S5 single source COM is beating the mixture COM by a reasonable margin. This contradicts that mixture helps. Could authors comment on this and highlight this in the main text.
  5. SigLIP-B is one of DiveUp’s training sources but is unseen by NAF and AnyUp-S, so Table 2(b), Figure 3, and the +3.44 gain over NAF do not test zero-shot transfer, even though they are presented that way. While I agree DiveUp is VFM-agnostic, SigLIP-B remains one of the models it is trained on, so the table should note that for fairness.
Questions for authors

I would hope the authors can answer and clarify the weaknesses I pointed out around difference from previous works and usefulness of COM. Also it would be good to add the teachers used by different AnyUp versions so its easy to track what in and out of train setup for each of them. Given the discrepancy in numbers I would like to understand if all of this is coming from different runs (in case that is true one should report averages with mean and std). Based on response to the weaknesses I am open to update my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.