acceptodds
Under review as a conference paper at ICLR 2027

CoordUp : Upsampling Any Feature Beyond the Megapixel Regime

Abstract

Vision foundation models produce semantically rich but spatially coarse features, limiting fine-detail recovery in dense prediction tasks. Existing feature upsamplers tackle this problem but are often tied to specific backbones or require materializing dense feature grids, making high-resolution inference costly. We introduce CoordUp, a lightweight, feature-agnostic upsampler that formulates feature upsampling as a coordinate-conditioned operator. Across five frozen backbones and three high-resolution datasets for semantic segmentation and depth estimation, it consistently improves boundary-sensitive metrics over previous encoder-agnostic upsamplers. It also transfers zero-shot to the final activations of DA3 and SAM3, improving boundary quality without retraining. Finally, its coordinate-based formulation allows CoordUp to scale gracefully with output resolution, enabling, for example, an 80 Mpx output using only 8.6 GB of GPU memory.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
feature upsampling, vision foundation models, encoder-agnostic, coordinate-based networks, high-resolution dense prediction, semantic segmentation, monocular depth estimation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_vkr6OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoordUp, a lightweight and feature-agnostic upsampler. It formulates upsampling as a coordinate-conditioned operator. For each target location, CoordUp builds a query vector from its coordinates and RGB values. The value vectors are taken directly from the original features, without projection, and the key map is from downsampled image, enabling transfer to unseen backbones. CoordUp processes coordinates in batches and eliminates the need to materalize full-resolution intermediate feature maps, enabling 8.6 GB peak memory at 80 Mpx output. The paper evaluates CoordUp on five frozen VFMs across three high-resolution datasets for semantic segmentation and depth estimation. Results show better boundary-sensitive metrics than AnyUp and NAF. CoordUp also improves boundary quality in a zero-shot setting on TFMs such as DA3 and SAM3.

Critical strengths
  1. The motivation is clear. Existing feature-agnostic upsamplers (e.g., AnyUp, NAF) materalize intermediate tensors at the target resolution. Their memory cost grows rapidly with output size. At high resolution (8–12 Mpx and above), this leads to OOM. CoordUp employs explicit control over memory usage, avoiding the need to materialize full-resolution feature maps.
  2. The design is logically self-consistent. It constructs the query from Fourier-encoded coordinates combined with local RGB appearance embeddings, encodes the downsampled image as a key feature map, and extracts a fixed-size local neighborhood from low-resolution feature maps as the value. The overall design is light-weight (approximately 466k trainable parameters), naturally supporting arbitrary output resolutions, sparse-region queries, and feature-agnostic transferability.
  3. The experiment results support the claims. The study covers five VFM backbones with diverse architectures and pre-training objectives, and extends to TFM backbones. Experiments are conducted on semantic segmentation and depth estimation, across three high-resolution datasets. Boundary-sensitive metrics improve consistently, with no evident backbone bias. Memory consumption and inference speed are quantitatively measured, confirming that CoordUp remains operational at 80 Mpx.
Critical weaknesses

Major Weaknesses

  1. The theoretical contribution is limited. The main advance lies in no need to materialize intermediate feature maps, thereby resolving the OOM when upsampling high-resolution map. Beyond this, however, the method lacks theoretical motivation and originality: extracting keys from images (NAF, CVPR 2026), combining RGB values with coordinates to form the query (LoftUp, ICCV 2025), and selecting local regions for training (LoftUp) are all design choices that have appeared in prior work. The authors do not justify why the specific combination should yield advantages over existing upsamplers.
  2. The experimental setup lacks fairness, and comparative experiments against the closest prior methods are missing. The resolutions of outputs are not kept identical within any one experiment, undermining the fairness of metric comparisons. Moreover, the paper does not compare against prior methods such as FeatUp (ICLR 2024) and LoftUp under a fixed backbone. Without these key comparisons, the position of CoordUp in VFM-upsampling family remains unclear.
  3. Global-accuracy improvements are limited, with gains concentrated in boundary regions; and the reported gains may be biased. On global metrics such as mIoU and AbsRel, CoordUp and NAF perform similarly (within 0.5 in most settings). Gains appear mainly in boundary metrics (e.g., BIoU, BF1). However, the paper notes that AnyUp and NAF, constrained by memory, cannot directly output full-resolution predictions; instead, they first output lower-resolution predictions and then compute metrics after bilinear interpolation. Bilinear interpolation itself blurs boundaries, introducing a performance penalty for the competitors and potentially amplifying CoordUp's advantage. The paper should discuss the potential bias introduced by this experimental setting.

Minor Weaknesses

  1. Factual accuracy. The description of LiFT is partly inaccurate with regard to its "without guidance" characterization. The description of UPLiFT is also imprecise: UPLiFT employs local attentional pooling to address semantic drift, not to reduce the quadratic cost of global attention. The latter is a side effect, not the main motivation.
  2. Evaluation is limited to very high-resolution datasets. It is unclear whether CoordUp still outperforms baselines (bilinear, NAF, AnyUp) at conventional resolutions (e.g., 0.5–1 Mpx). Since CoordUp queries coordinates in batches, its runtime advantage may not hold when full-resolution feature maps easily fit in GPU memory. Adding results on one standard dataset below 1 Mpx would clarify the general applicability of CoordUp.
  3. Writing quality. Some sentences are too long or structurally complex, which hinders comprehension. For example, “After all, the very purpose of this research direction is to capture fine-grained details, and with their abundance of small objects, thin structures, and minute details, high-resolution images are its most natural testbed.”
  4. The paper claims support queries over a small region of interest or a sparse set of points. However, all experiments in Section 4 evaluate full-grid upsampling, so this capability is not empirically validated.
Questions for authors

1.To what extent does CoordUp’s advantage in boundary metrics stem from its upsampling quality itself, versus the additional bilinear interpolation step applied to the lower-resolution outputs of AnyUp and NAF? This is central to assessing the fairness of the reported performance comparisons.

2.Have you evaluated CoordUp against earlier upsampling methods including FeatUp and LoftUp under a fixed backbone? Please clarify why these comparisons are included or omitted, as they are critical for positioning CoordUp in the full VFM upsampling landscape.

3.The paper identifies sparse coordinate querying as a key design advantage of the coordinate-conditioned formulation, but all experiments evaluate full-grid upsampling only. Has this capability been empirically validated, and what speed and memory benefits does it provide in typical sparse prediction scenarios?

3 comments

Q1 / W3. Bias from bilinear upsampling of AnyUp and NAF outputs

Regarding global metrics (Tabs. 1 and 2), we agree that CoordUp and NAF are close on mIoU for segmentation and that the gains concentrate at boundaries. Even though boundary pixels are a small fraction of each image, they are crucial for applications requiring the detection of fine details, such as image matting or editing. Nonetheless, we highlight that CoordUp does obtain the best A.Rel in depth estimation for all five VFMs on Synth4K (3,950 frames).

Regarding boundary metrics, to assess whether CoordUp's advantage comes from its upsampling quality rather than from the additional bilinear interpolation applied to the lower-resolution outputs of AnyUp and NAF, we conducted a controlled experiment in which all methods are evaluated using a tiled inference protocol at the native image resolution, thereby removing the need for any final interpolation (the new App. A.7). In tiled inference, the upsampler processes the native-resolution output in 512×512512 \times 512 tiles, the head is applied to each tile, and the resulting predictions are stitched together, so the full-resolution feature map is never stored. The results are reported in the new Tabs. 12 and 13 for VFMs and TFMs, respectively. In VFM linear probing (Tab. 12), native-resolution inference raises AnyUp's Test BIoU by 1.8–3.0 points and lowers NAF's by 0.5–1.8. CoordUp still achieves the best boundary segmentation metrics against these additional baselines, as well as the best combined depth-boundary error (accuracy + completeness) in every setting. In the zero-shot TFM experiments (Tab. 13), CoordUp outperforms the strongest competitor with SAM3 at native resolution by 1.1 (Val) and 1.0 (Test) BIoU points. For depth estimation with DA3, no tiling is needed, since every upsampler already runs at full resolution (Tab. 2).

Q2 / W2. FeatUp and LoftUp under a fixed backbone

In the paper, we compare CoordUp with AnyUp and NAF, which, like our method, are backbone-agnostic. In the new App. A.8, we extend the comparison to three backbone-specific upsamplers, FeatUp, LoftUp, and UPLiFT, on DINOv3-S+ (the new Tab. 14). FeatUp and LoftUp release no DINOv3 weights, so we trained both on DINOv3-S+ features following the official recipes. The backbone-specific methods improve over bilinear interpolation but remain below CoordUp on the boundary metrics. In the new App. A.8.1, we also run UPLiFT at the native input resolution with tiled inference (the new Tab. 15), which slightly lowers its BIoU (by about 1 point on both splits). CoordUp remains the best feature upsampler.

Q3 / w4. Sparse querying

We evaluate sparse coordinate querying for semantic segmentation on MaSS13K using semantic-edge-focused inference (SEFI, the new App. A.10.1 and the new Tab. 18), with the DINOv3-B linear probe of Tab. 1. The frozen CoordUp is first queried on a stride-8 grid, and the semantic head turns the resulting features into a coarse segmentation map, which is bilinearly upsampled to the native image resolution. Individual pixels are then queried only where a class boundary of this coarse map lies within 5 px of an edge in the RGB image. Neither CoordUp nor the head is retrained; only the set of queried coordinates changes. SEFI queries only 14.4% of the pixels and leaves segmentation quality essentially unchanged (mIoU −0.02 and BIoU +0.06 on Test), while inference is about 6.0× faster per image (7.5× for the upsampler and head alone). On the 64 Mpx images, SEFI takes 2.5 to 6.0 s, against about 32.5 s for the full grid. Peak memory is the same for both schemes, because it depends on the number of coordinates processed at once (50k per batch), not on the total number queried. This study shows the benefit of the coordinate-based formulation.

W1. Theoretical motivation and originality

CoordUp's novelty lies not in its individual components but in how it separates them, and this separation is what yields its advantages. In prior upsamplers, query and key are always coupled. In NAF, keys are pooled from the same full-resolution guidance features as the queries. In AnyUp, keys depend on the encoder features. In LoftUp, keys are the VFM features. In CoordUp, the key depends only on a low-resolution image I_key, and the query depends only on (x, y, I(x, y)). The memory advantage is therefore not an implementation trick but a direct consequence of this design. Encoder-agnosticism follows from it too, since keys never see the features (Tab. 5).

This separation also explains why the combination works. Keys are computed from I_key, whose resolution is tied to the feature grid (4× its size) rather than to the output resolution. As a result, keys and values are identical whether the target is 1 Mpx or 80 Mpx. The query encodes a single pixel, with a 1-pixel receptive field, so its input does not depend on pixel density either. No component therefore sees a different input distribution between training, where CoordUp only learns a ×2 upsampling, and inference, where it is applied at much larger factors (e.g., over ×20 per side when upsampling the 128×128 DINOv3 grid to native MaSS13K resolution). In contrast, upsamplers whose guidance encoder runs convolutions at the target resolution lack this property. As the output resolution grows, their kernels cover an increasingly small image region, so at inference they operate on content at a scale never seen during training. This is consistent with Tab. 2: on DA3 at 4K, CoordUp halves the boundary error of bilinear upsampling (2.053 → 1.042), whereas NAF degrades it (2.442).

w1 / w3. Writing issues

We fixed these issues in the paper.

w2. Conventional resolution

App. A.4 (Tab. 10) evaluates NYUv2 (640 × 480) with the Probe3D linear-probing protocol of NAF and JAFAR: CoordUp reaches 90.11 δ1, against 86.73 for NAF, 86.36 for AnyUp, and 86.10 for bilinear (baseline numbers from Chambon et al.). On runtime, CoordUp has an inference time comparable to NAF's when the dense map fits in memory (Fig. 4).

Thank you for rebuttal. I remain one concern about the performances of CoordUp in conventional resolution datasets. The rebuttal points out the evaluation comparison with NYUv2, but the paper also discusses the limitations of NYUv2. It raises the question why experiments on other standard conventional resolution datasets have not been considered. I consider this remaining concern acceptable as the authors have essentially addressed my other concerns. I will therefore raise my rating to weak accept, while still hoping for an evaluation comparison on standard conventional resolution datasets.

Reviewer_KZrHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

I think the motivation is good. When processing high-resolution images, the feature maps have low spatial resolution, so they need to be upsampled. This usually requires too much memory, so a query-based approach could reduce memory usage, although it might require more computation. In any case, the approach works, and the performance is good.

Critical strengths

The motivation is clear, and the paper covers the related work well, making it informative.

Critical weaknesses

Honestly, I would have been more convinced of its usefulness if it had substantially improved performance with DA3 or SAM3, but the gains were not that large.

As the authors probably already recognize, visually appealing PCA visualizations that clearly distinguish shapes do not necessarily correlate strongly with good performance on actual downstream tasks.

Honestly, I am not sure how useful feature upsampling research is at this point, unless the task specifically requires high-resolution features themselves. For example, if the task is not zero-shot, the effort spent training an upsampler could instead be spent building a dataset and training a model for the task. The proposed approach requires training the upsampler and, on top of that, training on the task dataset as well. To establish the value of this paper, the authors should demonstrate that:

1.The method outperforms models with task-specific decoders under the same budget.

  1. The method performs well on tasks where increasing the spatial resolution of the features is itself meaningful.

Honestly, the motivation is good, and the proposed solution makes sense. However, I still question whether this research direction is useful enough to warrant presentation at ICLR. My current score is 2 when considering timeliness, but I would give it a 3 based on the research itself, without considering timeliness.

The architecture is not novel. It appears to draw substantial architectural inspiration from NAF, and the contribution seems largely limited to adapting that architecture into a query-based approach.

Questions for authors

N/A

3 comments

Gains with DA3 and SAM3

The gains are small on global metrics but large on boundary metrics. Although boundary pixels make up only a small fraction of each image, they are crucial for applications that depend on fine details, such as image matting or editing. Below we report the relative gain of our method over the default way of using SAM3 or DA3, i.e., with bilinear upsampling, on the test sets of Tab. 2:

↑ = higher is better, ↓ = lower is better.

MetricBilinearCoordUpChange
SAM3, MaSS13KBIoU ↑13.1520.66+57%
BF1 ↑0.19550.3289+68%
mIoU ↑78.8479.62+0.78 pts
DA3, UnrealStereo4KϵPDBEacc\epsilon^{acc}_{PDBE} ↓2.0531.042−49%
DA3, Synth4KϵPDBEacc\epsilon^{acc}_{PDBE} ↓2.1111.400−34%
A.Rel ↓0.2050.206≈

CoordUp improves the quality of depth and semantic boundaries by 34% to 68%. It also outperforms the other upsamplers, as shown in Tab. 2. These gains require neither retraining of the host model nor labels.

Usefulness of feature upsampling

(1) Comparison with a task-specific decoder under the same budget (new App. A.11, Tab. 21). We agree that one alternative would be to collect a new dataset and train a model on it. Although labels are becoming easier to obtain thanks to automatic and semi-automatic annotation tools, annotating very-high-resolution images with very thin details remains difficult and expensive. This is reflected in how few datasets offer this level of detail: to the best of our knowledge, MaSS13K (CVPR 2025) is the only real-image semantic segmentation dataset with these characteristics. For depth, obtaining pixel-accurate annotations is even harder. Indeed, the benchmarks with ground truth of this quality, including UnrealStereo4K and Synth4K, are synthetic. Guided by this observation, we ran the following few-shot experiment to better understand the trade-off between collecting labelled data and using an upsampler.

We compare CoordUp + linear head with SETR-PUP, the progressive-upsampling decoder of SETR, both plugged into the same frozen DINOv3-S+ (about 29.2M parameters in total in both cases). SETR-PUP is trained from scratch on N ∈ {10, 100, 1,000, 11,348} labelled MaSS13K images; for CoordUp, only the linear head (3,463 parameters) is trained on the same images. Both are evaluated at native resolution on the validation set. Results are reported in the new Tab. 21 and repeated below for convenience:

NModelTrained paramsmIoUBIoUBF1
10SETR-PUP0.465 M62.098.990.1254
CoordUp + linear head0.003 M73.6124.990.3240
100SETR-PUP0.465 M80.7817.900.2278
CoordUp + linear head0.003 M82.3429.960.3568
1,000SETR-PUP0.465 M85.5821.230.2618
CoordUp + linear head0.003 M85.4331.490.3717
11,348SETR-PUP0.465 M87.0622.400.2757
CoordUp + linear head0.003 M85.9832.020.3750

CoordUp is the better option on every metric with up to 100 annotated samples. From 1,000 samples onwards, the decoder with more learnable parameters reaches a comparable or higher mIoU (85.58 vs. 85.43 at N = 1,000; 87.06 vs. 85.98 on the full dataset), while CoordUp remains clearly better on the boundary metrics at every N. Moreover, SETR-PUP learns its upsampling from labels, for one task and one encoder. CoordUp learns its upsampling once, without labels, in about 2.7 GPU-hours on a single A6000, and the same checkpoint serves every encoder.

(2) Tasks where higher feature resolution is itself meaningful.

We agree that feature upsampling is most useful when high-resolution features are needed. This is the case for tasks that require detecting fine details and accurate contours in very-high-resolution images, which is why our work focuses on this scenario: MaSS13K (8 to 64 Mpx) is a matting-level segmentation benchmark, while UnrealStereo4K and Synth4K provide pixel-accurate 4K depth. The experiments in our paper show that CoordUp achieves strong results in this setting.

Moreover, in this scenario large foundation models would require long inference times or could run out of memory (OOM), since memory usage scales quadratically with resolution. Feature upsampling therefore becomes even more important: it must recover not only the downsampling caused by the encoder stride, but also the input downsampling needed to run the foundation model at all. This is the main idea behind the design of CoordUp, in which the encoder input resolution is fixed. This highlight the relevance of our method.

Timeliness

We believe this research direction is timely, as shown by several recent publications at top venues: FeatUp (ICLR 2024), LoftUp (ICCV 2025, Oral), JAFAR (NeurIPS 2025), AnyUp (ICLR 2026, Oral), NAF (CVPR 2026, Highlight), and UPLiFT (CVPR 2026).

Novelty

CoordUp's novelty lies not in its individual components but in how it separates them, and this separation is what yields its advantages. In prior upsamplers, query and key are always coupled. In NAF, keys are pooled from the same full-resolution guidance features as the queries. In AnyUp, keys depend on the encoder features. In LoftUp, keys are the VFM features. In CoordUp, the key depends only on a low-resolution image I_key, and the query depends only on (x, y, I(x, y)). The memory advantage is therefore not an implementation trick but a direct consequence of this design. Encoder-agnosticism follows from it too, since keys never see the features (Tab. 5).

This separation also explains why the combination works. Keys are computed from I_key, whose resolution is tied to the feature grid (4× its size) rather than to the output resolution. As a result, keys and values are identical whether the target is 1 Mpx or 80 Mpx. The query encodes a single pixel, with a 1-pixel receptive field, so its input does not depend on pixel density either. No component therefore sees a different input distribution between training, where CoordUp only learns a ×2 upsampling, and inference, where it is applied at much larger factors (e.g., over ×20 per side when upsampling the 128×128 DINOv3 grid to native MaSS13K resolution). In contrast, upsamplers whose guidance encoder runs convolutions at the target resolution lack this property. As the output resolution grows, their kernels cover an increasingly small image region, so at inference they operate on content at a scale never seen during training. This is consistent with Tab. 2: on DA3 at 4K, CoordUp halves the boundary error of bilinear upsampling (2.053 → 1.042), whereas NAF degrades it (2.442).

The additional experiments strengthen the paper. Given the amount of public data available and how much fine-tuning a single GPU can handle, I’m still unsure when such a tight budget is necessary. I see the clearest practical value in the zero-shot setting and would emphasize approaches based on DA3, SAM3, or a VAE. Overall, I’m raising my score from 2 to 3.

Reviewer_dm1POpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoordUp, an image-guided feature upsampler, intended to be vision-backbone-agnostic. It uses coordinate-based queries to combine a local neighborhood of coarse feature vectors efficiently. Its pointwise design allows queries to be processed in batches and streamed through a task head without storing the full-resolution feature map. The upsampler is trained using DINOv3-B feature reconstruction and evaluated using it and other backbones on high-resolution semantic segmentation and depth estimation across five VFM configurations, with additional transfer experiments on frozen DA3 and SAM3. The authors report improved boundary-sensitive metrics and demonstrate inference at up to 80 megapixels within an 8.6 GB memory footprint in their SAM3 profiling setup.

Critical strengths
  1. CoordUp allows scaling to very high resolutions, well motivated by the intended use case of high-resolution prediction using TFMs such as SAM3. The pointwise query design allows feature batches to be streamed through a prediction head without storing the full-resolution feature map. The authors demonstrate an 80-megapixel output using 8.6 GB of GPU memory in their SAM3 profiling setup.
  2. The compact upsampler has very few parameters (466k) and demonstrates transfer across the tested backbones. A single checkpoint trained using DINOv3-B features is evaluated across five VFM configurations, supporting transfer across the tested feature spaces without backbone-specific upsampler retraining.
  3. Consistent segmentation-boundary improvements. In the reported VFM evaluation, CoordUp achieves the strongest BIoU and boundary F1 across all five backbones and both validation and test splits. Its test-set BIoU improvements over NAF range from 1.31 to 1.87 points, suggesting a useful benefit for fine spatial structures. We do note that while the A.rel dropped, however it does not appear to worsen over the baseline model, and we do not expect a feature upsampler to improve the quality of features.
  4. Great integration with frozen TFMs. The DA3 and SAM3 experiments demonstrate application to existing task-specific activations without retraining model weights.
  5. The paper is well written and easy to follow, and makes good use of the appendix with ample ablations that substantiate architectural decisions.
Critical weaknesses
  1. No VAE-based backbones are shown. CoordUp is positioned as a feature-agnostic upsampler, and should thus show its results on backbones such as SD1.5 VAE or more modern VAEs. Additionally, these results should be compared with those of UPLiFT, which does show this. UPLiFT's VAE results use a separately trained model with approximately 53.5M parameters and an additional refinement block, rather than direct transfer of its small visual-feature checkpoint. We do not expect that zero-shot transfer to VAE will perform well -- VAEs may have more complex feature spaces, requiring a larger (more parameters) CoordUp version, or training with VAE features specifically. Given the short rebuttal period, zero-shot inference should be shown at the minimum (even if it has subpar performance), with comparisons to UPLiFT-VAE. The RGB guidance available to each method should be specified: reconstruction with access to the original high-resolution RGB image is a different setting from latent super-resolution with only a low-resolution image available. A trained version, if needed, would make a strong addition to the manuscript in the future.
  2. No comparisons are present to backbone-specific upsamplers, such as UPLiFT. These would serve as a useful baseline to compare how this agnostic method would compare with an upsampler specifically trained for that backbone. In particular, comparisons with UPLiFT and (LoftUp or JAFAR). Please show quality metrics (e.g. semantic segmentation quality) as well as latency and memory (use UPLiFT's Fast Mode and see the question below about AnyUp's NATTEN mode).
  3. This paper appears to unify 3 key concepts from prior works: LoftUp (RGB/coordinate queries), AnyUp (backbone-agnostic transfer), and NAF (RGB-guided local filtering with raw feature values). These precedents overlap, as NAF also supports backbone-agnostic transfer. CoordUp's architectural distinction is its pointwise query encoder and separate construction of coarse keys, which enable independent query batches and streaming without storing the full-resolution feature map. The existing precedents limit its novelty to some degree, but that does not detract from the good science of combining these, ablation-guided architecture design, and final effective result.
  4. The main VFM comparisons do not clearly separate gains from the upsampling operator from gains enabled by higher output resolution and differences in prediction-head training. A matched-resolution comparison would help.
Questions for authors
  1. Could you quantitatively compare CoordUp with UPLiFT Fast Mode, trained for the same foundation-model backbone? Please include quality and output-resolution-versus-memory/latency curves under matched settings, clarifying dense feature storage versus streamed task predictions. Although UPLiFT is backbone-specific, this would clarify the quality and efficiency tradeoff relative to your transferable approach.

  2. Which AnyUp checkpoint, attention backend and query-chunking settings were used in Figure 4? In particular, was use_natten=True enabled? If not, could you include the NATTEN implementation and its supported memory-saving query chunking, reporting the resulting memory/latency tradeoff? Since the NATTEN version uses slightly different attention windows, please also report a quality comparison with the evaluated AnyUp configuration.

  3. Could you demonstrate zero-shot transfer of the frozen CoordUp upsampler to a VAE backbone, evaluating both qualitative and quantitative quality after decoding the upsampled latents? This would test the backbone-agnostic claim beyond the visual-representation and segmentation/depth features evaluated here. Please specify the RGB guidance available at inference and account for differences in available inputs when comparing with UPLiFT-VAE: using the original high-resolution RGB for guidance differs from latent super-resolution with only a low-resolution image available. Ideally, fine-tuning or full training for VAE latents, if needed, would be conducted and included, though we recognize this is not likely feasible during the rebuttal period. As such, for future manuscript revisions, this would greatly strengthen the claim of feature agnosticism.

  4. How does CoordUp behave when queried at a resolution higher than the original RGB guidance image? For example, a 2048×2048 feature grid from 1024×1024 RGB? How are RGB values obtained at subpixel query locations, and how does the resulting feature quality compare with interpolating the native-resolution CoordUp output?

  5. How many GPU-hours were required to train CoordUp itself for 50k steps, and on what GPU type and count? Please distinguish this from downstream prediction-head training.

I am happy to increase my rating should the above questions be addressed, in this order of importance:

  1. AnyUp implementation, UPLiFT Fast comparison
  2. VAE transfer and matched-resolution comparison
  3. Denser queries and GPU-hours -- useful for future work
3 comments

1. AnyUp implementation in Fig. 4 (Q2)

Fig. 4 uses the official AnyUp checkpoint with the NATTEN backend (use_natten=True) and no query chunking. Every AnyUp result in the paper uses NATTEN. In our TFM settings, AnyUp's standard attention cannot run: it builds its full window mask before chunking, which for 4K output on DA3 is a 3.26 TB allocation.

Regarding q_chunk_size, it was set to none in our experiments. We re-ran the experiment of Fig. 4 with q_chunk_size ∈ {none, 64, 32, 16, 8, 4}.

Output (Mpx)124163080
Time (s)
AnyUp, no chunking1.572.52failfailfailfail
AnyUp, q_chunk_size=642.293.818.66failfailfail
AnyUp, q_chunk_size=323.015.0811.78failfailfail
AnyUp, q_chunk_size=164.297.3317.28failfailfail
AnyUp, q_chunk_size=86.7711.7428.12failfailfail
AnyUp, q_chunk_size=412.0821.1050.90failfailfail

Chunking does make 4 Mpx feasible, at 31.2 GB, but substantially increases inference time. With any chunk size at 16 Mpx we experience OOM error.

We could run AnyUp through a memory-bounded tiled inference. See the new App. A.7 and A.9 for more details. Notably, in the DINOv3-S+ linear-probing pipeline at 8192², tiled AnyUp takes 261.5 s and 21.8 GB, against 26.3 s and 5.0 GB for CoordUp (Fig. 13).

2. Comparison with UPLiFT Fast mode (Q1, W2)

Quality (see the new App. A.8, Fig. 9, Tab. 14, Tab. 15). We use UPLiFT's DINOv3-S+/16 weights in Fast mode (fast=True, iters=4) and compared them in the DINOv3-S+ part of Tab. 1. We also add LoftUp and FeatUp, which release no DINOv3 weights, so we trained both on DINOv3-S+ with the official recipes. As we can see from Table 14, on depth estimation, UPLiFT has the best accuracy error on Synth4K (1.807 vs. 1.912), while CoordUp beats it on almost every other depth metric, and outperforms it on all MaSS13K segmentation metrics, especially boundary quality (31.16 vs. 22.68 BIoU on Test), despite using 41% fewer parameters (0.466M vs. 0.791M) and being backbone-agnostic rather than trained specifically for DINOv3-S+. We can also appreciate the better segmentation performance of CoordUp compared to UPLiFT in Fig.9. Moreover, to evaluate how the upsamplers behave when processing the same original high-resolution images, we adopt a tiled inference protocol and report the results in Tab. 15. In tiled inference, the upsampler processes the native-resolution output in 512×512512 \times 512 tiles, the head is applied to each tile, and the resulting predictions are stitched together, so the full-resolution feature map is never stored. A similar trend holds in this setting as well.

Efficiency (see the new App. A.9, A.9.1, Fig. 13, Tab. 17). We compare CoordUp with UPLiFT (FAST) in the original inference protocol and in a tiled inference for handling high-resolution images reporting results in Fig. 13 and Tab. 17. Even though the original UPliFT is the fastest option where it fits, it runs out of memory from 4096², the resolution that native MaSS13K images require. Tiled UPLiFT is faster than CoordUp up to 4096² with 2.4× more memory needed, whereas at 8192² CoordUp is 1.4× faster with 3.4× less memory.

3. VAE transfer (Q3, W1)

The new App. A.8.1 and Tab. 16 follow UPLiFT's latent super-resolution protocol: a 1024² image is downsampled to 256², encoded by the SD1.5 VAE into a 4×32×32 latent, upsampled 4×, decoded, and compared with the original. Every method, UPLiFT-VAE included, receives only the latent and the 256² image as guidance; none sees the 1024² original. As the reviewer expected, zero-shot CoordUp falls below UPLiFT-VAE. However, UPLiFT-VAE consists of a lightweight upsampler followed by a heavier refiner. We therefore replaced UPLiFT's upsampler with our frozen CoordUp while keeping UPLiFT's pretrained refiner, without any retraining. On super-resolution, this brings the FID close to that of UPLiFT-VAE (e.g., 5.55 vs. 5.29 on LHQ), showing that CoordUp can serve as a drop-in replacement within pretrained pipelines (qualitative results in the new Fig. 10). We applied the same approach to text-guided generation and observed similar trends (qualitative results in the new Figs. 11 and 12). A variant of CoordUp trained in the VAE latent space is left for future work, as the reviewer suggests.

4. Matched-resolution comparison (W4)

To better isolate the effect of the upsampling operator, we also run AnyUp and NAF at the same native resolution as CoordUp, using the tiled inference protocol mentioned above. Results on all datasets and backbones are reported in the new App. A.7 (the new Tabs. 12 and 13). In VFM linear probing (Tab. 12), native-resolution inference raises AnyUp's Test BIoU by 1.8–3.0 points and lowers NAF's by 0.5–1.8. CoordUp still achieves the best boundary segmentation metrics against these additional baselines, as well as the best combined depth-boundary error (accuracy + completeness) in every setting. In the zero-shot TFM experiments (Tab. 13), CoordUp outperforms the strongest competitor with SAM3 at native resolution by 1.1 (Val) and 1.0 (Test) BIoU points. For depth estimation with DA3, no tiling is needed, since every upsampler already runs at full resolution (Tab. 2).

5. Queries denser than the guidance (Q4)

To test CoordUp at an output resolution higher than the RGB guidance, we run the following experiment (new App. A.10.2). We take central 2048×2048 crops of the 500 MaSS13K Val images and downsample them to 1024×1024, which serves as the only RGB input. Using DINOv3-S+ with the frozen CoordUp and the linear head of Tab. 1 (no retraining), we query CoordUp on a 2048² grid, with each query reading the bilinearly upsampled RGB value at its location. We compare segmentation metrics (new Tab. 19) and feature quality (new Tab. 20) against: (A) processing 1024² coordinate queries followed by bilinear interpolation of the CoordUp output to 2048², (B) processing 2048² queries with the true 2048×2048 RGB crop as an upper-bound reference. Querying with denser queries on the interpolated RGB recovers some boundary detail: it slightly improves over (A) (+0.47 BIoU) but remains well below (B) (Tab. 19). The feature-quality metrics (Tab. 20) and the visualizations (Fig. 15) confirm this trend.

6. Training cost (Q5)

Training CoordUp for 50k steps takes about 2.7 GPU-hours on one NVIDIA A6000, including the frozen DINOv3-B forward passes that produce input and target features. We added this detail in Sec. 3.2.

Thank you for the detailed reply and the substantial set of new experiments, which address all my concerns.

AnyUp and UPLiFT Efficiency

Thank you for confirming the NATTEN backend and for the chunking and tiled-inference measurements. The streamed-versus-dense comparison in Appendix A.9 clearly shows where CoordUp's memory advantage comes from.

Backbone-Specific Upsamplers

Thank you for the UPLiFT, LoftUp and FeatUp comparisons. The native-resolution UPLiFT results show that CoordUp's boundary advantage holds against an upsampler trained for the same encoder. Since FeatUp and LoftUp were trained with shortened schedules, those feel less informative (LoftUp beats bilinear by 3mIoU in UPliFT's paper, but loses by 3 here, for example).

VAE Transfer

Thank you for running this. I am very impressed by the reconstruction quality of CoordUp + UPLiFT's refiner block (very clever use), given it was only trained on ViT style features.

The zero-shot result and the drop-in use with UPLiFT's refiner are useful evidence for the backbone-agnostic claim, and strong and substantial experiments are provided, thanks.

Matched-Resolution Comparison

Thank you for Appendix A.7. It resolves my concern: the bilinear resize explains part of the margin over AnyUp but none of the margin over NAF, and CoordUp remains best on boundary metrics.

Other Points

Thank you for the denser-query experiment and the training cost (2.7 GPU-hours). This is small :)

Two small nits for posterity's sake:

  1. Appendix A.8 says that UPLiFT uses iters=4 but Table 14 lists a 1024x1024 canvas; four 2x iterations from 128x128 grid would give 2048x2048.
  2. Please state the precision used for the CoordUp timings in Figure 13 and Table 17.

Overall, the rebuttal addresses my concerns, and I am raising my score to 4. Good work!

Comments on the paper

2 comments

We thank all reviewers for their constructive feedback. The revised manuscript adds nine new experiments. Due to space constraints, they are reported in five new appendices (A.7–A.11), with ten new tables (Tabs. 12–21) and several new figures:

  • Matched native-resolution comparison with AnyUp and NAF (A.7)
  • Comparison with backbone-specific upsamplers (FeatUp, LoftUp, UPLiFT) and VAE transfer (A.8)
  • Memory and latency against AnyUp, NAF and UPLiFT (A.9)
  • Sparse querying and queries denser than the RGB guidance (A.10)
  • Few-shot comparison with a task-specific decoder (A.11)

We also fixed the writing issues the reviewers pointed out. The titles of the new appendices are in red so they are easy to find. We hope these additions address the reviewers’ concerns, and we are happy to clarify anything further.

We sincerely thank all reviewers for their careful reading of our manuscript and for their constructive feedback, which has helped us further improve the paper. We are pleased that the additional results have addressed the reviewers’ main concerns. Thank you again for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.