- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CoordUp, a lightweight and feature-agnostic upsampler. It formulates upsampling as a coordinate-conditioned operator. For each target location, CoordUp builds a query vector from its coordinates and RGB values. The value vectors are taken directly from the original features, without projection, and the key map is from downsampled image, enabling transfer to unseen backbones. CoordUp processes coordinates in batches and eliminates the need to materalize full-resolution intermediate feature maps, enabling 8.6 GB peak memory at 80 Mpx output. The paper evaluates CoordUp on five frozen VFMs across three high-resolution datasets for semantic segmentation and depth estimation. Results show better boundary-sensitive metrics than AnyUp and NAF. CoordUp also improves boundary quality in a zero-shot setting on TFMs such as DA3 and SAM3.
Critical strengths
- The motivation is clear. Existing feature-agnostic upsamplers (e.g., AnyUp, NAF) materalize intermediate tensors at the target resolution. Their memory cost grows rapidly with output size. At high resolution (8–12 Mpx and above), this leads to OOM. CoordUp employs explicit control over memory usage, avoiding the need to materialize full-resolution feature maps.
- The design is logically self-consistent. It constructs the query from Fourier-encoded coordinates combined with local RGB appearance embeddings, encodes the downsampled image as a key feature map, and extracts a fixed-size local neighborhood from low-resolution feature maps as the value. The overall design is light-weight (approximately 466k trainable parameters), naturally supporting arbitrary output resolutions, sparse-region queries, and feature-agnostic transferability.
- The experiment results support the claims. The study covers five VFM backbones with diverse architectures and pre-training objectives, and extends to TFM backbones. Experiments are conducted on semantic segmentation and depth estimation, across three high-resolution datasets. Boundary-sensitive metrics improve consistently, with no evident backbone bias. Memory consumption and inference speed are quantitatively measured, confirming that CoordUp remains operational at 80 Mpx.
Critical weaknesses
Major Weaknesses
- The theoretical contribution is limited. The main advance lies in no need to materialize intermediate feature maps, thereby resolving the OOM when upsampling high-resolution map. Beyond this, however, the method lacks theoretical motivation and originality: extracting keys from images (NAF, CVPR 2026), combining RGB values with coordinates to form the query (LoftUp, ICCV 2025), and selecting local regions for training (LoftUp) are all design choices that have appeared in prior work. The authors do not justify why the specific combination should yield advantages over existing upsamplers.
- The experimental setup lacks fairness, and comparative experiments against the closest prior methods are missing. The resolutions of outputs are not kept identical within any one experiment, undermining the fairness of metric comparisons. Moreover, the paper does not compare against prior methods such as FeatUp (ICLR 2024) and LoftUp under a fixed backbone. Without these key comparisons, the position of CoordUp in VFM-upsampling family remains unclear.
- Global-accuracy improvements are limited, with gains concentrated in boundary regions; and the reported gains may be biased. On global metrics such as mIoU and AbsRel, CoordUp and NAF perform similarly (within 0.5 in most settings). Gains appear mainly in boundary metrics (e.g., BIoU, BF1). However, the paper notes that AnyUp and NAF, constrained by memory, cannot directly output full-resolution predictions; instead, they first output lower-resolution predictions and then compute metrics after bilinear interpolation. Bilinear interpolation itself blurs boundaries, introducing a performance penalty for the competitors and potentially amplifying CoordUp's advantage. The paper should discuss the potential bias introduced by this experimental setting.
Minor Weaknesses
- Factual accuracy. The description of LiFT is partly inaccurate with regard to its "without guidance" characterization. The description of UPLiFT is also imprecise: UPLiFT employs local attentional pooling to address semantic drift, not to reduce the quadratic cost of global attention. The latter is a side effect, not the main motivation.
- Evaluation is limited to very high-resolution datasets. It is unclear whether CoordUp still outperforms baselines (bilinear, NAF, AnyUp) at conventional resolutions (e.g., 0.5–1 Mpx). Since CoordUp queries coordinates in batches, its runtime advantage may not hold when full-resolution feature maps easily fit in GPU memory. Adding results on one standard dataset below 1 Mpx would clarify the general applicability of CoordUp.
- Writing quality. Some sentences are too long or structurally complex, which hinders comprehension. For example, “After all, the very purpose of this research direction is to capture fine-grained details, and with their abundance of small objects, thin structures, and minute details, high-resolution images are its most natural testbed.”
- The paper claims support queries over a small region of interest or a sparse set of points. However, all experiments in Section 4 evaluate full-grid upsampling, so this capability is not empirically validated.
Questions for authors
1.To what extent does CoordUp’s advantage in boundary metrics stem from its upsampling quality itself, versus the additional bilinear interpolation step applied to the lower-resolution outputs of AnyUp and NAF? This is central to assessing the fairness of the reported performance comparisons.
2.Have you evaluated CoordUp against earlier upsampling methods including FeatUp and LoftUp under a fixed backbone? Please clarify why these comparisons are included or omitted, as they are critical for positioning CoordUp in the full VFM upsampling landscape.
3.The paper identifies sparse coordinate querying as a key design advantage of the coordinate-conditioned formulation, but all experiments evaluate full-grid upsampling only. Has this capability been empirically validated, and what speed and memory benefits does it provide in typical sparse prediction scenarios?
3 comments
W1. Theoretical motivation and originality
CoordUp's novelty lies not in its individual components but in how it separates them, and this separation is what yields its advantages. In prior upsamplers, query and key are always coupled. In NAF, keys are pooled from the same full-resolution guidance features as the queries. In AnyUp, keys depend on the encoder features. In LoftUp, keys are the VFM features. In CoordUp, the key depends only on a low-resolution image I_key, and the query depends only on (x, y, I(x, y)). The memory advantage is therefore not an implementation trick but a direct consequence of this design. Encoder-agnosticism follows from it too, since keys never see the features (Tab. 5).
This separation also explains why the combination works. Keys are computed from I_key, whose resolution is tied to the feature grid (4× its size) rather than to the output resolution. As a result, keys and values are identical whether the target is 1 Mpx or 80 Mpx. The query encodes a single pixel, with a 1-pixel receptive field, so its input does not depend on pixel density either. No component therefore sees a different input distribution between training, where CoordUp only learns a ×2 upsampling, and inference, where it is applied at much larger factors (e.g., over ×20 per side when upsampling the 128×128 DINOv3 grid to native MaSS13K resolution). In contrast, upsamplers whose guidance encoder runs convolutions at the target resolution lack this property. As the output resolution grows, their kernels cover an increasingly small image region, so at inference they operate on content at a scale never seen during training. This is consistent with Tab. 2: on DA3 at 4K, CoordUp halves the boundary error of bilinear upsampling (2.053 → 1.042), whereas NAF degrades it (2.442).
w1 / w3. Writing issues
We fixed these issues in the paper.
w2. Conventional resolution
App. A.4 (Tab. 10) evaluates NYUv2 (640 × 480) with the Probe3D linear-probing protocol of NAF and JAFAR: CoordUp reaches 90.11 δ1, against 86.73 for NAF, 86.36 for AnyUp, and 86.10 for bilinear (baseline numbers from Chambon et al.). On runtime, CoordUp has an inference time comparable to NAF's when the dense map fits in memory (Fig. 4).
Thank you for rebuttal. I remain one concern about the performances of CoordUp in conventional resolution datasets. The rebuttal points out the evaluation comparison with NYUv2, but the paper also discusses the limitations of NYUv2. It raises the question why experiments on other standard conventional resolution datasets have not been considered. I consider this remaining concern acceptable as the authors have essentially addressed my other concerns. I will therefore raise my rating to weak accept, while still hoping for an evaluation comparison on standard conventional resolution datasets.
Q1 / W3. Bias from bilinear upsampling of AnyUp and NAF outputs
Regarding global metrics (Tabs. 1 and 2), we agree that CoordUp and NAF are close on mIoU for segmentation and that the gains concentrate at boundaries. Even though boundary pixels are a small fraction of each image, they are crucial for applications requiring the detection of fine details, such as image matting or editing. Nonetheless, we highlight that CoordUp does obtain the best A.Rel in depth estimation for all five VFMs on Synth4K (3,950 frames).
Regarding boundary metrics, to assess whether CoordUp's advantage comes from its upsampling quality rather than from the additional bilinear interpolation applied to the lower-resolution outputs of AnyUp and NAF, we conducted a controlled experiment in which all methods are evaluated using a tiled inference protocol at the native image resolution, thereby removing the need for any final interpolation (the new App. A.7). In tiled inference, the upsampler processes the native-resolution output in 512×512 tiles, the head is applied to each tile, and the resulting predictions are stitched together, so the full-resolution feature map is never stored. The results are reported in the new Tabs. 12 and 13 for VFMs and TFMs, respectively. In VFM linear probing (Tab. 12), native-resolution inference raises AnyUp's Test BIoU by 1.8–3.0 points and lowers NAF's by 0.5–1.8. CoordUp still achieves the best boundary segmentation metrics against these additional baselines, as well as the best combined depth-boundary error (accuracy + completeness) in every setting. In the zero-shot TFM experiments (Tab. 13), CoordUp outperforms the strongest competitor with SAM3 at native resolution by 1.1 (Val) and 1.0 (Test) BIoU points. For depth estimation with DA3, no tiling is needed, since every upsampler already runs at full resolution (Tab. 2).
Q2 / W2. FeatUp and LoftUp under a fixed backbone
In the paper, we compare CoordUp with AnyUp and NAF, which, like our method, are backbone-agnostic. In the new App. A.8, we extend the comparison to three backbone-specific upsamplers, FeatUp, LoftUp, and UPLiFT, on DINOv3-S+ (the new Tab. 14). FeatUp and LoftUp release no DINOv3 weights, so we trained both on DINOv3-S+ features following the official recipes. The backbone-specific methods improve over bilinear interpolation but remain below CoordUp on the boundary metrics. In the new App. A.8.1, we also run UPLiFT at the native input resolution with tiled inference (the new Tab. 15), which slightly lowers its BIoU (by about 1 point on both splits). CoordUp remains the best feature upsampler.
Q3 / w4. Sparse querying
We evaluate sparse coordinate querying for semantic segmentation on MaSS13K using semantic-edge-focused inference (SEFI, the new App. A.10.1 and the new Tab. 18), with the DINOv3-B linear probe of Tab. 1. The frozen CoordUp is first queried on a stride-8 grid, and the semantic head turns the resulting features into a coarse segmentation map, which is bilinearly upsampled to the native image resolution. Individual pixels are then queried only where a class boundary of this coarse map lies within 5 px of an edge in the RGB image. Neither CoordUp nor the head is retrained; only the set of queried coordinates changes. SEFI queries only 14.4% of the pixels and leaves segmentation quality essentially unchanged (mIoU −0.02 and BIoU +0.06 on Test), while inference is about 6.0× faster per image (7.5× for the upsampler and head alone). On the 64 Mpx images, SEFI takes 2.5 to 6.0 s, against about 32.5 s for the full grid. Peak memory is the same for both schemes, because it depends on the number of coordinates processed at once (50k per batch), not on the total number queried. This study shows the benefit of the coordinate-based formulation.