acceptodds
Under review as a conference paper at ICLR 2027

Differential Feature Upsampling

Abstract

We present DiffUp, a novel feature upsampler that exploits differential kernels to enhance details. Since feature upsamplers evolve from hand-crafted to content-aware designs, they excel at detail preservation yet still produce ambiguous boundaries. In this work, we uncover a boundary suppression phenomenon behind the de facto use of softmax kernel normalization: positive-only weights reduce boundary sharpness. Inspired by zero‑mean filters, we introduce signed weights and differential kernels to amplify the differences. However, introducing signed weights alone cannot guarantee the amplification behavior; their spatial distribution also matters. We show that the same differential formulation can enhance details while also executing local averaging. We therefore establish conditional guarantees for when and where the behaviors can take place. By embedding our insights into DiffUp, we show through controlled experiments and four dense prediction tasks that DiffUp outperforms or matches state-of-the-art dynamic upsamplers. Notably, it improves the boundary IoU on COCONut by up to 4.37% over bilinear interpolation. Our approach suggests yet another design principle for modern feature upsamplers, i.e., amplification over mere preservation.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Feature Upsampling, Dense Prediction, Softmax Function, Kernel Normalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Aut3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors observe that although existing upsampling algorithms are capable of preserving details to a relatively rich extent, they struggle to delineate boundaries clearly. Through their investigation, the authors find that the softmax kernel normalization method widely adopted in practice suffers from a boundary suppression phenomenon. To address this issue, they propose a novel feature upsampling algorithm, DiffUp, which employs differential kernels to enhance detailed features. Extensive downstream tasks demonstrate the effectiveness of the DiffUp module.

Critical strengths

(1) The authors provide a detailed and effective analysis of the problems confronting softmax normalization and the advantages of differential feature upsampling, which can relatively thoroughly substantiate the issue raised by the authors.
(2) The authors conduct sufficient comparative experiments on multiple datasets and across various tasks, demonstrating strong experimental completeness.

Critical weaknesses

(1) The authors' theoretical analysis relies on rather strong idealized assumptions, such as binary boundaries and constant feature values within local regions. Can the authors quantify the proportion of cases in which these assumptions actually hold in real network features? Meanwhile, in Theorem 1, the authors interpret softmax positive-weight aggregation as the root cause of boundary suppression; however, if the softmax weights are highly concentrated on the same side of the semantics, boundary blurring may be very weak. The authors need to further clarify which dynamic upsamplers and which boundary scenarios this theorem specifically applies to.
(2) Many of the experimental improvements reported in the paper are rather marginal, such as those on NYUx2 depth estimation and VisDrone detection, whereas the gains on COCONut are pronounced. The authors need to explain why the benefits vary so markedly across different tasks, and whether the model has specific downstream application scenarios. If balanced improvements across different downstream tasks cannot be achieved, is the title of the paper too broad? Moreover, do the various assumptions and derivations presented in the paper still hold?

Questions for authors

Please refer to the content of the Weakness section and provide detailed responses to each of the issues raised.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Aut3

We thank the reviewer for assessing the theoretical analysis and experimental coverage. We address the reviewer’s questions about the local assumptions, softmax weights that concentrate on the same side of a boundary, and variation in the gains across tasks. We have also added measurements on learned features and three-seed results for the main segmentation comparisons.

W1. How well can softmax preserve a boundary, and what does differential reassembly add?

We agree that softmax can preserve nearly all of the input contrast if its weights concentrate on the same side of the boundary. Let the two regions have constant feature values a<ba<b, and let α\alpha and β\beta be the total softmax weights assigned to the opposite region at the two output locations. The outputs are

a′=a+α(b−a),b′=b−β(b−a).a'=a+\alpha(b-a),\qquad b'=b-\beta(b-a).

The ratio of absolute output contrast to input contrast is therefore

∣b′−a′∣b−a=∣1−α−β∣.\frac{|b'-a'|}{b-a}=|1-\alpha-\beta|.

When α\alpha and β\beta are both close to zero, the ratio approaches one. The attenuation can thus be arbitrarily small, consistent with the analysis in the paper.

This calculation distinguishes the contrast the operator can express from the weights it learns in practice. In the two-region model, softmax can approach the input contrast by concentrating its weights on the same side. Nonnegative, unit-sum reassembly still bounds the attainable contrast by the input contrast. This matters for a weak edge where preserving the existing difference gives limited scope for improvement. Differential reassembly extends the range of local responses and can amplify that edge under an appropriate kernel configuration.

Optimization also matters. Although softmax can express nearly complete same-side concentration, that configuration can be difficult to reach consistently during training because of gradient smoothing. We therefore consider the influence of parameterization and optimization on the reassembly patterns a model learns to be relevant to the observed performance differences.

In the controlled experiment in the original paper, with the kernel generator held fixed, DiffUp improves on softmax reassembly by 2.11 mIoU points and 3.08 bIoU points at a boundary width of 1% of the image diagonal. The visualizations show improvements on thin structures and local edges, along with stronger gradients in the intermediate features. Our new analysis also gives a real example in which the output lies outside the range attainable by any convex combination of the features in the same sampled support. This connects the difference in expressivity to a response actually produced by a trained model.

The two theoretical configurations provide reference cases. Real features contain semantic, texture, and illumination changes of varying strength, and learned kernels can produce enhancement, averaging, and intermediate responses. Appendix C reports the distributions of the learned kernel coefficients across images and decoder stages, together with measurements of feature differences across boundaries and within regions.

W2. Why do gains vary across tasks, and what does this mean for the scope of the analysis and the title?

DiffUp acts directly on local feature differences. The effect on a task metric depends on how much boundary and detail representation limits performance in that setting. The dataset, input features, network configuration, and evaluation metric all affect that relationship.

For example, in our new three-seed SegFormer-B1 results on COCONut, DiffUp gains 0.53 mIoU and 0.81 bIoU points over FADE, with the larger gain on the boundary metric. On VisDrone, DiffUp achieves 27.6 mAP@50:95, close to FADE at 27.7. Segmentation bIoU directly evaluates a band around object contours. Detection AP also depends on classification, box localization, and confidence ranking, while the NYUv2 depth metrics aggregate errors or accuracy over all valid pixels. A change in local boundary representation can therefore affect these metrics to different degrees.

Object scale and structure, the boundary information already present in the input features, and how the prediction head uses that information also vary across settings. In some depth and detection configurations, boundary ambiguity may play a relatively small role in the final metric, which could explain the smaller gains. We will discuss these differences in the context of each task and its input features.

The same differential construction remains competitive across all four dense prediction tasks. Its ability to express enhancement, smoothing, and intermediate responses gives it scope to adapt across tasks. The title Differential Feature Upsampling refers to the local feature correction formed by a zero-sum differential kernel. The construction itself does not require a task-specific feature space. We will consider revising the title to make its focus on detail and boundary enhancement more explicit.

Thank you for your careful and detailed response. Although I am not very familiar with this field, I believe the novelty and workload of this paper have reached the level of ICLR. Good luck!

Reviewer_Syh6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Differential Feature Upsampling (DiffUp), a lightweight content-aware operator for dense prediction. It argues that conventional softmax-normalized upsampling uses positive, sum-to-one weights that tend to smooth feature differences across semantic boundaries. DiffUp instead combines an identity kernel with a zero-mean, signed differential kernel, enabling adaptive boundary sharpening while retaining smoothing behavior within homogeneous regions. Experiments across semantic segmentation, portrait matting, monocular depth estimation, and object detection show consistent improvements, with the largest gains on boundary-sensitive tasks.

Critical strengths
  • Useful insights and formulation The paper identifies an important limitation of conventional softmax-normalized upsampling kernels and introduces a simple differential formulation that enables local contrast amplification while preserving constant feature responses. The combination of a zero-mean, (L_1)-normalized differential kernel and an identity base kernel is interesting.

  • Convincing empirical improvements across multiple dense prediction tasks The paper evaluates DiffUp on semantic segmentation, image matting, depth estimation, and object detection, using representative architectures and competitive upsampling baselines. The improvements are particularly meaningful for boundary-sensitive tasks.

Critical weaknesses
  • More detailed ablations: Although the controlled comparison with softmax normalization is valuable, it remains unclear whether the improvements primarily arise from the specific zero-mean and (L_1)-normalized differential formulation or more generally from allowing signed aggregation weights. Comparisons against alternative signed-kernel normalization schemes, and ablations of the individual formulation components, would help establish the necessity of the proposed design.
Questions for authors
  • Although the authors provide various downstream tasks to show the effectiveness and advantages of the proposed work, I still wonder that if there are cases/applications/tasks that the proposed method are inferior or even damage the performance? In other words, what's the boundary of the proposed method, or in a more general sense, the feature upsampling paradigm?

  • Following the last question, I wonder how frequently do the theoretical conditions hold in practice? Could the authors quantify the proportion of learned kernels satisfying the conditions in theorems across different network layers or datasets? This would help establish whether the proposed theoretical mechanisms explain typical learned behavior or only particular examples.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Syh6 Part1

We thank the reviewer for assessing the differential construction, theoretical analysis, and evaluation across tasks. The reviewer’s questions prompted us to examine which components matter, what limits the operator's behavior, and how the theoretical cases relate to learned kernels. We have added normalization ablations, a control with an alternative signed parameterization, and an analysis across images and decoder stages.

W1. Component ablations and an alternative signed parameterization

Section 4.1 and Table 1 of the original submission compare softmax reassembly and DiffUp using the same kernel generator. With SegNet on Cityscapes, DiffUp raises mIoU from 60.62 to 62.73 and bIoU from 43.24 to 46.32 at a boundary width of 1% of the image diagonal. Following the reviewer’s suggestion, we also ran the following ablations with SegFormer-B1 on COCONut. The revised paper contains the full ablation table.

ConfigurationmIoUbIoU
Full DiffUp60.5043.84
Remove LayerNorm60.5643.76
Remove mean centering60.5143.82
Remove ℓ1\ell_1 normalization60.1543.32
Reduce amplitude factor to 159.6442.76
Decoder features only (no encoder guidance)60.0642.70
High-pass kernel 2δ−softmax(z)2\delta-\mathrm{softmax}(z)58.5340.13

The high-pass control keeps the DiffUp kernel generator and reassembly locations, as well as the identity path and a signed correction. It achieves 58.53 mIoU and 40.13 bIoU, compared with 60.50 and 43.84 for full DiffUp. The high-pass kernel has nonpositive off-center coefficients. DiffUp allows either sign at each off-center position, so the same reassembly operator can enhance local contrast or average neighboring features, depending on the input. The comparison supports the importance of how the signed coefficients are parameterized and placed.

The ℓ1\ell_1 normalization controls the magnitude of the differential correction. Removing it reduces mIoU by 0.35 points and bIoU by 0.52 points. LayerNorm and explicit mean centering partly overlap because both subtract the mean over the kernel coefficients. LayerNorm does so during standardization, before applying its learned per-channel affine transformation,

ui=γizi−μσ2+ϵ+βi.u_i=\gamma_i\frac{z_i-\mu}{\sqrt{\sigma^2+\epsilon}}+\beta_i.

The learned scales and biases can change the mean of the kernel coefficients. We therefore subtract the mean after the affine transformation to ensure that the differential kernel sums to zero and its correction vanishes on constant features. Removing LayerNorm gives 43.76 bIoU, and removing mean centering gives 43.82, compared with 43.84 for full DiffUp.

The default amplitude factor of 2 also has a theoretical motivation. Write the differential kernel as

d=a w−wˉ∥w−wˉ∥1+ϵ.d=a\,\frac{w-\bar w}{\|w-\bar w\|_1+\epsilon}.

Under the sign conditions of Theorem 3, the center differential coefficient is nonpositive and the off-center coefficients are nonnegative. Consequently,

−dc=∑q≠cdq=∥d∥12<a2.-d_c=\sum_{q\ne c}d_q=\frac{\|d\|_1}{2}<\frac{a}{2}.

Under the sign conditions of Theorem 3, any amplitude factor 0<a≤20<a\leq 2 guarantees that the effective kernel is nonnegative and sums to one. We use a=2a=2 because it is the largest factor for which this guarantee holds for every kernel satisfying those conditions. Reducing the factor to 1 retains the local averaging guarantee but reduces the magnitude of the differential correction and, in our COCONut ablation, lowers bIoU from 43.84 to 42.76.

AuthorsOpenReview ↗

Response to Reviewer Syh6 Part2

Q1. When is DiffUp likely to help, and what limits its effectiveness?

DiffUp predicts reassembly weights from the encoder and decoder features. The structures it can use, and the variations it can enhance, therefore depend on what those features represent. Our new visualizations show that different decoder branches represent the same scene differently and that DiffUp's responses vary across those branches.

The streetlight example in the original paper illustrates this dependence. A bright spot inside the lamp belongs to the same semantic region as its surroundings, yet it produces a strong local change in the feature map. Differential reassembly can further enhance that change. A response inside a semantic region can thus be traced to structure already present in the input to the upsampler.

DiffUp is most likely to help when prominent changes in the features coincide with boundaries or details relevant to the task. Its advantage can be smaller when illumination or texture dominates the local variation and correlates weakly with the target boundary. Both the task's sensitivity to boundaries and the quality of their representation in the input features affect the gain from differential upsampling.

Q2. How do the two theoretical cases relate to learned kernels?

Sections 4.1 and 4.2 analyze two idealized kernel configurations to show how the same differential reassembly operator can enhance contrast across a boundary or average features within a region. Figure 4 illustrates this relationship on a learned feature map. The kernel at the boundary has a contrastive pattern, the kernel in the interior has a smoothing pattern, and the kernel at the transition point exhibits an intermediate pattern.

We analyzed the learned kernel coefficients at all six DiffUp locations on 200 validation images. At a given distance from a ground-truth boundary, the net weight assigned to neighbors of a different class can be positive or negative. At U4 and U5, the distribution of the center differential coefficient concentrates near −1-1 farther inside regions.

This second observation connects the measurements to regional averaging. The effective center weight is 1+dc1+d_c. Under the zero-sum and magnitude constraints, a nonpositive center differential coefficient satisfies

∑q≠cmax⁡(−dq,0)≤1+dc.\sum_{q\ne c}\max(-d_q,0)\leq 1+d_c.

As dcd_c approaches −1-1, the total negative off-center weight approaches zero, and the effective kernel approaches a nonnegative weighted average of the neighbors.

We also measured differences on the features themselves. Relative to the input, after normalization by the input feature scale, mean cross-boundary differences increase at U0–U4 and mean within-region variation decreases at all six locations. Together with the intermediate kernel in Figure 4, these measurements show how the two idealized analyses help interpret the range of reassembly patterns learned on real features.

Thanks to the authors for their additional effort. With the newly added experimental results and discussion, my concerns on the ablations are mostly addressed. I will maintain my positive position on this submission, and I hope that the replied discussion could be included in future versions to further improve the soundness and comprehensiveness of the work.

Reviewer_2RNaOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DiffUp, a feature upsampling operator motivated by the contrast limitations of positive, softmax-normalized aggregation in existing dynamic upsamplers. It analyzes an idealized two-valued feature boundary and introduces a signed, zero-sum 'differential' kernel, bounded by L1 normalization and added to an identity (nearest-neighbor) base kernel, with kernels generated from both encoder and decoder features following FADE. Theoretical results show how particular weight patterns can sharpen boundaries or produce local averaging within regions. DiffUp is a drop-in operator trained end-to-end inside each task network, and a higher-capacity gated variant, DiffUp+, modulates the strength of the correction. The authors evaluate separately trained integrations on semantic segmentation, portrait matting, monocular depth estimation and object detection, reporting boundary-quality improvements and relatively small parameter/FLOP overhead for the standard operator.

Critical strengths
  1. A useful mathematical explanation of the design and its theoretical foundations is included. The analysis makes the distinction between positive averaging and signed contrast enhancement explicit, and identifies conditions under which the same differential construction can instead average neighboring features. The conditions are stated as assumptions, which helps separate the operator's capabilities from what training must learn.

  2. Helpful feature visualizations. Figures 3 and 5 show stronger feature-gradient magnitudes for DiffUp than for the softmax counterpart, supporting the claimed change in local feature responses (though greater gradient magnitude does not necessarily establish better upsampled representations or downstream usefulness). Figure 4 illustrates how the learned kernel pattern changes between interior, transition and boundary points.

  3. Empirical evidence that the reformulation provides some utility. Table 1 compares softmax and DiffUp with the same kernel generator in trained SegNet models, improving mIoU by 2.11 points and boundary IoU by 3.08 points at the 1% boundary width. Table 2 also shows gains over SAPA on SegFormer-B1 of 0.74 mIoU and 1.27 boundary-IoU points at the same reported additional parameter/FLOP cost. Segmentation and matting provide the strongest empirical support; the breadth across four task types is useful even though the depth and detection results are more mixed.

  4. A simple, well-documented operator. DiffUp is a drop-in replacement for existing upsampling layers, with the same reported overhead as SAPA (+0.11M parameters, +0.62 GFLOPs on SegFormer-B1) and lower overhead than FADE. The appendix gives per-task training and evaluation configurations.

Critical weaknesses
  1. The closest prior design is not compared, which narrows the novelty claim. The introduction states that prior dynamic upsamplers "ultimately follow the same procedure of softmax normalization" (lines 41-42), and therefore assemble features with positive weights only. However, FreqFusion (Chen et al., 2024), cited in Section 2, builds its adaptive high-pass filter as the identity kernel minus a softmax-normalized low-pass kernel and adds the filtered result residually to high-resolution encoder features (its Eqs 12 & 13). Its effective kernel is thus an identity base + a content-adaptive, zero-sum signed correction, which is the same form as Eq 7. DiffUp's distinction is that its correction can take any sign pattern, so one kernel can either sharpen or smooth, and it is applied while reassembling decoder features; FreqFusion's high-pass correction can only sharpen. FreqFusion is absent from all result tables. A matched comparison, for example on COCONut with SegFormer-B1, would show whether this added flexibility improves on the existing signed design.

  2. Gains over the strongest baselines are small and are reported without any run-to-run variability / CI intervals. Relative to FADE, DiffUp improves mIoU/bIoU by 0.23/0.65 points on SegFormer-B1 and 0.41/0.47 points on SegFormer-B5 (Tab 2), and reduces P3M-500-NP SAD from 10.558 to 10.544 (Tab 3). In detection, DiffUp's mAP50:95 of 27.6 is below FADE and DySample (27.7, tab 5); in depth, it is worse than FADE and DySample on Abs Rel and SILog. No seeds, standard deviations or confidence intervals are reported, so these orderings are difficult to distinguish from training variation. The headline numbers in the abstract and introduction use weaker references: 4.37 bIoU improvement over bilinear, and a 6.3% SAD reduction relative to DySample, compared with 0.13% relative to FADE. Repeated runs for the main comparisons and headline comparisons against the strongest baseline would make the empirical contribution stronger/clearer.

  3. The selective sharpening/averaging explanation needs model/dataset-level evidence. The theory establishes what happens under particular kernel sign patterns, while Figures 4 & 6 provide selected or clustered examples; Figure 6 uses a single image. This does not establish how the actual learned kernels sharpen near semantic borders and average within regions across the model and dataset. The introduction states that kernel visualizations "confirm that learned DiffUp kernels conform our expectations" (lines 60 &61), while Section 4.1 says that the negative-weight distribution "reflects complex local adaptations rather than strict boundary confinement." Distributions of kernel weights and signed cross-boundary mass versus distance to the nearest boundary, across held-out images and decoding stages (see question 2), would directly test this explanation and close the gap between theory and empirical evidence.

  4. The differential formulation itself is not ablated. Section 5.5 states that the effectiveness of the differential kernel formulation "has been validated in Section 4", but Table 1 compares only the complete DiffUp kernel with softmax, and Table 6 varies only predictor width, convolution type and the gate. Mean-centering, L1 normalization with the factor of 2, the identity base kernel and the layer normalization are never isolated. Controls such as softmax plus the same identity residual, or a FreqFusion-style identity-plus-high-pass kernel, delta + (delta - softmax), would show whether the unconstrained sign pattern is what produces the gains.

  5. The claim about model-agnostic upsamplers is not supported, and the frozen-feature comparison is missing. Section 2 (line 107) states that model-agnostic upsamplers "also suffer from the boundary ambiguity discussed above," but the paper gives no evidence of this in their native frozen-backbone setting. The SegFormer comparisons with FeatUp, LoftUp and NAF (Tab 2) fine-tune these methods jointly with the backbone and task head, so they do not show whether the signed-kernel formulation improves upsampling of fixed, pretrained representations. A controlled comparison against LoftUp, UPLiFT and NAF on a shared frozen DINOv2 or DINOv3 backbone would test whether the benefit persists when the upsampler must preserve an existing representation.

  6. Practical efficiency is not measured despite being an explicit design goal. Section 3.3 lists computational efficiency among the three design goals, and Tables 2 and 5 report parameters and FLOPs, but no inference latency, throughput or peak GPU memory is reported. Section 5.1 also notes that GPU memory limited the cross-paradigm comparison to SegFormer-B1, which makes memory a relevant measure. Measured latency and peak memory against the main baselines (see question 6) are needed to assess whether the reported gains are delivered efficiently in practice.

  7. DiffUp+ does not show a consistent benefit. It is worse than DiffUp on SegFormer-B5 and on every matting metric, and only modestly better elsewhere. The paper’s own ablation (Tab 6) suggests the gate adds little: adding it to an otherwise identical predictor improves mIoU by only 0.01 points, and a cheaper ungated variant (0.22M vs. 0.42M parameters) outperforms DiffUp+. Section 5.5’s claim of “stronger adaptive capability” is therefore not supported. Because DiffUp+ also changes predictor width and convolution type, its small detection gain cannot directly be attributed to the gate or is, at the least, confounded. The authors should test the regularization explanation for matting (Section 5.2) directly.

  8. From my understanding, delta-1, RMSE and log10 are computed over all valid pixels, so delta 1 does not specifically measure "prediction fidelity for pixels around object contours" (sec 5.3). What boundary-specific evidence supports the depth-sharpening interpretation, for example depth-boundary errors or edge F1 on depth discontinuities?

Questions for authors
  1. Beyond allowing any sign pattern, what does DiffUp's formulation provide over FreqFusion's identity-plus-high-pass kernel, and how do the two compare under a matched task and computational budget?

  2. Can you report learned kernel statistics as a function of distance to the nearest ground-truth boundary, across held-out images and decoding stages: the signed cross-boundary mass near boundaries, the share of center-negative/off-center-nonnegative kernels in interiors, and boundary contrast and interior variation normalized for feature scale?

  3. Can the effect of the signed formulation be separated from the identity residual base, for example by comparing with softmax plus the same identity residual (from weakness 4)?

  4. How variable are the main results across seeds? Please report the mean and standard deviation for the main comparisons in Tables 2 to 5.

  5. Please report inference latency and peak GPU memory for DiffUp and the main baselines (FADE, SAPA, DySample, the cross-paradigm methods in Table 2, and LoftUp/NAF/UPLiFT fast on fixed hardware, stating batch size, input/output resolutions, precision, warm-up, and whether timing covers the upsampler alone or the full pipeline.

  6. Have you tried DiffUp on VAE latent features? UPLiFT’s Appendix B1 / Figure 7 (https://arxiv.org/html/2601.17950v2#A2.SS1) shows blocky decoded images when its VAE refiner is removed and suggests that restrictive linear-combination aggregation contributes to the problem. Since DiffUp relaxes convex aggregation through signed weights, this seems a particularly relevant setting in which to test its proposed advantage. If you have tried it, how is decoded image quality affected, and is an additional refiner still needed?

My initial recommendation is weak rejection. The operator is well motivated mathematically, simple, and the controlled comparison with softmax is convincing. However, the contribution over FreqFusion’s existing signed kernel is unclear and the gains over the strongest baselines do not report variance. I will raise my score if the authors show 1) a matched comparison against FreqFusion or an identity with high pass control, and 2) seed variance for the main segmentation comparisons.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2RNa Part1

We thank the reviewer for the careful review and for suggesting specific experiments to distinguish our contribution from prior work and assess the reliability of the results. We have added a comparison with the full FreqFusion method, a high-pass control using the same kernel generator as DiffUp, and three-seed results for the main segmentation comparisons. We also report component ablations, an analysis of learned kernels across images and decoder stages, and whole-model inference measurements.

1. Stronger baselines and results across random seeds (W1, W2, Q1, Q4)

We trained the main SegFormer-B1 models on COCONut with three random seeds, using the same 160k training iterations and evaluation protocol as in the paper. The table reports the mean and standard deviation across seeds.Multi-seed experiments are still in progress for FeatUp, LoftUp, and NAF on SegFormer-B1 and for the SegFormer-B5 comparisons.

UpsamplermIoUbIoU
Bilinear57.99 ± 0.1238.89 ± 0.15
DySample59.44 ± 0.0940.82 ± 0.20
SAPA59.81 ± 0.0642.80 ± 0.17
FADE59.97 ± 0.1343.03 ± 0.05
FreqFusion60.18 ± 0.1142.34 ± 0.10
DiffUp60.50 ± 0.1743.84 ± 0.12
DiffUp+60.56 ± 0.0543.89 ± 0.07

DiffUp improves on FADE by 0.53 mIoU and 0.81 bIoU points, on SAPA by 0.69 and 1.04 points, and on FreqFusion by 0.32 and 1.50 points, respectively. Across all three comparisons, the gain is larger on bIoU. For FreqFusion, we followed the implementation guidance in the FreqFusion repository.

Following the reviewer’s suggestion, we also kept the kernel generator, reassembly locations, and training setup fixed and replaced the effective kernel with

KHP=2δ−softmax(z).K_{\mathrm{HP}}=2\delta-\mathrm{softmax}(z).

This high-pass control achieves 58.53 mIoU and 40.13 bIoU, respectively 1.97 and 3.71 points below the three-seed means for DiffUp. The control retains both the identity path and a signed high-pass correction. The comparison supports the practical value of the freedom DiffUp has in assigning signed coefficients across the reassembly kernel. Table 2 now reports the three-seed SegFormer-B1 comparisons, and the Introduction gives the gains over the stronger baselines.

2. Relationship to FreqFusion (W1, Q1, Q3)

We thank the reviewer for pointing out that both methods admit an identity-plus-zero-sum-correction interpretation. Section 2 of the original paper discusses FreqFusion. Its motivation comes from intra-class and inter-class feature similarity, whereas our starting point is how negative reassembly weights can enhance boundaries. The connection is clearest at the level of FreqFusion's high-pass branch.

Let p=softmax(z)p=\mathrm{softmax}(z). The effective kernel of that branch is 2δ−p2\delta-p, so every off-center coefficient is nonpositive. A single differential kernel in DiffUp can have both positive and negative off-center coefficients. Mean centering ensures that the correction sums to zero, and ℓ1\ell_1 normalization bounds its magnitude. This gives the same reassembly operator the capacity for both contrast enhancement and local averaging, as analyzed in Sections 4.1 and 4.2. The full FreqFusion method combines low-pass filtering, high-pass filtering, and offsets for feature fusion.

In our high-pass control, we observed that the learned softmax weights concentrated at the center. If pcp_c denotes the center weight, then

∥KHP−δ∥1=2(1−pc).\|K_{\mathrm{HP}}-\delta\|_1=2(1-p_c).

As pcp_c approaches one, the correction shrinks and the effective kernel approaches the nearest-neighbor (NN) base. The sign constraint on the off-center coefficients precludes nontrivial local averaging within this operator. Reducing the magnitude of the correction is one response available during training, consistent with the near-NN behavior we observed in the high-pass control. DiffUp can also change the signs and spatial placement of the neighboring coefficients to adapt to local structure.

We have narrowed the statement in the Introduction about which methods use softmax-normalized reassembly weights. In Related Work, we explicitly acknowledge FreqFusion's signed high-pass residual and distinguish the two methods by the signs allowed at off-center positions.

AuthorsOpenReview ↗

Response to Reviewer 2RNa Part2

3. Normalization, amplitude factor, and identity path (W4, Q3)

We have added the full component ablation to the revised paper. The relevant SegFormer-B1 results on COCONut are given below.

ConfigurationmIoUbIoU
Full DiffUp60.5043.84
Remove LayerNorm60.5643.76
Remove mean centering60.5143.82
Remove ℓ1\ell_1 normalization60.1543.32
Decoder features only (no encoder guidance)60.0642.70
Reduce amplitude factor to 159.6442.76

LayerNorm and explicit mean centering partly overlap because both subtract the mean over the kernel coefficients. LayerNorm does so during standardization, before applying its learned per-channel affine transformation,

ui=γizi−μσ2+ϵ+βi.u_i=\gamma_i\frac{z_i-\mu}{\sqrt{\sigma^2+\epsilon}}+\beta_i.

The learned scales and biases can change the mean of the kernel coefficients. We therefore subtract the mean after the affine transformation to ensure that the differential kernel sums to zero and its correction vanishes on constant features. In the ablation, removing LayerNorm or mean centering gives 43.76 or 43.82 bIoU, respectively, compared with 43.84 for full DiffUp. Removing ℓ1\ell_1 normalization reduces mIoU by 0.35 points and bIoU by 0.52 points.

We use an amplitude factor of 2, the largest value that guarantees a nonnegative, unit-sum effective kernel under the sign conditions of Theorem 3. A factor of 1 also satisfies this condition and gives a smaller differential correction, with bIoU decreasing by 1.08 points in our ablation.

For the identity path, the 2δ−softmax(z)2\delta-\mathrm{softmax}(z) control above implements the test suggested by the reviewer by retaining the same identity path and kernel generator.

4. Statistics of learned kernels and their effects on features (W3, Q2)

Following the reviewer’s suggestion, we analyzed all six DiffUp locations in one trained SegFormer-B1 on 200 COCONut validation images. We grouped kernels by distance to the nearest ground-truth boundary and measured feature differences across boundaries and within regions. We express distances in units of each module's input stride and normalize feature differences by the corresponding input feature scale.

The net weight assigned to neighbors of a different class varies with location, and its distribution becomes more concentrated near zero with increasing distance from the boundary. Farther inside regions, the center differential coefficient tends to concentrate closer to −1-1. This observation can be connected directly to the kernel construction. If dqd_q is the differential coefficient at position qq and cc denotes the center, then

Kc=1+dc,∑q≠cdq=−dc.K_c=1+d_c,\qquad \sum_{q\ne c}d_q=-d_c.

As dcd_c approaches −1-1, the effective center weight decreases and weight shifts to the neighbors. The zero-sum and ℓ1\ell_1 constraints also imply that, for dc≤0d_c\leq 0,

∑q≠cmax⁡(−dq,0)≤1+dc.\sum_{q\ne c}\max(-d_q,0)\leq 1+d_c.

Thus the total negative off-center weight also approaches zero, and the effective kernel approaches a nonnegative weighted average of neighboring features. This connects the observed center coefficients to the regional smoothing analyzed in the paper.

We also measured what reassembly does to the input features. Relative to the input, the scale-normalized mean feature difference across boundaries increases at U0–U4, while mean within-region variation decreases at all six locations. U5 receives shallow features rich in texture and exhibits stronger smoothing, with a pronounced decrease in within-region variation. These results illustrate how the same differential construction can produce different reassembly behavior across modules and local structures, as discussed in Section 4.

Appendix C presents the full analysis. A spatial permutation control holds the input features, the center coefficient, and the set of off-center coefficients fixed while varying their placement. A real boundary example shows the range of responses attainable by signed reassembly. PCA visualizations of the branch inputs show the local structures available to the different modules and provide context for the variation in their learned kernels.

AuthorsOpenReview ↗

Response to Reviewer 2RNa Part3

5. Measured latency, throughput, and memory (W6, Q5)

We measured the full SegFormer-B1 model on an RTX A6000 with batch size 1, FP32, TF32 disabled, and input and output resolutions of 512×512512\times512. Timing includes the backbone, decoder, restoration to the output resolution, and GPU argmax. We used GPU synchronization and ran three independent rounds, each with 20 warm-up and 100 timed passes. Data loading, preprocessing, host-to-device transfer, and test-time augmentation were outside the timed region.

UpsamplerLatency (ms)Throughput (images/s)Peak allocated GPU memory (MiB)
Bilinear9.57 ± 0.12104.52231.21
FADE11.94 ± 0.3083.78231.51
SAPA12.84 ± 0.2877.91231.61
DySample12.00 ± 0.3883.35231.40
DiffUp14.27 ± 0.2270.07231.66
FreqFusion21.26 ± 0.6747.03516.05
FeatUp18.80 ± 0.2653.18323.61
LoftUp77.46 ± 0.4912.912366.98
NAF23.24 ± 0.2143.02248.36

Latency is the mean ± standard deviation across the three rounds. Compared with FreqFusion, DiffUp improves both accuracy metrics while reducing latency by 32.9% and peak allocated GPU memory by 55.1%. These measurements complement the parameter and FLOP counts in the paper. They describe the methods as implemented in our downstream task network. The revised appendix also reports efficiency measurements for SegFormer-B5, depth estimation, and object detection.

6. Model-agnostic upsampling and frozen features (W5)

Section 2 distinguishes model-agnostic feature upsampling from joint training with a task network. Section 5.1 compares the methods under a common downstream training protocol, initializing model-agnostic upsamplers with their released pretrained weights when available. Model-agnostic methods typically reconstruct features from a frozen vision foundation model with a standalone network, without access to a multiscale decoder. DiffUp is a lightweight operator designed for multistage task decoders. Its kernel generator draws on both high-resolution encoder features and low-resolution decoder features. A frozen DINO backbone followed by a single 1×11\times1 linear head therefore falls outside its intended setting. We will revise the statement about boundary ambiguity in Section 2 (line 107 of the original submission) to distinguish our downstream comparison from evaluation in the native frozen-feature setting.

7. Role of DiffUp+ and differences across tasks (W7)

The gate in DiffUp+ gives the differential correction a location-dependent amplitude. We agree that predictor width and convolution type also change the capacity of the kernel generator. We will revise the reference to “regularization” to make clear what changes in DiffUp+: the model gains both local amplitude modulation and additional predictor capacity. The results for lightweight DiffUp already support the main conclusions of the paper. DiffUp+ is an optional extension.

8. Depth metrics and boundary quality (W8)

The reviewer correctly notes that δ1\delta_1 measures prediction accuracy over all valid depth pixels. We will revise the wording in Section 5.3 and describe Table 4 as evidence of depth prediction accuracy. The direct quantitative evidence concerning boundary quality comes from segmentation bIoU and the matting evaluation. The depth results provide complementary evidence of applicability to another dense prediction task.

9. VAE latent upsampling and the role of a refiner (Q6)

We thank the reviewer for suggesting this setting. Signed reassembly can produce responses outside the range of local convex combinations, which makes VAE latent upsampling an interesting application. We have not yet run VAE experiments. Reconstruction quality and the role of a refiner also depend on the input distribution expected by the pretrained decoder. We see this as a direction for future work, specifically to test how differential reassembly affects decoding quality and whether a refiner is needed.

Thank you for the detailed reply and the substantial set of new experiments, which address my main concerns.

FreqFusion and Seed Variance

Thank you for the three-seed results and the FreqFusion comparison. The gains over FADE and FreqFusion are well beyond seed variation, and the revised Introduction and Related Work now position the contribution accurately.

  • The high-pass control collapses toward the nearest-neighbor base and cannot average neighbors, so it does not fully isolate the effect of unconstrained signs. The FreqFusion comparison and Appendix C are the stronger evidence.

Component Ablations

Thank you. The ablation shows that the ℓ1\ell_1 normalization and amplitude factor matter, while removing LayerNorm or mean centering has little effect.

Learned Kernel Analysis

Thank you for Appendix C. The placement-permutation test resolves my concern about relying on selected examples.

Efficiency

Thank you for the measurements. The overhead is modest and clearly reported.

Frozen Features

Thank you; I agree that a frozen backbone with a linear head falls outside DiffUp's intended setting.

DiffUp+ and Depth

Thank you for revising Sections 5.2 and 5.3.

VAE Latents

Thank you. I agree this is a reasonable direction for future work.


Overall, the rebuttal addresses my main concerns, and I am raising my score to 3.

Comments on the paper

1 comment

We thank the reviewers for their time and constructive feedback. We have updated Table 2 with multi-run results for all SegFormer-B1 comparisons on COCONut, including FeatUp, LoftUp, and NAF. The updated results remain consistent with our conclusions. We appreciate the reviewers’ suggestions, which have helped us strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.