- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors observe that although existing upsampling algorithms are capable of preserving details to a relatively rich extent, they struggle to delineate boundaries clearly. Through their investigation, the authors find that the softmax kernel normalization method widely adopted in practice suffers from a boundary suppression phenomenon. To address this issue, they propose a novel feature upsampling algorithm, DiffUp, which employs differential kernels to enhance detailed features. Extensive downstream tasks demonstrate the effectiveness of the DiffUp module.
Critical strengths
(1) The authors provide a detailed and effective analysis of the problems confronting softmax normalization and the advantages of differential feature upsampling, which can relatively thoroughly substantiate the issue raised by the authors.
(2) The authors conduct sufficient comparative experiments on multiple datasets and across various tasks, demonstrating strong experimental completeness.
Critical weaknesses
(1) The authors' theoretical analysis relies on rather strong idealized assumptions, such as binary boundaries and constant feature values within local regions. Can the authors quantify the proportion of cases in which these assumptions actually hold in real network features? Meanwhile, in Theorem 1, the authors interpret softmax positive-weight aggregation as the root cause of boundary suppression; however, if the softmax weights are highly concentrated on the same side of the semantics, boundary blurring may be very weak. The authors need to further clarify which dynamic upsamplers and which boundary scenarios this theorem specifically applies to.
(2) Many of the experimental improvements reported in the paper are rather marginal, such as those on NYUx2 depth estimation and VisDrone detection, whereas the gains on COCONut are pronounced. The authors need to explain why the benefits vary so markedly across different tasks, and whether the model has specific downstream application scenarios. If balanced improvements across different downstream tasks cannot be achieved, is the title of the paper too broad? Moreover, do the various assumptions and derivations presented in the paper still hold?
Questions for authors
Please refer to the content of the Weakness section and provide detailed responses to each of the issues raised.
2 comments
Thank you for your careful and detailed response. Although I am not very familiar with this field, I believe the novelty and workload of this paper have reached the level of ICLR. Good luck!
Response to Reviewer Aut3
We thank the reviewer for assessing the theoretical analysis and experimental coverage. We address the reviewer’s questions about the local assumptions, softmax weights that concentrate on the same side of a boundary, and variation in the gains across tasks. We have also added measurements on learned features and three-seed results for the main segmentation comparisons.
W1. How well can softmax preserve a boundary, and what does differential reassembly add?
We agree that softmax can preserve nearly all of the input contrast if its weights concentrate on the same side of the boundary. Let the two regions have constant feature values a<b, and let α and β be the total softmax weights assigned to the opposite region at the two output locations. The outputs are
a′=a+α(b−a),b′=b−β(b−a).
The ratio of absolute output contrast to input contrast is therefore
b−a∣b′−a′∣=∣1−α−β∣.
When α and β are both close to zero, the ratio approaches one. The attenuation can thus be arbitrarily small, consistent with the analysis in the paper.
This calculation distinguishes the contrast the operator can express from the weights it learns in practice. In the two-region model, softmax can approach the input contrast by concentrating its weights on the same side. Nonnegative, unit-sum reassembly still bounds the attainable contrast by the input contrast. This matters for a weak edge where preserving the existing difference gives limited scope for improvement. Differential reassembly extends the range of local responses and can amplify that edge under an appropriate kernel configuration.
Optimization also matters. Although softmax can express nearly complete same-side concentration, that configuration can be difficult to reach consistently during training because of gradient smoothing. We therefore consider the influence of parameterization and optimization on the reassembly patterns a model learns to be relevant to the observed performance differences.
In the controlled experiment in the original paper, with the kernel generator held fixed, DiffUp improves on softmax reassembly by 2.11 mIoU points and 3.08 bIoU points at a boundary width of 1% of the image diagonal. The visualizations show improvements on thin structures and local edges, along with stronger gradients in the intermediate features. Our new analysis also gives a real example in which the output lies outside the range attainable by any convex combination of the features in the same sampled support. This connects the difference in expressivity to a response actually produced by a trained model.
The two theoretical configurations provide reference cases. Real features contain semantic, texture, and illumination changes of varying strength, and learned kernels can produce enhancement, averaging, and intermediate responses. Appendix C reports the distributions of the learned kernel coefficients across images and decoder stages, together with measurements of feature differences across boundaries and within regions.
W2. Why do gains vary across tasks, and what does this mean for the scope of the analysis and the title?
DiffUp acts directly on local feature differences. The effect on a task metric depends on how much boundary and detail representation limits performance in that setting. The dataset, input features, network configuration, and evaluation metric all affect that relationship.
For example, in our new three-seed SegFormer-B1 results on COCONut, DiffUp gains 0.53 mIoU and 0.81 bIoU points over FADE, with the larger gain on the boundary metric. On VisDrone, DiffUp achieves 27.6 mAP@50:95, close to FADE at 27.7. Segmentation bIoU directly evaluates a band around object contours. Detection AP also depends on classification, box localization, and confidence ranking, while the NYUv2 depth metrics aggregate errors or accuracy over all valid pixels. A change in local boundary representation can therefore affect these metrics to different degrees.
Object scale and structure, the boundary information already present in the input features, and how the prediction head uses that information also vary across settings. In some depth and detection configurations, boundary ambiguity may play a relatively small role in the final metric, which could explain the smaller gains. We will discuss these differences in the context of each task and its input features.
The same differential construction remains competitive across all four dense prediction tasks. Its ability to express enhancement, smoothing, and intermediate responses gives it scope to adapt across tasks. The title Differential Feature Upsampling refers to the local feature correction formed by a zero-sum differential kernel. The construction itself does not require a task-specific feature space. We will consider revising the title to make its focus on detail and boundary enhancement more explicit.