- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This study distinguishes three spatial phenomena: translating the entire scene, moving objects while keeping the background unchanged, and spatial patterns in the average prediction map. Experiments were conducted to test two ADE20K segmentation models by controlling for variations in the training layout, multiple detector distributions, and fine-tuning incorporating translation enhancements. The results reveal a significant directional influence of object position, and this influence varies by category. Even with improved scene translation scores after enhancement, the models still exhibit strong category-dependent spatial patterns.
Critical strengths
Comparing training results on uniform and correlated layouts clearly demonstrates the difference between the two transformations. Combined with explicit treatment of their different symmetries, this comparison shows that equivariance with scene-wide translation does not necessarily imply robustness when objects move independently of the background.
The report includes several useful details: conditional denominators, uncertainty estimation considering inter-class dependencies, disclosure of weak primitive probe responses, and three pairs of fine-tuning seeds. Contour magnitude and alignment contrast also complement the correlation analysis, enabling readers to assess the magnitude and direction of the observed effects.
Critical weaknesses
-
The enhancement analysis used different reference points for the two robustness metrics. Figure 6a assesses the change in scene offset penalty relative to standard fine-tuning, while Figure 6b and the associated contour statistics use the original pre-training checkpoint as a reference. This mismatch prevents us from directly assessing whether translational enhancement alters object placement dependencies (relative to other aspects of the matching training process). If standard fine-tuning also modifies these contours, then a high degree of consistency with the pre-training contours does not preclude significant enhancement effects. Therefore, the existing evidence suggests that better scene offset performance can be accompanied by persistent placement bias, but does not isolate the relative impact of enhancements on each outcome. Placement contrast, contour magnitude, and placement failure rate should be directly compared between paired standard and enhanced fine-tuning runs.
-
The proposed cross-class offsetting evidence does not use contrast, which is used to measure class-specific orientation. Equation (1) and Figure 2 define positional bias by the difference between top and bottom IoU, while the near-zero merged results in Section 4 compare the average IoU away from the center position with the IoU at the center position. A zero value for the second quantity does not imply an opposite preference between different classes. Consider a scenario where each category has an IoU of 0.8 at the top, 0.2 at the bottom, and 0.5 at the center: all categories share a vertical contrast of 0.6, but the average of the IoUs at the two off-center locations yields a zero relative difference at the center. Therefore, the pooled statistic cannot determine whether the cancellation occurs between different categories or merely between different locations within each object. Figure 2 supports the existence of differences in directional preference between categories, but quantifying cancellation requires using a uniform contrast throughout the analysis. Authors should use the same sample and weights to compare the signed mean of the vertical contrast for each category with a magnitude measure such as the mean absolute value.
Questions for authors
-
Have the clean-state and perturbation comparisons of zero-padded and circularly padded CNNs been replicated on independently trained models? If not, should the conclusions regarding their similar clean-state penalties and different perturbation responses be limited to the specific training run evaluated in this paper?
-
What are the effects of contour magnitude and test layout alignment after standard fine-tuning? When these measurements are directly paired with the results of translation-enhanced fine-tuning, does this enhancement weaken position dependence, remain essentially unchanged, or strengthen it?
2 comments
Response to Reviewer 9f9S (2/2): cancellation and replication (W2, Q1)
Pairs of values are DeepLabV3+ then SegFormer. Brackets are 95% intervals. Numbering refers to the revised PDF.
W2: cancellation on one contrast. Your counterexample is right: the off-centre-minus-centre average can vanish with no cancellation between classes. We replaced it with the comparison you proposed, on class means of the vertical contrast, with one sample and one set of weights (Section 4, Table 13, Figure 13). For the 20 classes with at least eight sources, weighted by source count:
| DeepLabV3+ | SegFormer | |
|---|---|---|
| Signed mean S | −0.010 [−0.048, 0.027] | −0.025 [−0.060, 0.011] |
| Magnitude M (weighted mean of absolute class means) | 0.062 | 0.065 |
| Horizontal floor M_h | 0.023 | 0.020 |
| M − M_h | 0.039 [0.023, 0.058] | 0.045 [0.025, 0.062] |
| Cancellation index K = 1 − abs(S) / M | 0.84 | 0.61 |
| Classes preferring top / bottom | 9 / 11 | 7 / 13 |
The class-level magnitude is about three times the horizontal floor, and the signed mean is indistinguishable from zero: opposite class directions cancel in the pooled mean. A within-class sign-flip test of M gives p = 0.0001 in both models (M averages 0.033 under that null). The result holds in all eight configurations per model (two class sets, two weightings, all 600 sources or the 400 test sources): every interval for S includes zero and every interval for M − M_h excludes it. Cancellation is less complete in SegFormer, whose pooled mean leans towards the bottom. The earlier statistic now appears only in Appendix F.
Q1: independently trained models. The replication is in progress. A second independently trained seed of the four CNN configurations is under way (three are trained, and the fourth is part-way through training), and we hope to post its clean penalties and its response to corruption here before the discussion period ends. Until it arrives, we agree that the two between-run comparisons should be limited to the evaluated runs, and the revision does so: Section 8 states that the near-equality of the two CNNs' clean penalties (0.156 with zero padding, 0.155 with circular padding) and their divergence under corruption are properties of single runs, and Section 5 that its intervals do not estimate seed variation. Further seeds, with a formal equivalence test at a margin fixed in advance at 0.03 IoU, are planned for the camera-ready version.
The main conclusions of Section 5 do not rest on comparing the two CNNs. They come from interventions within the same checkpoints, with intervals over 240 foreground instances.
- The equivariant CNN has a penalty of 0.155 [0.140, 0.172] relative to its own uniform-trained control.
- That penalty depends on the wraparound seam. With layouts reversed by reflection, which gives 0.160 for the same penalty, removing the seam from natural backgrounds leaves 0.038 [0.029, 0.046]. A seam on a layout-free texture produces 0.199 [0.178, 0.220], and removing that seam leaves −0.007 [−0.015, 0.001]. On natural backgrounds the zero-padded CNN keeps 97% of its penalty under the same manipulation (Figure 5, Table 20). The criteria were fixed before the test.
The seam also offers an account of the different responses to corruption that you ask about. A seam is a phase-coherent feature of the image, and phase replacement weakens it: its strength in the test images falls from 5.7 on clean images to 3.1, 2.0 and 1.0 at q = 0.25, 0.5 and 1 (Table 21), whereas a zero-padded border lies outside the image. This is consistent with the faster decline of the circular CNN's penalty. The new seed will show whether the divergence replicates, and we will repeat the seam test on it.
We hope these analyses resolve the two concerns behind your assessment of soundness, and we are glad to clarify anything further.
Response to Reviewer 9f9S (1/2): the paired comparison (W1, Q2)
Pairs of values are DeepLabV3+ then SegFormer. Brackets are 95% intervals. Numbering refers to the revised PDF.
Thank you for two precise and constructive criticisms. We agree with both, and we redid both analyses exactly as you specified. Both conclusions hold, and the evidence for them is now stronger.
W1: one reference for both metrics. Every comparison is now between an augmented fit and the standard fit with the same seed, for all three seed pairs (Section 7, Figure 6, Tables 9–11). The pretrained-referenced figures of the submission are replaced.
T is the relative reduction in profile magnitude divided by the relative reduction in scene-shift penalty: 1 if placement dependence fell as much as the shift penalty, 0 if it did not fall. In every seed pair T is at most 0.02 and the profile correlation is at least 0.951.
Q2: weakened, unchanged or strengthened? Essentially unchanged. The scene-shift penalty falls in every seed pair, by 38–55% on average, and no placement statistic changes by an amount distinguishable from zero: the profile magnitude is 3% and 4% higher and the mean aligned contrast 8% and 9% lower, each with an interval that includes zero.
Your point about the reference mattered for a second reason. Standard finetuning alone moves the profiles: their magnitude goes from 0.082 to 0.074 and from 0.081 to 0.065, their correlation with the pretrained profiles is 0.94–0.97, the mean aligned contrast goes from 0.045 in both pretrained models to 0.042 and 0.038, and agreement with the training layout from 71.6% and 69.2% to 70.4% and 68.4%. Only the paired comparison isolates the augmentation. Figure 6b now plots augmented against standard-finetuned profiles, and Figure 12 shows each seed pair.
Two additions that sharpen the result.