acceptodds
Under review as a conference paper at ICLR 2027

SAME OBJECT, DIFFERENT VERDICT: CLASS-SPECIFIC LOCATION PRIORS IN SEGMENTATION

Abstract

Moving an object can change a segmentation model’s verdict even when its pixels are unchanged. We study these class-specific location preferences through object placements, controlled training, and label-free prediction maps. On 400 reserved ADE20K sources, 65–67% of objects that reach IoU ≥ 0.5 somewhere fall below it at another placement in the same background. The effect’s direction agrees with the class’s training layout for 69–72% of objects with nonzero vertical contrast. Correlated training induces comparable placement penalties in zero-padded and exactly circularly equivariant CNNs: joint translation equivariance does not prevent learning object–background relationships. Architecture changes the preference’s persistence under corruption and its visibility in average prediction maps. Finally, translation augmentation reduces whole-scene shift penalties by 55% and 38% in DeepLabV3+ and SegFormer, while largely preserving their class-specific placement profiles. A better scene-shift score therefore does not certify object-placement robustness. Evaluating spatial reliability requires both, alongside the class-directed effects that pooled averages conceal.

Primary area:
interpretability and explainable AI
Keywords:
semantic segmentation, spatial priors, positional bias, visual robustness, distribution shift, translation equivariance, causal interventions, model diagnostics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9f9SOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This study distinguishes three spatial phenomena: translating the entire scene, moving objects while keeping the background unchanged, and spatial patterns in the average prediction map. Experiments were conducted to test two ADE20K segmentation models by controlling for variations in the training layout, multiple detector distributions, and fine-tuning incorporating translation enhancements. The results reveal a significant directional influence of object position, and this influence varies by category. Even with improved scene translation scores after enhancement, the models still exhibit strong category-dependent spatial patterns.

Critical strengths

Comparing training results on uniform and correlated layouts clearly demonstrates the difference between the two transformations. Combined with explicit treatment of their different symmetries, this comparison shows that equivariance with scene-wide translation does not necessarily imply robustness when objects move independently of the background.

The report includes several useful details: conditional denominators, uncertainty estimation considering inter-class dependencies, disclosure of weak primitive probe responses, and three pairs of fine-tuning seeds. Contour magnitude and alignment contrast also complement the correlation analysis, enabling readers to assess the magnitude and direction of the observed effects.

Critical weaknesses
  1. The enhancement analysis used different reference points for the two robustness metrics. Figure 6a assesses the change in scene offset penalty relative to standard fine-tuning, while Figure 6b and the associated contour statistics use the original pre-training checkpoint as a reference. This mismatch prevents us from directly assessing whether translational enhancement alters object placement dependencies (relative to other aspects of the matching training process). If standard fine-tuning also modifies these contours, then a high degree of consistency with the pre-training contours does not preclude significant enhancement effects. Therefore, the existing evidence suggests that better scene offset performance can be accompanied by persistent placement bias, but does not isolate the relative impact of enhancements on each outcome. Placement contrast, contour magnitude, and placement failure rate should be directly compared between paired standard and enhanced fine-tuning runs.

  2. The proposed cross-class offsetting evidence does not use contrast, which is used to measure class-specific orientation. Equation (1) and Figure 2 define positional bias by the difference between top and bottom IoU, while the near-zero merged results in Section 4 compare the average IoU away from the center position with the IoU at the center position. A zero value for the second quantity does not imply an opposite preference between different classes. Consider a scenario where each category has an IoU of 0.8 at the top, 0.2 at the bottom, and 0.5 at the center: all categories share a vertical contrast of 0.6, but the average of the IoUs at the two off-center locations yields a zero relative difference at the center. Therefore, the pooled statistic cannot determine whether the cancellation occurs between different categories or merely between different locations within each object. Figure 2 supports the existence of differences in directional preference between categories, but quantifying cancellation requires using a uniform contrast throughout the analysis. Authors should use the same sample and weights to compare the signed mean of the vertical contrast for each category with a magnitude measure such as the mean absolute value.

Questions for authors
  1. Have the clean-state and perturbation comparisons of zero-padded and circularly padded CNNs been replicated on independently trained models? If not, should the conclusions regarding their similar clean-state penalties and different perturbation responses be limited to the specific training run evaluated in this paper?

  2. What are the effects of contour magnitude and test layout alignment after standard fine-tuning? When these measurements are directly paired with the results of translation-enhanced fine-tuning, does this enhancement weaken position dependence, remain essentially unchanged, or strengthen it?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9f9S (1/2): the paired comparison (W1, Q2)

Pairs of values are DeepLabV3+ then SegFormer. Brackets are 95% intervals. Numbering refers to the revised PDF.

Thank you for two precise and constructive criticisms. We agree with both, and we redid both analyses exactly as you specified. Both conclusions hold, and the evidence for them is now stronger.

W1: one reference for both metrics. Every comparison is now between an augmented fit and the standard fit with the same seed, for all three seed pairs (Section 7, Figure 6, Tables 9–11). The pretrained-referenced figures of the submission are replaced.

Mean of three seed pairs, standard → augmentedDeepLabV3+SegFormer
Scene-shift penalty0.0517 → 0.0230 (−55.5%)0.0364 → 0.0227 (−37.5%)
RMS magnitude of class profiles0.0740 → 0.0760 (+2.7%)0.0647 → 0.0671 (+3.6%)
Change in magnitude, per seed pair+11.0%, −1.4%, −0.4%+3.8%, +4.0%, +3.1%
Transfer ratio T−0.05 [−0.28, 0.24]−0.10 [−0.67, 0.19]
Profile correlation, augmented vs standard, per pair0.955, 0.967, 0.9720.956, 0.963, 0.951
Mean aligned contrast of test objects0.0420 → 0.03860.0384 → 0.0351
Difference−0.0035 [−0.0087, 0.0019]−0.0033 [−0.0070, 0.0007]
Agreement with the training layout70.4% → 67.3%68.4% → 69.6%
Five-placement flip rate, difference in points+0.8 [−3.4, 5.4]−1.3 [−4.0, 2.7]

T is the relative reduction in profile magnitude divided by the relative reduction in scene-shift penalty: 1 if placement dependence fell as much as the shift penalty, 0 if it did not fall. In every seed pair T is at most 0.02 and the profile correlation is at least 0.951.

Q2: weakened, unchanged or strengthened? Essentially unchanged. The scene-shift penalty falls in every seed pair, by 38–55% on average, and no placement statistic changes by an amount distinguishable from zero: the profile magnitude is 3% and 4% higher and the mean aligned contrast 8% and 9% lower, each with an interval that includes zero.

Your point about the reference mattered for a second reason. Standard finetuning alone moves the profiles: their magnitude goes from 0.082 to 0.074 and from 0.081 to 0.065, their correlation with the pretrained profiles is 0.94–0.97, the mean aligned contrast goes from 0.045 in both pretrained models to 0.042 and 0.038, and agreement with the training layout from 71.6% and 69.2% to 70.4% and 68.4%. Only the paired comparison isolates the augmentation. Figure 6b now plots augmented against standard-finetuned profiles, and Figure 12 shows each seed pair.

Two additions that sharpen the result.

  • What the benchmark rewards. On the 80 scenes whose class stays fully visible under all four shifts, the penalty is at most 0.02 before augmentation (Table 11a). A border-free consistency score, the share of pixels that keep their label between overlapping interior crops displaced by 4 to 32 pixels, does not improve: 0.953 → 0.951 and 0.950 → 0.946, lower at every displacement, for the first seed pair (Table 11b; the other two pairs are still to be evaluated). The gain is therefore in the handling of truncation and zero-filled borders, which this augmentation trains on. Section 7 now states its claim for the scene-shift score: a better score brings neither placement robustness nor border-free consistency.
  • Crops in the original recipes. Both public checkpoints were already trained with random rescaling and cropping for 160,000 iterations. The training-aligned placement effect is what remains after that.
AuthorsOpenReview ↗

Response to Reviewer 9f9S (2/2): cancellation and replication (W2, Q1)

Pairs of values are DeepLabV3+ then SegFormer. Brackets are 95% intervals. Numbering refers to the revised PDF.

W2: cancellation on one contrast. Your counterexample is right: the off-centre-minus-centre average can vanish with no cancellation between classes. We replaced it with the comparison you proposed, on class means of the vertical contrast, with one sample and one set of weights (Section 4, Table 13, Figure 13). For the 20 classes with at least eight sources, weighted by source count:

DeepLabV3+SegFormer
Signed mean S−0.010 [−0.048, 0.027]−0.025 [−0.060, 0.011]
Magnitude M (weighted mean of absolute class means)0.0620.065
Horizontal floor M_h0.0230.020
M − M_h0.039 [0.023, 0.058]0.045 [0.025, 0.062]
Cancellation index K = 1 − abs(S) / M0.840.61
Classes preferring top / bottom9 / 117 / 13

The class-level magnitude is about three times the horizontal floor, and the signed mean is indistinguishable from zero: opposite class directions cancel in the pooled mean. A within-class sign-flip test of M gives p = 0.0001 in both models (M averages 0.033 under that null). The result holds in all eight configurations per model (two class sets, two weightings, all 600 sources or the 400 test sources): every interval for S includes zero and every interval for M − M_h excludes it. Cancellation is less complete in SegFormer, whose pooled mean leans towards the bottom. The earlier statistic now appears only in Appendix F.

Q1: independently trained models. The replication is in progress. A second independently trained seed of the four CNN configurations is under way (three are trained, and the fourth is part-way through training), and we hope to post its clean penalties and its response to corruption here before the discussion period ends. Until it arrives, we agree that the two between-run comparisons should be limited to the evaluated runs, and the revision does so: Section 8 states that the near-equality of the two CNNs' clean penalties (0.156 with zero padding, 0.155 with circular padding) and their divergence under corruption are properties of single runs, and Section 5 that its intervals do not estimate seed variation. Further seeds, with a formal equivalence test at a margin fixed in advance at 0.03 IoU, are planned for the camera-ready version.

The main conclusions of Section 5 do not rest on comparing the two CNNs. They come from interventions within the same checkpoints, with intervals over 240 foreground instances.

  • The equivariant CNN has a penalty of 0.155 [0.140, 0.172] relative to its own uniform-trained control.
  • That penalty depends on the wraparound seam. With layouts reversed by reflection, which gives 0.160 for the same penalty, removing the seam from natural backgrounds leaves 0.038 [0.029, 0.046]. A seam on a layout-free texture produces 0.199 [0.178, 0.220], and removing that seam leaves −0.007 [−0.015, 0.001]. On natural backgrounds the zero-padded CNN keeps 97% of its penalty under the same manipulation (Figure 5, Table 20). The criteria were fixed before the test.

The seam also offers an account of the different responses to corruption that you ask about. A seam is a phase-coherent feature of the image, and phase replacement weakens it: its strength in the test images falls from 5.7 on clean images to 3.1, 2.0 and 1.0 at q = 0.25, 0.5 and 1 (Table 21), whereas a zero-padded border lies outside the image. This is consistent with the faster decline of the circular CNN's penalty. The new seed will show whether the divergence replicates, and we will repeat the seam test on it.

We hope these analyses resolve the two concerns behind your assessment of soundness, and we are glad to clarify anything further.

Reviewer_uetEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates class-specific location preferences in semantic segmentation. Through object relocation, controlled training, and label-free prediction maps, the authors examine how segmentation performance depends on object placement and training-data spatial distributions. The paper also shows that translation equivariance and improved whole-scene shift robustness do not necessarily imply object-placement robustness.

Critical strengths
  1. Systematic experimental analysis. The paper connects object-placement evaluations, training-layout interventions, and spatial prediction diagnostics in a coherent experimental framework.
  2. Useful controlled comparisons. The correlated-versus-uniform training experiments and translation-augmentation study provide empirical evidence distinguishing whole-scene translation from object-placement sensitivity.
Critical weaknesses
  1. Limited novelty and incremental contribution. Object-placement sensitivity, contextual dependence, and position encoding have been extensively studied, including in Rosenfeld et al. (2018), Shetty et al. (2019), and Islam et al. (2020), which the paper already cites. Another relevant study is Modeling Visual Context is Key to Augmenting Object Detection Datasets (ECCV 2018), which demonstrates the importance of context-consistent object placement. The distinction between joint-translation equivariance and object-placement robustness also follows directly from their definitions. While the paper provides additional empirical evidence, its conceptual contribution beyond prior work remains insufficiently differentiated. The work offers limited new mechanistic insight into deployed models or actionable diagnostic value.
  2. Insufficient isolation of location effects. Relocating an object changes its surrounding context and may introduce compositing artifacts. The experiments therefore do not clearly separate absolute-location preferences from contextual dependence. The controlled training results demonstrate that imposed spatial correlations can induce placement sensitivity, but do not establish the same explanation for deployed models. The reported conditional flip rates also lack a suitable placement-control baseline.
  3. Limited experimental generality and diagnostic validation. The main experiments use only DeepLabV3+ and SegFormer-B2. The controlled architecture study relies on a single training seed, while the no-position transformer has very low segmentation accuracy. The label-free prediction maps are sensitive to the probe distribution, and their ability to predict actual placement failures remains unclear.
Questions for authors

Novelty. What is the main new insight beyond prior work on object transplantation, contextual dependence, and position encoding? How does the study differ from Dvornik et al.'s work on context-aware object placement?

Location versus context. Can the authors separate absolute-location effects from changes in surrounding context? Would controls using altered backgrounds or backgrounds without strong spatial structure change the conclusions?

Placement controls. How do the reported vertical effects compare with horizontal placement differences? How sensitive are the recognition and flip rates to compositing artifacts or boundary blending?

Statistical robustness. Are the controlled training results stable across multiple random seeds? Do the class-alignment findings remain significant after excluding the most frequent classes?

Generality and diagnostic utility. Do the findings hold for more recent segmentation architectures? Can the proposed label-free maps predict object-placement failures on natural images?

4 comments
AuthorsOpenReview ↗

Response to Reviewer uetE (1/3): what is new (W1, Q1)

Pairs of values are DeepLabV3+ then SegFormer. Numbering refers to the revised PDF.

Thank you for the detailed review. Your questions shaped most of the new controls. Every control you proposed came out in support of the paper's claims, and your request for mechanistic insight led to a new result. We answer in three parts.

The contribution. We agree that the distinction between joint-translation equivariance and placement robustness follows from the definitions. The paper's contribution is empirical and mechanistic: what trained segmenters actually do, and why the standard remedies do not reach it. The introduction is rewritten around this, and the new Table 1 sets the contributions against prior work.

  • Beyond transplanting and context studies (Rosenfeld et al.; Shetty et al.). These show that a transplanted object can disrupt a detector and that removing context objects changes predictions. They do not measure which direction of relocation hurts which class, or where that direction comes from. We do, in segmenters. The direction follows each class's training layout (69–72% of objects; p = 0.0001 and 0.0010). It appears on the vertical axis and not on the horizontal control axis. It is tied to the image frame and not to the scene, which your Q2 allowed us to test.
  • Beyond work on position encoding (Islam et al.; Kayhan & van Gemert; Alsallakh et al.). That work locates absolute position in padding and proposes boundary handling or equivariant design as the remedy. Kayhan & van Gemert report that circular padding prevents the exploitation of location, in a test with uniform backgrounds. Islam et al. (2021b) find circular padding often the second most position-informative type and conjecture that the value transition at the wrapped border is responsible. We test that conjecture by intervention and show its consequence: an exactly equivariant CNN learns a class–location association as strongly as a zero-padded one, and removing the seam removes most of its penalty on natural backgrounds and all of it on layout-free textures (Section 5, Figure 5).
  • Dvornik et al. show that pasting objects without regard to context hurts detector training, and they learn where to paste. That is a result about a training recipe. It does not measure what a trained model has learned about placement, or whether equivariance or shift augmentation removes it. The two studies are complementary, and Section 2 now discusses their paper. Our background control also adds a distinction that their setting does not address: context changes the outcome for individual objects, but the class-directed part of the effect is tied to position in the frame.

Mechanistic insight. The seam result is a mechanism, established by intervention with criteria fixed in advance. Circular padding makes the opposite borders of an image neighbours, and in any non-periodic image that creates a discontinuity along the frame. The equivariant CNN uses it as a position reference: its penalty falls from 0.160 to 0.038 when the seam is removed from natural backgrounds, appears at 0.199 on a layout-free texture that has a seam, and vanishes (−0.007) when that seam is removed. The zero-padded CNN and the absolute-position transformer keep their penalties. Equivariance therefore removes absolute position only for periodic inputs.

Diagnostic value. For deployed models the revision adds a diagnostic that needs no model at all. Training annotations alone identify the better half of the image for 38 of 49 and 37 of 48 objects whose verdict flips vertically. Section 8 turns the results into recommendations: report class-directed placement contrasts with a horizontal control next to shift scores, use label statistics to flag the classes at risk when framing changes, and act on where classes appear relative to the frame, not on equivariance.

AuthorsOpenReview ↗

Response to Reviewer uetE (2/3): location, context and placement controls (W2, Q2, Q3)

Pairs of values are DeepLabV3+ then SegFormer. Brackets are 95% intervals. Numbering refers to the revised PDF.

Location versus context (W2, Q2). We ran the controls you proposed on the 400 test sources (Section 4, Figure 3b, Tables 16–17).

  • Background turned upside down, object unchanged. This makes position in the frame and scene layout disagree. The training-aligned effect is unchanged: 0.045 to 0.041 in DeepLabV3+ and 0.045 to 0.046 in SegFormer. Splitting each object's contrast into a part that follows the frame and a part that follows the scene gives 0.043 [0.025, 0.063] and 0.045 [0.025, 0.066] for the frame, and 0.002 [−0.008, 0.014] and −0.001 [−0.014, 0.014] for the scene.
  • Backgrounds without spatial structure. On phase-randomised and mean-colour backgrounds the effect keeps its direction with no scene at all, at 26–44% of its size, and it is significant in three of the four cases. Objects are much easier to segment there. One reading, which we do not test, is that the prior matters most when the object is hard to segment.
  • What context does. Context matters for individual objects: the unsigned scene component is almost as large as the frame component. It carries no class direction.

So the conclusions hold and are sharper: the part of the placement effect that tracks the training layout is a prior on position in the image frame.

Placement-control baseline (W2, Q3). Among objects recognisable at a corner placement, a vertical move flips the verdict for 61.3% and 62.3%, and a horizontal move for 41.3% and 40.3%, although horizontal displacements are larger. The paired difference is 20.0 [11.4, 34.9] and 22.1 [15.0, 37.5] points (Figure 3a, Table 12). The horizontal analogue of our alignment test is at chance (50.0%, p = 0.56; 44.7%, p = 0.76). The horizontal axis thus measures generic sensitivity to moving a pasted object, about 40%, and the vertical excess is the class-directed effect. The revision reports the two components separately, as your comment on the conditional flip rates suggested.

Compositing artefacts and blending (Q3). We compare four composites: the hard paste, feathered boundaries with σ = 1.5 and 3 pixels, and Poisson blending (Table 18, Appendix F.6). In that order, softer compositing makes objects harder to recognise: 85, 71, 61 and 15 recognisable objects for DeepLabV3+, and 82, 72, 60 and 18 for SegFormer. The directional result is unaffected. Agreement with the training layout is 71.6%, 76.2%, 77.3% and 73.5% for DeepLabV3+, and 69.2%, 71.4%, 73.8% and 75.0% for SegFormer, all significant. Flip rates do not fall either: 64.7%, 66.2%, 77.0% and 93.3%, and 67.1%, 70.8%, 75.0% and 77.8%. Vertical moves flip more verdicts than horizontal moves under every composite. A sharp paste boundary is therefore not what produces the effect.

Deployed models (W2). You note that controlled training does not by itself establish the explanation for deployed models. We agree, and the revision adds three lines of evidence for them: alignment with the training layout, no alignment on the horizontal axis (which flip-augmented training makes uninformative), and anchoring to the frame. A direct intervention is in progress as well: copy-paste finetuning of DeepLabV3+ with random placement, against a control that pastes the same objects at their original height, so that the two arms differ only in the class–height statistics. Both arms are trained, and we hope to post the placement evaluation here before the discussion period ends.

AuthorsOpenReview ↗

Response to Reviewer uetE (3/3): robustness, diagnostics and generality (W3, Q4, Q5)

Pairs of values are DeepLabV3+ then SegFormer. Numbering refers to the revised PDF.

Frequent classes (Q4). Removing the 1, 3, 5 or 10 classes with the most test sources leaves agreement at 73.0–77.8% for DeepLabV3+ (p ≤ 0.0005) and 64.2–71.8% for SegFormer (p ≤ 0.020). With each class counted once, mean per-class agreement is 77.7% and 65.5% (Table 14, Appendix F.3). The map alignment of Section 6 is similarly stable: 59.3–61.2% and 72.1–74.5% after removing up to ten classes (p ≤ 0.0005).

Seeds (W3, Q4). A second seed of the four CNN configurations is in progress, and we hope to post it here before the discussion period ends. Further seeds, including the transformers, are planned for the camera-ready version, and the revision states which comparisons are single-run results (Sections 5 and 8). The seam result does not depend on comparing runs: it is an intervention within the trained checkpoints, with intervals over 240 foreground instances.

Transformer without positional embeddings (W3). We agree. At an IoU of 0.051 its zero penalty cannot be told apart from a floor effect, and the revision presents it as a floor-level check of the symmetry argument, not as an architecture comparison. We also report relative penalties: the absolute-embedding transformer loses 60% of its favoured-layout IoU under reversal, against 33% and 31% for the two CNNs.

Do the maps predict placement failures? (W3, Q5) We tested this at class level and at object level, on pasted objects in natural backgrounds (Section 6, Table 7). The maps carry the right direction in every analysis. The class-level rank correlation is 0.21 (p = 0.13) and 0.53 (p = 0.014), and the maps identify the better half for 65.3% (p = 0.023) and 63.6% (p = 0.14) of the flipping objects whose class has a usable map. They also track the frame-anchored component from your Q2 (0.35, p = 0.048; 0.65, p = 0.004). They do not outperform the training-label density, which is right for 77.6% and 81.8% of the same objects. Each model passes one of our two tests fixed in advance, not both. The revision therefore presents the maps as an exploratory, label-free view of class–location structure, with a symmetry null for when they must be flat, and recommends the label statistics as the practical predictor. On sensitivity to the probe distribution, we agree: the paper says that no single noise distribution is a universal readout, and the seam explains one case, since stationary probes are periodic and lack the cue that the circular CNN uses.

Recent architectures (W3, Q5). We will run larger and more recent segmenters (Mask2Former and a recent plain-ViT segmenter) for the camera-ready version. Each needs its own codebase, a verified reproduction of its reported accuracy and, for mask-classification models, a different rule for per-pixel class scores, which we could not complete in the discussion period. Those models may show different effect sizes, and this is noted in Section 8. Most of the principles the paper demonstrates do not depend on the model: how to measure the effect and separate it from generic sensitivity and from context; why pooled scores hide it; that exact equivariance does not prevent it, and by what mechanism; and that a better scene-shift score does not remove it. Its two ingredients are also general: a training set in which classes have characteristic positions, and a model that can locate the image frame. Newer segmenters are trained on the same ADE20K layout, and those we will add use explicit positional encodings, which give more direct access to position than either model we tested. We therefore expect the effect in them, at a size that the camera-ready runs will measure.

We hope the new controls answer your questions, and we are glad to clarify anything further.

Thank you for the detailed rebuttal and new controls. However, I remain concerned that the experiments do not cleanly isolate absolute-location effects, and the mechanistic findings lack generalization to modern segmentation architectures, and the revised paper is difficult to follow (there are not even revision marks here). I will maintain my rating.

Reviewer_eFnfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This is an analysis paper -- it studies how the output of semantic segmentaiton model changes when the position of objects change. The authors study this using four experiments:

  1. Authors paste objects onto different positions in ADE20K dataset. Of the objects that reach IoU ≥ 0.5 at one position, 65–67% fall below 0.5 at a different position.
  2. The authors then do a controlled training where each class gets a preferred quadrant. A zero-padded CNN and an exactly circular-equivariant CNN learn almost the same excess layout penalty. The authors find that joint translation equivariance does not prevent learned object background dependence.
  3. The authors average class preds over phase randomized scenes and stationary noise. A symmetry proposition gives a null result for equivariant models. On deployed models, the map direction agrees with the training layout for 60-75% of classes.
  4. Finetuning with imagea nd label shifts decreases scene shoft penalty.But class specific placement profiles stay the same.
Critical strengths

The paper is an interesting and an informative ablation study paper aim to understand segmentaiton networks. The authors conduct several controlled experiments. The paper also clearly defines its scope, reports statistics carefully.

Critical weaknesses

The main issue is the limited novelty. Many of the findings in the paper is already known in some form to the vision community. Many of these findings make intuitive sense and "context and location dependence in recognition is a well known topic". Some of the experiments seem way too artificially constructed (like the pasting scenes). The model is not trained on such pasting images, so there can be domain shift. For an analysis paper, I would expect more models to be evaluated. The paper only considered DeepLabV3+ R50 and SegFormer-B2 which are fairly old at this point. Modern SOTA models are missing. The paper also does not provide good recommendations on how to improve the model. This might be a valuable insight to the community.

Questions for authors

How can the findings in the paper help train a better model? One of the natural things that come from experiments is trying to see if we can do some augmentation for copy paste. Will that help at scale? Can we get more experiments on SOTA models?

2 comments
AuthorsOpenReview ↗

Response to Reviewer eFnf (1/2): novelty and the pasting test

Pairs of values are DeepLabV3+ then SegFormer. Numbering refers to the revised PDF.

Thank you for the review. We answer your four concerns in two parts.

Novelty. We agree that context and location dependence are well known. The paper's contribution is not that location matters. It is three things that were not known, and the rewritten introduction and the new Table 1 set each against prior work.

  1. Which way, and why. In segmenters, moving an object hurts in a direction that depends on its class and follows where that class appears in the training set. The effect is tied to the image frame and not to scene context: it is unchanged when the background is turned upside down. Opposite class directions largely cancel in pooled metrics, so standard evaluations do not show it.
  2. Equivariance is not the cure. The remedy that prior work proposes for position dependence is boundary handling or equivariant design. An exactly shift-equivariant CNN nevertheless learns a class–location association as strongly as a zero-padded one, and we identify how: it reads the frame from the wraparound seam that circular padding creates in any non-periodic image.
  3. The standard score does not see it. Translation augmentation improves the scene-shift score by 38–55% and leaves the class placement profiles unchanged.

Pasting is artificial, and the models were not trained on pasted images. This is a fair concern, and the revision addresses it in three ways.

  • We quantify the cost. In their original images, 208 and 199 of the 400 components are recognisable. Pasted, 85 and 82 are (Tables 2 and 15). The paper presents the test as a controlled stress test, not as an estimate of ordinary validation error.
  • The design holds the cost constant. Every comparison is between placements of the same object on the same background, so the domain shift of pasting is the same on both sides. It can lower every score. It cannot by itself create a direction that depends on the class and follows that class's training layout.
  • Three controls confirm this. First, turning the background upside down changes the object's surroundings at every position, and the class-directed effect is unchanged (0.045 to 0.041, and 0.045 to 0.046). It follows the frame, not the pasted surroundings. Second, the horizontal axis has the same pasting but no training layout to align with: agreement is 50.0% and 44.7% there, and vertical moves flip about 20 points more verdicts. Third, with feathered and Poisson-blended boundaries, agreement with the training layout is 71–77%, against 69–72% for the hard paste (Table 18).

In addition, the controlled models are trained on composites, so pasting is in-distribution there. The effect appears, and the training layout switches it on and off.

AuthorsOpenReview ↗

Response to Reviewer eFnf (2/2): more models, and how to train a better model

Pairs of values are DeepLabV3+ then SegFormer. Numbering refers to the revised PDF.

More and newer models. We will run larger and more recent segmenters (Mask2Former and a recent plain-ViT segmenter) for the camera-ready version. Each needs its own codebase, a verified reproduction of its reported accuracy and, for mask-classification models, a different rule for per-pixel class scores, which we could not complete in the discussion period. Those models may show different effect sizes, as mentioned in Section 8.

Most of the principles the paper demonstrates do not depend on the model, for two reasons. First, they rest on interventions, not on a survey of models: controlled training creates and removes the prior; the seam test identifies the equivariant model's cue; paired fine-tuning shows that a better scene-shift score does not bring placement robustness; the background controls show that the deployed effect follows the frame. Second, the two ingredients of the effect are general: a training set in which classes have characteristic positions, and a model that can locate the image frame. Newer segmenters are trained on the same ADE20K layout, and those we will add use explicit positional encodings, which give more direct access to position than either model we tested. We therefore expect the effect in them, at a size that the camera-ready runs will measure.

How can the findings help train a better model? The revision adds a paragraph on implications for practice (Section 8). The results support four concrete statements.

  • Evaluate differently. Report class-directed placement contrasts next to whole-image shift scores, with the horizontal control, and test for cancellation. A pooled shift score can look good while a class-directed effect remains.
  • Find the classes at risk without running a model. The direction in which relocation hurts a class is predictable from training annotations alone, for 77–78% of the objects whose verdict flips vertically. Label statistics flag the classes at risk when deployment framing differs from training.
  • Know where the lever is. In the controlled study, models trained with uniform placement show no penalty, in every architecture, and exact equivariance leaves the penalty in place. Augmentations that move the whole image, whether shifts or crops, do not change where each class appears in the frame: translation augmentation left the profiles unchanged, and both public checkpoints were already trained with random crops for 160,000 iterations.
  • Know the trade-off. The prior helps when test layouts match training and hurts when they do not. For the zero-padded CNN: 0.490 IoU on the favoured layout against 0.390 for the uniform-trained model, and 0.330 against 0.386 on the reversed layout. An intervention should be judged on both.

Would copy-paste augmentation help? This is the natural next experiment, and we are running it. We finetuned DeepLabV3+ with copy-paste under two placement policies: a random position, and the object's original height with a random horizontal position. The two arms paste the same objects and differ only in whether each class keeps its height statistics, so their difference isolates the effect of the training layout. It also bears on an open question in the literature: Dvornik et al. find context-free placement harmful for detection, and Ghiasi et al. find random placement sufficient. Both arms are trained, and they reach the same clean validation mIoU: 42.55 with random placement and 42.57 with original height (one seed). The placement evaluation is running, and we hope to post it here before the discussion period ends. This is one model, one seed and a short finetune: it can show whether the lever moves the prior in a deployed model, not what copy-paste does at scale.

We hope the revision answers your concerns, and we are glad to clarify anything further.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General response (1/2): what the paper contributes

We thank the three reviewers for careful and constructive reviews. The reviews describe the experiments as systematic and the statistical reporting as careful, and they ask two things: whether the findings go beyond prior work, and whether specific analyses hold up. This comment answers the first question. The next one summarises the new evidence on the second.

The question. Spatial robustness is usually assessed and improved in one way: measure sensitivity to shifts of the whole image, and reduce it with augmentation, anti-aliasing or equivariant design. We ask what that toolkit leaves in place when an object occurs somewhere unusual within a scene.

Three results.

  1. Segmenters carry a class-specific prior on position in the image frame. Moving an object changes its segmentation in a direction that depends on its class and follows where that class appears in the training set. The effect appears on the vertical axis, where classes have training layouts, and not on the horizontal control axis. It is unchanged when the background is turned upside down, so it is tied to the frame and not to scene context. Because classes prefer opposite directions, it largely cancels in pooled metrics, which is why standard evaluations do not show it.
  2. Exact translation equivariance does not remove it, and we show why. An exactly shift-equivariant CNN acquires a placement penalty comparable to that of its zero-padded twin. It reads the frame from the wraparound seam that circular padding creates in any non-periodic image: removing the seam removes most of the penalty on natural backgrounds and all of it on layout-free textures. Equivariance removes absolute position only for periodic inputs, and natural images are not periodic.
  3. The standard scene-shift score neither detects nor certifies it. Translation augmentation improves the scene-shift score by 38–55% and leaves the class placement profiles unchanged. A closer look shows that the score rewards the handling of truncated, zero-filled borders.

How this extends prior work (new Table 1).

EstablishedThis paper adds
Transplanted objects disrupt detectors, and context drives recognition (Rosenfeld et al. 2018; Shetty et al. 2019; Dvornik et al. 2018).In segmenters the failure has a direction for each class. It follows the training layout, it is anchored to the image frame and not to scene context, and it largely cancels in pooled metrics.
CNNs encode absolute position through padding and boundaries; boundary handling or equivariant design is the proposed remedy (Islam et al. 2020; Kayhan & van Gemert 2020; Alsallakh et al. 2021).That remedy is not sufficient. An exactly equivariant CNN learns a comparable placement penalty (0.155 against 0.156), and an intervention identifies its cue: the wraparound seam of non-periodic images.
Shift robustness improves with augmentation, anti-aliasing or equivariant design (Azulay & Weiss 2019; Zhang 2019; Rojas-Gomez et al. 2024).A better scene-shift score does not reduce the dependence on object placement.
Vertical class priors exist and can be exploited for accuracy (Choi et al. 2020).Their cost when an object is elsewhere, and their persistence under standard robustness interventions.

How the claims are established. By matched comparisons in which one thing changes at a time: the same object on the same background at different placements, the same scene with its background turned upside down, training runs that differ only in where classes are placed, test images with and without the seam, finetuning runs paired by seed. Uncertainty is clustered by class or by object instance, and no analysis treats pixels as replicates.

Why it matters.

  • For evaluation. A pooled shift score can look good while a class-directed placement effect remains. The paper gives the contrasts, the horizontal control and the cancellation statistics that expose it.
  • For architecture. Equivariance is not a guarantee against location priors in real images, and the seam explains why.
  • For data and deployment. In controlled training, uniform placement removes the penalty and equivariance does not. In the deployed models, training annotations alone predict the direction in which relocation hurts for 77–78% of the objects whose verdict flips vertically, so label statistics flag the classes at risk when framing changes.
AuthorsOpenReview ↗

General response (2/2): what the revision adds

Pairs of values are DeepLabV3+ then SegFormer. Brackets are 95% intervals. Numbering refers to the revised PDF.

We ran every analysis the reviews asked for that fits in the discussion period. All reuse the paper's data, splits and statistical conventions. Each of the paper's three central claims held under every control.

Question raisedResultWhere
Specific effect, or generic sensitivity to moving a pasted object? (uetE)Among objects recognisable at a corner placement, vertical moves flip the verdict for 61% and 62%, and horizontal moves for 41% and 40%, although horizontal displacements are larger. Difference: 20.0 [11.4, 34.9] and 22.1 [15.0, 37.5] points. On the horizontal axis, alignment with the training layout is at chance.Section 4, Figure 3a
Location or context? (uetE)With the background upside down, the training-aligned effect is unchanged: 0.045 to 0.041, and 0.045 to 0.046.Section 4, Figure 3b
Compositing artefacts? (uetE, eFnf)With feathered or Poisson-blended boundaries, agreement with the training layout is 71–77% (hard paste: 69–72%).Table 18
Frequent classes? (uetE)Alignment holds without the ten classes with the most test sources (77.8%, p = 0.0002; 64.2%, p = 0.020).Table 14
Is cancellation measured on the right statistic? (9f9S)Signed class mean: −0.010 [−0.048, 0.027] and −0.025 [−0.060, 0.011]. Magnitude: 0.062 and 0.065, about three times the horizontal floor.Section 4, Table 13
Is augmentation compared with the right reference? (9f9S)Against matched standard finetuning, the scene-shift penalty falls by 55% and 38% and profile magnitude changes by +3% and +4%. Transfer ratio (1 would be full transfer): −0.05 [−0.28, 0.24] and −0.10 [−0.67, 0.19].Section 7, Figure 6
What does pasting cost? (eFnf)208 and 199 of 400 components are recognisable in their original image, 85 and 82 when pasted. The controls above show that this cost does not produce the class-directed result.Tables 2 and 15
Do the label-free maps predict failures? (uetE)They carry the right direction but add no detectable information beyond training-label statistics. The paper recommends the label statistics as the predictor and presents the maps as exploratory.Section 6, Table 7

Two further results.

  • A mechanism for the equivariant CNN. By intervention, with criteria fixed in advance: removing the wraparound seam cuts the circular CNN's penalty from 0.160 to 0.038 on natural backgrounds; a seam on a layout-free texture produces 0.199, and removing it leaves −0.007. The zero-padded CNN keeps its penalty. This tests a conjecture of Islam et al. (2021b) and replaces the explanation that the submission offered for this model (Section 5, Figure 5).
  • What the scene-shift benchmark rewards. Agreement between predictions on overlapping border-free crops does not improve with augmentation (0.953 → 0.951 and 0.950 → 0.946, first seed pair). The benchmark's gain comes from truncation and zero-filled borders, so the score certifies neither placement robustness nor border-free consistency. This adds to the case for reporting placement contrasts alongside it (Section 7, Table 11).

Still to come. We hope to post three results in this thread before the discussion period ends, time permitting: a second training seed for the controlled CNNs (9f9S, uetE); copy-paste finetuning of DeepLabV3+ with random placement against a height-preserving control, a direct training intervention on a deployed model (eFnf, uetE); and the border-free check for the other two seed pairs. Larger and more recent segmenters, and further seeds, are planned for the camera-ready version. The responses to uetE and eFnf explain why the paper's principles do not depend on them.

What changed in the paper. The experiments, data and reported results of the submission are retained. Two analyses are redone as 9f9S specified. The revision adds the controls above and states the contribution more clearly. By section: the abstract and introduction are rewritten around the contributions, and Section 2 gains Table 1; Section 3 gains a paragraph on controls; Section 4 gains Figure 3 and three paragraphs (horizontal control, frame versus scene, cancellation); Section 5 gains the seam test and Figure 5; Section 6 is shortened, with two figures moved to Appendix D, adding the test of the maps as predictors; Section 7 uses the paired comparison throughout; Section 8 adds implications for practice and limits. Appendices F and G are new.

AuthorsOpenReview ↗

Follow-up comment 1 of 3: second training seed, with the seam test

Update: a second training seed for the controlled CNNs, and the seam test on it (9f9S, uetE)

Brackets are 95% intervals over the 240 held-out foreground instances. Numbering refers to the updated PDF: Appendices H and I are new, and every earlier number is unchanged.

A second, independently trained seed of the four CNN configurations reproduces both comparisons that 9f9S asks about in Q1, the clean penalties of the two CNNs and their response to corruption, and the seam mechanism with them. Exact equivariance again leaves the prior in place, and for the same reason.

The new seed changes the weight initialisation, the data order and the sampled training compositions. The recipe is unchanged, and the new checkpoints are scored on exactly the test images of the first seed, with the two seeds paired composition by composition (Appendix H, Tables 22 and 23). In the table, q is the share of Fourier phase pairs replaced.

Excess layout penalty LCleanq = 0.25q = 0.5
Zero padding, first seed0.156 [0.140, 0.173]0.150 [0.135, 0.166]0.079 [0.070, 0.089]
Zero padding, second seed0.153 [0.133, 0.173]0.126 [0.110, 0.141]0.071 [0.061, 0.081]
Circular padding, first seed0.155 [0.139, 0.172]0.085 [0.072, 0.098]0.024 [0.017, 0.033]
Circular padding, second seed0.143 [0.127, 0.158]0.082 [0.070, 0.094]0.025 [0.017, 0.033]
Zero minus circular, first seed0.001 [−0.013, 0.014]0.066 [0.050, 0.082]0.055 [0.044, 0.066]
Zero minus circular, second seed0.010 [−0.006, 0.026]0.044 [0.029, 0.059]0.046 [0.036, 0.057]
  1. An exactly equivariant CNN again learns a penalty close to that of its zero-padded twin. The clean penalty L is 0.153 with zero padding and 0.143 with circular padding, after 0.156 and 0.155 in the first seed. In each seed the paired difference has a 90% interval inside the equivalence margin of 0.03 set before the second seed was trained: [−0.010, 0.012] in the first seed and [−0.003, 0.023] in the second. Across seeds and reversals the equivariant CNN acquires between three quarters and all of the zero-padded penalty. Its smallest shares are in the second seed under the axis-specific reversals of Appendix G (0.092 against 0.119 vertically, 0.107 against 0.129 horizontally; Table 23). A test across seeds needs more than two, and further seeds are planned for the camera-ready version.
  2. The divergence under corruption replicates. At q = 0.25 and 0.5 the zero-padded CNN keeps the larger penalty in both seeds, and every interval for the difference excludes zero. Relative to favoured-layout IoU, the loss under reversal again rises with corruption for zero padding (31% to 43%; first seed 33% to 47%) and falls for circular padding (29% to 12%; first seed 31% to 9%). Between seeds, only the zero-padded penalty at q = 0.25 changes detectably, by −0.025 [−0.041, −0.008], which narrows the gap there from 0.066 to 0.044.
  3. The seam mechanism replicates. The seam test on the new checkpoints uses the same images and the same three criteria, set before the first test, and all three are met. Removing the seam removes 83% of the circular CNN's horizontal penalty on natural backgrounds and all of its penalty on the layout-free texture, while the zero-padded CNN is unaffected.
First seedSecond seed
Circular CNN, natural backgrounds, diagonal reversal: seam → seam removed0.160 → 0.038 [0.029, 0.046]0.136 → 0.032 [0.024, 0.040]
Circular CNN, natural backgrounds, horizontal reversal: seam → seam removed0.115 → 0.018 [0.011, 0.026]0.107 → 0.018 [0.010, 0.025]
Circular CNN, layout-free texture, diagonal reversal: seam → seam removed0.199 [0.178, 0.220] → −0.007 [−0.015, 0.001]0.203 [0.181, 0.226] → 0.003 [−0.005, 0.010]
Zero-padded CNN, natural backgrounds: share of its penalty kept without the seam97%99%

In the PDF. Section 5 gives the second-seed values and Appendix H the full results, and Section 8 no longer lists the two comparisons as single-run results. The update also repositions two labels in Figure 4 that overlapped the plot, with the plotted values unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.