acceptodds
Under review as a conference paper at ICLR 2027

Same Dice, Different Mistakes: Label Coarsening Relocates Segmentation Errors

14 views

Abstract

Segmentation metrics such as Dice and IoU report how much a model gets wrong, but not what it gets wrong: they discard the identity of the pixels that receive false positives. That is harmless only if false positives are interchangeable; they are not. A multiple sclerosis (MS) lesion detector that flags a benign white-matter hyperintensity makes a different, clinically consequential mistake than one that flags ordinary tissue; a car detector that fires on a truck differs from one that fires on road. Training pipelines erase this distinction by collapsing every non-target structure into one background class. Prior work has shown that keeping such structures separate can raise Dice. What has not been shown is that coarsening changes the composition of a model's errors even when their quantity and Dice are held fixed: when coarse- and fine-labeled models make the same number of mistakes, do they make the same kinds of mistakes? We show they do not, and name the phenomenon semantic error relocation (SER). We introduce Semantic False-Positive Concentration (SFPC), the fraction of target false positives that fall on a pre-specified hidden negative subclass, and a protocol that compares models only after matching their total false-positive burden on validation data, then under an exact equal-error-budget audit in which any SFPC difference is a relocation of fixed error mass. On MS3SEG, in a pre-registered confirmation across three 3D architectures and three seeds, binary supervision places 20.96% of its false positives on normal white-matter hyperintensities versus 13.81% under four-class supervision (+7.15 percentage points, 95% CI 6.04–8.31, 39/40 patients), while total false positives are indistinguishable (1280 vs. 1283 voxels) and binary Dice is slightly higher (0.626 vs. 0.617). This is about one additional benign region falsely flagged per patient, a 266× enrichment over the subclass's area. An independent Cityscapes replication reproduces the effect: +7.24 points more car false positives land on trucks and buses (95% CI 5.09–9.59) at statistically identical Dice (0.865 vs. 0.867). Controlled follow-ups rule out prediction-head size as the cause and trace the shift to the training objective the taxonomy induces: reweighting the hidden class alone reproduces part of the effect, with any further contribution from explicit class identity left open. The practical conclusion is direct: whenever fine labels exist at evaluation time, even if deployment needs only a binary mask, reporting SFPC alongside overlap metrics exposes a difference in what errors mean that Dice cannot see.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
semantic segmentation, label granularity, evaluation metrics, error analysis, hidden stratification, segmentation evaluation, medical image segmentation, semantic error relocation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yamfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates an important limitation of conventional segmentation evaluation: two models can achieve similar Dice scores and make comparable numbers of false-positive predictions, yet distribute those errors across clinically or semantically different regions.

The authors study how collapsing fine-grained negative classes into a single background category during training changes the semantic composition of segmentation errors. They call this phenomenon Semantic Error Relocation (SER).

The paper introduces Semantic False-Positive Concentration (SFPC), defined as the proportion of target false positives assigned to a prespecified hidden negative subclass: \mathrm{SFPC}_H=\frac{\mathrm{FP}H}{\mathrm{FP}{\mathrm{all}}}.

The central contribution is not simply introducing a new segmentation metric, but demonstrating a controlled relationship between training-label granularity and the semantic destination of prediction errors.

The evaluation has several strengths: validation-selected false-positive matching, an exact equal-FP-budget audit, an MS3SEG confirmation study, cross-domain replication on Cityscapes, and controls investigating the role of prediction-head size and loss weighting.

Critical strengths

The empirical question is clear and important, the matched-error design addresses a major confound, the MS3SEG confirmation is strong at the patient level, and the independent Cityscapes replication provides evidence that the phenomenon is not restricted to one medical imaging task. The authors also avoid overclaiming that explicit semantic class identity has been proven to cause the effect.

Critical weaknesses

The principal metric is a simple error-composition proportion closely related to established class-conditional false-positive analyses. The exact-budget audit relies on ground-truth negative locations and is therefore diagnostic rather than deployable. Additionally, the training objectives differ between binary and fine supervision, making it difficult to identify which part of the effect originates from label granularity itself rather than the loss induced by that granularity.

There is also a noteworthy statistical interpretation issue in the mechanism controls: the H-Weighted versus H-Union comparison has a bootstrap confidence interval excluding zero but a nonsignificant Wilcoxon test. The authors explicitly explain this divergence in Appendix G; nevertheless, the interpretation of the resulting evidence could be substantially clearer.

Table 1 presents the SFPC results but does not report FP counts and Dice alongside them. Because the manuscript's central claim is that error composition changes while error quantity remains approximately controlled, these quantities should appear together.

The paper's most important mechanistic finding is that increasing the loss weight on hidden negatives reduces SFPC even without giving them explicit class identities. This substantially refines the paper's interpretation, yet the results appear relatively late.

The paper partly merges two distinct conclusions. The first is well supported: changing the supervision regime changes the semantic concentration of false positives. The second is mechanistic: this relocation arises from changes in loss weighting, class competition, or explicit class identity. The current mechanistic evidence only partially decomposes these factors.

SFPC itself is mathematically simple and closely related to existing class-conditional error statistics: \mathrm{SFPC}_H=\frac{\text{FPs on hidden class }H}{\text{all target FPs}}. Its usefulness is clear, but the metric is not a fundamentally new statistical estimator. The experimental matching framework adds more novelty than the formula. The Bayes-posterior observation in Equation (3) is also straightforward: merging negative labels leaves the target-class event unchanged. Its value is explanatory rather than theoretical novelty.

Questions for authors

Section 4.1; Appendix A. In MS3SEG, the binary models optimize target/non-target Dice plus balanced negative log-likelihood, whereas the fine models use masked four-class cross-entropy plus foreground Dice. Therefore, the experiment changes both the output taxonomy and the objective induced by that taxonomy. The authors recognize this problem and conduct useful follow-up controls on Cityscapes. However, these do not fully establish the same mechanism in MS3SEG.

Section 3.3, Equation (4). The exact-budget experiment ranks scores at ground-truth negative locations and selects the highest-scoring BB locations. This guarantees equal FP counts, but it relies on information unavailable during deployment. The authors correctly label it as a diagnostic rather than a deployable thresholding method. The concern is interpretative: these experiments compare the composition of high-scoring negatives, not the complete deployment-time segmentation predictions.

Sections 5.1 and 5.3. The MS3SEG FP difference is very small: −2.3 voxels, with a confidence interval of [−54.3, 52.8]. Cityscapes is less tightly matched. Its aggregate FP counts are 2,294.7 versus 2,055.1, and one architecture has a 19.7% test-time FP mismatch despite validation matching. The reported p=0.703p=0.703 does not establish equivalence of FP burden.

Section 5.4; Appendix G, Table 4. For H-Weighted Binary minus H-Union at the exact 1× budget, the reported mean SFPC difference is: \Delta=+2.18\text{ pp},\quad 95%\ \mathrm{CI}=[0.80,3.69], \quad p_{\mathrm{Wilcoxon}}=0.694. The bootstrap confidence interval excludes zero, while the Wilcoxon signed-rank test is nonsignificant. This is not necessarily a computational inconsistency. The authors explain that only 52 images have positive paired differences, 68 have negative differences, and 17 are ties. The bootstrap interval measures the paired mean, while the signed-rank test assesses a different aspect of the paired distribution. However, the manuscript's conclusion that the additional class-identity contribution is not reliably supported requires more nuance.

Sections 5.1 and 6; Appendix D. The paper interprets nWMH as a semantically adjacent negative subclass. However, some false positives may arise because nWMH and MS lesions share image intensity, anatomical location, or proximity to lesion boundaries. The existing paired-patient comparisons successfully control for differences in patient anatomy between supervision regimes. They do not, however, identify which properties of nWMH make them susceptible to relocation. The authors already address component size and distance from ventricles. A further analysis would be valuable.

Section 4.1; Appendix I. The authors report that the matched-error endpoint was frozen before it was examined on folds 3–4, while conventional argmax metrics from those folds had already been seen during development. The appendix also explains that some supporting panels became visible during checkpoint recovery. The procedural transparency is commendable, and the large effect in 39 of 40 confirmation patients remains persuasive. Nevertheless, the evidence should not be described as equivalent to a completely untouched, prospectively preregistered external confirmation.

Section 5.1; Appendix D. The manuscript reports approximately one additional falsely detected nWMH component per patient under binary supervision. This is a useful operational interpretation of SFPC. However, component detections are defined using an overlap threshold of at least 10%, and nWMH annotations may be ambiguous in small or periventricular regions. The authors provide relevant robustness checks, but the clinical impact remains hypothetical.

3 comments

Thank you for the exceptionally careful reading, and we really appreciate the time you spent on reviewing our work. References identify highlighted PDF paragraphs/captions; brackets give 95% CIs.

W1 — Metric, audit and objectives. SFPC is descriptive; exact budgets audit ranked negatives. The contribution is the supervision intervention and controlled composition comparison. Corrected MRI methods retain auxiliary-Dice and target-scaling differences (pp. 3–5, R02/R05/R06).

W2 — Bootstrap/Wilcoxon divergence. We retain H-Weighted/H-Union mean +2.18 points [0.80,3.69] and prespecified p=0.694: 52 positive, 68 negative and 17 tied differences, median zero. Mean-bootstrap and signed-rank inference differ (Q4); the new test does not replace the original (p. 8, R11).

W3 — Joint reporting. Tables 1–2 report Dice/FPs/SFPC together. New validation-selected Marginal/Fine values are 0.8661/0.8635 Dice, 2310.1/2209.0 FPs, 22.64%/19.50% SFPC; captions distinguish approximate matching from exact budgets (pp. 7/9, R09/R34).

W4 — Weighting appeared late. The abstract/contributions now foreground it; Figure 3 shows original and new controls. The weighting reduction is 1.48 points [0.70,2.29], Holm p=0.0000593 at exact 1×. Successive contrasts are not independent causal shares (pp. 1/3/8, R01/R02/R37).

W5 — Finding versus mechanism. New Cityscapes holds weights, normalization, target Dice, four active logits and paired training rules fixed. Adding conditional negative-class likelihood lowers exact-budget SFPC by 2.11 points [0.70,3.52], p=0.0054, pooled across six pairs/137 images (p. 8, R12).

MRI controls show a different dependence: target-only Fine SFPC 15.98%, original Binary 13.80%, original Fine 11.04%. The 4.94-point reduction [4.20,5.80] between Fine objectives reflects joint changes to auxiliary Dice and target coefficient, not an isolated identity effect. This one-seed result is explicit in Results and Table 7 (pp. 9/18–19, R36/R25/R32).

W6 — SFPC/Bayes novelty. Neither is claimed as a new estimator or theorem. The controlled supervision–error-composition relationship reveals a selection trade-off: Binary has higher Dice yet greater nWMH-component overlap (pp. 2–3, R35/R03).

Q1 — MRI losses/transfer. Both core branches use masked unweighted CE plus foreground Dice; “balanced NLL” was a description error. CE averages valid batch voxels. Dice averages patch/class losses with smoothing 10−610^{-6}, excluding background/invalid voxels and retaining absent classes. Top-level coefficients are 1. Binary uses target Dice; Fine averages ventricle/nWMH/target, giving target coefficient 1/3 (p. 13, R19). Appendix I records provenance limits. Table 7's patient-level CSVs/reproduction code support W5's MRI-specific joint-Dice interpretation; Cityscapes transfer is not assumed (pp. 17/19, R21/R33).

Q2 — Exact-budget interpretation. The audit requires ground-truth negatives and target size. It fixes FP counts only, not Dice or sensitivity, and yields no deployable threshold. Frozen validation-selected predictions provide separate operating-point evidence (pp. 4/9, R05/R34).

Q3 — Matching versus equivalence. We do not interpret p=0.703 as equivalence. MRI FP difference is −2.3 [−54.3,52.8], with means 1280.3/1282.6. Original Cityscapes means are 2294.7/2055.1, difference +239.6 [−96.3,729.2], including 19.7% mismatch in one architecture. Exact budgets provide exact count control; no post-hoc equivalence claim is made (pp. 7/10, R09/R17).

Q4 — Statistical interpretation. The interval supports a mean difference; p=0.694 gives no signed-rank support for a location shift. Positive/negative differences average +8.37/−2.01 points. Neither “no effect” nor uniform benefit follows. New models test a different intervention on the same images, not preregistered independent replication or evidence for every pair (pp. 8/18, R11/R23).

Q5 — Appearance/anatomy. Pairing controls patient anatomy across regimes. The original large/non-periventricular analysis remains positive: +15.55 points in nWMH false-positive rate, p=0.0032, among 21 eligible patients. This does not isolate intensity or distance to an abWMH boundary as the cause (pp. 6/19, R39/R26).

Q6 — Confirmation provenance. We use “internally frozen endpoint,” disclose prior argmax/recovery-output visibility, and label full OOF as overlapping robustness evidence (p. 10, R16).

Q7 — Components/practical consequences. The +1.06 count uses annotated components of at least five voxels and at least 10% prediction overlap. Abstract/Results consistently say “overlap”; clinical harm was not measured. The cost equation retains application-dependent preference and sensitivity. MS review, radiotherapy and robot navigation are distinguished from proven downstream outcomes (pp. 1/9–10, R01/R15/R27/R28).

We appreciate your detailed guidance and reconsideration.

Thank you for your detailed response, I am happy to keep my rate as 3.

Thank you very much for reviewing our response and revisions. We appreciate your continued positive assessment and feedback. If any questions or concerns remain about the revised manuscript, evidence, or its interpretation, we would be more than happy to address them.

Reviewer_DaWvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how label coarsening affects the distribution of false positives in semantic segmentation models. To this end, the authors distinguish between semantically related hard negatives and other background classes. They then train and compare models with fine-grain labels and coarser labels that merge hard negatives with other background classes. In addition, the authors introduce Semantic False-Positive Concentration (SFPC) to analyze the distribution of false positives on the predefined hard negative cases. They show that the distribution of false positives shifts between models trained on fine- and coarse-labelled when controlled for segmentation accuracy or number of false positives.

Critical strengths
  • Interesting phenomenon: shows the importance of identifying semantically related hard negatives and using the class for training.
  • Careful experimental design: evaluation across two datasets, including a matched false-positive count and a matched segmentation score setting.
  • Highlights that aggregate scores can hide the distribution shift of semantic errors.
Critical weaknesses
  • Limited novelty: The work builds upon BackSplit, which already established that subdivisions of the background classes can impact the segmentation performance. This paper shows that the subdivision can shift the semantic errors, but the authors provide limited explanation for this observation.

  • Unclear practical relevance: If fine-grained labels are available, it is unclear why a practitioners should abandon them during training. If they are not, it is unclear how these should be obtained during evaluation. The paper does not describe a practical scenario or an implication for which their empirical observation becomes relevant.

  • Change in training objective: The comparison between coarse and fine-labelled classes did not only change the subdivision, but also the training objective (Dice + NLL for coarse, 4-class entropy + foreground Dice for fine). Furthermore, the experiments show that label weighting explains part of the effect, but leaves open how other effects (class distribution of selected classes, etc.) are affecting the outcome. Therefore, it remains unclear whether the observed effect solely comes from the class division.

Questions for authors
  • In which practical settings would coarse supervision be preferred over fine-grained if the labels are available?

  • Could the authors clarify why different loss formulations are used in the compared settings and how this impact the analysis?

  • What explains the semantic shift of false positives after coarsening, given that the Bayes-optimal target posterior keeps unchanged? Can the authors provide further evidence of the underlying mechanism (e.g. changes in loss, gradient, ...)?

As the influence of label subdivision is not fully controlled, the change of loss is potentially a confounding factor, a concrete mechanism that explains the observation is not clear, and practical relevance remains uncertain, I am currently leaning towards rejecting the paper.

2 comments

We really appreciate the time you spent on reviewing our work. Your concern prompted the MRI code audit and a new 12-model Cityscapes control. References identify highlighted PDF paragraphs/captions; brackets give 95% CIs.

W1 — Novelty beyond BackSplit. Binary has higher MRI Dice by 0.0092 [0.0041,0.0144], yet overlaps 1.06 more annotated nWMH components per patient at nearly equal FP counts. BackSplit studies background subdivision; we examine where residual FPs land after controlling their number. Original exact-1× gaps are +2.73 MRI and +4.30 Cityscapes points. Related work distinguishes this selection trade-off from subdivision and semantic-error analysis; the SFPC ratio is not claimed as a new estimator (pp. 2–3, R35/R03).

W2 — Practical relevance. A legacy target-only detector can be audited on a smaller, finely annotated cohort. Fine labels are needed for that audit, not throughout the original training set. This reveals subclass errors hidden by overlap scores and supports targeted review or selection under application-specific costs. MS lesion review is the immediate example; radiotherapy contour errors on critical organs and robot false-traversable predictions are proposed applications requiring downstream validation (pp. 9–10, R14/R15/R27/R28).

W3 — Objective and class-distribution confounding. The new Cityscapes comparison uses the same images, pixel-weight map, normalization, target Dice, active outputs and paired training rules. Adding conditional negative-class likelihood lowers exact-budget SFPC by 2.11 points [0.70,3.52], p=0.0054, pooled across six pairs and 137 images. This identifies a contribution beyond changed pixel weighting (pp. 6/8, R07/R12).

MRI controls identify a different objective dependence: Fine with target-only Dice gives exact-1× SFPC 15.98%, original Binary 13.80%, and original Fine 11.04%. The 4.94-point reduction [4.20,5.80] between Fine objectives changes auxiliary Dice and target scaling together. Finer likelihood labels alone therefore do not reproduce the original MRI improvement. We interpret this one-seed control separately from Cityscapes; Table 7 retains all five regimes and both endpoints, with patient-level CSVs and a reproduction script (pp. 9/18–19, R36/R25/R33).

Q1 — Why coarse labels if fine labels exist? We do not recommend discarding them. Coarsening is the scientific intervention; legacy binary training with a richer audit subset is its practical counterpart. SFPC complements overlap, FP counts and sensitivity; costs determine preference (p. 9, R14/R15).

Q2 — Why different losses, and what follows? The MRI audit corrects the submitted “balanced NLL versus CE” description. Both core branches use masked unweighted CE plus foreground Dice; the balanced-NLL helper is unused. Dice is applied to each regime's foreground classes: target only for Binary, versus target/ventricle/nWMH averaged for Fine. Thus auxiliary Dice and target coefficient (1 versus 1/3) differ. Numerical results are unchanged. Appendix A gives complete masking, unit CE weights and top-level coefficients, valid-voxel batch normalization, patch/class averaging, absent-class handling and smoothing 10−610^{-6} (pp. 5/13, R06/R19).

New Cityscapes holds these factors fixed: other/car/truck/bus weights (0.25,1,1,1), batch normalizer Z=max⁡(1,∑n,imniwni)Z=\max(1,\sum_{n,i}m_{ni}w_{ni}), per-image target Dice with coefficient/smoothing 1, and four active logits. Marginal uses −log⁡(1−pt)-\log(1-p_t) on negatives; Fine uses −log⁡py-\log p_y. Appendix J gives both complete equations (p. 17, R22).

Exact-budget Marginal/Fine SFPC is 20.58%/18.48%. Validation-selected matching gives +3.14 points [1.35,4.89], Holm p=0.0000219, with Dice 0.8661/0.8635 and FPs 2310.1/2209.0. Test matching is approximate; the primary exact-budget audit fixes FP counts (p. 9, R34).

Q3 — Why can errors shift with an unchanged Bayes target posterior? The target event is unchanged, but the fitted objective changes. For a negative pixel with the same fine-class probabilities:

−log⁡py=−log⁡(1−pt)−log⁡py1−pt,y≠t.-\log p_y=-\log(1-p_t)-\log\frac{p_y}{1-p_t},\quad y\ne t.

The added term classifies among negatives. Under softmax it has zero direct derivative with respect to the target logit, but changes negative-logit and shared-feature gradients. Finite shared networks can therefore learn different target rankings. The new experiment supports this objective-level contribution without identifying a unique representation mechanism (pp. 4/17, R04/R22).

We retain the earlier mean +2.18 [0.80,3.69] and prespecified p=0.694. The new +2.11 estimate has similar direction/magnitude but tests a different intervention on the same images; it is revision-stage follow-up evidence, not preregistered independent replication or an effect established for every pair (pp. 8/18, R11/R23).

We hope the revision and responses address your concerns, and would welcome any remaining questions.

Thank you again for your time and constructive review. Your questions about the training objectives were particularly valuable: they prompted our MRI code audit and the additional controlled Cityscapes experiments, helping us make the methods and interpretation more precise.

As the discussion period draws to a close, we would appreciate any feedback on whether our response and revised manuscript address your concerns. If any questions remain unresolved, we would welcome the opportunity to clarify them or provide additional information.

We appreciate your careful consideration and the improvements your feedback has helped us make to the paper.

Reviewer_s5HBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether coarse- and fine-grained segmentation supervision produce different semantic distributions of false positives (FPs), even when the total FP burden is matched. The authors identify a phenomenon called Semantic Error Relocation (SER) and propose a metric named Semantic False-Positive Concentration (SFPC) to quantify the proportion of target FPs falling on negative subclasses. They evaluate SER on MS3SEG and Cityscapes using matched-FP and exact equal-FP-budget protocols. Additional experiments examine the effects of loss weighting and explicit class supervision on this phenomenon.

Critical strengths
  1. Interesting empirical question: The paper highlights how conventional overlap metrics may overlook differences in the semantic composition of segmentation errors. The authors further observe this effect in both medical and natural-image segmentation tasks, suggesting that it may be a more general phenomenon worth investigating beyond a single application.

  2. Careful experimental design: The matched-FP and exact-budget analyses are useful for separating error quantity from error composition. The paired evaluation, multiple architectures and seeds, and appropriate statistical analysis strengthen the empirical evidence, increasing confidence in the reported phenomenon.

Critical weaknesses
  1. Limited novelty and unclear practical significance.

My main concern is whether SER provides sufficiently new knowledge beyond existing findings on fine-grained background supervision and hard-negative reweighting. Prior work such as BackSplit has already studied the effects of subdividing background classes, while Zhou and Beyerer (2023) and Bernhard et al. (2024) have emphasized semantic and fine-grained error analysis. Although the matched-FP comparison is a useful addition, different supervision objectives can naturally lead to different confidence rankings and error distributions on hard negatives. Furthermore, a lower SFPC does not necessarily imply a more desirable error distribution, since the consequences of different FPs depend on the application context. Specifically, the MS3SEG component-level analysis suggests potential clinical relevance, but the impact on actual diagnostic decisions is not evaluated. Similarly, in Cityscapes, depending on the downstream task, falsely predicting car pixels on roads or sky could be more consequential than confusing trucks or buses with cars, or vice versa. Therefore, while the paper demonstrates that error composition changes, it remains unclear whether these changes translate into practically preferable model behavior.

  1. Insufficient clarity of training objectives and underlying mechanisms.

The training objectives are central to interpreting SER, but their complete mathematical formulations are not provided in the paper. In particular, changes in label granularity also affect loss weighting and class competition, making it difficult to identify which factors drive the observed differences. The mechanism experiments partially address this issue. Hidden-negative reweighting already reduces SFPC, while the additional effect of explicit class identity remains inconclusive. Furthermore, Coarse-PM matches the nominal parameter count, but its additional output logits do not participate in the binary loss, limiting the informativeness of this capacity control. Consequently, although the experiments provide convincing evidence of different error distributions under the studied supervision regimes, the underlying mechanisms remain insufficiently understood, limiting the explanatory depth of the findings.

Questions for authors
  1. The mechanism experiments show that hidden-negative reweighting can partially reproduce SER, while the additional effect of explicit class identity remains inconclusive. What specific new insight does SER provide beyond the known effects of fine-grained supervision and hard-negative reweighting?
  2. Could the authors provide the complete mathematical formulations of the training losses, including their weighting and normalization? Additionally, what does the Coarse-PM experiment effectively control for when the additional output logits do not participate in the binary loss?
  3. The MS3SEG models achieve an average Dice of approximately 0.62. Does SER persist as segmentation performance improves on the same dataset? Could the authors provide evidence using stronger models or checkpoints at different performance levels, if available?
  4. Could the authors clarify under what practical circumstances the observed differences in error composition would meaningfully affect model evaluation or selection, given that the relative costs of false positives on different negative classes are application-dependent?
3 comments

Thank you for identifying novelty and practical value as the central questions. The revision adds controlled evidence beyond pixel weighting and a concrete model-selection consequence. References identify highlighted PDF paragraphs/captions; brackets give 95% CIs.

W1 — Novelty and practical significance. On 40 MRI confirmation patients, Binary has higher Dice by 0.0092 [0.0041,0.0144], yet overlaps 1.06 more nWMH components per patient [0.76,1.39] at nearly equal FP counts. The confirmation SFPC gap is +7.15 points [6.04,8.31], positive in 39/40 patients. Dice-based selection can therefore favor more errors on this subclass. BackSplit studies background subdivision; Zhou–Beyerer and Bernhard et al. study semantic error analysis. We connect supervision changes to error composition while controlling FP quantity. Novelty lies in this experimental relationship and protocol, not the SFPC ratio. Original exact-1× gaps are +2.73 MRI and +4.30 Cityscapes points. Lower SFPC is preferable only under appropriate application costs (pp. 2–3, R35/R03; p. 7, R09; p. 9, R15).

W2 — Objectives, mechanism and capacity. The new Cityscapes comparison shares pixel weights, normalization, target Dice, four active logits, paired initialization and training rules. Coarse-PM controlled nominal head parameters only; unused logits did not match active supervised capacity. Figure 3 distinguishes the original controls from the new intervention (pp. 6–8, R07/R10/R37). MRI controls show sensitivity to the joint Dice objective: target-only Fine SFPC is 15.98%, original Binary 13.80%, and original Fine 11.04%; auxiliary Dice and target scaling change together (p. 9, R36).

Q1 — Beyond reweighting? Across two architectures, three paired seeds and 137 images, adding conditional negative-class likelihood reduces exact-budget SFPC by 2.11 points [0.70,3.52], p=0.0054. Validation-selected matching gives +3.14 [1.35,4.89], Holm p=0.0000219. These are pooled effects across six fitted pairs. We retain the earlier +2.18 [0.80,3.69] and prespecified p=0.694; the new intervention uses the same cohort and is not an independent replication (p. 8, R11/R12; p. 18, R23).

Q2 — Complete losses and Coarse-PM. The code audit corrects the submitted MRI “balanced NLL” description; its helper is not used by the core branch. Both regimes use masked unweighted CE plus foreground Dice; numerical results are unchanged. For valid mask mnim_{ni}, patch count NN, foreground set FRF_R and softmax pRp^R:

dnkR=1−2∑imnipnikR1[yniR=k]+ϵ∑imnipnikR+∑imni1[yniR=k]+ϵ,ϵ=10−6.d^R_{nk}=1-\frac{2\sum_i m_{ni}p^R_{nik}\mathbf1[y^R_{ni}=k]+\epsilon}{\sum_i m_{ni}p^R_{nik}+\sum_i m_{ni}\mathbf1[y^R_{ni}=k]+\epsilon},\quad\epsilon=10^{-6}.

LR=−∑n,imnilog⁡pni,yniRRmax⁡(1,∑n,imni)+1N∣FR∣∑n∑k∈FRdnkR.L_R=-\frac{\sum_{n,i}m_{ni}\log p^R_{ni,y^R_{ni}}}{\max(1,\sum_{n,i}m_{ni})}+\frac{1}{N\lvert F_R\rvert}\sum_n\sum_{k\in F_R}d^R_{nk}.

CE weights and top-level coefficients are 1. Binary uses target Dice; Fine averages target/ventricle/nWMH Dice, so target coefficients are 1 and 1/3. Dice averages patch/class losses, retaining absent foreground classes and excluding background; invalid voxels enter neither term. Entirely invalid batches raise an error (p. 13, R19).

New Cityscapes weights are (0.25,1,1,1) for other/car/truck/bus, with batch normalizer Z=max⁡(1,∑n,imniwni)Z=\max(1,\sum_{n,i}m_{ni}w_{ni}) and per-image target Dice, coefficient/smoothing 1. Marginal sums negative probabilities; Fine uses subclass identity. Both activate all four logits, addressing Coarse-PM's limitation. Appendices A/E/J give full objectives; Appendix I states historical source-provenance limits (pp. 13/15/17, R19/R20/R21/R22).

Q3 — Higher MRI Dice? Within the original protocol, Binary Dice spans 0.5894–0.6518, with positive SFPC gaps +7.43/+7.30/+6.72 points, all CIs above zero (p. 18, R24). Completed residual-encoder runs reach Marginal/Fine Dice 0.7232/0.7242 on fold 3 (three seeds) and 0.7445/0.7405 on fold 4 (one seed), with 20 patients per fold. Exact-1× gaps are −0.36 [−0.80,0.15] and +0.11 [−0.48,0.72]. These use shared target-only Dice and all valid negatives without distance exclusions; Dice uses separate validation-selected Dice-optimal thresholds. Architecture, likelihood objectives and pipeline changed together, so these results cannot isolate higher Dice's effect (p. 18, R30/R31).

Q4 — When does composition matter? At fixed per-case FP budget BB, the cost difference is B(cH−cO)ΔSFPCHB(c_H-c_O)\Delta\mathrm{SFPC}_H; preference reverses with relative costs. A finely annotated audit set can support MS lesion review and model selection. Radiotherapy contour errors on critical organs and robot false-traversable predictions are additional proposed uses requiring downstream validation. Selection should also consider sensitivity and spatial consequences (pp. 9–10, R15/R27/R28).

We appreciate your reconsideration and welcome remaining questions.

Thank you for your detailed response, for adding the exact loss equations, and for running the additional experiments. However, looking at the new numbers you reported, my initial concerns seem to be actually confirmed.

First, on MS3SEG, the loss equations and new controls (W2, Q2) show that the original Fine model explicitly included an auxiliary Dice loss term on the hidden negative class (nWMH). When you control for this by using target-only Dice for Fine, the SFPC goes up to 15.98%, which is actually worse than Binary (13.8%). This shows to me that the MS3SEG effect was really driven by the changed Dice objective rather than the finer cross-entropy labels. Second, regarding model strength (Q3), when you use stronger models on MS3SEG that reach 0.72–0.74 Dice with a shared target-only Dice loss, the SER effect completely disappears.

With the new Cityscapes experiment (a difference of 2.11), these results mainly confirm my original point that binary and multi-class losses penalise and weight background regions differently, rather than showing a surprising new phenomenon beyond prior work like BackSplit. The practical guidance for model design also remains limited since the cost of different false positives depends entirely on the downstream task.

I really appreciate your transparency in reporting these new controls, and I will keep my score as 2.

Thank you for recognizing our transparency. We are committed to testing our claims thoroughly and reporting the results accurately.

1. What our study adds beyond BackSplit. BackSplit shows that subdividing the background can improve target segmentation, supported by statistical-efficiency theory and Dice, HD95 and NSD evaluations. Our question is different: when the number of false positives is fixed, does supervision change which negative subclasses receive those errors? Improvements in segmentation performance do not answer this fixed-burden composition question.

Our MRI results show why the distinction matters. On the 40 confirmation patients, Binary has higher Dice (0.6263 versus 0.6172), yet overlaps 1.06 more annotated nWMH components per patient [0.76, 1.39], at nearly identical total FP counts (1280.3 versus 1282.6). Fine would therefore rank below Binary by Dice, but above it by errors on this prespecified subclass. In the separate exact-1× audit, Binary's SFPC is 2.73 percentage points higher at identical FP counts; this diagnostic audit does not also fix Dice. All intervals here are 95% confidence intervals.

Our added finding is that the Dice-preferred supervision regime can have more errors on a prespecified subclass at closely matched FP burden. BackSplit's performance comparisons do not establish this result. Validation-selected matching and the separate exact-budget audit distinguish error quantity from composition. The contribution is the controlled evidence and evaluation consequence; SFPC provides a simple summary.

2. What the mechanism controls establish. The Cityscapes follow-up holds the pixel-weight map, normalization, target Dice and four active outputs fixed. Adding conditional negative-class likelihood reduces pooled exact-budget SFPC by 2.11 percentage points [0.70, 3.52], Wilcoxon p=0.0054, across six paired fits and 137 images. The objective changes, as you note. The result establishes that a change in explicit scalar pixel weights is not required for this compositional shift. This is a pooled Cityscapes finding, not a mechanism established for MRI or every individual model pair.

The MRI controls identify a different dependency. In the separate one-seed exact-1× control, Fine with target-only Dice gives 15.98% SFPC, versus 13.80% for Binary and 11.04% for original Fine. Finer CE labels with target-only Dice therefore do not reproduce the original Fine model's SFPC reduction. The comparison changes both auxiliary foreground Dice and the target-Dice coefficient (1 versus 1/3), so it does not isolate the nWMH Dice term alone.

The stronger residual-encoder comparisons yield near-zero exact-1× gaps: -0.36 points [-0.80, 0.15] on fold 3 with three seeds and +0.11 [-0.48, 0.72] on fold 4 with one seed, with 20 patients per fold. They do not establish persistence in that setting. Architecture, objectives and pipeline changed together, preventing attribution of the near-zero gaps to higher Dice alone. These limits refine the claim without removing the measured disagreement between evaluation criteria.

3. Why task-dependent costs make this evaluation useful. At equal FP counts, use prespecified per-voxel costs cH>cOc_H>c_O for errors on critical structures HH versus other tissue OO. Then lower SFPCH\mathrm{SFPC}_H means lower total FP cost. With other relevant outcomes held equal, this favors the model whose errors fall on lower-cost tissue, even when Dice ties.

Radiotherapy offers an additional proposed application. Two target contours can have equal Dice and FP volume, yet mistakenly include different healthy structures. A false positive involving a sensitive organ, such as the spinal cord, may have greater consequences if it affects treatment planning and increases unintended radiation exposure. On annotated evaluation cases, organ-specific FP counts and SFPC reveal whether errors concentrate in critical structures, informing model assessment and contour-review priorities. Actual risk depends on the treatment plan and dose distribution; an FP voxel is not automatically an irradiated voxel.

As related motivation, Marquez et al., Diagnostics, 2024 found that geometric agreement could be misleading about dosimetric agreement for some organ-at-risk contours near treatment targets. That study examined organ-at-risk contours under existing plans; it motivates anatomy-aware evaluation without validating SFPC or demonstrating clinical benefit from our experiments.

We respectfully ask you to reconsider the paper on this specific contribution: controlled evidence that supervision can change the semantic distribution of residual errors, even when total FP burden is fixed, together with a measured disagreement between Dice and subclass-error preferences. This provides an evaluation perspective beyond establishing that background subdivision can improve segmentation performance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Comment to all the reviewers and the area chair

Thank you very much for the time you spent on reviewing our work and for the constructive reviews. We have revised the manuscript around four changes. References below identify highlighted PDF paragraphs and captions; all reported intervals are 95% confidence intervals.

1. A concrete evaluation consequence. On the 40 MRI confirmation patients, Binary has higher Dice by 0.0092 [0.0041,0.0144], yet overlaps 1.06 more annotated nWMH components per patient at nearly equal FP counts. Thus a Dice-based choice can favor a model with more errors on a clinically relevant negative subclass. The new abstract and introduction lead with this trade-off. The original exact-1× SFPC gaps are +2.73 points on MRI and +4.30 on Cityscapes; these are distinguished from the approximately 7-point validation-matched gaps (p. 1, paragraph R01; p. 2, paragraph R35).

2. New evidence beyond changed pixel weights. Twelve new Cityscapes models form six paired fits across two architectures and three seeds. The two objectives use identical pixel weights, normalization and target-Dice loss, and both train all four output logits. Adding conditional negative-class likelihood lowers exact-budget SFPC by 2.11 points [0.70,3.52], p=0.0054, pooled over all six pairs and 137 images. Figure 3 shows the original controls alongside this follow-up on the same evaluation cohort; the original tests remain reported (p. 8, paragraph R12; p. 8, paragraph R37).

3. Complete objectives and a corrected MRI description. The reviewers’ questions prompted an audit of the archived core training code. This confirmed that both MRI regimes used masked, unweighted cross-entropy plus foreground Dice. We correct the submitted description of Binary as using “balanced NLL”: a balanced-NLL helper exists in the source but is not called by the core MS3SEG training branch. Binary uses target Dice, whereas Fine averages target, ventricle and nWMH Dice, giving target-Dice coefficients of 1 and 1/3, respectively. The revised equations specify weights, masks, normalization, smoothing and averaging. This correction concerns the description of the executed objectives; the reported numerical results are unchanged (p. 5, paragraph R06; p. 13, paragraph R19).

4. Joint evaluation and practical use. Tables 1–2 report Dice, FPs and SFPC together. The MRI-control table has a patient-level evidence supplement and reproduction script; Appendix K retains the completed stronger-model evaluations with their seed coverage and distinct endpoints. MS lesion review illustrates the measured error trade-off, with radiotherapy and robot navigation retained as additional proposed applications. The cost equation states when subclass errors should influence model selection (p. 7, paragraph R09; p. 9, paragraph R34; p. 19, paragraph R33; p. 18, paragraph R30; p. 9, paragraph R15; p. 10, paragraph R27; p. 10, paragraph R28).

The central contribution is a controlled empirical relationship between supervision and the semantic destinations of residual errors. We appreciate your reconsideration in light of the corrected methods, new controls and clearer scope.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.