- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates an important limitation of conventional segmentation evaluation: two models can achieve similar Dice scores and make comparable numbers of false-positive predictions, yet distribute those errors across clinically or semantically different regions.
The authors study how collapsing fine-grained negative classes into a single background category during training changes the semantic composition of segmentation errors. They call this phenomenon Semantic Error Relocation (SER).
The paper introduces Semantic False-Positive Concentration (SFPC), defined as the proportion of target false positives assigned to a prespecified hidden negative subclass: \mathrm{SFPC}_H=\frac{\mathrm{FP}H}{\mathrm{FP}{\mathrm{all}}}.
The central contribution is not simply introducing a new segmentation metric, but demonstrating a controlled relationship between training-label granularity and the semantic destination of prediction errors.
The evaluation has several strengths: validation-selected false-positive matching, an exact equal-FP-budget audit, an MS3SEG confirmation study, cross-domain replication on Cityscapes, and controls investigating the role of prediction-head size and loss weighting.
Critical strengths
The empirical question is clear and important, the matched-error design addresses a major confound, the MS3SEG confirmation is strong at the patient level, and the independent Cityscapes replication provides evidence that the phenomenon is not restricted to one medical imaging task. The authors also avoid overclaiming that explicit semantic class identity has been proven to cause the effect.
Critical weaknesses
The principal metric is a simple error-composition proportion closely related to established class-conditional false-positive analyses. The exact-budget audit relies on ground-truth negative locations and is therefore diagnostic rather than deployable. Additionally, the training objectives differ between binary and fine supervision, making it difficult to identify which part of the effect originates from label granularity itself rather than the loss induced by that granularity.
There is also a noteworthy statistical interpretation issue in the mechanism controls: the H-Weighted versus H-Union comparison has a bootstrap confidence interval excluding zero but a nonsignificant Wilcoxon test. The authors explicitly explain this divergence in Appendix G; nevertheless, the interpretation of the resulting evidence could be substantially clearer.
Table 1 presents the SFPC results but does not report FP counts and Dice alongside them. Because the manuscript's central claim is that error composition changes while error quantity remains approximately controlled, these quantities should appear together.
The paper's most important mechanistic finding is that increasing the loss weight on hidden negatives reduces SFPC even without giving them explicit class identities. This substantially refines the paper's interpretation, yet the results appear relatively late.
The paper partly merges two distinct conclusions. The first is well supported: changing the supervision regime changes the semantic concentration of false positives. The second is mechanistic: this relocation arises from changes in loss weighting, class competition, or explicit class identity. The current mechanistic evidence only partially decomposes these factors.
SFPC itself is mathematically simple and closely related to existing class-conditional error statistics: \mathrm{SFPC}_H=\frac{\text{FPs on hidden class }H}{\text{all target FPs}}. Its usefulness is clear, but the metric is not a fundamentally new statistical estimator. The experimental matching framework adds more novelty than the formula. The Bayes-posterior observation in Equation (3) is also straightforward: merging negative labels leaves the target-class event unchanged. Its value is explanatory rather than theoretical novelty.
Questions for authors
Section 4.1; Appendix A. In MS3SEG, the binary models optimize target/non-target Dice plus balanced negative log-likelihood, whereas the fine models use masked four-class cross-entropy plus foreground Dice. Therefore, the experiment changes both the output taxonomy and the objective induced by that taxonomy. The authors recognize this problem and conduct useful follow-up controls on Cityscapes. However, these do not fully establish the same mechanism in MS3SEG.
Section 3.3, Equation (4). The exact-budget experiment ranks scores at ground-truth negative locations and selects the highest-scoring locations. This guarantees equal FP counts, but it relies on information unavailable during deployment. The authors correctly label it as a diagnostic rather than a deployable thresholding method. The concern is interpretative: these experiments compare the composition of high-scoring negatives, not the complete deployment-time segmentation predictions.
Sections 5.1 and 5.3. The MS3SEG FP difference is very small: −2.3 voxels, with a confidence interval of [−54.3, 52.8]. Cityscapes is less tightly matched. Its aggregate FP counts are 2,294.7 versus 2,055.1, and one architecture has a 19.7% test-time FP mismatch despite validation matching. The reported does not establish equivalence of FP burden.
Section 5.4; Appendix G, Table 4. For H-Weighted Binary minus H-Union at the exact 1× budget, the reported mean SFPC difference is: \Delta=+2.18\text{ pp},\quad 95%\ \mathrm{CI}=[0.80,3.69], \quad p_{\mathrm{Wilcoxon}}=0.694. The bootstrap confidence interval excludes zero, while the Wilcoxon signed-rank test is nonsignificant. This is not necessarily a computational inconsistency. The authors explain that only 52 images have positive paired differences, 68 have negative differences, and 17 are ties. The bootstrap interval measures the paired mean, while the signed-rank test assesses a different aspect of the paired distribution. However, the manuscript's conclusion that the additional class-identity contribution is not reliably supported requires more nuance.
Sections 5.1 and 6; Appendix D. The paper interprets nWMH as a semantically adjacent negative subclass. However, some false positives may arise because nWMH and MS lesions share image intensity, anatomical location, or proximity to lesion boundaries. The existing paired-patient comparisons successfully control for differences in patient anatomy between supervision regimes. They do not, however, identify which properties of nWMH make them susceptible to relocation. The authors already address component size and distance from ventricles. A further analysis would be valuable.
Section 4.1; Appendix I. The authors report that the matched-error endpoint was frozen before it was examined on folds 3–4, while conventional argmax metrics from those folds had already been seen during development. The appendix also explains that some supporting panels became visible during checkpoint recovery. The procedural transparency is commendable, and the large effect in 39 of 40 confirmation patients remains persuasive. Nevertheless, the evidence should not be described as equivalent to a completely untouched, prospectively preregistered external confirmation.
Section 5.1; Appendix D. The manuscript reports approximately one additional falsely detected nWMH component per patient under binary supervision. This is a useful operational interpretation of SFPC. However, component detections are defined using an overlap threshold of at least 10%, and nWMH annotations may be ambiguous in small or periventricular regions. The authors provide relevant robustness checks, but the clinical impact remains hypothetical.
3 comments
Thank you for your detailed response, I am happy to keep my rate as 3.
Thank you very much for reviewing our response and revisions. We appreciate your continued positive assessment and feedback. If any questions or concerns remain about the revised manuscript, evidence, or its interpretation, we would be more than happy to address them.
Thank you for the exceptionally careful reading, and we really appreciate the time you spent on reviewing our work. References identify highlighted PDF paragraphs/captions; brackets give 95% CIs.
W1 — Metric, audit and objectives. SFPC is descriptive; exact budgets audit ranked negatives. The contribution is the supervision intervention and controlled composition comparison. Corrected MRI methods retain auxiliary-Dice and target-scaling differences (pp. 3–5, R02/R05/R06).
W2 — Bootstrap/Wilcoxon divergence. We retain H-Weighted/H-Union mean +2.18 points [0.80,3.69] and prespecified p=0.694: 52 positive, 68 negative and 17 tied differences, median zero. Mean-bootstrap and signed-rank inference differ (Q4); the new test does not replace the original (p. 8, R11).
W3 — Joint reporting. Tables 1–2 report Dice/FPs/SFPC together. New validation-selected Marginal/Fine values are 0.8661/0.8635 Dice, 2310.1/2209.0 FPs, 22.64%/19.50% SFPC; captions distinguish approximate matching from exact budgets (pp. 7/9, R09/R34).
W4 — Weighting appeared late. The abstract/contributions now foreground it; Figure 3 shows original and new controls. The weighting reduction is 1.48 points [0.70,2.29], Holm p=0.0000593 at exact 1×. Successive contrasts are not independent causal shares (pp. 1/3/8, R01/R02/R37).
W5 — Finding versus mechanism. New Cityscapes holds weights, normalization, target Dice, four active logits and paired training rules fixed. Adding conditional negative-class likelihood lowers exact-budget SFPC by 2.11 points [0.70,3.52], p=0.0054, pooled across six pairs/137 images (p. 8, R12).
MRI controls show a different dependence: target-only Fine SFPC 15.98%, original Binary 13.80%, original Fine 11.04%. The 4.94-point reduction [4.20,5.80] between Fine objectives reflects joint changes to auxiliary Dice and target coefficient, not an isolated identity effect. This one-seed result is explicit in Results and Table 7 (pp. 9/18–19, R36/R25/R32).
W6 — SFPC/Bayes novelty. Neither is claimed as a new estimator or theorem. The controlled supervision–error-composition relationship reveals a selection trade-off: Binary has higher Dice yet greater nWMH-component overlap (pp. 2–3, R35/R03).
Q1 — MRI losses/transfer. Both core branches use masked unweighted CE plus foreground Dice; “balanced NLL” was a description error. CE averages valid batch voxels. Dice averages patch/class losses with smoothing 10−6, excluding background/invalid voxels and retaining absent classes. Top-level coefficients are 1. Binary uses target Dice; Fine averages ventricle/nWMH/target, giving target coefficient 1/3 (p. 13, R19). Appendix I records provenance limits. Table 7's patient-level CSVs/reproduction code support W5's MRI-specific joint-Dice interpretation; Cityscapes transfer is not assumed (pp. 17/19, R21/R33).
Q2 — Exact-budget interpretation. The audit requires ground-truth negatives and target size. It fixes FP counts only, not Dice or sensitivity, and yields no deployable threshold. Frozen validation-selected predictions provide separate operating-point evidence (pp. 4/9, R05/R34).
Q3 — Matching versus equivalence. We do not interpret p=0.703 as equivalence. MRI FP difference is −2.3 [−54.3,52.8], with means 1280.3/1282.6. Original Cityscapes means are 2294.7/2055.1, difference +239.6 [−96.3,729.2], including 19.7% mismatch in one architecture. Exact budgets provide exact count control; no post-hoc equivalence claim is made (pp. 7/10, R09/R17).
Q4 — Statistical interpretation. The interval supports a mean difference; p=0.694 gives no signed-rank support for a location shift. Positive/negative differences average +8.37/−2.01 points. Neither “no effect” nor uniform benefit follows. New models test a different intervention on the same images, not preregistered independent replication or evidence for every pair (pp. 8/18, R11/R23).
Q5 — Appearance/anatomy. Pairing controls patient anatomy across regimes. The original large/non-periventricular analysis remains positive: +15.55 points in nWMH false-positive rate, p=0.0032, among 21 eligible patients. This does not isolate intensity or distance to an abWMH boundary as the cause (pp. 6/19, R39/R26).
Q6 — Confirmation provenance. We use “internally frozen endpoint,” disclose prior argmax/recovery-output visibility, and label full OOF as overlapping robustness evidence (p. 10, R16).
Q7 — Components/practical consequences. The +1.06 count uses annotated components of at least five voxels and at least 10% prediction overlap. Abstract/Results consistently say “overlap”; clinical harm was not measured. The cost equation retains application-dependent preference and sensitivity. MS review, radiotherapy and robot navigation are distinguished from proven downstream outcomes (pp. 1/9–10, R01/R15/R27/R28).
We appreciate your detailed guidance and reconsideration.