acceptodds
Under review as a conference paper at ICLR 2027

Should This Case Be Adapted? Prediction Fragmentation Controls Test-Time Adaptation

Abstract

Episodic test-time adaptation (TTA) resets a frozen segmenter to source weights M₀ on each case and adapts for a fixed number of steps. A fixed horizon conflates a cohort-level question, how far to adapt, with an irreducibly per-case one, whether this case should be adapted at all. Cohort means average over that per-case decision: on cross-vendor cardiac MRI the mean ΔDice from adaptation is statistically indistinguishable from zero while 58.7% of cases are individually made worse. We quantify this harm as harmful accepted area (HA), the harmful fraction of the edited area a controller actually deploys. Held-out tuning gives a stronger baseline than a fixed horizon, but the budget it selects does not transfer on either of the two main medical benchmarks, and no global budget can condition on the case. We show that prediction fragmentation—the disagreement geometry between M₀ and the adapted mask Mₖ—predicts HA with no labels and no additional backward passes at decision time, at comparable strength on three benchmarks (Spearman ρ 0.50–0.60), at a quarter of gradient-norm's latency. A case-level router built on it cuts HA from 0.228 to 0.139 on a benchmark that took no part in its design, with that design frozen and only its cut-points recalibrated there. On the cardiac benchmark the design was selected on, the same router cuts HA from 0.129 to 0.013 at matched Dice and 1.10 deployed updates, against the retrospective-best budget identified post hoc from evaluation labels, and reduces that 58.7% to 20.0%—an upper bound we quantify. Where the retained cases are not net-helped (as on prostate), the router still cuts HA but concedes accuracy, a boundary we report explicitly. Thresholds are calibrated once on a labeled split disjoint from evaluation; deployment decisions use no labels and no gradients. The template ports across architecture and domain (nnU-Net→SegFormer, Cityscapes→ACDC) with coordinate, thresholds and per-bucket actions instantiated per domain.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time adaptation, episodic test-time adaptation, reliability, label-free monitoring, medical image segmentation, semantic segmentation, distribution shift

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Hs4xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper is about TTA for image segmentation. They found that TTA can make harmful edits. Figure 1 show that averages can hide damage to individual cases (mIOU fine but HA not). Plus more changes, and more scattered ones, go with higher HA. Built upon this finding, they use a label-free signal to measure how much the mask changed and propose a per-case router. The results show that their router can decrease HA.

Critical strengths
  1. The question is good and should be studied in medical cases.
  2. The results in table3&4 show that their router can decrease HA.
Critical weaknesses
  1. The paper is Difficult to follow despite careful reading. This contains 3 aspects: 1st, terms and prior work are used without being explained. GraTa-cos, appears with no definition. It took me a while to know it comes from a prior work. The paper introduces a large set of its own words and uses them in previous section without define them in words, like COQR; FragStop;FragSubset;All the Tier like A,A′,B,C; TENT8; retrospective; deployable. Several of them are defined only in a table footnote or in the appendix, makes it hard to follow in main text. 2nd, there is no figure that shows the overview. Most of the description are in appendix. 3rd, AI-Interpretation is often defensive. Honesty about limits is good. However, the limits here are scattered across main text and appendix, and the reader is never told in one place what the paper actually shows. After careful reading I cannot say whether the authors think the signal predicts harm, or the router is useful, or only that the question exists.
  2. HA gives free points for changing little or nothing, and the router benefits from this rather than spotting harmful TTA. For HA, small changes are not counted, and a case with no changes scores 0. The router undo almost all changes and thus could obtain huge HA decrease. This is shown very clear in table34, where mIOU almost has no change and HA has small increase. The kept changes are not better, only fewer. This matters because the paper's main claim is that the signal detects harmful adaptation.
Questions for authors

Questions:

  1. Prediction using only measurable cases. Please report how well each signal predicts HA using only cases where at least one change is large enough to be counted.
  2. Please compare the router and others under a harm measure that does not reward changing nothing (also random selection of the same number of undone cases should be compared).
  3. I read several appendix tables and found that (a)the router isn't always better than RANDOM; (b) the proposed method seems not useful in some cases. here are examples: for (a), tableS9, it seems comparable to random?; for (b), tableS5, "ours" is worse with lower Dice and higher HA compared to MEMO and CoTTA.
  4. Sec. 7 concludes "risky adaptation is easier to identify than beneficial adaptation". This seems not tested in the main paper, where is it come from?
  5. The information in table caption isn't helpful for readers to capture the claims. examples: table5, why the 3 numbers are textbf? what is rolled back? "which is what hides the 58.7%" what does this mean?.It is best to clear these information in caption.

Suggestions (also affect my score):

  1. Please add a method figure to the main text.
  2. State the conclusions plainly.
  3. Explain prior work and terms where they were first used.
2 comments
AuthorsOpenReview ↗

Response to Reviewer Hs4x

We thank the reviewer for the careful reading; answers follow in order.

W1. Readability. We accept all three points. In words: the source model predicts (M0); one adaptation step gives a second prediction; the disagreement between the two masks is summarised by two label-free quantities, the share of changed voxels (delta) and the number of changed regions (n_reg); frozen cut-points on one of them map each case to deploy, shrink or roll back. Ground truth enters only the offline evaluation, never the online decision; label visibility during design: Appendix S10 and the corrections below. It shows that the problem exists (on vendor B fixed-4 changes mean Dice by +0.0001 [-0.0012, +0.0014] while 135 of 230 cases fall; replicated in pooled vendors C+D, not used for router design, though earlier M0/TENT diagnostics existed) and that HA is a reasonable but coarse measure of it. It does not establish a positive harm ranking by the signal on voxel-level measures, nor that the router is useful compared with not adapting on cardiac data.

W2 / Q2. HA rewards changing little; alternative measure; matched random. Short answer: on HA at matched quota the router beats random on three of four medical benchmarks; on the voxel-level measure below it does not. That measure: net error voxels relative to M0 per ground-truth foreground voxel; M0 scores 0 and a helpful edit scores below 0, so abstaining is not optimal by construction (class-aware on M&Ms, binary elsewhere).

BenchmarkRouterRandom, same actions and bucketsRandom at least as good
M&Ms vendor B (selected; depths 3/2/0, no shrink)-0.0001-0.0009 [-0.0026, +0.0008]80%
Vendors C+D (frozen)+0.0000-0.0013 [-0.0028, +0.0002]96%
Prostate158 (frozen; depths 3/2/0, no shrink)-0.0053-0.0070 [-0.0094, -0.0046]92%
Prostate OOD-all (deployed; corrected-rerun masks)-0.0042-0.0261 [-0.0456, -0.0059]98%

Random brackets: 2.5/97.5 percentiles of 10,000 assignments. On HA at the same quota it beats random on cardiac data and Prostate OOD-all, not on Prostate158 at common depths (0.161 vs 0.164). "The kept changes are not better, only fewer" is mostly right: on M&Ms the router retains a scorable edit in 7% of cases vs 21% under random; among scorable cases its HA is lower, but no advantage is established on voxel-level measures.

Q1. Measurable cases only. M&Ms, step-1 signal vs step-1 HA, cases with at least one scorable region (167 of 230; subject bootstrap):

Signalall casesmeasurable only
delta0.49 [0.39, 0.59]0.26 [0.11, 0.40]
n_reg0.53 [0.44, 0.61]0.19 [0.03, 0.34]
1 - augmentation agreement0.32 [0.20, 0.44]0.18 [0.03, 0.33]
gradient norm0.33 [0.20, 0.45]0.16 [-0.02, 0.33]
entropy (corrected)-0.48 [-0.57, -0.37]-0.22 [-0.38, -0.07]
1 - MSP (corrected)-0.49 [-0.59, -0.38]-0.24 [-0.39, -0.08]

As a top-HA-tertile ranker, delta falls from AUROC 0.74 to 0.59 [0.50, 0.67].

Q3. (a) Yes. In Table S9 the fixed-retain substitute control does not establish better HA than matched random in rain, snow or night, fog being the exception; on Prostate158 no advantage is established at common depths (W2). (b) Also correct for MEMO: on OOD-hard (Table S5) MEMO has higher Dice and lower HA than both of our budget-axis controllers.

Q4. The Sec. 7 sentence rests only on two AUROC columns of Table S14, and the favourable column uses a target sharing its construction with the signal. We will remove it.

Q5. Captions will state the claim, define every bolded entry and term ("rolled back" = the source prediction is deployed), and not depend on text elsewhere.

Shared corrections to the submission. (i) Per Appendix S10, the coordinate, middle action and low-bucket depth were chosen with the vendor-B evaluation split visible; 0.129 to 0.013 will be labelled a selected-design result, not the headline. (ii) The entropy and MSP baselines read a probability map with its in-plane axes swapped (masks, Dice, HA, delta, n_reg unaffected). Corrected: M&Ms Table S8 entropy HA 0.0537 to 0.1186; the printed MSP 0.0567 used the opposite rollback orientation, 0.0572 is the bug-affected value in the stated orientation, corrected to 0.1221; both exceed every matched random assignment. Prostate OOD-all (printed 0.1044/0.1030 reproduce from the affected signals): corrected selectors reach HA 0.163 vs 0.099 for the router, Dice +0.013 [0.002, 0.025] and net errors -0.044 [-0.084, -0.007] relative to it, no difference from adapting every case detected; HA/Dice from archived actions, net errors from corrected-rerun masks. (iii) Frozen transfer test on the previously evaluated Prostate158 cohort (126 cases, criteria fixed in advance): FragSubset did not establish its prespecified advantage, Dice vs M0 +0.0029 [-0.0020, +0.0074], vs FragStop +0.0009 [-0.0011, +0.0030]. (iv) Revision: overview figure; define terms.

Thank you for running the additional experiments. I’m keeping my score at 2. The authors mostly acknowledge my weakness and questions (like rebuttal of W1,Q3,Q4).

HA at matched quota the router beats random on 3/4 medical benchmarks. On the voxel-level metric, however, the results do not establish an advantage over matched random ("on the voxel-level measure below it does not"). Can the router tell which TTA edits are worth keeping?

In the rebuttal, the authors mostly acknowledge the writing issue, however, I didn't see a modified version of the paper. Since "risky adaptation is easier to identify than beneficial adaptation" is still in the paper(along with others). The concern about writing is not resolved. I also still find the paper difficult to follow.

Reviewer_AC84OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the reliability of episodic test-time adaptation (TTA) for semantic segmentation, arguing that fixed adaptation budgets can conceal substantial case-level deterioration even when average segmentation accuracy remains unchanged. The authors introduce Harmful Accepted Area (HA), a source-referenced metric that quantifies the proportion of modified regions classified as harmful relative to the original prediction. They then propose prediction fragmentation, characterized by the connected-component structure and extent of disagreement between the original and adapted predictions, as a label-free indicator of adaptation-induced harm. Based on this signal, the paper develops several controllers, including adaptive stopping, parameter-subset modulation, and a case-level router that determines whether to retain, restrict, or revert adaptation. Experiments on cardiac MRI, prostate MRI, and adverse-condition driving segmentation demonstrate reductions in HA, although the effectiveness of the proposed routing strategy varies substantially across benchmarks.

Critical strengths
  1. The paper highlights a meaningful limitation of evaluating TTA primarily through cohort-averaged segmentation metrics. The observation that adaptation can degrade individual predictions despite negligible changes in average Dice is important, particularly for medical applications where reliability at the individual-case level matters. The distinction between selecting a global adaptation budget and deciding whether adaptation should be retained for a particular case is well motivated.
  2. A computationally efficient approach to monitoring adaptation behavior.
Critical weaknesses
  1. As noted in Appendix S10, the authors evaluated multiple controller variants on the M&Ms vendor-B test split, using these results to choose the fragmentation coordinate, middle-bucket action, and adaptation depth. Because the decision policy’s architecture was selected based on test outcomes, the reported HA reduction (0.129 to 0.013) is not an unbiased estimate of generalization performance. Subsequent recalibration on a disjoint split does not erase this initial dependence. While I appreciate the transparency, disclosure alone is insufficient. The authors must provide a genuinely independent evaluation where all design choices are made exclusively on development/validation data to substantiate their claims.

  2. Matched-quota experiments reveal that HA reductions mainly come from rejecting more updates. While fragmentation outperforms baselines on M&Ms, this is the very dataset used to design the controller. On independent benchmarks, the gains are marginal or non-existent: it ties with baselines on Prostate OOD-all, matches random routing on ACDC (except in fog), and shows a negligible difference (0.161 vs. 0.164) against random on zero-shot Prostate158. Ultimately, the authors must disentangle the benefit of simply rejecting updates from the actual incremental value of the fragmentation signal.

  3. HA does not measure the fraction of harmful edits it claims to; it measures the area of net-harmful connected components. Because each component is classified by its aggregate error, one containing 49% harmful changes contributes nothing, while one containing 51% contributes its full area — making HA a function of spatial grouping at least as much as of prediction quality. Two additional choices compound the problem: binarizing predictions into foreground and background silently excludes clinically meaningful inter-class confusions (e.g., myocardium vs. ventricular blood pool), and the minimum-component-size threshold allows small harmful edits to yield HA = 0. The sensitivity analysis addresses threshold dependence but not these semantic limitations. The authors should validate HA against a direct voxel-level harmful-edit fraction, class-aware error changes, and standard region- or boundary-sensitive metrics; without this, it remains unclear whether reported HA reductions reflect genuine improvements in segmentation reliability.

  4. The proposed router fails to reliably identify cases where adaptation is actually beneficial. Predicting the geometric risk of a modification is not equivalent to predicting its practical utility, and the experiments expose this gap. On M&Ms, the router matches the original model's Dice primarily by rejecting almost all modifications (only 17/230 cases adapt), offering no practical advantage over simply retaining the source model (which already yields HA=0 at zero cost). On Prostate OOD-all, it reduces HA but actively degrades Dice (0.752 to 0.739), sacrificing the very accuracy gains adaptation is meant to provide. Ultimately, the controller acts more as a change-suppressor than a utility-predictor. The authors must explicitly evaluate case-level adaptation benefit and demonstrate a meaningful accuracy–reliability trade-off against both the unadapted model and fixed-budget baselines.

  5. The evaluation across adaptation objectives fails to establish the generality of the proposed mechanism. The primary medical experiments rely on a modified EATA (using an unweighted penalty instead of Fisher-weighted regularization, with inactive entropy filtering), alongside altered versions of CoTTA and SAR. While transparent, these deviations complicate fair comparisons with established TTA methods. Supplementary GraTa and DeYO experiments further expose limitations: the correlation-band criterion is met in only 1 of 6 configurations, and matched-random controls reproduce 69%–86% of the router’s HA reduction, severely undermining the claim of fragmentation-specific control. Ultimately, the proposed signal appears highly sensitive to the adaptation objective and update magnitude. The authors must avoid conflating the mere ability to attach a router to various optimization trajectories with evidence that fragmentation serves as a generally effective risk-ranking mechanism.

Questions for authors

Given that the M&Ms evaluation split was used to select several components of the controller design, can the authors provide a completely independent evaluation in which the fragmentation coordinate, routing policy, bucket actions, and adaptation depths are finalized without access to evaluation outcomes?

What incremental benefit does fragmentation provide beyond reducing the number of adapted cases? Can the authors report matched-rollback and matched-computation comparisons on additional independent benchmarks, particularly given the null ranking results on Prostate158 and ACDC?

How would the method perform under a genuinely deployment-oriented selection protocol that allows the calibration procedure to choose no adaptation as a valid global action? In particular, what practical advantage does the router provide on M&Ms when the source model already achieves essentially identical Dice with zero HA and no adaptation steps?

2 comments
AuthorsOpenReview ↗

Response to Reviewer AC84

We thank the reviewer for the careful reading; answers follow in order.

W1 / Q1. Independent evaluation, all design choices fixed in advance. With everything frozen, the HA selection effect reproduces on new vendors C+D; the Dice benefit does not. Recalibration does not remove the dependence; 0.129 to 0.013 will be labelled a selected-design result. The new evaluation on vendors C and D fixes coordinate, policy, actions, depths and cut-points from vendor B, criteria written before the run: HA 0.016 vs 0.193 for the deployable fixed-2 (-0.177 [-0.220, -0.137]); no Dice difference from M0 detected (+0.00001 [-0.00008, +0.00011]); Dice below fixed-2 (-0.0026 [-0.0054, -0.0002]). Limits: C and D are new centres of the same dataset, with earlier source-model and TENT diagnostics but no router results.

W2 / Q2. Rejection vs the incremental value of the signal. Matched quotas equalise deployed depth, not signal-computation cost. Random assignment is the direct control (router vs random HA; share of random assignments at least as good): M&Ms vendor B 0.013 vs 0.061, 0%; vendors C+D 0.016 vs 0.068, 0%; Prostate OOD-all 0.099 vs 0.133, 1.0%; Prostate158 (cut-points refit on 32 cases; common depths 3/2/0, no shrink; deployed HA 0.139) 0.161 vs 0.164, 39%; ACDC substitute control beats random only in fog (0.5%; rain/snow/night 37%/93%/9%). So the signal adds to rejection on HA for cardiac data and on Prostate OOD-all; not on Prostate158 at common depths or on three of four ACDC conditions. Much of the cardiac advantage is that the router retains cases whose edits stay below the scoring size (C+D: scorable edit in 10% of cases vs 26% under random; HA among scorable cases 0.162 vs 0.265; exploratory split). On net errors relative to M0 (M0 = 0, helpful edits below zero; class-aware on M&Ms, binary elsewhere; M&Ms/Prostate158 at depths 3/2/0 without shrink) the router is not better than random anywhere (M&Ms/C+D/Prostate158/OOD-all: 80%/96%/92%/98%).

W3. What HA measures. HA is the share of edited area in changed regions that reach the size threshold and are net harmful, not the fraction of harmful edits. On M&Ms (case level, step 1, subject bootstrap) HA correlates with harmful/edited voxels at 0.66 [0.56, 0.74], with the class-aware harmful fraction at 0.64 [0.53, 0.73], with Dice decrease at 0.58 [0.46, 0.69] and with HD95 increase at 0.35 [0.24, 0.45]. The size threshold matters most: in 202 of 591 observations with HA = 0 net binary errors increased, in 86 of 329 with HA > 0 they decreased. HA is a useful but coarse summary.

W4 / Q3. Benefit, and no adaptation as a valid action. On M&Ms 130 of 230 cases are rolled back and 100 keep an adapted checkpoint; 17 is the number of those 100 with a scorable edit. Against the deployable budget, rollback removes harmed and helped cases at about its base rate (harmed rolled back / helped rolled back / rollback rate / net Dice effect): M&Ms fixed-2 63% / 53% / 57% / -0.0013; Prostate OOD-all fixed-1 41% / 53% / 45% / -0.0130; Prostate158 fixed-3 35% / 29% / 31% / -0.0022. "Change-suppressor rather than utility-predictor" is accurate and we will adopt it. With no adaptation as an option, no Dice benefit of the router over M0 is detected on M&Ms; in an exploratory five-split protocol across three benchmarks (prostate arm: OOD-hard), calibration Dice never selects k = 0 among k in {0,...,4} (15 of 15 cells; +0.0010 on M&Ms, +0.0035 on Prostate158 over M0). FragSubset was run with its rules frozen on the 126 Prostate158 evaluation cases (criteria fixed in advance; a previously used benchmark); its prespecified advantage was not established: Dice over M0 +0.0029 [-0.0020, +0.0074]; against FragStop, Dice +0.0009 [-0.0011, +0.0030], HA +0.029 [0.003, 0.059] (+0.027 [-0.0001, 0.057] against a second execution of that comparator; no multiplicity correction).

W5. Generality across objectives. The medical experiments use episodic variants of EATA, CoTTA and SAR. Across six GraTa/DeYO configurations, one meets the correlation band; matched random reproduces 69-86% of the HA reduction. This does not establish a general risk ranker.

Shared corrections to the submission. (i) Selected-design labelling, as in W1. (ii) The entropy and MSP baselines read a probability map with its in-plane axes swapped (masks, Dice, HA, delta, n_reg unaffected). Corrected: M&Ms Table S8 entropy HA 0.0537 to 0.1186; the printed MSP 0.0567 used the opposite rollback orientation, 0.0572 is the bug-affected value in the stated orientation, corrected to 0.1221; both exceed every matched random assignment. Prostate OOD-all: corrected selectors reach HA 0.163 vs 0.099 for the router, Dice +0.013 [0.002, 0.025] and net errors -0.044 [-0.084, -0.007] relative to it; HA/Dice from archived actions, net errors from corrected-rerun masks. (iii) FragSubset transfer test: see W4. (iv) Revision: overview figure, terms defined at first use, M0 and matched random as primary comparators.

Thanks for the new experiments -- the frozen C+D evaluation settles my test-set selection concern, and the HA gain over matched random routing is real if incremental.

Still, the key question remains: does routing prevent meaningful case-level failures? Mean Dice on C+D is flat versus the source and below fixed-2, and the class-aware net-error analysis doesn't beat random routing either. So the HA story is stronger, but I'd rather see fewer large per-case Dice drops or class-specific errors, plus the actual adaptation cost. Rating stays at 2.

Reviewer_FfeSOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on the reliability aspect in episodic test-time adaptation (TTA) in which the model is reset to initial source weights. The authors argue that stable accuracy can hide catastrophic adaptation failures in individual cases. To address this, the authors introduce harmful accepted area (HA) to measure harmful prediction changes, and propose prediction fragmentation based on how much the original and adapted masks geometrically disagree as a label-free way to indicate risky adaptations, and their proposed framework decides whether to adapt or roll back. The authors evaluated their method on segmentation benchmarks including cardiac (M&Ms), prostate, and ACDC and showed a HA drop e.g. (0.129 to 0.013) on M&Ms with stable Dice scores. However, the authors acknowledge performance trade-offs and generalization limits.

Critical strengths

The authors are much more transparent about their limitations than in typical submissions we see these days, which to me is a strong positive point and makes the work informative. Also, looking beyond accuracy in TTA is important and they tackle overlooked problems in this paper.

Critical weaknesses

First, I have to admit the paper is not readable and there are ambiguities in the draft.

  • My first comment is related to contradiction with respect to the main claim. In Appendix S10, it is written that the routers were tuned using the M&Ms test split, which heavily biases that reported headline 90% HA reduction. When tested independently on Prostate158, the fragmentation ranking performed no better than random assignment. This makes me doubt whether fragmentation is better at case-selection than simpler baselines.

  • The harm metric (HA) and the fragmentation signal both rely on the same thresholded disagreement components. If a case has no components large enough to hit the threshold, then both metrics score a zero, no? More importantly, the fragmentation signal doesn't actually predict case-level Dice drops well, if I am not wrong. On M&Ms, the router somehow matches the unadapted source model. It seems the method is good at predicting its own specific metric, but struggles to consistently improve the actual segmentation quality.

Questions for authors
  • Can you show a frozen routing policy, without access to the final evaluation split, could outperform random or confidence-based selection at matched rollback rates or not? Also what exactly caused the ranking advantage to vanish on Prostate158, despite a HA reduction? Authors need to clarify these points.

  • If you remove cases with zero disagreement, how much fragmentation-HA correlation is actually left? Would your conclusions hold if harm was evaluated using independent clinical metrics or simple pixel-level errors?

  • On M&Ms, the router matches the unadapted source model. Could you compare this against a calibrated baseline with no adaptation as an option?

1 comment
AuthorsOpenReview ↗

Response to Reviewer FfeS

We thank the reviewer for the careful reading; answers follow in order.

Q1. Frozen evaluation and matched controls.

Short answer: yes for HA selection. With every design choice frozen, the router on new vendors C+D beats all 10,000 matched random assignments (HA 0.016 vs 0.068); this does not extend to Dice or net errors. We agree that the vendor-B design was selected with its evaluation split visible (Appendix S10), so 0.129 to 0.013 is not independent validation.

On vendors C+D (200 frames, 100 subjects) we carried over the vendor-B policy and cut-points unchanged, with criteria written before the run; C/D labels were used only for scoring. Earlier M0/TENT predictions existed there; no router had been run. All selectors use the same actions and bucket sizes (109/43/48).

SelectorHADice
Frozen router0.016 [0.006, 0.029]0.8279
Matched random0.068 [0.049, 0.088]0.8286
Entropy0.1350.8294
MSP0.1370.8294
Source model M000.8279

Random brackets: 2.5/97.5 percentiles of 10,000 assignments. Paired HA differences: -0.118 [-0.161, -0.079] vs entropy, -0.121 [-0.163, -0.080] vs MSP. Two of five prespecified criteria passed, both on HA selection. On net errors, 96% of random assignments are at least as good. Router Dice does not differ detectably from M0 (+0.00001 [-0.00008, 0.00011]); vs fixed-2, HA falls by 0.177 but Dice by 0.0026 [-0.0054, -0.0002].

On Prostate158 the design was frozen but cut-points were refit on 32 calibration cases. The matched-random test uses common depths 3/2/0 without the middle-bucket shrink: HA 0.161 vs 0.164, 39% of random assignments at least as good. No ranking advantage is established there.

A descriptive explanation: on C+D 10% of cases carry a scorable deployed edit vs 26% under random; an exploratory, order-dependent decomposition attributes 49-80% of the gap to scorable frequency. On Prostate158 every case is scorable at steps 1-3, and step-1 rho(delta,HA) is 0.12 [-0.06, 0.29]. This does not cover the deployed shrink policy.

Q2. Removing zeros; independent harm measures.

The association remains, but weakens substantially. No M&Ms observation has zero disagreement; the relevant filter is whether any changed component reaches 16 voxels, which 41% of pooled observations fail.

rho with HA (M&Ms, subject bootstrap)AllScorable only
delta, pooled0.52 [0.43, 0.59]0.20 [0.06, 0.33]
n_reg, pooled0.60 [0.53, 0.66]0.135 [0.003, 0.257]
delta, step 10.49 [0.39, 0.59]0.26 [0.11, 0.40]

Among scorable step-1 cases augmentation disagreement (0.18) and gradient norm (0.16) are of similar size.

Without component grouping or size thresholds, step-1 delta correlates with harmful/edited voxels at -0.05 [-0.20, 0.11], class-aware net error change at -0.09 [-0.26, 0.08], Dice decrease at -0.10 [-0.26, 0.07] and HD95 increase at -0.20 [-0.33, -0.04]. On Prostate OOD-all its correlation with net error change is -0.32 [-0.54, -0.07], again opposite in sign. A positive harm-ranking signal is not established on these endpoints.

Q3. Calibration with no adaptation available.

In an exploratory five-split protocol on three benchmarks (prostate arm: OOD-hard), calibration Dice selects k>0 in all 15 cells when k=0,...,4 is allowed; gains over M0 are +0.0010 (M&Ms) and +0.0035 (Prostate158). On vendor B neither fixed-2 (+0.0015 [-0.0001, 0.0031]) nor the router (+0.00001) establishes a Dice gain over M0; fixed budgets gain on the prostate benchmarks and pooled C+D. We withdraw the implication that the cardiac router is preferable to M0.

Shared corrections to the submission. (i) Per Appendix S10, the coordinate, middle action and low-bucket depth were chosen with the vendor-B evaluation split visible; 0.129 to 0.013 will be labelled a selected-design result, not the headline. (ii) The entropy and MSP baselines read a probability map with its in-plane axes swapped (masks, Dice, HA, delta, n_reg unaffected). Corrected: M&Ms Table S8 entropy HA 0.0537 to 0.1186; the printed MSP 0.0567 used the opposite rollback orientation, 0.0572 is the bug-affected value in the stated orientation, corrected to 0.1221; both exceed every matched random assignment. Prostate OOD-all (printed 0.1044/0.1030 reproduce from the affected signals): corrected selectors reach HA 0.163 vs 0.099 for the router, Dice +0.013 [0.002, 0.025] and net errors -0.044 [-0.084, -0.007] relative to it, no difference from adapting every case detected; HA/Dice from archived actions, net errors from corrected-rerun masks. (iii) Frozen transfer test on the previously evaluated Prostate158 cohort (126 cases, criteria fixed in advance): FragSubset did not establish its prespecified advantage, Dice vs M0 +0.0029 [-0.0020, +0.0074], vs FragStop +0.0009 [-0.0011, +0.0030]. (iv) Revision: overview figure, terms defined at first use, M0 and matched random as primary comparators, HA beside voxel-level measures.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.