- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper is about TTA for image segmentation. They found that TTA can make harmful edits. Figure 1 show that averages can hide damage to individual cases (mIOU fine but HA not). Plus more changes, and more scattered ones, go with higher HA. Built upon this finding, they use a label-free signal to measure how much the mask changed and propose a per-case router. The results show that their router can decrease HA.
Critical strengths
- The question is good and should be studied in medical cases.
- The results in table3&4 show that their router can decrease HA.
Critical weaknesses
- The paper is Difficult to follow despite careful reading. This contains 3 aspects: 1st, terms and prior work are used without being explained. GraTa-cos, appears with no definition. It took me a while to know it comes from a prior work. The paper introduces a large set of its own words and uses them in previous section without define them in words, like COQR; FragStop;FragSubset;All the Tier like A,A′,B,C; TENT8; retrospective; deployable. Several of them are defined only in a table footnote or in the appendix, makes it hard to follow in main text. 2nd, there is no figure that shows the overview. Most of the description are in appendix. 3rd, AI-Interpretation is often defensive. Honesty about limits is good. However, the limits here are scattered across main text and appendix, and the reader is never told in one place what the paper actually shows. After careful reading I cannot say whether the authors think the signal predicts harm, or the router is useful, or only that the question exists.
- HA gives free points for changing little or nothing, and the router benefits from this rather than spotting harmful TTA. For HA, small changes are not counted, and a case with no changes scores 0. The router undo almost all changes and thus could obtain huge HA decrease. This is shown very clear in table34, where mIOU almost has no change and HA has small increase. The kept changes are not better, only fewer. This matters because the paper's main claim is that the signal detects harmful adaptation.
Questions for authors
Questions:
- Prediction using only measurable cases. Please report how well each signal predicts HA using only cases where at least one change is large enough to be counted.
- Please compare the router and others under a harm measure that does not reward changing nothing (also random selection of the same number of undone cases should be compared).
- I read several appendix tables and found that (a)the router isn't always better than RANDOM; (b) the proposed method seems not useful in some cases. here are examples: for (a), tableS9, it seems comparable to random?; for (b), tableS5, "ours" is worse with lower Dice and higher HA compared to MEMO and CoTTA.
- Sec. 7 concludes "risky adaptation is easier to identify than beneficial adaptation". This seems not tested in the main paper, where is it come from?
- The information in table caption isn't helpful for readers to capture the claims. examples: table5, why the 3 numbers are textbf? what is rolled back? "which is what hides the 58.7%" what does this mean?.It is best to clear these information in caption.
Suggestions (also affect my score):
- Please add a method figure to the main text.
- State the conclusions plainly.
- Explain prior work and terms where they were first used.
2 comments
Thank you for running the additional experiments. I’m keeping my score at 2. The authors mostly acknowledge my weakness and questions (like rebuttal of W1,Q3,Q4).
HA at matched quota the router beats random on 3/4 medical benchmarks. On the voxel-level metric, however, the results do not establish an advantage over matched random ("on the voxel-level measure below it does not"). Can the router tell which TTA edits are worth keeping?
In the rebuttal, the authors mostly acknowledge the writing issue, however, I didn't see a modified version of the paper. Since "risky adaptation is easier to identify than beneficial adaptation" is still in the paper(along with others). The concern about writing is not resolved. I also still find the paper difficult to follow.
Response to Reviewer Hs4x
We thank the reviewer for the careful reading; answers follow in order.
W1. Readability. We accept all three points. In words: the source model predicts (M0); one adaptation step gives a second prediction; the disagreement between the two masks is summarised by two label-free quantities, the share of changed voxels (delta) and the number of changed regions (n_reg); frozen cut-points on one of them map each case to deploy, shrink or roll back. Ground truth enters only the offline evaluation, never the online decision; label visibility during design: Appendix S10 and the corrections below. It shows that the problem exists (on vendor B fixed-4 changes mean Dice by +0.0001 [-0.0012, +0.0014] while 135 of 230 cases fall; replicated in pooled vendors C+D, not used for router design, though earlier M0/TENT diagnostics existed) and that HA is a reasonable but coarse measure of it. It does not establish a positive harm ranking by the signal on voxel-level measures, nor that the router is useful compared with not adapting on cardiac data.
W2 / Q2. HA rewards changing little; alternative measure; matched random. Short answer: on HA at matched quota the router beats random on three of four medical benchmarks; on the voxel-level measure below it does not. That measure: net error voxels relative to M0 per ground-truth foreground voxel; M0 scores 0 and a helpful edit scores below 0, so abstaining is not optimal by construction (class-aware on M&Ms, binary elsewhere).
Random brackets: 2.5/97.5 percentiles of 10,000 assignments. On HA at the same quota it beats random on cardiac data and Prostate OOD-all, not on Prostate158 at common depths (0.161 vs 0.164). "The kept changes are not better, only fewer" is mostly right: on M&Ms the router retains a scorable edit in 7% of cases vs 21% under random; among scorable cases its HA is lower, but no advantage is established on voxel-level measures.
Q1. Measurable cases only. M&Ms, step-1 signal vs step-1 HA, cases with at least one scorable region (167 of 230; subject bootstrap):
As a top-HA-tertile ranker, delta falls from AUROC 0.74 to 0.59 [0.50, 0.67].
Q3. (a) Yes. In Table S9 the fixed-retain substitute control does not establish better HA than matched random in rain, snow or night, fog being the exception; on Prostate158 no advantage is established at common depths (W2). (b) Also correct for MEMO: on OOD-hard (Table S5) MEMO has higher Dice and lower HA than both of our budget-axis controllers.
Q4. The Sec. 7 sentence rests only on two AUROC columns of Table S14, and the favourable column uses a target sharing its construction with the signal. We will remove it.
Q5. Captions will state the claim, define every bolded entry and term ("rolled back" = the source prediction is deployed), and not depend on text elsewhere.
Shared corrections to the submission. (i) Per Appendix S10, the coordinate, middle action and low-bucket depth were chosen with the vendor-B evaluation split visible; 0.129 to 0.013 will be labelled a selected-design result, not the headline. (ii) The entropy and MSP baselines read a probability map with its in-plane axes swapped (masks, Dice, HA, delta, n_reg unaffected). Corrected: M&Ms Table S8 entropy HA 0.0537 to 0.1186; the printed MSP 0.0567 used the opposite rollback orientation, 0.0572 is the bug-affected value in the stated orientation, corrected to 0.1221; both exceed every matched random assignment. Prostate OOD-all (printed 0.1044/0.1030 reproduce from the affected signals): corrected selectors reach HA 0.163 vs 0.099 for the router, Dice +0.013 [0.002, 0.025] and net errors -0.044 [-0.084, -0.007] relative to it, no difference from adapting every case detected; HA/Dice from archived actions, net errors from corrected-rerun masks. (iii) Frozen transfer test on the previously evaluated Prostate158 cohort (126 cases, criteria fixed in advance): FragSubset did not establish its prespecified advantage, Dice vs M0 +0.0029 [-0.0020, +0.0074], vs FragStop +0.0009 [-0.0011, +0.0030]. (iv) Revision: overview figure; define terms.