Should This Case Be Adapted? Prediction Fragmentation Controls Test-Time Adaptation

Episodic test-time adaptation (TTA) resets a frozen segmenter to source weights M₀ on each case and adapts for a fixed number of steps. A fixed horizon conflates a cohort-level question, how far to adapt, with an irreducibly per-case one, whether this case should be adapted at all. Cohort means average over that per-case decision: on cross-vendor cardiac MRI the mean ΔDice from adaptation is statistically indistinguishable from zero while 58.7% of cases are individually made worse. We quantify this harm as harmful accepted area (HA), the harmful fraction of the edited area a controller actually deploys. Held-out tuning gives a stronger baseline than a fixed horizon, but the budget it selects does not transfer on either of the two main medical benchmarks, and no global budget can condition on the case. We show that prediction fragmentation—the disagreement geometry between M₀ and the adapted mask Mₖ—predicts HA with no labels and no additional backward passes at decision time, at comparable strength on three benchmarks (Spearman ρ 0.50–0.60), at a quarter of gradient-norm's latency. A case-level router built on it cuts HA from 0.228 to 0.139 on a benchmark that took no part in its design, with that design frozen and only its cut-points recalibrated there. On the cardiac benchmark the design was selected on, the same router cuts HA from 0.129 to 0.013 at matched Dice and 1.10 deployed updates, against the retrospective-best budget identified post hoc from evaluation labels, and reduces that 58.7% to 20.0%—an upper bound we quantify. Where the retained cases are not net-helped (as on prostate), the router still cuts HA but concedes accuracy, a boundary we report explicitly. Thresholds are calibrated once on a labeled split disjoint from evaluation; deployment decisions use no labels and no gradients. The template ports across architecture and domain (nnU-Net→SegFormer, Cityscapes→ACDC) with coordinate, thresholds and per-bucket actions instantiated per domain.