acceptodds
Under review as a conference paper at ICLR 2027

Dreaming Deeper: Stable Recurrent Medical Image Segmentation Beyond Training Rollouts

Abstract

Medical image segmentation in resource-constrained settings requires compact models that preserve accuracy under practical inference budgets. Recurrent refinement allows such models to increase test-time computation without adding parameters, but refiners trained on short rollouts can lose accuracy at greater depths. We attribute this limitation to a mismatch between recurrent states encountered during training and extended inference. We introduce VD-Refine, a compact U-Net-style architecture with virtual-depth (VD) supervision to bridge this gap. VD geometrically extrapolates late feature states from a short training rollout, avoiding the forward cost of their long prefixes. Initial-state-conditioned FiLM normalization controls channel-moment drift, while multi-start training supervises corrections at several real depths. VD-Refine outperforms compact baselines across BraTS, LiTS, and DRIVE. A single checkpoint supports multiple inference depths. On BraTS, VD reduces mean case-level late degradation, while a separate patch-level diagnostic shows stable accuracy far beyond the training rollout. Our analytic inference directly decodes an extrapolated state, approaching 32-step BraTS accuracy at about half the FLOPs. Our anonymous project page: https://anonymous-vd-refine.github.io/VD-Refine.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
medical image segmentation, recurrent refinement, virtual-depth supervision, test-time compute

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bthUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies recurrent refinement for compact medical segmentation networks, where a weight-tied block is applied K times to a decoder feature and K can be chosen at test time. Refiners trained on short rollouts tend to lose accuracy at larger K. The proposed VD-Refine trains with eight-step rollouts and adds virtual-depth supervision: from the states at steps 4, 6 and 8 it fits a geometric decay, extrapolates a state for a nominal depth of up to 30, and supervises two real refinement steps from that state. This is combined with a normalization whose scale and shift depend on the initial state (FiLM), two-step training windows from several real depths, and a five-phase training schedule. The same extrapolation gives an "analytic" inference mode that decodes the limit of the fitted series after eight steps. On BraTS, LiTS and DRIVE, a 4.4M-parameter model reaches 85.3, 79.7 and 82.4 Dice at K=32, slightly above nnU-Net, U-Mamba and other baselines. On BraTS, adding virtual depth raises Dice at K=32 from 84.7 to 85.3 and changes the difference between K=8 and K=32 from -0.14 to +0.07 points. A patch-level test shows stable Dice up to K=1024.

Critical strengths
  1. The idea is simple and well motivated. Late states of a recurrent refiner are estimated from three states of a short rollout, and only two real steps are back-propagated from there, so supervision at larger depths adds little training cost. The same construction gives a second inference mode for free.
  2. The analysis is careful and candid. The paper reports state norms and update sizes up to K=1024, perturbation recovery, the error of the extrapolated states against real ones, anchor sensitivity and paired case-level intervals. It also says where its evidence is limited, for example the intervals that include zero in Table 5, the patch-level nature of the K=1024 result and the heavier degradation tails on LiTS.
  3. The method and schedule are specified in detail (Algorithm 1, Appendix A), and an anonymous repository with code, fixed splits and per-case reference results is provided. The evaluation covers 3D MRI, 3D CT and 2D fundus images and reports parameters, FLOPs and runtime.
Critical weaknesses
  1. The evidence that virtual-depth supervision is what makes deeper inference work is limited. Its effect is tested on BraTS only, by comparing A9 with A10 in Table 2. Most of the 0.53-point gain at K=32 is already present at K=8 (84.87 versus 85.19), inside the real training rollout, so it is not specifically a late-depth effect. The control that the method is meant to replace is missing: training from real late states obtained by a gradient-free rollout, with its accuracy and training time. By Table 1, one Refiner step costs about 10 G FLOPs against about 170 G for the rest of the network, so this cost may be modest, and it is never measured. With the decay reported in Table 7 (0.746 per step), the fitted series is 90% complete at depth 16, so virtual starts sampled between 15 and 30 are nearly the same state. On LiTS, where no model without virtual depth is reported, Dice drops by 0.11 points from K=8 to K=32 (Table 5).
  2. The paper targets resource-constrained settings, but efficiency is argued with parameters and FLOPs. In the measured runtime of Table 1, nnU-Net (23 ms) is two to three times faster than VD-Refine (52 to 68 ms), and no memory or CPU measurement is given. Depth scaling offers little beyond the training rollout: on BraTS, going from K=8 to K=32 doubles the FLOPs for +0.07 points. At small K the model is worse than the same backbone without a Refiner (83.2 versus 84.7 at K=0 on BraTS, and about 82.0 versus 82.3 up to K=8 on DRIVE), so the usable range of depths is narrow. Analytic inference costs the same as K=8, and its difference from K=8 is not significant on BraTS and LiTS (Table 5), so being close to 32-step accuracy at half the FLOPs holds equally for plain K=8. The only significant gain is on DRIVE, and the repository's reference curve for DRIVE (82.33 at K=8, 82.35 analytic) does not show it.
Questions for authors
  1. How were the baselines trained (framework, number of epochs, patch size, test-time augmentation, checkpoint selection)? nnU-Net's liver Dice on LiTS (91.5) is lower than usually reported. How were the dataset-specific choices made (updating the backbone in the B phases for LiTS and DRIVE, final versus best checkpoint, augmentation), and on which split?

  2. Could the authors train with real late starts (gradient-free rollouts to depths 14 to 30) and report accuracy and training time next to virtual depth, and also with copied or noise-perturbed d8 as the start? How does the method compare with training from fixed points found by an accelerated solver, as in the deep equilibrium models cited in Section 2? Is there a result without virtual depth on LiTS and DRIVE?

  3. In Table 2, virtual depth raises Dice at K=8 by about 0.3 points, inside the training rollout. What explains this? What values does q take during training, and does the sampled depth matter once the series has saturated?

4 comments

Thank you for the careful review and for suggesting that we measure the training cost directly. The main changes are marked in blue in the revised paper. Tables were renumbered in the revision; where you cited a table, we also give its number in the submission.

On BraTS, we compared four ways to construct a late training start: running the Refiner to that depth, or estimating the state from the eight-step prefix in three ways. Appendix C.6 (Table 8) reports Dice and the time per training step for all four. We also measured the decay factor qq at four points during training (Appendix C.6) and added no-VD controls on LiTS and DRIVE (Appendix B.3, Table 4).

Weakness 1 & Q2: Real late starts and alternative starts, with accuracy and training time

All four variants start from the same 50-epoch Stage-A checkpoint. In Table 8, the Geometric row is the full model and reports the mean of three training runs; Copy, Linear and Exact are single runs. Only the detached start differs: Exact continues the real rollout from d8d_8 to the sampled depth ρ\rho (up to 30), while Copy, Linear and Geometric use only the eight-step prefix. Start-depth sampling, branch weights, the two supervised correction steps and the rest of the schedule are the same. We did not train from a noise-perturbed d8d_8 or from a solver-derived fixed point.

StartReal prefixDice@8 (%)Dice@32 (%)Δ8→32\Delta_{8\rightarrow32} (pp)B1 step (ms)B2 step (ms)
Copy885.1785.17+0.01501.75±0.62501.75\pm0.62501.47±0.45501.47\pm0.45
Linear885.0985.14+0.04502.00±0.79502.00\pm0.79501.66±0.35501.66\pm0.35
Exact8–3085.2085.24+0.04503.74±2.28503.74\pm2.28510.89±7.38510.89\pm7.38
Geometric885.1985.26+0.07502.24±0.55502.24\pm0.55502.19±0.64502.19\pm0.64

Differences are computed from unrounded values.

Accuracy. At K=32K=32, Geometric (85.26%) and Exact (85.24%) differ by 0.02 pp, and Copy and Linear are 0.09 and 0.12 pp lower. These gaps are within the run-to-run SD of the full model (0.19 pp at K=32K=32, Table 1), and Copy, Linear and Exact were trained once, so they do not show that geometric extrapolation improves final Dice. All four variants exceed the no-VD model A9 (84.73%, single run), including Copy, which uses no extrapolation. Most of the gain over A9 therefore appears to come from the added late-start supervision branch, not from the particular start construction. The construction matters more for how closely the start matches the real state: on the full model, the mean relative feature error at depth 30 is 0.139 for Geometric, 0.216 for Copy and 0.688 for Linear (Figure 5).

Training cost. From a cached d8d_8, reaching the deepest start (ρ=30\rho=30) takes 22 gradient-free Refiner updates, 26.30±0.1526.30\pm0.15 ms in total, or 26.0% of a K=0K=0 forward pass (101.02±0.14101.02\pm0.14 ms). Over a full training step including the optimizer update (H100 PCIe; matched weights and optimizer state; 3 warm-ups, 20 repetitions), Exact takes 1.50 ms longer than Geometric in B1 and 8.69 ms longer in B2, so Geometric is only about 0.3% and 1.7% faster per step. As you expected, this cost is modest at our depths. Geometric reuses d4,d6,d8d_4,d_6,d_8, so its cost does not depend on the start depth; Exact needs ρ−8\rho-8 sequential updates per sample, so its cost grows with the target depth.

Inference. Decoding the analytic estimate instead of running 32 real steps reduced BraTS batch-1 patch latency from 68.44 to 51.82 ms (−24.3%), with full-volume Dice of 85.24% versus 85.26% (Table 1).

Two further revisions are used in the replies below: full-volume BraTS inference now runs through K=256K=256 (Appendix B.4, Figure 10), and Table 1 now also reports the no-Refiner LiteMB-UNet as a three-run mean ± SD (the submission gave a single run).

Q3 & Weakness 1: Why does VD raise Dice at K=8K=8, and what is its late-depth effect?

The A9→A10 gain at K=32K=32 is not purely a late-depth effect. In Table 2, adding VD raises BraTS Dice at K=8K=8 from 84.87% to 85.19% (+0.32 pp) and turns the K=8→32K=8\rightarrow32 change from −0.14 to +0.07 pp (+0.21 pp); together these give the 0.53 pp difference at K=32K=32. A10 is the mean of three training runs; A9 is a single run.

Source of the K=8K=8 gain. The two supervised updates after a detached late start train the same Refiner that is used at K=8K=8, so this branch also changes shallow predictions. The late-start controls (Table 8) point to the branch itself rather than to geometric extrapolation: Dice at K=8K=8 is 85.09–85.20% for all four starts, including Copy, which reuses d8d_8, compared with 84.87% for A9. Copy, Linear, Exact and A9 are single runs, so this is indicative rather than conclusive. Figure 13 (Appendix C.5) fits this reading. After Stage A, Refiner gradients from real two-step windows starting at d1d_1, d3d_3 and d6d_6 are closely aligned (median cosine 0.962 and 0.966 for neighboring windows, 0.887 for early versus late), so supervision at later states likely moves the shared parameters in a direction that also helps shallow predictions. Figure 13 measures gradients on real windows, not on the virtual branch.

Late-depth effect at the case level (Table 5). For each of 74 BraTS volumes, we took the largest Dice drop below its own eight-step score over the listed depths (zero if none). Values are percentage points, without VD → with VD.

Inference stepsMean95th percentileMaximum
16, 320.234 → 0.0571.426 → 0.3063.010 → 0.733
16, 32, 64, 128, 2560.277 → 0.0601.518 → 0.3404.840 → 0.733

Through 256 steps, the worst drop grew without VD but stayed at 0.733 pp with VD.

On LiTS and DRIVE, the full model (three-run mean) is 3.07 pp (tumor +4.73 pp) and 0.12 pp above a single no-VD run at K=32K=32 (Appendix B.3, Table 4). From K=8K=8 to K=32K=32, the LiTS no-VD run loses 1.46 pp (78.13% → 76.67%), compared with 0.11 pp for the full model.

Q3 & Weakness 1: Values of qq during training, and the sampled depth after saturation

qq during training (new, Appendix C.6). The per-step value of 0.746 that you cited (Table 7 in the submission, now Table 10) was fitted on the 74 held-out BraTS volumes. Here we recomputed qq on the same 64 augmented training patches per dataset at four checkpoints. qq is defined over two steps (u8≈q u6u_8\approx q\,u_6); the table gives medians.

DatasetB1 startB1 endB2 startB2 endPer-step r=qr=\sqrt{q} (B2 end)
BraTS0.56570.54780.55240.55250.743
LiTS0.60660.58110.58170.58120.762
DRIVE0.69910.70740.71150.70550.840

All 768 unclipped estimates lie in [0.5376, 0.7347], below the clipping bound of 0.95, so clipping does not affect the reported values. The medians change little across checkpoints (BraTS stays near 0.55).

Saturation. By start depth ρ\rho, the fitted series has reached a fraction 1−rρ−81-r^{\rho-8} of its limit (Eq. 4): 90.7/99.1/99.9% at depths 16/24/30 on BraTS, 88.6/98.7/99.7% on LiTS and 75.2/93.9/97.8% on DRIVE. As you noted, on BraTS and LiTS the starts beyond about depth 20 are nearly the same state.

Starts are drawn from {8,…,14}\{8,\dots,14\} in B1 and also from {15,…,30}\{15,\dots,30\} in B2 (Algorithm 1; supervised endpoints 10–16 and 17–32). On BraTS, the first range covers 0–83.1% of the fitted displacement and the second only 87.5–99.9%, so the deeper B2 starts add little variety. DRIVE decays more slowly (r=0.840r=0.840) and is still about 2% short of the limit at depth 30. We did not vary the sampling range to measure its effect on training.

Whether the trained Refiner stays stable at larger real depths is a separate question, addressed by Figures 3 and 10 and by 64 further real updates from d8d_8, d32d_{32} and d^∞\hat d_\infty (Appendix C.8).

Weakness 2: Efficiency, usable depth range, and analytic inference

Measured runtime (Table 1, H100, BraTS configuration). nnU-Net takes 23.40 ± 0.06 ms per patch; VD-Refine takes 68.44 ± 1.59 ms at K=32K=32 and 51.82 ± 0.03 ms with analytic inference, so nnU-Net is 2.2–2.9× faster. Our efficiency claim concerns choosing the depth, and using analytic inference, within the same recurrent model. We do not claim to be faster than nnU-Net. Among compact baselines, analytic inference is faster than MedNeXt-S (67.03 ms) and LightM-UNet (254.74 ms) and has higher Dice on all three datasets; UNeXt is much faster (5.37 ms), but its Dice is 1.7–10.6 pp lower than that of analytic inference.

Depth scaling. On BraTS, going from 8 to 32 steps doubles FLOPs for +0.07 pp. Equations 3, 5 and 6 average losses against the same label across sampled depths and do not reward a deeper prediction for beating a shallower one, so training does not ask accuracy to keep rising with depth. What VD changes is how much accuracy is lost at greater depth. On BraTS, it turns the K=8→32K=8\rightarrow32 change from −0.14 to +0.07 pp and reduces the mean per-case drop over K∈{16,32}K\in\{16,32\} from 0.234 to 0.057 pp; over K∈{16,…,256}K\in\{16,\dots,256\} on full volumes, the reduction is from 0.277 to 0.060 pp. Averaged over the 74 held-out volumes, full-volume Dice stays within 0.11 pp of its K=8K=8 value through K=256K=256 (Appendix B.4); the K=1024K=1024 result is patch-level only. On DRIVE, mean Dice rises from 82.01% at K=8K=8 to 82.36% at K=32K=32. On LiTS, the full model changes by −0.11 pp from K=8K=8 to K=32K=32 (Figure 3). A single no-VD run loses 1.46 pp over the same range (78.13% → 76.67%) and is 3.07 pp below the three-run mean of the full model at K=32K=32 (Appendix B.3).

Low KK. You are right that stopping early does not always beat the backbone alone. Table 1 now reports the no-Refiner backbone as a three-run mean: 84.81% on BraTS and 82.18% on DRIVE (the submission gave single-run values of 84.73% and 82.26%). On BraTS, decoding before the first refinement step gives 83.23%, below the backbone, and eight steps reach 85.19%. On DRIVE, eight steps are still below the backbone (82.01% vs. 82.18%); 32 steps reach 82.36%. If the inference budget allows only a few steps, our results do not support choosing VD-Refine over the backbone by default. The stability results measure how much accuracy the model loses when run beyond eight steps, not how well it performs when stopped early.

Analytic inference vs. K=8K=8. In Table 6 (Table 5 in the submission; three-run means, paired over held-out cases), the difference between analytic inference and K=8K=8 is not significant on BraTS (+0.046 pp) or LiTS (+0.101 pp). We agree that, on these two datasets, "near 32-step accuracy at about half the FLOPs" holds equally for plain K=8K=8. Analytic inference estimates the 32-step result at the cost of K=8K=8: 85.24 vs. 85.26% on BraTS, 79.95 vs. 79.73% on LiTS and 82.34 vs. 82.36% on DRIVE, with no significant difference from 32 steps on any of the three datasets. DRIVE is the only dataset where analytic inference is significantly above K=8K=8 (+0.341 pp). These significance tests resample held-out cases and do not include run-to-run variation.

DRIVE values in the repository. We had released an older DRIVE results file by mistake. It has been replaced by results/drive-depth-results.csv, which matches the Table 1 means and standard deviations, and the reproduction notes and the project-page depth curve have been updated.

Q1: Data splits, baseline training protocol, and dataset-specific settings

Splits. We re-checked the splits_final.json files against the test-case IDs and confirmed that the training, validation and test sets are pairwise disjoint on all three datasets.

DatasetTrain/Val/TestSplit
BraTS 2020236/59/7474 of the 369 official labeled cases held out for testing; fold 0 of nnU-Net's five-fold split on the remaining 295
LiTS92/13/26131 official labeled CTs split ~7:1:2 with random seed 2026
DRIVE16/4/20Official 20 training / 20 test images; fold 0 on the official training set for train/val

The DRIVE validation IDs are 24, 29, 34 and 37, matching nnU-Net's five-fold assignment with random_state=12345. LiTS follows the split ratio of LightM-UNet, but not necessarily its case IDs, and is evaluated on 26 internally held-out labeled cases rather than on the 70 official challenge test cases.

nnU-Net on LiTS. We retrained nnU-Net on LiTS three times with different random seeds, and the revised Table 1 reports the mean: liver Dice 92.65% (LiTS average 77.64%), replacing the single-run 91.54% in the submission. VD-Refine remains above nnU-Net on LiTS (79.73% at K=32K=32). Published LiTS results use different test sets or splits, so they are not directly comparable with ours.

Baseline implementations. We re-checked the model code, trainer and training log of each baseline. nnU-Net runs in a local nnU-Net v2 installation; SegResNet uses the MONAI network inside nnU-Net v2; MedNeXt-S uses the official network code inside our training framework; U-Mamba Enc/Bot use the released networks with local trainers; LightM-UNet uses the authors' released implementation. UNeXt is reimplemented from the original paper, with the 2D model on DRIVE and our 3D extension on BraTS and LiTS.

Baseline training settings. On BraTS, nnU-Net was trained for 100 epochs with SGD and PolyLR, without deep supervision, and evaluated with the best checkpoint and TTA. UNeXt was trained for 100 epochs on all three datasets, MedNeXt-S for 100 epochs on BraTS and LiTS, and LightM-UNet for 100 epochs on LiTS with best-checkpoint selection. The baseline training plans use 1283128^3 patches on BraTS and LiTS, 512×512512\times512 patches on DRIVE and a batch size of 2; the shared trainer runs 250 training and 50 validation minibatches per epoch. Augmentation consists of random rotation, scaling, mirroring, Gaussian noise and blur, brightness and contrast changes, low-resolution simulation and gamma; rotation and scaling are each applied with probability 0.2, with a scaling range of 0.7–1.4.

VD-Refine training and checkpoint selection. On BraTS, only the shared Refiner core is updated during the B phases. On LiTS and DRIVE, the encoder, decoder and prediction head are also updated, FiLM stays frozen, and a K=0K=0 loss keeps supervising the initial proposal. For BraTS and DRIVE we use the checkpoint with the highest exponentially averaged validation pseudo-Dice at K=8K=8; for LiTS we use the final checkpoint. Within each run, this checkpoint is used at every inference depth. We did not run a separate comparison of alternative B-phase update rules, or of best versus final checkpoints.

TTA. VD-Refine uses TTA on BraTS and DRIVE but not on LiTS (Section 4.1): on large LiTS CT volumes, TTA multiplies the number of sliding-window passes per volume. On LiTS, VD-Refine is evaluated without TTA at every depth and in both inference modes.

Reviewer_g5cWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper proposes VD-Refine, a lightweight recurrent U-Net-style medical image segmentation model.
  • The model repeatedly applies a shared Refiner to increase test-time computation without increasing parameter count.
  • To avoid degradation when inference goes beyond the short rollout used during training, the authors introduce:
    • Virtual-depth supervision: geometrically extrapolate late feature states and train local corrections from those estimated states.
    • Initial-state-conditioned FiLM normalization: reduce feature-statistic drift across recurrent steps.
    • Multi-start supervision: train corrections from several real states within the short rollout.
  • Experiments are conducted on BraTS, LiTS, and DRIVE, with comparisons against compact and larger segmentation baselines.
  • The paper reports improved average Dice, reduced late-depth degradation, and an analytic inference mode that approximates deeper real rollouts at lower FLOPs.
  • The main evidence is based on ablations within the proposed lightweight U-Net family, full-cohort segmentation results, and additional patch-level feature-dynamics diagnostics.
Critical strengths
  • The paper identifies a real failure mode in recurrent refinement.
    Short-rollout training can cause accuracy degradation at deeper inference depths. The proposed virtual-depth supervision is a plausible and technically coherent response to this problem.

  • The ablation and diagnostic analysis are reasonably thorough.
    The paper separates the effects of normalization, multi-start training, and virtual-depth supervision, and evaluates both average accuracy and case-level late degradation. This provides some evidence that VD contributes beyond the overall training schedule.

Critical weaknesses
  • The practical accuracy improvement is small.
    On BraTS, the main benefit beyond the eight-step rollout is only about 0.07 Dice points from K=8 to K=32. The paper mainly demonstrates preservation of performance rather than a substantial improvement from additional inference computation. The gains over strong compact baselines are also modest in some settings.

  • The scope and generality of the contribution are insufficiently established.
    The method is evaluated primarily within one lightweight U-Net-style architecture. It is unclear whether the proposed virtual-depth supervision is broadly useful or mainly effective for this particular recurrent decoder and training recipe. The 1024-step stability claim is based on a patch-level diagnostic rather than complete-case evaluation, and the reported runtime advantage is less compelling than the FLOP reduction suggests.

Questions for authors
  • How many independent training runs and random seeds were used for each method and ablation? Please report comparable variance or confidence intervals for all competing baselines, especially where the improvements are below one Dice point.

  • Did you compare VD supervision against training with the same compute budget but with real late states, longer rollouts, or other late-state approximations? This comparison would clarify whether geometric extrapolation is specifically responsible for the observed gains.

  • Does VD-Refine retain its advantage when inserted into a substantially different segmentation architecture, such as a transformer- or state-space-based encoder-decoder? If not evaluated, please clarify whether the intended contribution is architecture-specific.

  • The paper reports patch-level stability through 1024 steps. How does the method behave on complete volumes or cases at these depths, particularly for the cases showing the largest late-depth degradation?

5 comments

Thank you for suggesting that we look beyond patch averages and check what happens to individual volumes as inference runs for more steps. The main changes in the revised paper are marked in blue.

We extended inference to 256 steps on all held-out BraTS volumes, compared Dice changes on patches with those on full volumes, and measured the largest Dice drop of each case with and without virtual-depth supervision (Appendix B.4). We also compared alternative starting states during training and related early changes in update direction to errors in our state estimates (Appendix C.6).

Accuracy beyond eight steps

On BraTS, Dice rose by only 0.07 percentage points (pp), from 85.19% at K=8K=8 to 85.26% at K=32K=32 (Table 2). The more useful difference is that individual cases lost less Dice at greater depth with VD than without it. Multi-start and VD supervision apply the same segmentation loss to predictions from different starts (Eqs. 3, 5–6) and do not reward a deeper prediction for beating a shallower one, so the training targets stability across depth rather than continued gains.

For each of the 74 held-out BraTS volumes, we measured the largest drop below the same model's K=8K=8 Dice across K∈{16,32,64,128,256}K\in\{16,32,64,128,256\}, counting no drop as zero (Table 5). Without and with VD, the mean drop was 0.277 and 0.060 pp; the 95th percentile fell from 1.518 to 0.340 pp, and the largest drop from 4.840 to 0.733 pp. Over K=16K=16 and 3232 only, the means were 0.234 and 0.057 pp.

From K=8K=8 to 3232, VD-Refine rose from 82.01% to 82.36% on DRIVE but lost 0.11 pp on LiTS (Figure 3). The LiTS model without VD lost 1.46 pp (78.13% to 76.67%). At K=32K=32, Table 4 compares the full model's three-run mean with a single no-VD run per dataset: 79.73% vs. 76.67% on LiTS (+3.07 pp overall, +4.73 pp for tumor) and 82.36% vs. 82.24% on DRIVE (+0.12 pp).

With analytic inference, VD-Refine is 0.259 pp above MedNeXt-S on BraTS, 3.306 pp on LiTS and 0.147 pp on DRIVE, with 20.7% fewer parameters in the BraTS configuration (4.40M). On BraTS and DRIVE these margins are at most about 1.6 times VD-Refine's across-run SD (0.16 and 0.21 pp), and MedNeXt-S was trained once, so we do not claim an advantage over MedNeXt-S there; the LiTS margin is about three times the SD (1.08 pp). Adding the Refiner also did not always help: on DRIVE, 82.01% at K=8K=8 is below the backbone alone (82.18%).

Q1: Independent training runs and uncertainty

LiteMB-UNet and VD-Refine were each trained three times per dataset with different seeds (Table 1, mean ± SD). The full model, reported as A10 in Table 2 and as Geometric in Table 8, and the paired comparisons in Table 6 also use three-run means. The other ablations (A1–A9), the Copy, Linear and Exact controls in Table 8 and the LiTS/DRIVE no-VD controls are single runs, so comparisons between them and the full model set a single run against a three-run mean. The revised LiTS nnU-Net score averages three seeds; the other baselines were trained once, and we cannot report across-seed uncertainty for them. Within each run, the same checkpoint is used at all depths and in both inference modes. The CIs in Tables 5 and 6 resample held-out cases and do not include run-to-run variation.

Q2: Real late states, longer rollouts, and alternative starts

We did not match total training FLOPs or wall-clock time, and we did not train a separate model with a longer rollout throughout training. On BraTS, we compared four ways of constructing the start state while keeping the supervision, sampled depths and training schedule the same (§4.4; Appendix C.6, Table 8).

All four variants start from the same epoch-50 checkpoint (end of Stage A) and follow the same 100-epoch schedule. In Table 8, Geometric is the full model and reports the mean of three training runs; Copy, Linear and Exact are single runs. In B1 the start depth ρ\rho is sampled uniformly from 8–14; in B2 it is sampled from 8–14 or 15–30 with probabilities 0.6 and 0.4 (Algorithm 1). Each variant detaches the start state, takes two real updates and supervises the prediction at S=ρ+2S=\rho+2. Copy reuses d8d_8; Linear uses Pd0[d8+ρ−82(d8−d6)]P_{d_0}[d_8+\frac{\rho-8}{2}(d_8-d_6)]; Geometric uses Eq. 4; Exact runs the Refiner from d8d_8 to ρ\rho without gradients to obtain the real state.

StartReal prefixDice@8 (%)Dice@32 (%)Δ8→32\Delta_{8\rightarrow32} (pp)B1 step (ms)B2 step (ms)
Copy885.1785.17+0.01501.75±0.62501.75\pm0.62501.47±0.45501.47\pm0.45
Linear885.0985.14+0.04502.00±0.79502.00\pm0.79501.66±0.35501.66\pm0.35
Exact8–3085.2085.24+0.04503.74±2.28503.74\pm2.28510.89±7.38510.89\pm7.38
Geometric885.1985.26+0.07502.24±0.55502.24\pm0.55502.19±0.64502.19\pm0.64

Dice changes use unrounded scores.

At K=32K=32, all four variants score above the no-VD model A9 (84.73%, single run), including Copy, which uses no extrapolation. Geometric scores 85.26% and Exact 85.24%; the 0.09 and 0.12 pp margins over Copy and Linear are within the full model's run-to-run SD (0.19 pp), and the three controls were trained once, so they do not show that geometric extrapolation improves Dice. Most of the gain over A9 therefore appears to come from supervising two extra updates from a late start, whichever way that start is built. Geometry does give a closer approximation of the real state: on the full model, the mean relative feature error against d30d_{30} is 0.139 for Geometric, 0.216 for Copy and 0.688 for Linear (Figure 5).

Starting from a cached d8d_8, Exact needs 22 sequential updates to reach d30d_{30} (26.30±0.1526.30\pm0.15 ms, or 26.0% of a 101.02 ms K=0K=0 forward pass); Geometric needs none. Over a full training step, however, Exact takes only 1.50 ms longer in B1 and 8.69 ms longer in B2 (H100 PCIe, matched weights and optimizer state, 3 warm-ups, 20 repetitions), so Geometric shortens a step by about 0.3% and 1.7%. We did not measure end-to-end training time.

What the late-start branch trains

The two updates after the detached start train the same Refiner used at K=8K=8, so supervision from a late start also changes shallow predictions. This matches Table 8: Dice at K=8K=8 is 85.09–85.20% for all four starts versus 84.87% for A9. Figure 13 (Appendix C.5) shows that Refiner gradients from real early (d1,d3d_1,d_3), middle (d3,d5d_3,d_5) and late (d6,d8d_6,d_8) windows are closely aligned (median cosine 0.962 and 0.966 for neighboring windows, 0.887 for early versus late). It does not measure gradients from the virtual branch.

In Appendix C.6 we estimated qq, the decay factor over two updates, on 64 augmented training patches per dataset at four checkpoints. The table gives the median at each checkpoint and the per-update factor r=qr=\sqrt{q} at the end of B2.

DatasetB1 startB1 endB2 startB2 endB2-end r=qr=\sqrt{q}
BraTS0.56570.54780.55240.55250.743
LiTS0.60660.58110.58170.58120.762
DRIVE0.69910.70740.71150.70550.840

All 768 raw estimates lie within [0.5376,0.7347][0.5376,0.7347], so none is clipped, and the medians change little across checkpoints.

By depth ρ\rho, the geometric start covers a fraction 1−rρ−81-r^{\rho-8} of its estimated displacement. With the B2-end medians, depths 16, 24 and 30 cover 90.7%, 99.1% and 99.9% on BraTS; 88.6%, 98.7% and 99.7% on LiTS; and 75.2%, 93.9% and 97.8% on DRIVE. On BraTS, the B1 starts cover 0–83.1% of the displacement and the extra B2 range (15–30) only 87.5–99.9%, so sampling deeper there adds little variety. This describes the training starts; behavior under more real steps at inference is shown in Figures 3 and 10 and Appendix C.8.

Q3: Does VD transfer to other architectures?

We did not test VD in a Transformer or state-space encoder–decoder. BraTS (3D MRI), LiTS (3D CT) and DRIVE (2D fundus images) all use the same LiteMB-UNet family with a weight-tied Refiner, so the experiments vary the dataset and imaging modality but not the architecture family.

The geometric estimate uses the changes in the Refiner state over a short rollout to approximate a later state. Another decoder could also reuse a block across steps, but weight sharing alone does not tell us whether the same extrapolation would approximate its later states, or whether training from those estimates would improve segmentation. Our claims are therefore limited to weight-tied recurrent decoders of the kind evaluated here; we do not claim that VD transfers to other architecture families without further experiments.

Full-volume accuracy and inference time on BraTS

We ran all 74 held-out BraTS volumes through K=256K=256 and compared them with the 500 fixed patches of the patch-level diagnostic. No full volume was run at K=1024K=1024, including the cases with the largest Dice drops.

Mean Dice (%) at selected depths (Appendix B.4, Figure 10):

ScopeK=8K=8K=32K=32K=64K=64K=128K=128K=256K=256Δ\Delta: 256 vs. 8 (pp)
500 fixed patches85.06685.15285.16885.16985.170+0.104
74 full volumes85.19385.25885.27885.27685.282+0.089

At every measured depth through K=256K=256, both means stay within 0.11 pp of their own K=8K=8 values. Individual volumes change more: across K∈{16,32,64,128,256}K\in\{16,32,64,128,256\}, the largest drop below a volume's K=8K=8 score is 0.733 pp with VD and 4.840 pp without it (Table 5). From K=8K=8 to K=256K=256, the mean change on a case's patches correlates with the change on its full volume (Pearson r=0.675r=0.675; Figure 10b). On patches, mean Dice stays between 85.15% and 85.18% at the sampled depths from K=32K=32 to K=1024K=1024.

BraTS runtime. On an H100, replacing 32 real steps with analytic inference reduces FLOPs from 494.26 to 251.09 G and batch-1 patch latency from 68.44 to 51.82 ms (−24.3%), with full-volume mean Dice of 85.26% and 85.24%, respectively (Table 1). On the 10 prespecified BraTS cases, analytic inference saves 0.87 s per volume over 32 real steps (4.01 vs. 4.88 s; −17.8%) and takes about as long as eight real steps (4.00 s). These times include reading, preprocessing, inference, resampling and export (Appendix D.2, Table 11).

Compared with nnU-Net. Our BraTS model has 4.40M parameters against nnU-Net's 31.20M, but both of our inference modes take 2.2–2.9 times as long per patch as nnU-Net (23.40 ms, Table 1). Our efficiency claim concerns choosing the depth within the same model, not speed relative to nnU-Net.

Reviewer_E3mdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VD-Refine, a compact recurrent segmentation model trained to remain useful beyond its short training rollouts. A shared refiner updates a decoder feature state, with channel statistics controlled by initial-state-conditioned FiLM. The main addition is virtual-depth supervision: a geometric estimate constructed from states at depths 4, 6, and 8 initializes two supervised correction steps. Training alternates updates to the refiner and the surrounding network. The same checkpoint supports real recurrent inference and direct decoding of an extrapolated state. Experiments cover BraTS, LiTS, and DRIVE, with detailed BraTS ablations and trajectory diagnostics.

Critical strengths
  1. The BraTS ablation shows a measurable benefit from adding virtual-depth supervision.
  2. The inference options are evaluated carefully. The continuation experiment also measures residual feature motion after extrapolation, keeping the interpretation of analytic inference distinct from an exact recurrent fixed point.
Critical weaknesses
  1. Geometric extrapolation is not isolated from additional supervision. A9 versus A10 shows that adding the VD branch helps, but Figure 5’s evaluation-only comparisons do not establish that geometric extrapolation drives this improvement. Matched training controls using copied, linear, and exact late starts, together with training-cost measurements, are needed to establish its accuracy–cost advantage.
  2. The failure modes of geometric extrapolation remain unclear. Equation (4) assumes aligned, geometrically decaying updates through a single clipped coefficient, but the paper does not examine how violations of these assumptions affect the estimated states.
Questions for authors

See weakness.

3 comments

We thank the reviewer for asking us to isolate geometric extrapolation from additional supervision and examine its failure modes. A9–A10 does not isolate these effects; we added matched controls, timings, and assumption analysis in Appendix C.6 (blue).

  • Accuracy. With matched supervision, Geometric gives the highest Dice of the four late starts, matching Exact and exceeding Copy and Linear.
  • Cost. Geometric needs no Refiner updates beyond the eight-step prefix; Exact's cost grows linearly with the start depth.
  • Failure modes. In training, an assumption violation mainly shifts where the Refiner practices. In analytic inference, the directly decoded estimate matches 32 real steps on all three datasets.

1. Isolation from additional supervision

Controls. The four runs share depth sampling, branch weights, and two supervised correction steps, differing only in the start: Copy reuses d8d_8; Linear repeats the last displacement; Geometric uses Eq. (4); Exact continues the real gradient-free rollout to dρd_\rho.

StartReal prefixDice@8Dice@32Δ8→32 (pp)B1 step (ms)B2 step (ms)
Copy885.1785.17+0.01501.75501.47
Linear885.0985.14+0.04502.00501.66
Exact8–3085.2085.24+0.04503.74510.89
Geometric885.1985.26+0.07502.24502.19

Accuracy. Without VD (A9) the model reaches 84.73% at K=32K=32 (Table 2); all four VD variants exceed 85.1%. Geometric reaches the highest Dice@32 and the largest 8→32 gain: +0.09 pp over Copy, +0.12 pp over Linear, on par with Exact (+0.02 pp).

State fidelity. At ρ=30\rho=30 (Figure 5), relative feature error is 0.139 for Geometric, 0.216 for Copy, and 0.688 for Linear.

Cost. Exact needs ρ−8\rho-8 extra sequential Refiner updates per sample ($\approx$1.2 ms each; 26.3 ms to reach d30d_{30}). This adds 1.50 ms per training step in B1 and 8.69 ms in B2, where deeper starts are sampled; Geometric's step time stays flat (502.24 vs 502.19 ms). The overhead is modest at our depths ($\le$1.7%) but grows with start depth and Refiner cost. At inference, the same estimator cuts BraTS patch latency from 68.44 ms (32 real steps) to 51.82 ms ($-$24.3%; Table 1).

In conclusion, geometric extrapolation matches Exact's accuracy, improves state fidelity over Copy and Linear, and avoids extra Refiner updates.

2. Failure modes of the geometric assumption

Eq. (4) is used in training and in analytic inference; a violation affects them differently.

Training: the estimate decides where the Refiner practices, not its target. The extrapolated state is a detached start, z0=stopgrad(Pd0(d~ρ))z_0=\mathrm{stopgrad}(P_{d_0}(\tilde d_\rho)), followed by two real updates z2=Rθ2(z0)z_2=R_\theta^2(z_0) supervised by segmentation labels (Eq. 5). No gradient flows through extrapolation. A violation shifts the start away from the true deep state without changing the target or update rule. Geometric starts train as well as exact starts (Section 1).

Inference: the estimate is decoded directly. Analytic inference fits qq from d4,d6,d8d_4,d_6,d_8 and decodes the limit d^∞\hat d_\infty with no further Refiner steps (Eq. 7), so its accuracy directly tests the assumption. It matches 32 real steps on all three datasets:

BraTSLiTSDRIVE
Real K=32K=32, Dice (%)85.2679.7382.36
Analytic d^∞\hat d_\infty, Dice (%)85.2479.9582.34

How violations affect the estimated state. Eq. (4) assumes aligned updates that shrink at a constant rate. From the first eight real steps (uk=dk−dk−2u_k=d_k-d_{k-2}) we measure direction change 1−cos⁡(u6,u8)1-\cos(u_6,u_8) and decay-rate change ∣∥u8∥/∥u6∥−∥u6∥/∥u4∥∣\big|\|u_8\|/\|u_6\|-\|u_6\|/\|u_4\|\big| on one held-out patch per BraTS case. Both correlate strongly with the error at d30d_{30} (Spearman 0.940 and 0.798): larger departures give less accurate estimates. By tertile of direction change, relative error against the same real d32d_{32}:

Direction changenDirect d^32\hat d_{32}Two real updates from d^30\hat d_{30}
Low250.1170.080
Medium240.1320.091
High250.1740.130

In every group, the two real updates move the state closer to the real trajectory (all 74 cases). The coefficient qq never reached either clipping bound, so clipping was inactive here.

In conclusion, assumption violations increase estimation error, but two real updates reduce it in all 74 cases. Analytic inference also matches 32-step accuracy across all three datasets.

We thank the reviewer again for these insightful comments.

Thanks to the author's efforts, most of my issues have been resolved.

We sincerely thank the reviewer for reconsidering and updating the rating!

We greatly appreciate your time, constructive feedback, and support, which have helped us improve our paper!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.