- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies recurrent refinement for compact medical segmentation networks, where a weight-tied block is applied K times to a decoder feature and K can be chosen at test time. Refiners trained on short rollouts tend to lose accuracy at larger K. The proposed VD-Refine trains with eight-step rollouts and adds virtual-depth supervision: from the states at steps 4, 6 and 8 it fits a geometric decay, extrapolates a state for a nominal depth of up to 30, and supervises two real refinement steps from that state. This is combined with a normalization whose scale and shift depend on the initial state (FiLM), two-step training windows from several real depths, and a five-phase training schedule. The same extrapolation gives an "analytic" inference mode that decodes the limit of the fitted series after eight steps. On BraTS, LiTS and DRIVE, a 4.4M-parameter model reaches 85.3, 79.7 and 82.4 Dice at K=32, slightly above nnU-Net, U-Mamba and other baselines. On BraTS, adding virtual depth raises Dice at K=32 from 84.7 to 85.3 and changes the difference between K=8 and K=32 from -0.14 to +0.07 points. A patch-level test shows stable Dice up to K=1024.
Critical strengths
- The idea is simple and well motivated. Late states of a recurrent refiner are estimated from three states of a short rollout, and only two real steps are back-propagated from there, so supervision at larger depths adds little training cost. The same construction gives a second inference mode for free.
- The analysis is careful and candid. The paper reports state norms and update sizes up to K=1024, perturbation recovery, the error of the extrapolated states against real ones, anchor sensitivity and paired case-level intervals. It also says where its evidence is limited, for example the intervals that include zero in Table 5, the patch-level nature of the K=1024 result and the heavier degradation tails on LiTS.
- The method and schedule are specified in detail (Algorithm 1, Appendix A), and an anonymous repository with code, fixed splits and per-case reference results is provided. The evaluation covers 3D MRI, 3D CT and 2D fundus images and reports parameters, FLOPs and runtime.
Critical weaknesses
- The evidence that virtual-depth supervision is what makes deeper inference work is limited. Its effect is tested on BraTS only, by comparing A9 with A10 in Table 2. Most of the 0.53-point gain at K=32 is already present at K=8 (84.87 versus 85.19), inside the real training rollout, so it is not specifically a late-depth effect. The control that the method is meant to replace is missing: training from real late states obtained by a gradient-free rollout, with its accuracy and training time. By Table 1, one Refiner step costs about 10 G FLOPs against about 170 G for the rest of the network, so this cost may be modest, and it is never measured. With the decay reported in Table 7 (0.746 per step), the fitted series is 90% complete at depth 16, so virtual starts sampled between 15 and 30 are nearly the same state. On LiTS, where no model without virtual depth is reported, Dice drops by 0.11 points from K=8 to K=32 (Table 5).
- The paper targets resource-constrained settings, but efficiency is argued with parameters and FLOPs. In the measured runtime of Table 1, nnU-Net (23 ms) is two to three times faster than VD-Refine (52 to 68 ms), and no memory or CPU measurement is given. Depth scaling offers little beyond the training rollout: on BraTS, going from K=8 to K=32 doubles the FLOPs for +0.07 points. At small K the model is worse than the same backbone without a Refiner (83.2 versus 84.7 at K=0 on BraTS, and about 82.0 versus 82.3 up to K=8 on DRIVE), so the usable range of depths is narrow. Analytic inference costs the same as K=8, and its difference from K=8 is not significant on BraTS and LiTS (Table 5), so being close to 32-step accuracy at half the FLOPs holds equally for plain K=8. The only significant gain is on DRIVE, and the repository's reference curve for DRIVE (82.33 at K=8, 82.35 analytic) does not show it.
Questions for authors
-
How were the baselines trained (framework, number of epochs, patch size, test-time augmentation, checkpoint selection)? nnU-Net's liver Dice on LiTS (91.5) is lower than usually reported. How were the dataset-specific choices made (updating the backbone in the B phases for LiTS and DRIVE, final versus best checkpoint, augmentation), and on which split?
-
Could the authors train with real late starts (gradient-free rollouts to depths 14 to 30) and report accuracy and training time next to virtual depth, and also with copied or noise-perturbed d8 as the start? How does the method compare with training from fixed points found by an accelerated solver, as in the deep equilibrium models cited in Section 2? Is there a result without virtual depth on LiTS and DRIVE?
-
In Table 2, virtual depth raises Dice at K=8 by about 0.3 points, inside the training rollout. What explains this? What values does q take during training, and does the sampled depth matter once the series has saturated?
4 comments
Q3 & Weakness 1: Why does VD raise Dice at , and what is its late-depth effect?
The A9→A10 gain at is not purely a late-depth effect. In Table 2, adding VD raises BraTS Dice at from 84.87% to 85.19% (+0.32 pp) and turns the change from −0.14 to +0.07 pp (+0.21 pp); together these give the 0.53 pp difference at . A10 is the mean of three training runs; A9 is a single run.
Source of the gain. The two supervised updates after a detached late start train the same Refiner that is used at , so this branch also changes shallow predictions. The late-start controls (Table 8) point to the branch itself rather than to geometric extrapolation: Dice at is 85.09–85.20% for all four starts, including Copy, which reuses , compared with 84.87% for A9. Copy, Linear, Exact and A9 are single runs, so this is indicative rather than conclusive. Figure 13 (Appendix C.5) fits this reading. After Stage A, Refiner gradients from real two-step windows starting at , and are closely aligned (median cosine 0.962 and 0.966 for neighboring windows, 0.887 for early versus late), so supervision at later states likely moves the shared parameters in a direction that also helps shallow predictions. Figure 13 measures gradients on real windows, not on the virtual branch.
Late-depth effect at the case level (Table 5). For each of 74 BraTS volumes, we took the largest Dice drop below its own eight-step score over the listed depths (zero if none). Values are percentage points, without VD → with VD.
| Inference steps | Mean | 95th percentile | Maximum |
|---|---|---|---|
| 16, 32 | 0.234 → 0.057 | 1.426 → 0.306 | 3.010 → 0.733 |
| 16, 32, 64, 128, 256 | 0.277 → 0.060 | 1.518 → 0.340 | 4.840 → 0.733 |
Through 256 steps, the worst drop grew without VD but stayed at 0.733 pp with VD.
On LiTS and DRIVE, the full model (three-run mean) is 3.07 pp (tumor +4.73 pp) and 0.12 pp above a single no-VD run at (Appendix B.3, Table 4). From to , the LiTS no-VD run loses 1.46 pp (78.13% → 76.67%), compared with 0.11 pp for the full model.
Q3 & Weakness 1: Values of during training, and the sampled depth after saturation
during training (new, Appendix C.6). The per-step value of 0.746 that you cited (Table 7 in the submission, now Table 10) was fitted on the 74 held-out BraTS volumes. Here we recomputed on the same 64 augmented training patches per dataset at four checkpoints. is defined over two steps (); the table gives medians.
| Dataset | B1 start | B1 end | B2 start | B2 end | Per-step (B2 end) |
|---|---|---|---|---|---|
| BraTS | 0.5657 | 0.5478 | 0.5524 | 0.5525 | 0.743 |
| LiTS | 0.6066 | 0.5811 | 0.5817 | 0.5812 | 0.762 |
| DRIVE | 0.6991 | 0.7074 | 0.7115 | 0.7055 | 0.840 |
All 768 unclipped estimates lie in [0.5376, 0.7347], below the clipping bound of 0.95, so clipping does not affect the reported values. The medians change little across checkpoints (BraTS stays near 0.55).
Saturation. By start depth , the fitted series has reached a fraction of its limit (Eq. 4): 90.7/99.1/99.9% at depths 16/24/30 on BraTS, 88.6/98.7/99.7% on LiTS and 75.2/93.9/97.8% on DRIVE. As you noted, on BraTS and LiTS the starts beyond about depth 20 are nearly the same state.
Starts are drawn from in B1 and also from in B2 (Algorithm 1; supervised endpoints 10–16 and 17–32). On BraTS, the first range covers 0–83.1% of the fitted displacement and the second only 87.5–99.9%, so the deeper B2 starts add little variety. DRIVE decays more slowly () and is still about 2% short of the limit at depth 30. We did not vary the sampling range to measure its effect on training.
Whether the trained Refiner stays stable at larger real depths is a separate question, addressed by Figures 3 and 10 and by 64 further real updates from , and (Appendix C.8).
Weakness 2: Efficiency, usable depth range, and analytic inference
Measured runtime (Table 1, H100, BraTS configuration). nnU-Net takes 23.40 ± 0.06 ms per patch; VD-Refine takes 68.44 ± 1.59 ms at and 51.82 ± 0.03 ms with analytic inference, so nnU-Net is 2.2–2.9× faster. Our efficiency claim concerns choosing the depth, and using analytic inference, within the same recurrent model. We do not claim to be faster than nnU-Net. Among compact baselines, analytic inference is faster than MedNeXt-S (67.03 ms) and LightM-UNet (254.74 ms) and has higher Dice on all three datasets; UNeXt is much faster (5.37 ms), but its Dice is 1.7–10.6 pp lower than that of analytic inference.
Depth scaling. On BraTS, going from 8 to 32 steps doubles FLOPs for +0.07 pp. Equations 3, 5 and 6 average losses against the same label across sampled depths and do not reward a deeper prediction for beating a shallower one, so training does not ask accuracy to keep rising with depth. What VD changes is how much accuracy is lost at greater depth. On BraTS, it turns the change from −0.14 to +0.07 pp and reduces the mean per-case drop over from 0.234 to 0.057 pp; over on full volumes, the reduction is from 0.277 to 0.060 pp. Averaged over the 74 held-out volumes, full-volume Dice stays within 0.11 pp of its value through (Appendix B.4); the result is patch-level only. On DRIVE, mean Dice rises from 82.01% at to 82.36% at . On LiTS, the full model changes by −0.11 pp from to (Figure 3). A single no-VD run loses 1.46 pp over the same range (78.13% → 76.67%) and is 3.07 pp below the three-run mean of the full model at (Appendix B.3).
Low . You are right that stopping early does not always beat the backbone alone. Table 1 now reports the no-Refiner backbone as a three-run mean: 84.81% on BraTS and 82.18% on DRIVE (the submission gave single-run values of 84.73% and 82.26%). On BraTS, decoding before the first refinement step gives 83.23%, below the backbone, and eight steps reach 85.19%. On DRIVE, eight steps are still below the backbone (82.01% vs. 82.18%); 32 steps reach 82.36%. If the inference budget allows only a few steps, our results do not support choosing VD-Refine over the backbone by default. The stability results measure how much accuracy the model loses when run beyond eight steps, not how well it performs when stopped early.
Analytic inference vs. . In Table 6 (Table 5 in the submission; three-run means, paired over held-out cases), the difference between analytic inference and is not significant on BraTS (+0.046 pp) or LiTS (+0.101 pp). We agree that, on these two datasets, "near 32-step accuracy at about half the FLOPs" holds equally for plain . Analytic inference estimates the 32-step result at the cost of : 85.24 vs. 85.26% on BraTS, 79.95 vs. 79.73% on LiTS and 82.34 vs. 82.36% on DRIVE, with no significant difference from 32 steps on any of the three datasets. DRIVE is the only dataset where analytic inference is significantly above (+0.341 pp). These significance tests resample held-out cases and do not include run-to-run variation.
DRIVE values in the repository. We had released an older DRIVE results file by mistake. It has been replaced by results/drive-depth-results.csv, which matches the Table 1 means and standard deviations, and the reproduction notes and the project-page depth curve have been updated.
Q1: Data splits, baseline training protocol, and dataset-specific settings
Splits. We re-checked the splits_final.json files against the test-case IDs and confirmed that the training, validation and test sets are pairwise disjoint on all three datasets.
| Dataset | Train/Val/Test | Split |
|---|---|---|
| BraTS 2020 | 236/59/74 | 74 of the 369 official labeled cases held out for testing; fold 0 of nnU-Net's five-fold split on the remaining 295 |
| LiTS | 92/13/26 | 131 official labeled CTs split ~7:1:2 with random seed 2026 |
| DRIVE | 16/4/20 | Official 20 training / 20 test images; fold 0 on the official training set for train/val |
The DRIVE validation IDs are 24, 29, 34 and 37, matching nnU-Net's five-fold assignment with random_state=12345. LiTS follows the split ratio of LightM-UNet, but not necessarily its case IDs, and is evaluated on 26 internally held-out labeled cases rather than on the 70 official challenge test cases.
nnU-Net on LiTS. We retrained nnU-Net on LiTS three times with different random seeds, and the revised Table 1 reports the mean: liver Dice 92.65% (LiTS average 77.64%), replacing the single-run 91.54% in the submission. VD-Refine remains above nnU-Net on LiTS (79.73% at ). Published LiTS results use different test sets or splits, so they are not directly comparable with ours.
Baseline implementations. We re-checked the model code, trainer and training log of each baseline. nnU-Net runs in a local nnU-Net v2 installation; SegResNet uses the MONAI network inside nnU-Net v2; MedNeXt-S uses the official network code inside our training framework; U-Mamba Enc/Bot use the released networks with local trainers; LightM-UNet uses the authors' released implementation. UNeXt is reimplemented from the original paper, with the 2D model on DRIVE and our 3D extension on BraTS and LiTS.
Baseline training settings. On BraTS, nnU-Net was trained for 100 epochs with SGD and PolyLR, without deep supervision, and evaluated with the best checkpoint and TTA. UNeXt was trained for 100 epochs on all three datasets, MedNeXt-S for 100 epochs on BraTS and LiTS, and LightM-UNet for 100 epochs on LiTS with best-checkpoint selection. The baseline training plans use patches on BraTS and LiTS, patches on DRIVE and a batch size of 2; the shared trainer runs 250 training and 50 validation minibatches per epoch. Augmentation consists of random rotation, scaling, mirroring, Gaussian noise and blur, brightness and contrast changes, low-resolution simulation and gamma; rotation and scaling are each applied with probability 0.2, with a scaling range of 0.7–1.4.
VD-Refine training and checkpoint selection. On BraTS, only the shared Refiner core is updated during the B phases. On LiTS and DRIVE, the encoder, decoder and prediction head are also updated, FiLM stays frozen, and a loss keeps supervising the initial proposal. For BraTS and DRIVE we use the checkpoint with the highest exponentially averaged validation pseudo-Dice at ; for LiTS we use the final checkpoint. Within each run, this checkpoint is used at every inference depth. We did not run a separate comparison of alternative B-phase update rules, or of best versus final checkpoints.
TTA. VD-Refine uses TTA on BraTS and DRIVE but not on LiTS (Section 4.1): on large LiTS CT volumes, TTA multiplies the number of sliding-window passes per volume. On LiTS, VD-Refine is evaluated without TTA at every depth and in both inference modes.
Thank you for the careful review and for suggesting that we measure the training cost directly. The main changes are marked in blue in the revised paper. Tables were renumbered in the revision; where you cited a table, we also give its number in the submission.
On BraTS, we compared four ways to construct a late training start: running the Refiner to that depth, or estimating the state from the eight-step prefix in three ways. Appendix C.6 (Table 8) reports Dice and the time per training step for all four. We also measured the decay factor q at four points during training (Appendix C.6) and added no-VD controls on LiTS and DRIVE (Appendix B.3, Table 4).
Weakness 1 & Q2: Real late starts and alternative starts, with accuracy and training time
All four variants start from the same 50-epoch Stage-A checkpoint. In Table 8, the Geometric row is the full model and reports the mean of three training runs; Copy, Linear and Exact are single runs. Only the detached start differs: Exact continues the real rollout from d8 to the sampled depth ρ (up to 30), while Copy, Linear and Geometric use only the eight-step prefix. Start-depth sampling, branch weights, the two supervised correction steps and the rest of the schedule are the same. We did not train from a noise-perturbed d8 or from a solver-derived fixed point.
Differences are computed from unrounded values.
Accuracy. At K=32, Geometric (85.26%) and Exact (85.24%) differ by 0.02 pp, and Copy and Linear are 0.09 and 0.12 pp lower. These gaps are within the run-to-run SD of the full model (0.19 pp at K=32, Table 1), and Copy, Linear and Exact were trained once, so they do not show that geometric extrapolation improves final Dice. All four variants exceed the no-VD model A9 (84.73%, single run), including Copy, which uses no extrapolation. Most of the gain over A9 therefore appears to come from the added late-start supervision branch, not from the particular start construction. The construction matters more for how closely the start matches the real state: on the full model, the mean relative feature error at depth 30 is 0.139 for Geometric, 0.216 for Copy and 0.688 for Linear (Figure 5).
Training cost. From a cached d8, reaching the deepest start (ρ=30) takes 22 gradient-free Refiner updates, 26.30±0.15 ms in total, or 26.0% of a K=0 forward pass (101.02±0.14 ms). Over a full training step including the optimizer update (H100 PCIe; matched weights and optimizer state; 3 warm-ups, 20 repetitions), Exact takes 1.50 ms longer than Geometric in B1 and 8.69 ms longer in B2, so Geometric is only about 0.3% and 1.7% faster per step. As you expected, this cost is modest at our depths. Geometric reuses d4,d6,d8, so its cost does not depend on the start depth; Exact needs ρ−8 sequential updates per sample, so its cost grows with the target depth.
Inference. Decoding the analytic estimate instead of running 32 real steps reduced BraTS batch-1 patch latency from 68.44 to 51.82 ms (−24.3%), with full-volume Dice of 85.24% versus 85.26% (Table 1).
Two further revisions are used in the replies below: full-volume BraTS inference now runs through K=256 (Appendix B.4, Figure 10), and Table 1 now also reports the no-Refiner LiteMB-UNet as a three-run mean ± SD (the submission gave a single run).