Dreaming Deeper: Stable Recurrent Medical Image Segmentation Beyond Training Rollouts

Medical image segmentation in resource-constrained settings requires compact models that preserve accuracy under practical inference budgets. Recurrent refinement allows such models to increase test-time computation without adding parameters, but refiners trained on short rollouts can lose accuracy at greater depths. We attribute this limitation to a mismatch between recurrent states encountered during training and extended inference. We introduce VD-Refine, a compact U-Net-style architecture with virtual-depth (VD) supervision to bridge this gap. VD geometrically extrapolates late feature states from a short training rollout, avoiding the forward cost of their long prefixes. Initial-state-conditioned FiLM normalization controls channel-moment drift, while multi-start training supervises corrections at several real depths. VD-Refine outperforms compact baselines across BraTS, LiTS, and DRIVE. A single checkpoint supports multiple inference depths. On BraTS, VD reduces mean case-level late degradation, while a separate patch-level diagnostic shows stable accuracy far beyond the training rollout. Our analytic inference directly decodes an extrapolated state, approaching 32-step BraTS accuracy at about half the FLOPs. Our anonymous project page: https://anonymous-vd-refine.github.io/VD-Refine.