StreamRefine: Learning When to Stop by Occasionally Continuing in Medical Image Translation

3D medical image translation synthesizes a target modality while preserving patient anatomy. Iterative refinement can improve predictions, but useful refinement varies across modality pairs and cases, motivating learned stopping. However, when stopping also truncates training trajectories, it hides the outcomes of later refinements and removes their direct supervision. Poorly trained later refinements can then reinforce premature stopping. In this work, we propose StreamRefine, which learns when to stop without stopping refinement learning. It models translation as an autoregressive stream of 3D refinement states conditioned on the source and preceding model predictions. Our Continue-to-Check training occasionally continues beyond proposed stops. These continuations train otherwise skipped refinements and teach a lightweight head whether further refinement is still worthwhile, balancing translation improvement, anatomical drift, and computation cost. At inference, the head decides when to stop without additional checks or ground truth targets. We evaluate StreamRefine across diverse 3D medical translation tasks for translation quality, anatomy-related performance, and computational efficiency. Further analyses show that occasional continuation strengthens later refinements, contributing to translation quality gains and better stopping decisions.