acceptodds
Under review as a conference paper at ICLR 2027

StreamRefine: Learning When to Stop by Occasionally Continuing in Medical Image Translation

Abstract

3D medical image translation synthesizes a target modality while preserving patient anatomy. Iterative refinement can improve predictions, but useful refinement varies across modality pairs and cases, motivating learned stopping. However, when stopping also truncates training trajectories, it hides the outcomes of later refinements and removes their direct supervision. Poorly trained later refinements can then reinforce premature stopping. In this work, we propose StreamRefine, which learns when to stop without stopping refinement learning. It models translation as an autoregressive stream of 3D refinement states conditioned on the source and preceding model predictions. Our Continue-to-Check training occasionally continues beyond proposed stops. These continuations train otherwise skipped refinements and teach a lightweight head whether further refinement is still worthwhile, balancing translation improvement, anatomical drift, and computation cost. At inference, the head decides when to stop without additional checks or ground truth targets. We evaluate StreamRefine across diverse 3D medical translation tasks for translation quality, anatomy-related performance, and computational efficiency. Further analyses show that occasional continuation strengthens later refinements, contributing to translation quality gains and better stopping decisions.

Primary area:
generative models
Keywords:
Medical image synthesis, Autoregressive generation, Generative models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DUxzOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents StreamRefine, a framework for adaptive refinement in 3D medical image translation. The method uses Continue-to-Check training, which occasionally continues beyond a proposed stopping point so that later refinement states can still receive supervision. A lightweight stopping head predicts whether further refinement is useful based on translation improvement, anatomical change, and computation cost. The method is evaluated on BraTS 2024, SynthRAD 2025, and AutoPET across multiple translation tasks. The experiments compare StreamRefine with existing translation methods and matched internal baselines, and further analyze the effects of continuation, anatomy-aware stopping, later-step learning, and inference efficiency.

Critical strengths
  1. Continue-to-Check occasionally extends training beyond a proposed stop, so later refinement states remain supervised while the stopping head also receives useful targets. The Naive, Head-only, Refiner-only, and Joint comparisons directly examine these two effects.
  2. Experiments span BraTS, SynthRAD, and AutoPET with multiple translation directions, matched internal baselines, downstream segmentation, and efficiency measurements. The results are consistent across these settings and support the claimed balance between translation quality and computational cost.
Critical weaknesses
  1. Tables 1–2 list only modest improvements over the matched internal baselines, and StreamRefine is not best on every reported metric. The gains are relatively small considering the additional components introduced by the method.
  2. PIR is used to define the anatomy-aware stopping target across several modalities, but its direct validation against independent structural measures is only reported for one BraTS translation direction. Its reliability outside this setting remains less clear.
Questions for authors
  1. How does the stopping policy affect the refinement process during training?
  2. How much of the gain comes from adaptive stopping, and how much comes from later-step supervision?
4 comments
AuthorsOpenReview ↗

Response to Reviewer DUxz (Part 1/2)

We appreciate the reviewer’s recognition of CtC’s dual role in later-refinement and stopping-head training, as well as our broad evaluation and controlled analyses. Some questions, particularly regarding the effects of stopping and later-step supervision, may stem from aspects of our existing analyses that were not sufficiently clear, which we clarify below. We have also extended PIR validation to two additional tasks and further discussed the modest gains and limitations.

Summary of Key Concerns and Main Responses

Key ConcernsMain Responses
Modest improvements over internal baselines (W1)Clarified the joint quality–efficiency benefits and discussed limitations (Appendix F).
PIR reliability beyond the original validation setting (W2)Added independent PIR validation on additional tasks (Appendix E.2).
Effect of stopping on refinement learning (Q1)Clarified how stopping affects later-step supervision and refinement learning (Sec. 4.4; Appendix E.1).
Contributions of adaptive stopping and later-step supervision (Q2)Disentangled their contributions to refinement quality and inference efficiency (Secs. 4.4–4.5).

W1: Modest improvements.

Thank you for highlighting this issue. We agree that the improvements over the matched internal baselines in Tables 1–2 are modest, and that StreamRefine does not achieve the best result on every reported metric. We would like to clarify their value in two respects.

  • The gains span translation fidelity, anatomy-related performance, and inference efficiency. Tables 1–3 and Fig. 4 show a consistent overall trend toward a better balance among these objectives across the evaluated tasks. For instance, on BraTS T2w→T2f, StreamRefine improves both PSNR and downstream Dice over Fixed-K Refine while reducing the average number of refinement steps by approximately 47%. These joint benefits help explain the value of the additional training components. Moreover, inference requires neither continuation checks nor PIR computation, using only a lightweight head to determine when to stop (Sec. 3.4).

  • Controlled analyses explain how the proposed mechanism contributes to these gains. Secs. 4.4–4.5 (Figs. 5–7) on pages 8-9 provide quantitative and qualitative evidence for the roles of later-step supervision and adaptive stopping. Fixed-depth and shared-prefix experiments show that occasional continuation strengthens later refinements, while stopping-regret analyses and step-by-step visualizations show how the learned head selects useful refinements and helps avoid unnecessary anatomical changes. Together, these analyses support the complementary contributions of learning to refine further and learning when to stop.

We have added a discussion of the modest gains and their limitations to Appendix F (page 32). Taken together, the joint benefits and controlled analyses support the practical value of these gains within the evaluated settings. We appreciate your perspective and welcome further discussion and suggestions for strengthening this assessment.


W2: Additional validation of PIR.

Thank you for raising this important concern. To further assess the generalizability of PIR, we have expanded its validation beyond the original BraTS T2w→T2f setting to two additional tasks, covering different translation directions and imaging modalities. The new results are provided in Appendix E.2 (Table 21, page 28), where we examine the agreement between PIR error and two independent segmentation-based structural measures.

Table R1: Independent validation of the PIR structural proxy across three translation tasks.

TaskStructural measureState-level ρ\rhoStep-change ρ\rho
BraTS T2w→T2fDice error0.630.51
HD950.680.54
BraTS T1n→T1cDice error0.600.47
HD950.640.51
AutoPET CT→PETDice error0.550.43
HD950.600.48

The results demonstrate consistent positive associations between PIR and independent structural errors across all evaluated tasks. Although the correlation strengths vary moderately across modalities, the overall trend remains consistent. This additional evidence strengthens the validity of PIR as an anatomy-aware stopping proxy beyond the original BraTS setting.

AuthorsOpenReview ↗

Response to Reviewer DUxz (Part 2/2)

Q1: The effect of stopping policy on the refinement process.

Thank you for this insightful observation. During training, the stopping policy determines which refinement states are executed and receive direct supervision. Naive early stopping truncates training trajectories, leaving later refinements undertrained and potentially reinforcing premature stopping. Our Continue-to-Check training addresses this by occasionally continuing beyond proposed stops, allowing otherwise skipped refinements to receive direct supervision (Sec. 3.3, pages 4–5).

We clarify that we have already analyzed this effect experimentally in the current manuscript:

  • Training supervision (Appendix E.1, Table 20, page 27): Under matched training FLOPs, StreamRefine increases the execution frequency of Step 4 from 15.6% to 28.2% and its direct supervision count from 105k to 169k, compared with Naive Stop.

  • Refinement quality (Sec. 4.4, Fig. 5, pages 7–8): With stopping disabled during evaluation, StreamRefine improves final-step PSNR by 0.35 dB over Naive Stop. The shared-prefix analysis further confirms that most of this improvement comes from stronger later refinements rather than better earlier states.

These analyses demonstrate that the stopping policy affects refinement learning by controlling access to later-state supervision, while occasional continuation prevents useful later refinements from being undertrained. We believe these analyses clarify how the stopping policy affects refinement learning during training. We would also welcome any suggestions for additional analyses that could further illustrate this effect.


Q2: Gains from adaptive stopping and later-step supervision.

We appreciate your helpful comment. Later-step supervision primarily improves refinement quality, whereas adaptive stopping mainly improves inference efficiency while preserving quality. Our current manuscript disentangles these contributions through controlled experiments in Secs. 4.4–4.5 (pages 7–9).

  • Later-step supervision (Sec. 4.4, Fig. 5, page 8): With stopping disabled and refinement depth fixed, Head-only, Refiner-only, and Joint improve PSNR over Naive Stop by 0.07, 0.28, and 0.35 dB, respectively. This indicates that direct later-step supervision accounts for most of the refinement-quality improvement, while improved stopping feedback provides a complementary benefit.

  • Adaptive stopping (Sec. 4.5, page 9): For the same CtC-trained refiner, adaptive stopping reduces the average refinement steps from 4.00 to 2.13, with only a 0.08 dB PSNR decrease compared with full-depth refinement. Thus, its main contribution is improving inference efficiency while largely preserving translation quality.

Overall, these analyses show that later-step supervision primarily improves refinement quality, whereas adaptive stopping mainly improves the quality–efficiency trade-off. We believe our existing experiments separate their respective contributions and welcome suggestions for further analysis.


We thank the reviewer again for the constructive comments, which have helped us strengthen the validation and clarify the contributions of our paper. We hope these responses help clarify the points raised and welcome any further questions or discussion.

Thank you for the detailed response. These address some of my concerns and help clarify the role of each component. I am keeping my original score and raising my confidence to 3.

Thank you for your thoughtful follow-up and for taking the time to reconsider our responses. We appreciate your constructive comment and are glad that our clarifications helped address some of your concerns.

Reviewer_dZFXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents StreamRefine, an adaptive iterative framework for 3D medical image translation. The model represents translation as an autoregressive stream of 3D latent refinement states and learns a lightweight head that decides whether further refinement is worthwhile. Its stopping target, termed Continue Benefit, combines potential translation improvement, estimated anatomical drift, and additional computation cost.

The main proposed training strategy, Continue-to-Check, occasionally overrides a predicted stopping decision and continues the trajectory to the maximum refinement horizon. These continued trajectories provide supervision both for the stopping head and for later refinement states that would otherwise be skipped. The authors further use inverse reach-probability weighting to correct for the lower observation frequency of later states.

The method is evaluated on BraTS 2024, SynthRAD 2025, and AutoPET, covering ten paired 3D translation directions. The evaluation includes reconstruction metrics, FID, downstream segmentation performance, inference cost, ablations, and analyses of later-state learning and stopping decisions. StreamRefine generally provides modest improvements over fixed-horizon and post-hoc stopping baselines while reducing the average number of refinement steps.

Critical strengths
  1. The paper identifies a meaningful interaction between learned stopping and refinement learning The observation that early stopping removes not only information needed to train the stopping policy but also direct supervision for later refinement states is sensible and relevant. Continue-to-Check offers a simple way to partially address both effects using the same continued trajectories. This is a useful perspective on adaptive iterative generation, even though I am less convinced that the resulting technical contribution is sufficiently novel.

  2. The empirical analysis is relatively thorough The paper evaluates multiple datasets, modalities, and translation directions, and includes matched internal baselines using the same backbone. The fixed-depth, shared-prefix, stopping-regret, and hidden-benefit analyses are helpful for separating gains from improved stopping decisions and gains from better-trained later refinements. Evaluating tumor/lesion ROIs and downstream segmentation is also more informative than relying only on global reconstruction metrics.

Critical weaknesses
  1. Limited methodological novelty Continue-to-Check mainly combines stochastic continuation, supervision of otherwise skipped states, and inverse-probability weighting—all closely related to established ideas in randomized truncation and learned stopping. The paper does not clearly show why this formulation is substantially different from simpler stochastic full-rollout training under the same compute budget.

  2. Limited demonstrated significance and generality The improvements over matched internal baselines are generally modest, while the main experiments use a maximum horizon of only four refinement states. The method also requires paired data to construct stopping targets, and PIR is independently validated on only one BraTS translation direction, limiting confidence in broader applicability.

  3. Presentation and clarity Figures 1 and 2 are overly schematic, slogan-driven, and repetitive, giving the presentation a strongly template-like style. The exposition also shifts abruptly among medical motivation, streaming generation, stopping theory, and implementation details, making the precise technical contribution harder to identify.

Questions for authors
  1. How does Continue-to-Check differ substantially from existing stochastic continuation or randomized-truncation methods?
  2. How does StreamRefine compare with full-continuation training under strictly matched training budgets across all datasets?
  3. Does PIR reliably measure anatomical changes beyond the single BraTS translation direction on which it is validated?
  4. Does the method remain effective for substantially longer refinement horizons than (K_max=4)?
5 comments
AuthorsOpenReview ↗

Response to Reviewer dZFX (Part 1/3)

We appreciate the reviewer's recognition of the meaningful interaction between learned stopping and refinement learning, as well as our extensive evaluation and mechanism analyses. To address the concerns, we have clarified CtC's distinction from related methods, added compute-matched and cross-task PIR experiments, and further discussed the practical gains, limitations, and presentation.

Key Concerns and Main Responses

Key ConcernsMain Responses
Methodological novelty and related methods (W1, Q1)Clarified how CtC differs from existing training strategies (Appendix F).
Compute-matched comparison (W1, Q2)Added GPU-hour-matched comparisons with Full Continuation (Appendix D.6).
Modest performance gains (W2)Explained the joint quality–efficiency benefits and discussed their limitations (Secs. 4.4–4.5; Appendix F).
PIR reliability across tasks (W2, Q3)Added independent PIR validation on two additional tasks (Appendix E.2).
Short refinement horizon (W2, Q4)Clarified effectiveness up to eight steps and acknowledged longer-horizon limitations (Appendices D.5 and F).

W1.

  • Differences from existing methods: Please see Q1.

  • Matched-compute budget comparison: Please see Q2.


W2.

  • Modest improvement: Thank you for highlighting this issue. We agree that the improvements over the matched internal baselines are modest, and that StreamRefine is not best on every reported metric. We would like to clarify why we still believe these gains are meaningful within the evaluated settings.

    • The gains are modest, but they are multi-dimensional.

      • Tables 1–3 and Fig. 4 show that the improvements are not limited to a single metric, but reflect a better overall balance among translation fidelity, anatomy-related performance, and inference efficiency.

      • For example, on BraTS T2w→T2f, StreamRefine improves both PSNR and downstream Dice over Fixed-K Refine while reducing the average number of refinement steps by approximately 47%. This joint improvement helps justify the additional components.

    • The paper also explains where these gains come from.

      • Secs. 4.4–4.5 (Figs. 5–7) provide controlled quantitative and qualitative analyses showing that the gains arise from two complementary effects, namely stronger later refinements enabled by occasional continuation and better stopping decisions learned by the stopping head.

      • These analyses help explain why StreamRefine improves the overall quality-efficiency trade-off, rather than only reporting small end-metric differences.

We have also added a brief discussion of the modest gains and their interpretation in Appendix F (page 32). Overall, while the improvements are indeed modest, we believe their consistency across multiple evaluation aspects, together with the mechanism analysis, supports the practical value of the proposed method. We appreciate this perspective and welcome further discussion.

  • Short refinement states: Please see Q4.

  • Paired-data requirement: Thank you for pointing this out. We acknowledge that StreamRefine requires paired source–target data to supervise refinement and construct stopping targets. However, paired supervision is widely adopted in medical image translation, including representative methods [1-6]. Our experiments follow this established setting to ensure fair comparisons. We agree that extending StreamRefine to weakly paired or unpaired settings is valuable but would require alternative supervision for stopping targets. We have added this limitation and future direction to Appendix F (Discussion).

  • Additional validation of PIR: Please see Q3.


W3: Presentation and clarity.
Thank you for your suggestion on presentation and clarity. We would like to clarify these two points below.

  • Figure presentation: These figures serve complementary purposes, where Figure 1 presents the motivation and supporting quantitative evidence, while Figure 2 explains the backbone, stopping objective, and training strategy. Their consistent visual design connects the motivation with the proposed method.

  • Exposition and technical contribution: To clarify the main thread, stopping can also truncate training, hiding later outcomes and leaving later refinements undertrained. Continue-to-Check uses the same occasional continuations to supervise the stopping head and train otherwise skipped refinements. The autoregressive stream provides the refinement framework, while the medical setting motivates the anatomy-aware stopping objective.

AuthorsOpenReview ↗

Response to Reviewer dZFX (Part 2/3)

Q1: Differences from existing methods.

Thank you for this fair and constructive comment. Our proposed CtC differs in how continuation is used, as the same completed trajectory provides missing stopping targets and directly trains later refinements that an early stop would otherwise skip. Randomized truncation primarily targets gradient estimation; CtC focuses on how learned stopping removes both learning signals. We have expanded Appendix F (pages 31, 33) and added the following comparison table.

Table R1: Comparison with related training strategies.

ApproachMain objectiveTraining mechanism
Randomized truncationEfficient, unbiased gradient estimationRandomly truncate computation and reweight sampled contributions.
Learned stopping: PonderNetLearn adaptive computationLearn halting probabilities with prediction losses and a halting prior.
Full Continuation (ϵ=1\epsilon=1)Complete trajectory supervisionAlways complete trajectories and train both the refiner and stopping head.
Stochastic full rolloutSample complete trajectoriesRandomly select full rollouts; supervision and weighting vary.
StreamRefine (CtC)Learn stopping and later refinementOccasionally continue beyond proposed stops to supervise both the stopping head and later refinements, with reach correction.

For stochastic full rollout, the distinction depends on the implementation, where the sampling scheme alone does not specify stopping targets, supervision, or weighting. Our controlled analyses in Secs. 4.4–4.5 examine the two learning effects, while Appendix D.6 (page 25) provides the separate compute-matched comparison with Full Continuation (ϵ=1\epsilon=1). We hope these clarifications and comparisons help distinguish CtC from existing methods. We welcome any further questions or discussion on this point.


Q2: Compute-matched Comparison.

Thank you for your insightful suggestion. Following your recommendation, we have added a compute-matched training budget comparison in Appendix D.6 (Table 19, pages 25-27) on two representative tasks: BraTS T2w→T2f and AutoPET CT→PET. Both methods use identical architectures and training configurations, with optimization updates adjusted to ensure strictly matched training GPU-hours. Full Continuation (ϵ=1\epsilon=1) always trains to Kmax⁡=4K_{\max}=4, while StreamRefine (ϵ=0.15\epsilon=0.15) occasionally continues beyond proposed stops. Both use learned adaptive stopping at inference.

Table R2: Comparison with full-continuation training under matched training budgets.

TaskMethodTrain GPU-hUpdatesPSNR ↑FID ↓Dice ↑Avg. Steps ↓
BraTS T2w→T2fFull Continuation31.9465.7k21.4746.80.7602.25
StreamRefine31.94100.0k21.6843.50.7682.13
AutoPET CT→PETFull Continuation29.8069.7k27.3749.90.5732.69
StreamRefine29.80100.0k27.6547.50.5802.57

As shown above, StreamRefine consistently outperforms full continuation under identical training budgets, improving PSNR by 0.21 and 0.28 dB on BraTS and AutoPET, respectively, while achieving lower FID, higher Dice, and fewer inference steps. These findings demonstrate that occasional continuation utilizes limited training compute more effectively than full-continuation training, providing a better quality-efficiency trade-off.


Q3: Additional validation of PIR.

Thank you for your constructive comment. As suggested, we have extended the independent PIR validation to two additional translation tasks: BraTS T1n→T1c and AutoPET CT→PET, as reported in Appendix E.2 (Table 21, page 28). We evaluate Spearman correlations between PIR error and independent segmentation-based structural errors (Dice error and HD95), at both the refinement-state and adjacent-step levels.

Table R3: Independent validation of the PIR structural proxy across three translation tasks.

TaskStructural measureState-level ρ\rhoStep-change ρ\rho
BraTS T2w→T2fDice error0.630.51
HD950.680.54
BraTS T1n→T1cDice error0.600.47
HD950.640.51
AutoPET CT→PETDice error0.550.43
HD950.600.48

Across all three tasks, PIR exhibits moderate positive correlations with both independent structural measures at the state and adjacent-step levels. These results support its usefulness as a structural proxy across the evaluated settings.

AuthorsOpenReview ↗

Response to Reviewer dZFX (Part 3/3)

Q4: Short refinement states.

Thank you for raising this thoughtful question. Our current manuscript already demonstrates that StreamRefine remains effective when Kmax⁡K_{\max} is increased from 4 to 6 and 8, as shown in Appendix D.5, Table 18 (pages 25–26), while substantially longer horizons remain untested. Specifically, Appendix D.5, Table 18 compares Kmax⁡∈{4,6,8}K_{\max}\in\lbrace 4,6,8\rbrace on BraTS T2w→T2f and T1n→T1c. Longer horizons yield modest quality gains, with no consistent improvement across PSNR and Dice when increasing the cap from six to eight steps. At Kmax⁡=8K_{\max}=8, the average inference depths are 2.67 and 3.46, respectively, indicating selective use of the additional refinement capacity.

These results support effectiveness beyond the default four-step horizon within the tested range. However, most cases still stop within four steps, so they do not establish applicability to models requiring dozens of iterations. We acknowledge this scope limitation in Appendix F (page 32), where longer or dynamically expandable horizons are discussed as future directions.


We are grateful for the reviewer's thoughtful comments and suggestions. We believe the additional experiments and clarifications strengthen our work. We would be happy to answer any further questions and welcome continued discussion.


References

[1] Zhu X, et al. Introducing 3D representation for dense volume-to-volume translation via score fusion. ICML, 2025.

[2] Ha J, et al. Multi-resolution guided 3d gans for medical image translation. WACV, 2025.

[3] Arslan F, et al. Self-consistent recursive diffusion bridge for medical image translation. MedIA, 2025.

[4] Pang H, et al. Cascaded diffusion model and segment anything model for medical image synthesis. PR, 2026.

[5] Rassmann S, et al. Regression is all you need for medical image translation. IEEE TMI, 2026.

[6] Dorent R, et al. Unified cross-modal medical image synthesis with hierarchical mixture of product-of-experts. IEEE TPAMI, 2026.

Reviewer_GDNkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes StreamRefine, an autoregressive 3D medical image translation framework that dynamically learns per-case stopping criteria for iterative refinement. To resolve the feedback loop of premature stops caused by rollout truncation during training, it introduces Continue-to-Check, which periodically extends trajectories to supervise late-stage refinement states and train a lightweight benefit-prediction head. The stopping policy optimizes a Continue Benefit objective, balancing reconstruction quality, computational cost, and anatomical drift via a Pair-Invariant Relations metric.

Critical strengths
  1. The core motivation is clear and addresses an understudied tension between training efficiency and representation learning in adaptive generative models.

  2. Grounding the stopping objective in an anatomy-aware metric (PIR) aligns well with medical imaging constraints, preventing models from trading structural fidelity for marginal appearance gains.

  3. The empirical evaluation is thorough, demonstrating consistent improvements across multiple 3D modalities alongside diagnostic experiments.

Critical weaknesses
  1. For full volumes processed via sliding windows, the model aggregates local window scores using a simple weighted average to make a single global stopping decision (Eq. 19). Because each local score looks ahead to its own best future step, averaging these individual peaks can misjudge the overall volume's actual need. The authors acknowledge in Appendix C.4 (Eq. 20-21) that this average of per-window maxima upper-bounds the benefit of continuing all windows to a shared depth, but the size of this gap and its effect on the final stopping depth are not measured. Consequently, in scans with mixed complexity, simple soft-tissue regions risk being over-refined while intricate lesions may be stopped prematurely.

  2. The framework is validated primarily on a short horizon of four steps. The appendix compares K_max of 4, 6 and 8 on two BraTS tasks, where longer horizons give only modest gains and most cases still stop within four steps, so it remains unclear if this approach can generalize to models requiring dozens of iterative steps. Scaling to deeper trajectories could inflate gradient variance during reach-correction reweighting, since later states are observed less often, potentially destabilizing model training.

Questions for authors
  1. The paper argues in Appendix C.2 that independent stopping causes inconsistencies between overlapping windows. Have you explored allowing sliding windows to terminate independently once their local benefit falls below zero, and then reusing cached representations or frozen states for those converged windows while continuing to refine incomplete patches?

  2. In Figure 7, stopping decisions are presented alongside total predicted and hindsight benefit values (B^k\hat{B}_k and Bk∗B_k^*). However, because the Continue Benefit objective balances translation error reduction against structural degradation, could you provide qualitative visualizations (e.g., intermediate generated crops alongside spatial error maps) that explicitly disentangle these two components step-by-step?

2 comments
AuthorsOpenReview ↗

Response to Reviewer GDNk (Part 1/2)

We appreciate the reviewer's positive assessment of our motivation, anatomy-aware stopping design, and extensive experimental evaluation. Following the reviewer's suggestions, we have added sliding-window aggregation and independent-stopping experiments, and further clarified the refinement-horizon limitations, gradient variance, and qualitative analyses.

Key Concerns and Main Responses

Key ConcernsMain Responses
Volume-level aggregation gap (W1)Added analyses of aggregation gaps and stopping disagreements (Appendix E.6).
Short refinement horizon and gradient variance (W2)Clarified effectiveness up to eight steps, reach correction, and longer-horizon limitations (Appendices B.4, D.5, and F).
Independent-window stopping (Q1)Added cached-state independent stopping experiments, supporting synchronized stopping for quality and overlap consistency (Appendix E.6).
Visualization of stopping components (Q2)Clarified existing qualitative and structural analyses, and acknowledged the remaining visualization limitation (Sec. 4.5; Appendix E.2).

W1: Please see Q1.


W2: Short refinement horizon.

Thank you for highlighting this limitation. We acknowledge that our current experiments are limited to horizons up to Kmax⁡=8K_{\max}=8 and do not establish generalization to models requiring dozens of refinement steps. As reported in Appendix D.5, Table 18 on page 26, extending the horizon from four to six/eight steps yields modest quality gains on two BraTS tasks. At Kmax⁡=8K_{\max}=8, 17.3% and 26.0% of cases continue beyond four steps, indicating that the additional refinement capacity is used selectively. We agree that these results do not replace evaluation on tasks requiring substantially deeper trajectories, consistent with the horizon limitation discussed in Appendix F (page 32).

Regarding gradient variance, CtC makes a single continuation decision and, when selected, executes the entire remaining trajectory (Sec. 3.3, Eq. (3)). Consequently, every state beyond the proposed stop has reach probability ϵ\epsilon and inverse-reach weight 1/ϵ1/\epsilon, independent of depth (Appendix B.4, page 17). The weights therefore do not compound across steps. We nevertheless acknowledge the sampling variance introduced by inverse weighting; our implementation uses normalization by Kmax⁡K_{\max} and global gradient clipping to control gradient magnitudes. These measures do not guarantee training stability over dozens of steps, which remains to be evaluated.

AuthorsOpenReview ↗

Response to Reviewer GDNk (Part 2/2)

Q1: Sliding window and volume-level aggregation.

Thank you for this constructive feedback. We agree that quantifying the aggregation gap and testing independent stopping are important for evaluating our volume-level stopping design. We have added both experiments in Appendix E.6 (Tables 23–24, pages 29-31).

Both experiments use frozen StreamRefine checkpoints on BraTS T2w→T2f and AutoPET CT→PET, with Kmax⁡=4K_{\max}=4, 192×192×96192\times192\times96 windows, and 25% overlap. No additional training is required.

  • Aggregation gap and stopping disagreement. On identical completed trajectories, we compare the weighted average of local hindsight maxima (Local-max) with the maximum aggregated benefit at a common future depth (Shared-depth). We measure their benefit gap, per-state continue/stop disagreement (Decision Dis.), and final stopping-depth disagreement (Stop Dis.).

Table R1: Analysis of sliding-window aggregation gaps and stopping disagreements.

TaskComparisonMean Gap ↓P90 Gap ↓Decision Dis. (%) ↓Stop Dis. (%) ↓
BraTS T2w→T2fLocal-max vs. Shared-depth0.0200.0745.37.0
BraTS T2w→T2fStreamRefine vs. Shared-depth——13.315.0
AutoPET CT→PETLocal-max vs. Shared-depth0.0340.0957.010.0
AutoPET CT→PETStreamRefine vs. Shared-depth——15.718.0

The aggregation approximation changes final stopping depths in at most 10% of volumes relative to Shared-depth. Its nonnegative gap can favor additional continuation. The learned policy’s higher disagreement additionally reflects benefit-prediction errors. Importantly, Shared-depth isolates the max–average discrepancy rather than providing an exact fused-volume oracle.

  • Independent-window stopping with cached states. As suggested, we implement an alternative in which each window stops when its predicted benefit becomes non-positive. Its final latent prediction is cached and reused while active windows continue. Updated and cached predictions are combined using the same Gaussian blending, and the fused state supplies the subsequent refinement history.

Table R2: Comparison of synchronized and independent-window stopping.

TaskStopping strategyPSNR ↑ROI PSNR ↑Dice ↑Avg. W-Steps ↓Overlap MAE ↓Latency (s/vol) ↓
BraTS T2w→T2fIndependent + Cached States21.6018.300.7631.890.02919.7
BraTS T2w→T2fStreamRefine (Synchronized)21.6818.370.7682.130.02720.8
AutoPET CT→PETIndependent + Cached States27.5717.750.5742.290.03831.5
AutoPET CT→PETStreamRefine (Synchronized)27.6517.820.5802.570.03233.8

Avg. W-Steps measures the average refinement depth per window, and Overlap MAE measures disagreement between overlapping predictions before blending.

As reported, independent stopping reduces refinement steps by approximately 11% and slightly improves inference speed. However, synchronized stopping achieves better reconstruction quality, Dice, and overlap consistency. These results support synchronized stopping, while independent stopping offers a computationally efficient alternative.


Q2: Additional qualitative results.

Thank you for this helpful suggestion. We agree that separately visualizing translation error reduction and structural degradation would make the stopping mechanism clearer. Our structural term measures weighted local relation discrepancies in latent space (Appendix B.2), so a corresponding spatial map would represent a structural proxy and require careful anatomical interpretation.

Fig. 7(b) contrasts image-only and anatomy-aware hindsight benefits, which favor different decisions at k=2k=2. This illustrates the structural term’s effect on stopping, although it does not spatially separate translation error and structural drift. We acknowledge this limitation. The anatomy-aware comparison in Sec. 4.5 and independent PIR validation in Appendix E.2 provide complementary quantitative evidence.


We appreciate the reviewer's detailed feedback, which has helped us better examine our stopping strategy and clarify its limitations. We believe these additional analyses and explanations strengthen our work, and would be glad to address any further questions or suggestions through continued discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

I. Acknowledgments

We sincerely appreciate the AC and all three reviewers for their thoughtful assessments and valuable suggestions. Their feedback has guided us in strengthening our empirical validation, clarifying methodological points, and better discussing the scope and limitations of our work. In this general response, we highlight the recognized strengths, address the main concerns, and summarize the revisions, with further details provided in our individual responses.


II. Key Strengths

We appreciate the reviewers’ recognition of the following strengths:

  • Meaningful motivation: Addresses the important interaction between learned stopping and later-refinement learning (GDNk, dZFX).

  • Effective training and stopping design: Connects anatomy-aware stopping with refinement learning through CtC (GDNk, DUxz).

  • Extensive evaluation and analyses: Provides broad multi-dataset experiments and informative mechanism analyses (GDNk, dZFX, DUxz).


III. Summary of Key Concerns and Main Responses

Key ConcernsReviewersMain Responses
Methodological novelty and related methodsdZFXClarified CtC's distinction from related training strategies, emphasizing its dual supervision of stopping and later refinements.
Fairness under matched training budgetsdZFXAdded GPU-hour-matched comparisons showing better quality and fewer inference steps than Full Continuation.
Modest gains and practical valuedZFX, DUxzAcknowledged modest gains and clarified the joint benefits for translation quality, anatomical preservation, and efficiency.
PIR reliability across tasksdZFX, DUxzExtended independent PIR validation to additional tasks, showing consistent agreement with structural measures.
Sliding-window aggregation and independent stoppingGDNkAdded aggregation-gap and independent-stopping analyses, highlighting the quality–efficiency trade-off.
Limited refinement horizonGDNk, dZFXClarified existing eight-step results and acknowledged longer-horizon limitations.

IV. Additional Clarifications

We further clarify several points already addressed in the original submission:

  • Contributions of stopping and later-step supervision. The original submission already disentangles these contributions in Secs. 4.4–4.5, where later-step supervision mainly improves refinement quality, while adaptive stopping improves inference efficiency. Appendix E.1 further analyzes later-state supervision.

  • Continuation weights and gradient variance. As explained in Appendix B.4, CtC uses a single continuation decision, so inverse-reach weights do not compound across steps, although sampling variance remains.

  • Training and inference requirements. As stated in Sec. 3.4, PIR and continuation checks are used only during training. Inference requires neither paired targets nor PIR computation.


V. Revisions and Updates

We have incorporated the additional experiments and clarifications into our revised manuscript, with all modifications highlighted in purple\color{#3C3CA3}{\text{purple}} for ease of reference. The main updates are summarized below:

  • Appendix D.6, Table 19: comparisons with Full Continuation under matched training GPU-hours on two tasks.
  • Appendix E.2, Table 21: expanded independent PIR validation covering three translation tasks across two datasets.
  • Appendix E.6, Tables 23–24: analyses of aggregation gaps, stopping disagreements, and independent-window stopping with cached states.
  • Appendix F, Table 27: comparison with related training strategies, together with expanded discussions of modest gains and paired-data dependence.

We are grateful to the AC and all reviewers for their thoughtful comments, which have helped us improve and strengthen our work. We hope our additional evidence and explanations provide useful context for the reviewers' concerns, and we welcome any further suggestions or discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.