acceptodds
Under review as a conference paper at ICLR 2027

Mixture-of-Depth Rectified Flow for Heterogeneous Multimodality

Abstract

Hierarchical Rectified Flow models data generation through a hierarchy of coupled ordinary differential equations (ODEs), where each depth solves a flow-matching problem over progressively transformed target directions. We show that the ambiguity of the conditional target-direction distribution varies across both time and hierarchy depth, and that increasing depth progressively concentrates this distribution, making the corresponding regression problem simpler. This heterogeneous multimodality implies that a fixed hierarchy depth can allocate computation inefficiently: some space-time regions require deeper hierarchical modeling, while others can be sufficiently modeled at shallow depth. To address this, we propose _Mixture-of-Depth Rectified Flow_ (MoD-RF), which adaptively selects the hierarchy depth at each outer integration step through a sequential stopping mechanism trained from depth-dependent velocity reconstruction quality, without ground-truth depth labels. Under a fixed number of function evaluations (NFEs), MoD-RF allocates deeper computation only where needed while preserving cheaper shallow updates elsewhere. Experiments on synthetic 1D and 2D distributions, MNIST, CIFAR-10, and ImageNet-32 demonstrate improved quality-compute trade-offs over standard rectified flow and fixed-depth hierarchical inference, with particularly strong gains on CIFAR-10 and ImageNet-32.

Primary area:
generative models
Keywords:
generative model, flow matching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rm4uOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Mixture-of-Depth Rectified Flow (MoD-RF), an adaptive extension of hierarchical rectified flow. The paper makes three main contributions: 1) a Gaussian-mixture analysis of hierarchical target directions; 2. a learned sequential depth selector which predicts stopping probabilities over hierarchy levels, and is trained using weights derived from a coarse reconstruction of the original velocity; 3. adaptive-depth inference where at each outer integration step, the method selects a hierarchy depth, fixes that depth and the sampled deeper source states, solves the selected hierarchy, and updates the outer state. Experiments cover two one-dimensional synthetic configurations, a two-dimensional two-moons task, MNIST, CIFAR-10, and ImageNet-32, and the paper reports better Wasserstein distances and FID scores than standard rectified flow and fixed-depth hierarchical baselines.

Critical strengths
  1. This paper's adaptive-computation strategy for hierarchical rectified flow is reasonable and useful
  2. The paper provides both solid theoretical analysis (both in Corollary 1 and Corollary 2), and also practical implementations for training (supervise the depth selector using the prediction quality achieved at each depth, as in Eq. (15))
Critical weaknesses
  1. The paper only reported total NFE budgets, which may accounts depth-specific predictor evaluations, it would be good to also report overall computational cost budget.
  2. The experiment in the paper mainly uses figures, it would be good to also provide numerical tables and uncertainty estimates for quantitative comparison.
Questions for authors
  1. In sec.3.3.1, the reconstruction error normalization factor (\sigma_d) is set as 0.5, how sensitive are generation quality to this parameter?
  2. The Image dataset for experiment is relatively small, how about the algorithms' performance on other dataset such as IMAGENET 256*256.
4 comments
AuthorsOpenReview ↗

Response to Reviewer rm4u (Part 1/2)

We thank the reviewer for their time and feedback. We also thank the reviewer for highlighting our the applications of adaptive computation strategy, our solid theoretical analysis and proposed implementations. We answer questions next.

Q1. The paper only reported total NFE budgets, which may accounts depth-specific predictor evaluations, it would be good to also report overall computational cost budget.

Great suggestion. We adopt the number of function evaluations (NFEs) as our primary computational cost metric, as NFEs constitute a major computational expense in flow-matching-based generative models. To provide a more comprehensive assessment of computational efficiency and answer the reviewer’s question, we have additionally included end-to-end wall-clock measurements and component-wise computation analysis in Appendix E of the revised manuscript.

1. Overall computational cost. In Appendix E.1 (Table 8, lines 1512-1520), we report the total batch inference time, the wall-clock shares of the selector and HRF experts, and their average execution time per call. These measurements include all forward passes and solver operations on MNIST, CIFAR-10, and ImageNet-32, using a batch size of 1,000. The numerical results are summarized below.

DatasetNFEBatch timeSelector shareExpert shareSelector ms/callExpert ms/call
MNIST9931.7 s0.38%98.30%6.1254
MNIST30798.0 s0.37%98.30%6.1248
CIFAR-107518.5 s0.72%97.60%6.7216
CIFAR-1023055.6 s0.71%97.50%6.8216
ImageNet-327126.5 s0.54%98.10%7.2335
ImageNet-3221881.3 s0.50%98.20%6.9329

As shown, the selector accounts for only 0.37%–0.72% of total inference time, while the HRF experts account for 97.5%–98.3%. Moreover, each selector call is approximately 32-48 times faster than an expert call, confirming that adaptive depth selection introduces negligible additional overhead relative to nested integration.

2. Performance versus wall-clock time. Beyond the component-wise analysis, Appendix E.2 (Figure 15, lines 1503–1508) presents end-to-end FID versus wall-clock-time trade-offs on MNIST, CIFAR-10, and ImageNet-32. These comparisons account for all selector evaluations, expert computations, nested integration, and solver overhead. The results show that MoD-RF maintains a favorable quality-computation trade-off over standard RF and fixed-depth HRF at comparable inference times, consistent with the NFE-based comparisons.

Together, these additional results demonstrate that MoD-RF's performance improvements persist when evaluated using actual inference time, rather than relying solely on NFE as a computational proxy.

Q2. The experiment in the paper mainly uses figures, it would be good to also provide numerical tables and uncertainty estimates for quantitative comparison.

We thank the reviewer for this constructive suggestion. To facilitate quantitative comparisons and better assess the robustness of our results, we have added numerical FID scores with mean and standard deviation in Appendix D.1 (Table 4, lines 1080-1093) of the revised manuscript. These results complement the FID-NFE curves in Figure 6 and provide detailed comparisons between MoD-RF, RF, and HRF-2 across multiple inference budgets on MNIST, CIFAR-10, and ImageNet-32.

Numerical results. We reproduce the results from Table 4 below. All values are reported as mean FID ± standard deviation, where lower is better.

DatasetNFEMoD-RF (Ours)RFHRF-2
MNIST3051.744 ± 0.0022.601 ± 0.0411.941 ± 0.017
5121.718 ± 0.0072.589 ± 0.0401.929 ± 0.044
10271.696 ± 0.0092.584 ± 0.0401.940 ± 0.037
CIFAR-1023024.67 ± 0.1236.66 ± 0.1826.44 ± 0.28
38523.99 ± 0.2236.68 ± 0.1125.68 ± 0.13
77223.70 ± 0.1436.76 ± 0.1325.28 ± 0.08
ImageNet-3221823.49 ± 0.1933.40 ± 0.1725.74 ± 0.12
36523.18 ± 0.1733.42 ± 0.1724.92 ± 0.22
73322.73 ± 0.1633.45 ± 0.1724.39 ± 0.25

As shown, MoD-RF achieves the lowest mean FID at every reported NFE budget across all three datasets, with standard deviations comparable to those of the baselines. Furthermore, MoD-RF consistently improves as the inference budget increases, whereas RF remains nearly unchanged. These numerical results reinforce the findings in Section 4.3 (Figure 6) and provide additional evidence that the quality-computation advantages of adaptive depth allocation are consistent across the evaluated settings.

(Continue from the next comment)

AuthorsOpenReview ↗

Response to Reviewer rm4u (Part 2/2)

Q3. In sec.3.3.1, the reconstruction error normalization factor (\sigma_d) is set as 0.5, how sensitive are generation quality to this parameter?

To investigate the impact of the reconstruction error normalization factor σd\sigma_d, we conduct additional ablation studies with $\sigma_d \in $ {0.1,0.25,0.5,1.0,2.00.1,0.25,0.5,1.0,2.0} on the 1to5 and 2to2 datasets, keeping the stopping threshold fixed at δ=0.5\delta=0.5. The results are presented in Appendix D.4.2 (Figure 14 and Table 7) of the revised manuscript.

Generation quality and computational efficiency. As shown in Figure 14, σd=0.5\sigma_d=0.5 consistently achieves the best quality-computation trade-off at moderate and high NFE budgets across both datasets. For example, at 800 NFEs, MoD-RF with σd=0.5\sigma_d=0.5 achieves Wasserstein distances of 0.0103 and 0.0125 on 1to5 and 2to2, respectively, substantially outperforming both smaller (σd=0.1\sigma_d=0.1) and larger (σd=1.0\sigma_d=1.0) settings. Although larger values can be competitive at very low NFE budgets on 2to2, σd=0.5\sigma_d=0.5 provides the strongest overall performance as inference computation increases.

Effectiveness of adaptive depth allocation. Importantly, Table 7 reveals that these performance differences are closely related to how computation is allocated across hierarchy depths. Smaller σd\sigma_d values produce sharper supervision signals and substantially increase depth-3 usage without corresponding improvements in generation quality. Conversely, larger values (σd≥1\sigma_d \ge 1) lead to exclusively selecting depth 2, effectively removing adaptive depth variation. In contrast, σd=0.5\sigma_d=0.5 achieves a more effective allocation by primarily using depth 2 while selectively invoking depth 3, demonstrating the benefit of allocating additional hierarchical computation only to a subset of states.

Overall, these results support our central motivation: effective generation does not require uniformly deeper computation, but rather an appropriate allocation of hierarchy depth. The consistent advantage of σd=0.5\sigma_d=0.5 across both datasets motivates its use as the default setting in our experiments.

Q4. The Image dataset for experiment is relatively small, how about the algorithms' performance on other dataset such as IMAGENET 256*256

We thank the reviewer for highlighting the importance of evaluating MoD-RF on larger-scale image generation tasks. To answer, we have extended our experiments to ImageNet-256 (256×256256 \times 256 resolution) in the revised manuscript and observed similarly outstanding performances across NFE ranges (Section 4.3, Figure 7, lines 421-425).

Experimental setup. Unlike our 32×3232 \times 32 experiments conducted in pixel space, ImageNet-256 generation is performed in the latent space of the SDXL autoencoder, representing images as 4×32×324 \times 32 \times 32 latents. We independently train HRF-1 and HRF-2 experts for 150K steps, freeze their EMA weights, and train a lightweight Transformer depth selector for 20K steps using the same one-step velocity-reconstruction objective. During unconditional generation, the selector adaptively chooses between depths 1 and 2 at each outer Euler step, with five inner steps for depth 2. Generated latents are decoded into images and evaluated using FID-50k, with RF and HRF-2 compared at matched NFE budgets.

Results on ImageNet-256. As shown in Section 4.3 (Figure 7), MoD-RF consistently achieves lower FID than standard RF and fixed-depth HRF-2 across all evaluated NFE budgets. While RF exhibits nearly constant FID as the inference budget increases, MoD-RF continues to improve and maintains the most favorable FID-NFE trade-off throughout the evaluated range. This demonstrates that adaptive depth allocation remains effective for substantially higher-resolution image generation.

Furthermore, we investigate whether the improvements persist with stronger underlying experts by comparing models trained for 100K and 150K steps (Section 4.4, lines 483-489; Figure 7). Although additional training improves all methods, MoD-RF consistently maintains the lowest FID at every evaluated NFE under both training budgets, indicating that adaptive hierarchy depth provides complementary benefits as the underlying experts become better optimized.

Together, these additional experiments provide evidence that MoD-RF's quality–computation advantages extend beyond low-resolution benchmarks to higher-resolution ImageNet-256 generation, supporting the scalability and practical relevance of adaptive hierarchical modeling.

AuthorsOpenReview ↗

Follow-up on our Responses

We thank the reviewer again for their constructive feedback and positive assessment. In our rebuttal and revision, we have provided wall-clock comparisons to assess overall computational costs, added numerical results with uncertainty estimates, clarified the sensitivity to the normalization parameter σd\sigma_d, and extended our experiments to ImageNet-256 using latent flow matching to demonstrate practical relevance.

We would greatly appreciate it if the reviewer could review these additional results and clarifications and let us know whether they address the main concerns. We would be happy to address any remaining questions.

Thanks the authors for detailed experiment results on component-wise computation analysis, numerical results with uncertainty estimates, and sensitivity analysis. My concerns have been addressed and I'll keep my positive rating

Reviewer_dhesOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses potentially inefficient computation allocation in Hierarchical Rectified Flow (HRF) with a fixed hierarchy depth. Through a Gaussian-mixture case study, the authors show that ambiguity in the target-direction distribution varies across states, time, and hierarchy depth. Motivated by this observation, they propose MoD-RF, which uses a sequential stopping selector at each outer integration step to choose between standard RF and deeper nested HRF. The selector receives soft supervision from one-step velocity reconstruction quality at each depth, without requiring ground-truth depth labels. Experiments cover one- and two-dimensional synthetic distributions, MNIST, CIFAR-10, and ImageNet-32. Under matched average NFE budgets, the method mainly outperforms fixed-depth baselines in the moderate- and high-budget regimes, with gains also observed after enlarging the depth-one RF backbone.

Critical strengths
  1. An intuitive idea addressing a meaningful problem. Selecting HRF depth according to the current sample and generation time, rather than applying deep computation uniformly, offers a reasonable direction for improving the quality–compute trade-off in hierarchical generation.
  2. Inexpensive construction of selector supervision. The method reuses existing depth-specific predictors, obtaining each coarse velocity estimate with one forward pass and a simple conversion. This provides a training signal without full nested sampling or ground-truth depth labels.
Critical weaknesses
  1. It is unclear whether the selector's training signal reflects the actual depth required. Training relies on one-step velocity reconstruction error at initial hierarchical states, whereas inference uses full nested integration; these need not correspond. The signal may reflect differences in prediction accuracy across networks rather than identify states that genuinely require deeper hierarchies. The loss also does not explicitly account for computation cost, so it does not establish that the selector will prefer a shallower depth when it is sufficiently accurate.
  2. Actual depth-selection behavior is not adequately validated. The paper does not report the usage proportions of different depths or how selection varies across time and samples. In particular, it is unclear whether depth choices differ substantially between one-dimensional synthetic distributions and image distributions, or whether complex distributions lead to selecting the deepest hierarchy almost all the time, reducing the value of adaptive computation savings. Controls such as random depth selection are also missing, making it difficult to determine whether the gains genuinely come from allocating computation adaptively to difficult states.
  3. Key settings and uncertainty in the results are insufficiently documented. The appendix gives examples of inner integration step counts but does not fully specify the actual integration steps used in each experiment. The stopping threshold is fixed at 0.5, without examining whether generation quality and computation cost remain stable as it changes. Image results are mainly presented as FID curves, without reporting variability across runs or the number of samples used to compute FID, limiting reproducibility and assessment of robustness.
  4. The evaluation scope remains limited. Image experiments focus on MNIST, CIFAR-10, and ImageNet-32, leaving scalability to higher resolutions or larger systems unclear. The method also does not consistently outperform fixed-depth HRF under very low NFE budgets, so its practical value for generation with few network evaluations requires further validation.
Questions for authors

See "Weakness".

7 comments
AuthorsOpenReview ↗

Response to Reviewer dhes (Part 1/4)

We thank the reviewer for their time and feedback. We also thank the reviewer for highlighting the importance of improving quality-compute tradeoffs in hierarchical integration as well as the simplicity and efficiency of our method. We answer questions next.

Q1. It is unclear whether the selector's training signal reflects the actual depth required. Training relies on one-step velocity reconstruction error at initial hierarchical states, whereas inference uses full nested integration; these need not correspond. The signal may reflect differences in prediction accuracy across networks rather than identify states that genuinely require deeper hierarchies. The loss also does not explicitly account for computation cost, so it does not establish that the selector will prefer a shallower depth when it is sufficiently accurate.

We address the reliability of one-step depth supervision and the preference for computationally efficient depth selection through additional analyses in the revised manuscript.

1. Validity of one-step reconstruction supervision. We emphasize that one-step velocity reconstruction in MoD-RF can approximate the quality of each depth-specific expert without significant performance loss, while preserving simulation-free training. To examine this training-inference approximation, we compare our objective with a rollout-based variant that performs 3 nested Euler steps per inner level while keeping all pretrained experts frozen during training. As shown in Appendix D.4.1 (Table 6), one-step supervision matches or outperforms rollout-based supervision at every evaluated NFE budget across the 1to5, 2to2, and moon datasets. A possible explanation is that stochastic nested rollouts introduce additional variance into the supervision signal, particularly at greater depths. We provide the result table (Table 6 in revision) as follows.

DatasetNFEMoD-RF (Ours)3-step Rollout
1to52000.0104 ± 0.00060.0105 ± 0.0008
1to53400.0091 ± 0.00130.0126 ± 0.0005
1to55100.0098 ± 0.00210.0141 ± 0.0009
1to58500.0104 ± 0.00140.0156 ± 0.0002
1to512000.0115 ± 0.00140.0173 ± 0.0005
2to22500.0139 ± 0.00140.0185 ± 0.0010
2to23700.0139 ± 0.00040.0168 ± 0.0008
2to25000.0130 ± 0.00080.0153 ± 0.0010
2to28500.0125 ± 0.00070.0133 ± 0.0004
2to211500.0127 ± 0.00020.0130 ± 0.0008
moon5000.0879 ± 0.00520.1412 ± 0.0068
moon8300.0687 ± 0.00530.1383 ± 0.0078
moon11500.0731 ± 0.00330.1389 ± 0.0068

2. Preference for shallower computation. We emphasize that MoD-RF prefers shallower depths and allocate deeper flows to states with complex velocity distribution, although our training objective does not explicitly penalize computation.

This is because the inference procedure enforces a structural preference for shallower depths. Specifically, MoD-RF selects the first depth satisfying sη,d>δs_{\eta,d} > \delta, avoiding deeper integration whenever an earlier stopping condition is met (Section 3.3.2, Algorithm 1). We further analyze this mechanism in Appendix D.3 (Figures 12-13), showing that the stopping threshold controls the allocation between shallow and deep experts. Importantly, complex intersection regions, highlighted by the red boxes, consistently receive deeper computation than the surrounding regions. This shows that the threshold controls the overall computational budget while the selector preserves its preference for allocating deeper flows to more difficult regions. We nevertheless consider computation-aware training a valuable direction for future work.

Together, these results support the effectiveness of one-step supervision and demonstrate how computation is controlled through sequential stopping, without requiring ground-truth depth labels or rollout-based training.

(Continue in the next comment)

AuthorsOpenReview ↗

Response to Reviewer dhes (Part 2/4)

Q2. Actual depth-selection behavior is not adequately validated. The paper does not report the usage proportions of different depths or how selection varies across time and samples. In particular, it is unclear whether depth choices differ substantially between one-dimensional synthetic distributions and image distributions, or whether complex distributions lead to selecting the deepest hierarchy almost all the time, reducing the value of adaptive computation savings. Controls such as random depth selection are also missing, making it difficult to determine whether the gains genuinely come from allocating computation adaptively to difficult states.

Great suggestion. We have added depth-selection analyses in Appendix D.2 to examine whether MoD-RF learns meaningful state-dependent allocation and whether its improvements arise from adaptive computation rather than simply combining pretrained experts.

1. Alignment with velocity ambiguity. In Appendix D.2.2 (Figures 9-10), we compare selected depths with an independent ambiguity measure, the empirical conditional velocity variance Var[x1−x0∣xt]\mathrm{Var}[x_1-x_0\mid x_t], which is not provided to the selector during training. On 2to2, depth-3 selections concentrate predominantly in high-variance trajectory-intersection regions, while shallower depths are preferred elsewhere. Similar spatial preferences are observed on moon, demonstrating that deeper computation is preferentially allocated to more ambiguous states. This pattern is further consistent across different inference-time stopping thresholds δ\delta, as shown in Figure 12.

2. Depth usage across datasets and time. We further analyze routing behavior on image datasets in Appendix D.2.3 (Figure 11). Across MNIST, CIFAR-10, and ImageNet-32, the selector exhibits a clear time-dependent transition from depth 1 to depth 2, occurring later on MNIST than on CIFAR-10 and ImageNet-32. Moreover, these routing patterns remain stable across different outer-step discretizations (N=20N=20 and N=60N=60), indicating that depth selection adapts consistently across inference budgets rather than following a fixed global switching step.

3. Comparison with random depth selection. To isolate the benefit of learned routing, we introduce a random-depth baseline using the same frozen experts and sampling procedure. As reported in Appendix D.2.1 (Table 5), MoD-RF consistently achieves lower Wasserstein distance at comparable NFE budgets. For example, on 2to2, MoD-RF achieves 0.0139 compared with 0.0388 for random allocation in the 370–407 NFE range.

Together, these analyses show that MoD-RF learns a meaningful state-dependent depth-allocation policy that responds to local ambiguity and improves generation quality over random selection, rather than simply benefiting from access to multiple depth-specific experts.

DatasetNFE rangeMoD-RF (Ours)Random
1to5300–3400.0091 ± 0.00130.0249 ± 0.0020
1to5340–3700.0091 ± 0.00130.0217 ± 0.0019
1to5500–5180.0098 ± 0.00210.0148 ± 0.0016
2to2370–4070.0139 ± 0.00040.0388 ± 0.0013
2to2592–6250.0120 ± 0.00050.0308 ± 0.0011
2to2875–9610.0129 ± 0.00050.0246 ± 0.0007

(Continue in the next comment)

AuthorsOpenReview ↗

Response to Reviewer dhes (Part 3/4)

Q3. Key settings and uncertainty in the results are insufficiently documented. The appendix gives examples of inner integration step counts but does not fully specify the actual integration steps used in each experiment. The stopping threshold is fixed at 0.5, without examining whether generation quality and computation cost remain stable as it changes. Image results are mainly presented as FID curves, without reporting variability across runs or the number of samples used to compute FID, limiting reproducibility and assessment of robustness.

We thank the reviewer for these constructive suggestions. To improve reproducibility and evaluate the robustness of MoD-RF, we have expanded the experimental details and added several ablation studies in the revised manuscript (in blue text).

1. Sensitivity to the stopping threshold. We conduct additional experiments to investigate how the stopping threshold δ\delta affects depth allocation and generation quality (Appendix D.3, Figures 12–13). As shown in Figure 13, MoD-RF transitions smoothly from HRF-1 at δ=0\delta=0 to HRF-3 at δ=1\delta=1, with intermediate thresholds providing state-dependent mixtures of experts. The best performance is generally achieved around δ=0.5\delta=0.5, balancing shallow and deep computation. Furthermore, Figure 12 shows that regions with greater velocity ambiguity consistently receive deeper computation across different thresholds, demonstrating that the selector preserves its state-dependent allocation behavior.

2. Integration schedules and NFE accounting. We provide the complete integration schedules in Appendix C.3 (lines 1054-1073). Specifically, we use [N1,10,10][N_1, 10, 10] for all low-dimensional datasets, [N1,10][N_1, 10] for MNIST, and [N1,5][N_1, 5] for CIFAR-10 and ImageNet-32, where N1N_1 denotes the number of outer Euler steps. We vary the NFE budget by sweeping N1N_1 while keeping the inner integration steps fixed. Since MoD-RF selects the hierarchy depth independently at each outer step, its total NFE depends on the selected depths. For fair comparisons, we measure MoD-RF's average NFE and evaluate the fixed-depth baselines at the closest achievable budget, subject to integer integration-step counts.

3. Numerical results with uncertainty. In addition to the variability bands shown in the FID-NFE curves, we provide numerical mean FID and standard deviation values in Appendix D.1 (Table 4, lines 1080-1093). All image-generation results are evaluated using 50K generated samples, averaged across 5 different seeds, as specified in Section 4.3 (lines 411-414). We reproduce the numerical results below for clarity, which are added to the revised manuscript.

Dataset/ScheduleNNFEMoD-RF (Ours)RFHRF-2
MNIST / [N,10]603051.744 ± 0.0022.601 ± 0.0411.941 ± 0.017
1005121.718 ± 0.0072.589 ± 0.0401.929 ± 0.044
20010271.696 ± 0.0092.584 ± 0.0401.940 ± 0.037
CIFAR-10 / [N,5]6023024.67 ± 0.1236.66 ± 0.1826.44 ± 0.28
10038523.99 ± 0.2236.68 ± 0.1125.68 ± 0.13
20077223.70 ± 0.1436.76 ± 0.1325.28 ± 0.08
ImageNet-32 / [N,5]6021823.49 ± 0.1933.40 ± 0.1725.74 ± 0.12
10036523.18 ± 0.1733.42 ± 0.1724.92 ± 0.22
20073322.73 ± 0.1633.45 ± 0.1724.39 ± 0.25

As shown, MoD-RF achieves the lowest mean FID at every reported NFE budget across all three datasets, with standard deviations comparable to those of the baselines. Moreover, its FID consistently improves as the inference budget increases, whereas RF remains nearly unchanged. These results, together with the additional threshold ablations and detailed integration schedules, provide stronger evidence for the robustness and reproducibility of MoD-RF.

(Continue in the next comment)

Reviewer_5oYsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

MoD-RF replaces the fixed hierarchy depth of hierarchical rectified flow with depth-specific predictors and a learned stopping policy trained without depth labels, adapting to spatially and temporally varying target-direction ambiguity. Experiments on synthetic, MNIST, CIFAR-10 and ImageNet-32 evaluate FID or Wasserstein distance against NFE.

Critical strengths

1.The adaptive-depth motivation is clear. HRF's results suggest that velocity ambiguity varies with state and depth.

  1. Nested predictor evaluations and NFE budgets are explicitly reported at matched budgets, and the limitations cover added complexity, supervision quality, and the ImageNet-32 ceiling, noting HRF2 can win on MNIST at very small NFE.
Critical weaknesses
  1. The core mechanism is weakly supported. The depth supervision is computed at the initial state during training but applied to intermediate trajectory states at inference, and this mismatch is never analyzed.

The theory only holds for Gaussian mixtures and only in expectation.

Also, at low NFE budgets the method loses to fixed-depth HRF, so the adaptive overhead can be a net burden? pleasse explain.

  1. Only the depth predictors inside the nested solve are counted, while the stopping Transformer runs extra forward passes per step and its cost is never clearly included.

No wall-clock comparison is provided either, so the matched-budget claim is hard to verify.

  1. The evaluation is far from modern generative modeling. The largest experiment is 32x32 images, it is hard to judge whether the conclusions matter for real generation tasks.
Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 5oYs (Part 1/3)

We thank the reviewer for their time and feedback. We also thank the reviewer for highlighting our research motivation of addressing varying velocity ambiguity with state and depth; and the clear experiment designs. We answer questions next.

Q1. The core mechanism is weakly supported. The depth supervision is computed at the initial state during training but applied to intermediate trajectory states at inference, and this mismatch is never analyzed.

There might be some confusion. We clarify that the depth selector is not trained solely on the initial outer state. As detailed in the revised manuscript (Section 3.3.1, Eq. (14), lines 264–269), during training, we sample an outer time t∼U(0,1)t \sim U(0,1) and construct $ x_t^{(1)} = (1-t)x_0^{(1)}+tx_1 $. Only the deeper hierarchical states are initialized from their source distributions. Specifically, the selector receives ([xt(1),x0(2)],[t,0d−1])([x_t^{(1)},x_0^{(2)}],[t,\mathbf{0}^{d-1}]).

This construction is consistent with inference, where the selector observes the current outer state and time, while deeper hierarchical states are freshly initialized before each nested solve (Section 3.3.2, lines 301–310; Appendix A). Thus, the selector is trained on outer states spanning the entire interpolation trajectory, rather than only at t=0t=0.

We have clarified this distinction in the revised Eq. (14) and its accompanying description. The remaining difference is that training uses interpolated states constructed from source-target pairs, whereas inference uses numerically generated outer states, as in standard flow matching. We additionally investigate the related one-step reconstruction versus nested-integration approximation in Appendix D.4.1 (Table 6), where one-step supervision matches or outperforms rollout-based supervision across the evaluated synthetic datasets.

Q2.The theory only holds for Gaussian mixtures and only in expectation.

We thank the reviewer for raising this important point. Our theoretical analysis consists of two complementary results with different levels of generality and directly supports our proposal.

First, Corollary 1 is a Gaussian-mixture case study, providing a closed-form characterization of how the conditional target-direction distribution varies across hierarchy depth and time (Section 3.1; Appendix B.1). This analysis offers an interpretable example of heterogeneous multimodality rather than restricting MoD-RF to Gaussian-mixture targets.

Second, Corollary 2 admits a more general interpretation beyond Gaussian mixtures. As shown in Appendix B.2, its proof relies only on the tower property of conditional expectation and the non-negativity of conditional variance. Consequently, the expected posterior-concentration result holds for any target distribution admitting a latent mixture-component variable, provided the hierarchical conditioning information is nested. It does not require Gaussian mixture components.

Importantly, the guarantee holds in expectation, not pointwise. This distinction directly motivates MoD-RF: although additional hierarchical information increases posterior concentration on average, the ambiguity reduction can vary considerably across individual states. Consequently, a fixed hierarchy depth may allocate unnecessary computation to already-simple states, while more ambiguous states may benefit from deeper modeling. MoD-RF addresses this heterogeneity through state-dependent depth selection (Section 3.2).

We emphasize that our theoretical results motivate adaptive hierarchy depth rather than guarantee optimal depth selection for arbitrary distributions. The empirical effectiveness of this strategy is further supported by the depth-allocation and velocity-ambiguity analyses in Appendix D.2.

(Continue in the next comment)

AuthorsOpenReview ↗

Response to Reviewer 5oYs (Part 2/3)

Q3. At low NFE budgets the method loses to fixed-depth HRF, so the adaptive overhead can be a net burden? please explain

Note, the performance gap at low NFE budgets primarily arises from limited outer integration steps rather than the computational overhead of the depth selector. Following the hierarchical integration protocol of HRF [1], the total NFE budget is distributed across hierarchy levels. For example, with a fixed depth-3 schedule of [N1,10,10][N_1,10,10], a budget of 200 NFEs permits only 2 outer integration steps, whereas standard RF can allocate all 200 evaluations to outer integration. Although MoD-RF adaptively selects the hierarchy depth and can use more outer steps under the same budget, its advantage is less pronounced when the budget is too small to sufficiently resolve the outer trajectory. As the NFE budget increases, adaptive depth allocation becomes more effective, as demonstrated in Section 4.2 (Figure 5) and Section 4.3 (Figure 6) in our revised version.

To verify that the depth selector itself does not introduce substantial computational overhead, we additionally report end-to-end wall-clock measurements and component-wise computation costs in Appendix E.1 (Table 8, lines 1512-1520). The measurements include all selector and expert evaluations, using a batch size of 1000.

As shown, the selector accounts for only 0.37%–0.72% of total inference time, confirming that its additional computational overhead is negligible relative to the hierarchical experts. Moreover, Appendix E.2 (Figure 15) provides end-to-end wall-clock-time versus FID comparisons, demonstrating that MoD-RF maintains a favorable quality - computation trade-off when all inference costs are included.

Table 1. End-to-end wall-clock measurements and component computation costs

DatasetNFEBatch timeSelector shareExpert shareSelector ms/callExpert ms/call
MNIST9931.7 s0.38%98.30%6.1254
MNIST30598.0 s0.37%98.30%6.1248
CIFAR-107518.5 s0.72%97.60%6.7216
CIFAR-1023055.6 s0.71%97.50%6.8216
ImageNet-327126.5 s0.54%98.10%7.2335
ImageNet-3221881.3 s0.50%98.20%6.9329

Reference

[1] Zhang et al., Towards Hierarchical Rectified Flow, ICLR 2025.

Q4. Only the depth predictors inside the nested solve are counted, while the stopping Transformer runs extra forward passes per step and its cost is never clearly included. No wall-clock comparison is provided either, so the matched-budget claim is hard to verify.

Great suggestion. We adopt the number of function evaluations (NFEs) as our primary computational cost metric, as NFEs constitute a major computational expense in flow-matching-based generative models. To provide a more comprehensive assessment of computational efficiency and answer the reviewer’s question, we have additionally included end-to-end wall-clock measurements and component-wise computation analysis in Appendix E of the revised manuscript.

Specifically, Appendix E.1 (Table 8, lines 1512-1520) reports the total inference time and the computational breakdown between the selector and HRF experts on MNIST, CIFAR-10, and ImageNet-32, using a batch size of 1,000. All selector and expert forward passes are included. The results show that the selector accounts for only 0.37%–0.72% of total inference time, whereas the HRF experts account for 97.5%–98.3%, confirming that the additional selector overhead is negligible relative to the nested integration cost.

Furthermore, Appendix E.2 (Figure 15, lines 1540-1550) presents end-to-end FID versus wall-clock time comparisons on three image datasets. These results demonstrate that MoD-RF maintains a favorable quality–computation trade-off over RF and fixed-depth HRF at comparable inference times, even after accounting for selector overhead and other solver operations. Together, these additional analyses validate that MoD-RF's improvements are not artifacts of excluding selector computation from the NFE budget.

(Continue in the next comment)

AuthorsOpenReview ↗

Response to Reviewer 5oYs (Part 3/3)

Q5. The evaluation is far from modern generative modeling. The largest experiment is 32x32 images, it is hard to judge whether the conclusions matter for real generation tasks.

We thank the reviewer for highlighting the importance of evaluating MoD-RF on larger-scale generation tasks. To answer this question, we have extended our experiments to ImageNet-256 (256$\times$256 resolution) in the revised manuscript (Section 4.3, lines 421–425; Figure 7).

Experimental setup. Unlike our 32×3232 \times 32 experiments conducted in pixel space, ImageNet-256 generation is performed in the latent space of the SDXL autoencoder, representing images as 4×32×324\times 32 \times 32 latents. We independently train HRF-1 and HRF-2 experts for 150K steps, freeze their EMA weights, and train a lightweight depth selector for 20K steps.

Experimental results. As shown in Figure 7, MoD-RF consistently achieves lower FID than both standard RF and fixed-depth HRF2 across all evaluated NFE budgets. While RF exhibits nearly constant FID as the inference budget increases, both HRF2 and MoD-RF benefit from additional computation, with MoD-RF maintaining the best FID-NFE trade-off throughout the evaluated range.

Furthermore, we investigate whether these improvements persist with stronger underlying models by increasing expert training from 100K to 150K steps, while keeping the architectures fixed (Section 4.4, lines 483-489; Figure 7). Additional training improves the generation quality of all methods, yet MoD-RF continues to achieve the lowest FID at every evaluated NFE under both training budgets.

Together, these additional experiments provide evidence that MoD-RF's quality-computation advantages extend to higher-resolution image generation and persist with more extensively trained experts.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttals

Dear AC and Reviewers,

We sincerely thank the AC and all three reviewers for their thoughtful feedback and constructive suggestions. We particularly appreciate Reviewer dhes for raising their recommendation to Acceptance, Reviewer rm4u for retaining their Acceptance score, and Reviewer 5oYs for recognizing the significance of our research problem and experimental design. We summarize the key additions and improvements below.

1. Scalability to large-scale image generation (Reviewers 5oYs, dhes, rm4u). To answer questions of practical relevance beyond low-resolution images, we conducted new ImageNet-256 (256×256256\times256) generation experiments using latent flow matching (Appendix C.2). MoD-RF consistently achieves the lowest FID across all evaluated NFE budgets compared with RF and fixed-depth HRF-2. We further investigated scalability with longer expert training on ImageNet-256, demonstrating that adaptive depth allocation remains beneficial with stronger underlying models (Section 4.4, Figure 7).

2. Validity of depth supervision and theoretical justification (Reviewers 5oYs, dhes). We clarified that the selector is trained on outer states spanning the entire interpolation trajectory, rather than only the initial state, addressing concerns about training-inference consistency (Section 3.3.1, Eq. (14)). We additionally compared our simulation-free one-step supervision with a three-step nested rollout variant, showing that one-step reconstruction matches or outperforms rollout-based training across all evaluated synthetic settings (Appendix D.4.1, Table 6). We also clarified the scope of our theoretical analysis: Corollary 1 provides an interpretable Gaussian-mixture case study, while the expected posterior-concentration result in Corollary 2 extends beyond Gaussian mixtures under nested hierarchical conditioning (Section 3.1, Appendix B.1-B.2).

3. Effectiveness and interpretability of adaptive depth selection (Reviewer dhes). We introduced a random-depth selection baseline using identical frozen experts and sampling procedures, demonstrating that learned depth allocation consistently achieves better generation quality at comparable computational budgets (Appendix D.2.1, Table 5). Furthermore, independent conditional velocity-variance analyses show that deeper computation concentrates in more ambiguous trajectory-intersection regions (Appendix D.2.2, Figures 9-10). Additional analyses on MNIST, CIFAR-10, and ImageNet-32 reveal meaningful, time-dependent depth-selection patterns across image datasets (Appendix D.2.3, Figure 11). These results provide direct evidence that MoD-RF learns state-dependent computation allocation rather than merely benefiting from multiple pretrained experts.

4. Hyperparameter sensitivity and robustness (Reviewers dhes, rm4u). We conducted additional ablations on the stopping threshold δ\delta and reconstruction normalization factor σd\sigma_d. The threshold experiments demonstrate how MoD-RF smoothly adjusts computational depth while preserving its preference for ambiguous regions (Appendix D.3, Figures 12–13). The normalization-factor ablation further shows that the default σd=0.5\sigma_d=0.5 achieves a favorable quality-computation trade-off through selective deeper computation (Appendix D.4.2, Figure 14 and Table 7).

5. Computational efficiency and low-NFE behavior (Reviewers 5oYs, dhes, rm4u). We added end-to-end wall-clock measurements and component-wise computational breakdowns (Appendix E.1, Table 8). Across MNIST, CIFAR-10, and ImageNet-32, the depth selector accounts for only 0.37%-0.72% of total inference time, confirming its negligible overhead. Additional FID versus wall-clock-time comparisons demonstrate that MoD-RF maintains favorable quality-computation trade-offs beyond NFE-based evaluation (Appendix E.2, Figure 15). We also clarified that performance limitations at extremely low NFE budgets primarily arise from insufficient outer integration steps rather than selector overhead (Sections 4.2-4.3, Figures 5-6).

6. Reproducibility and quantitative uncertainty (Reviewers dhes, rm4u). We expanded the implementation details, integration schedules, and NFE accounting procedures (Appendix C.2–C.3). We additionally provided numerical FID results with means and standard deviations across five random seeds, evaluated using 50k generated samples (Appendix D.1, Table 4; Appendix C.2). These results consistently show MoD-RF achieving lower mean FID than RF and HRF-2 across the reported image-generation settings, with comparable variability.

Overall, we believe these substantial revisions have addressed all reviewer questions and significantly strengthened the manuscript. We hope that the additional results, detailed responses, and updated reviewer assessments will be considered in the final evaluation of our submission.

Thank you again for your time and consideration.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.