acceptodds
Under review as a conference paper at ICLR 2027

Learning Conditional Source Distribution via Flow Reversal for Temporal Flow Matching

Abstract

We introduce CNP-Flow, a flow matching framework for temporal generation that learns conditional source distributions through flow reversal. Whereas standard conditional flow matching (FM) incorporates conditioning through the vector field and draws source samples from a standard Gaussian, CNP-Flow uses a conditional noise predictor (CNP) to produce an isotropic Gaussian source for each temporal condition. The CNP is supervised by source samples obtained through flow reversal, which maps observed targets backward through a pretrained FM model. A three-stage pipeline pretrains the FM model, trains the CNP, and fine-tunes the FM model using the learned source distribution, while preserving the FM backbone architecture. Across video prediction, video interpolation, and 7-DoF Franka robot motion planning, CNP-Flow consistently improves generation quality. It also matches baseline performance with fewer function evaluations.

Primary area:
generative models
Keywords:
flow matching, flow reversal, generative models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ixz1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CNP-Flow, a flow-matching framework that learns condition-dependent source distributions for temporal generation. Rather than relying on a fixed standard Gaussian source, the method introduces a Conditional Noise Predictor (CNP) that predicts the mean and variance of an isotropic Gaussian distribution conditioned on temporal observations. The proposed training procedure consists of three stages. First, a standard conditional flow-matching model is pretrained. Second, target samples are mapped backward through the pretrained flow to construct supervision for the conditional source predictor. Finally, the flow-matching model is fine-tuned using samples from the learned conditional source distributions. The authors evaluate CNP-Flow on video prediction, video interpolation, and 7-DoF Franka robot motion planning. Additional experiments investigate its applicability to higher-resolution physical world modeling. The results demonstrate substantial improvements in video-generation metrics and promising gains in trajectory validity and end-effector accuracy.

Critical strengths
  1. An interesting approach to learning conditional source distributions through flow reversal. Although condition-dependent source modeling has been investigated previously, the use of a pretrained flow model to construct explicit supervision for source prediction offers a meaningful methodological distinction. The three-stage procedure is relatively straightforward and can be applied without modifying the underlying flow-matching backbone.

  2. Strong experimental improvements on temporal generation benchmarks. The reported improvements are substantial, particularly on KTH video prediction. The method also outperforms several competing approaches that use structured or learned source distributions.

  3. Demonstrated applicability across different generative tasks. The evaluation covers video prediction, interpolation, and robot motion planning. The additional experiments on the ACWM-Phys benchmark provide evidence that the method can also benefit a larger flow-matching transformer under both in-distribution and out-of-distribution settings.

  4. Useful ablations and practical efficiency improvements. The authors investigate the contributions of the predicted source mean, source variance, flow-reversal supervision, and backbone fine-tuning. The experiments indicate that source learning contributes to the improvements even when the pretrained flow backbone remains frozen.

Critical weaknesses
  1. The novelty relative to existing conditional-source learning methods requires clearer justification. The central idea of learning condition-dependent source distributions for flow matching has already been explored in recent studies, including CAR-Flow (NeurIPS 2025) and Better Source, Better Flow (CSFM, 2026). The main distinctive element of CNP-Flow is therefore its flow-reversal-based supervision. However, the paper could more clearly explain why this supervision provides fundamentally different information from directly learned conditional-source distributions and under which circumstances the additional reversal stage is necessary.

  2. The theoretical explanation does not fully establish why flow reversal should produce a useful nontrivial conditional source distribution For an ideal conditional flow that exactly transports a standard Gaussian source to the conditional data distribution, reversing samples from that distribution should recover the original standard Gaussian source. Therefore, the proposed condition-dependent structure in the reversed source distribution appears to depend on properties of the approximate pretrained model, imperfect distribution matching, or finite numerical integration. The paper does not sufficiently explain this distinction or characterize how the benefit of flow-reversal supervision changes as the pretrained flow model becomes more accurate.

  3. The contribution of source learning versus simpler conditional initialization strategies remains insufficiently characterized. The ablations compare a standard Gaussian, a deterministic predicted mean, and the complete learned Gaussian. However, simpler alternatives, such as directly predicting a conditional source mean from the temporal observations or using a condition-dependent Gaussian fitted without flow reversal, would help clarify the importance of the proposed supervision mechanism.

  4. The motion-planning results are promising but relatively limited in scope. Although the method improves the valid trajectory fraction and end-effector errors, the reported success rate changes only marginally.

Questions for authors

Q1. Relationship to prior conditional-source learning methods. CAR-Flow and CSFM have already investigated condition-dependent source distributions for flow matching. Could the authors clarify the specific advantages of flow-reversal-based supervision over directly optimizing a conditional source distribution? In particular, what properties of the reversed source samples cannot be learned effectively through existing source-alignment or source-regularization objectives?

Q2. Necessity of flow-reversal supervision. Could the authors compare CNP-Flow with a conditional Gaussian source predictor trained through a simpler direct regression or source-alignment objective, using the same network capacity and training budget? This would help clarify whether the improvements come specifically from flow reversal or more generally from learning a condition-dependent source initialization.

Q3. Effect of the source-distribution family. The CNP predicts an isotropic Gaussian source distribution. How sensitive is performance to this assumption, especially when multiple distinct futures are plausible under the same temporal condition? Would a diagonal-covariance Gaussian or a mixture model provide additional benefits?

3 comments

We thank the reviewer insightful comments. We address the concerns below.

[W1, W3 Q1, Q2] What properties of the reversed source samples cannot be learned effectively through existing source-alignment or source-regularization objectives? Why not use a simpler direct regression or source-alignment objective?

The key insight is that Flow Reversal provides explicit supervision of source locations that are already aligned with real targets through the pretrained FM, rather than requiring the model to discover suitable sources from scratch. Unlike source-alignment or regularization objectives, which indirectly encourage favorable source distributions, Flow Reversal directly identifies source samples that map back to real targets. These samples provide a more informative learning signal, transforming an indirect source-search problem into supervised distribution fitting. While existing objectives may also learn suitable distributions, Flow Reversal offers a more direct learning signal that can facilitate optimization. To show the impact of flow-reversal supervision, we compare CNP-Flow with two simpler conditional-source learning objectives: direct source–target regression and source alignment. All methods use the same fine-tuning budget. To avoid collapse, we add a KL regularization term with coeficient λKL\lambda_{\mathrm{KL}}.

KTH 10→30 Results

MethodPSNR ↑SSIM ↑FVD ↓
Direct Regression (λKL=10−4\lambda_{\mathrm{KL}}=10^{-4})30.270.90077.86
Direct Regression (λKL=1\lambda_{\mathrm{KL}}=1)33.340.93446.38
Direct Regression (λKL=100\lambda_{\mathrm{KL}}=100)33.190.93153.18
Source-Alignment (λKL=10−4\lambda_{\mathrm{KL}}=10^{-4})29.460.89086.50
Source-Alignment (λKL=1\lambda_{\mathrm{KL}}=1)33.530.93646.25
Source-Alignment (λKL=100\lambda_{\mathrm{KL}}=100)33.020.93150.20
CNP-Flow (Ours)34.250.94344.76

We tested both arms with different KL regularization. Even with their best-performing settings, both MSE regression and cosine alignment are underperform compared with CNP-Flow in PSNR, SSIM, and FVD. Unlike these geometric alignment objectives, CNP-Flow uses flow reversal to provide explicit source supervision derived from the pretrained FM's transport dynamics.

[W2] Why the reverse distribution is not standard Gaussian? How the benefit of flow-reversal supervision changes as the pretrained flow model becomes more accurate?

We agree with the reviewer that, under an ideal conditional FM with exact transport, Flow Reversal should recover the standard Gaussian distribution. However, in practice, the reversed source distribution may deviate from the standard Gaussian due to pretrained model approximation errors and numerical integration errors. Since the underlying conditional data distribution and exact transport map are generally intractable, precisely disentangling and quantifying these error sources is challenging. Nevertheless, we empirically investigate how the quality of the Stage-1 pretrained FM affects downstream CNP-Flow performance by evaluating checkpoints from different training stages while keeping the Stage-3 training budget fixed.

Stage-1 / Stage-3 IterationsFVD ↓PSNR ↑SSIM ↑
30k / 50k66.1820.190.839
150k / 50k62.3620.190.840
350k / 50k61.1020.160.841

The results show that longer Stage-1 pretraining consistently improves downstream FVD (66.18 → 62.36 → 61.10), while CNP-Flow remains effective even with relatively early pretrained checkpoints.

To investigate the impact of pretrained FM quality, we evaluate different Stage-1 checkpoints while fixing the Stage-3 training budget at 50k iterations.

Total iterationsFixd GaussianCNP-Flow (Ours)
80k88.4566.18
200k68.5762.36
400k66.2261.10

CNP-Flow consistently outperforms the fixed-Gaussian baseline across all checkpoints, achieving relative FVD improvements of 25.2%, 9.1%, and 7.7%, respectively. These results demonstrate that the benefits of CNP-Flow persist across different levels of pretrained FM quality.

[Q3] How sensitive is performance to source distribution, especially when multiple distinct futures are plausible under the same temporal condition?

We adopt an isotropic Gaussian for its simplicity and computational efficiency. To examine whether more expressive source distributions improve generation quality, we compare diagonal Gaussian, covariance, and a four-component Gaussian mixture (GMM) on BAIR64 1->15. All variants use the same training budgets. We also report LPIPS diversity to evaluate whether more expressive distributions better capture multiple plausible futures.

Source DistributionPSNR ↑SSIM ↑FVD ↓LPIPS Diversity ↑
Isotropic Gaussian20.150.83961.170.0686
Diagonal Gaussian20.190.84061.190.0686
Covariance20.130.83960.370.0676
GMM20.190.84061.880.0681

All variants achieve comparable generation quality, with only minor differences across PSNR, SSIM, and FVD. Although covariance improves FVD slightly, it requires approximately 71% additional Stage-2 training time. Moreover, all the other more complex Gaussian variants do not improve LPIPS diversity over our isotropic Gaussian (0.0681 vs. 0.0686). These results show that the isotropic Gaussian is sufficient to provide source expressiveness and maintain a balance between generation quality and computational efficiency.

[W4] The motion-planning results are promising but relatively limited in scope.

Our motion-planning experiment is intended to demonstrate the applicability of CNP-Flow beyond video generation, rather than solely maximizing task-level success rates. In motion planning, success rates are already near saturation (96% vs. 97%), because a task is considered successful if at least one valid trajectory is found among 50 candidates. However, CNP-Flow improves the valid trajectory fraction from 84.9% to 88.5%, and reduces end-effector position and orientation errors by approximately 50%. These results demonstrate that conditional source learning can improve the feasibility and accuracy of robotic motion-planning trajectory beyond video generation.

Thank you for the author's detailed reply. I think it has basically resolved my questions, and I have no further questions.

Reviewer_phLsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CNP-Flow, a method designed to improve the performance of Flow Matching (FM) in temporal generation tasks by learning a condition-dependent source distribution. While conventional conditional FM typically employs a fixed standard Gaussian source distribution—incorporating conditional information solely through the vector field—this approach introduces a Conditional Noise Predictor (CNP) that predicts the mean and variance of an isotropic Gaussian distribution based on temporal conditions.

The training process consists of three stages: first, the FM model is pre-trained, and "Flow Reversal" is used to map real target data back to the source space to construct supervised training samples; next, the CNP is trained using a combination of negative log-likelihood (NLL) and KL-divergence regularization; finally, the FM model is fine-tuned using the predicted conditional source distribution, while the core architecture remains unchanged. The method was evaluated on tasks including video prediction, video frame interpolation, and 7-DoF Franka robot motion planning. Experimental results demonstrate significant improvements: on the KTH 10→30 video prediction task, the FVD score decreased from 180.00 to 44.76, and the success rate for Franka robot trajectory planning increased from 84.9% to 88.5%. Furthermore, the model achieves superior generation quality while requiring fewer function evaluations.

Critical strengths
  1. The proposed method is valuable and easy to integrate. It extends conditional modeling to the source distribution of Flow Matching (FM) models, leveraging the "flow reversal" property of pre-trained models to construct supervisory signals. This approach requires no modifications to the underlying FM backbone architecture; instead, it simply adjusts the starting point of the generation process via a relatively straightforward conditional Gaussian model, offering an alternative pathway for improving conditional generation.

  2. The experimental results are robust, and the ablation studies provide further validation. The method demonstrates performance gains across tasks such as video prediction, video frame interpolation, and robotic motion planning—with particularly notable improvements in long-sequence prediction on the KTH dataset. Figure 4 illustrates the method’s superior quality even with a limited number of function evaluations, while Table 4 confirms the individual contributions of the conditional mean, variance, and FM fine-tuning. Furthermore, the experiments on robotic planning show that the conditional source distribution can yield improvements even without fine-tuning the FM model itself. These findings provide strong evidence for the practical value of the proposed method.

Critical weaknesses

The theoretical motivation behind the core method lacks a sufficient explanation. The paper constructs a condition-dependent source distribution via the inverse mapping of a pre-trained flow-based model (FM) and utilizes this as a key basis for improving time-series generation. However, for an invertible FM capable of precisely matching the true conditional data distribution, the conditional marginal source distribution—obtained by mapping real samples back through the inverse transformation—should theoretically revert to a standard Gaussian distribution. Consequently, the observed condition-dependent structures may, in part, stem from the distribution approximation errors of the pre-trained model. The paper fails to adequately clarify the origin of these structures or their specific relationship to the observed improvements in generation performance. Existing theoretical frameworks, which primarily address the statistical properties and locality of the source distribution, are insufficient to fully substantiate the authors' account of the underlying mechanism. As a result, the applicability of this method—given varying levels of quality in pre-trained models—remains unclear.

Questions for authors

Regarding the conditional source distribution learned via "flow reversal" and its underlying mechanism: for a reversible Conditional Flow Matching (FM) model that precisely matches the target distribution, the conditional marginal source distribution—obtained by mapping real data back through the inverse transformation—should recover a standard Gaussian distribution. How do the authors account for the conditional correlation structures observed in practice? To what extent do these structures reflect distribution approximation errors stemming from the pre-trained model? Could the authors analyze the evolution of the inverse source distribution—and its subsequent impact on CNP performance—by utilizing FM checkpoints from various stages of the training process? Such an analysis would help clarify the method's mechanism and determine whether its benefits remain consistent across pre-trained models of varying quality.

3 comments

We thank the reviewer insightful comments. We address the concerns below.

[W1, Q1] How do the authors account for the conditional correlation structures observed in practice?

We agree that an ideal conditional FM model should recover the standard Gaussian through flow reversal. For an invertible transport map Tθ,cT_{\theta,c}, the inverse source distribution qθ=(Tθ,c−1)♯pdata(⋅∣c)q_\theta=(T_{\theta,c}^{-1})\sharp p_{\mathrm{data}}(\cdot\mid c) satisfies (Tθ,c)♯qθ=pdata(⋅∣c)(T_{\theta,c}) \sharp q_\theta=p_{\mathrm{data}}(\cdot\mid c). Thus, qθq_\theta equals the standard Gaussian if and only if the pretrained flow perfectly matches the target distribution. However, practical pretrained models may exhibit imperfect conditional transport because of approximation error and numerical error. Since approximation errors in the pretrained FM may vary across temporal conditions, the reversed source distribution can exhibit condition-dependent deviations from the standard Gaussian. Similar phenomenon can be found in PYoCo [1] Figure 2.

[1] Ge et al. Preserve your own correlation: A noise prior for video diffusion models. ICCV 2023.

[Q1] To what extent do these structures reflect distribution approximation errors stemming from the pre-trained model? Could the authors analyze the evolution of the inverse source distribution—and its subsequent impact on CNP performance—by utilizing FM checkpoints from various stages of the training process?

The observed conditional structures may partly reflect distribution approximation errors in the pretrained FM, as imperfect conditional transport can introduce condition-dependent deviations in the reversed source distribution. However, precisely disentangling model approximation errors from numerical integration errors is challenging, since the underlying conditional data distribution and exact transport map are generally intractable. To empirically investigate the role of pretrained FM quality, we evaluate CNP-Flow using checkpoints from different training stages while keeping the Stage-3 training budget fixed.

Stage-1 / Stage-3 IterationsFVD ↓PSNR ↑SSIM ↑
30k / 50k66.1820.190.839
150k / 50k62.3620.190.840
350k / 50k61.1020.160.841

The results show that longer Stage-1 pretraining consistently improves downstream FVD (66.18 → 62.36 → 61.10), while CNP-Flow remains effective even with relatively early pretrained checkpoints.

To investigate the impact of pretrained FM quality, we evaluate different Stage-1 checkpoints while fixing the Stage-3 training budget at 50k iterations.

Total iterationsFixed GaussianCNP-Flow (Ours)
80k88.4566.18
200k68.5762.36
400k66.2261.10

CNP-Flow consistently outperforms the fixed-Gaussian baseline across all checkpoints, achieving relative FVD improvements of 25.2%, 9.1%, and 7.7%, respectively. The benefits of CNP-Flow persist across pretrained FMs of varying quality.

Thanks to the authors for their response and clarifications. I will keep my score.

We thank the reviewer for your comments. We would like to know whether you decided to keep the same score because our response did not fully address your concerns. We would be happy to address any remaining concerns. Otherwise, we would appreciate it if you could reconsider raising your score.

Reviewer_TSheOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CNP-Flow, a flow matching framework that learns condition-dependent source distributions to improve generation quality and temporal consistency. Unlike standard conditional flow matching, which relies on a fixed Gaussian source and incorporates conditions primarily through the vector field, CNP-Flow adapts the source distribution to different temporal conditions. Specifically, the method adopts a three-stage training pipeline that first pretrains a conditional flow matching model, then trains a Conditional Noise Predictor (CNP) to estimate the mean and variance of a Gaussian source distribution using samples obtained through flow reversal, and finally fine-tunes the flow matching model with the learned source distribution. The authors also provide theoretical analyses characterizing the learned source distribution and the locality of source samples under similar temporal conditions. Extensive experiments on video prediction, video interpolation, and robotic motion planning demonstrate improvements in generation quality and temporal consistency over existing methods, while additional evaluations show enhanced sampling efficiency with fewer function evaluations.

Critical strengths

The paper presents an interesting approach to temporal generation by learning condition-dependent source distributions rather than relying on a fixed Gaussian prior. In particular, using flow reversal to obtain source supervision is a reasonable and effective design that can be integrated into existing flow matching models without modifying their backbone architectures. Another strength is the extensive experimental evaluation across video prediction, video interpolation, and robotic motion planning, where CNP-Flow achieves consistent improvements over existing methods. For example, FVD decreases from 180.00 to 44.76 on KTH video prediction. The ablation studies further support the effectiveness of flow reversal supervision, learned source distributions, and backbone fine-tuning, while the results also demonstrate improved sampling efficiency under limited NFEs.

Critical weaknesses

My main concern is that the theoretical analysis does not fully explain why learning a condition-dependent source distribution improves generation quality. For an ideal conditional flow matching model, reversing the learned flow should recover the original Gaussian source distribution, so it remains unclear why the reversed source samples would provide a better distribution for generation. Moreover, the theoretical results mainly establish properties of Gaussian moment matching and source locality, without directly demonstrating improved transport accuracy or generation quality. Another concern is the additional training cost. CNP-Flow requires flow reversal, CNP training, and further fine-tuning of the flow matching backbone. Although the ablation studies demonstrate the effectiveness of source learning, a comparison with a fixed-source model trained for an equivalent computational budget would help clarify how much of the improvement comes from the proposed source distribution rather than additional optimization.

Questions for authors
  1. For an ideal conditional flow matching model, flow reversal should recover the original Gaussian source distribution. It is better to explain why the reversed source samples provide a more suitable condition-dependent distribution and how this relates to errors in the pretrained model.

  2. CNP-Flow requires additional flow reversal, CNP training, and backbone fine-tuning. Could the authors compare it with a fixed-source FM trained under the same total computational budget to better isolate the benefits of source learning?

  3. The CNP models each conditional source distribution as an isotropic Gaussian. Please justify this assumption and investigate whether more expressive distributions, such as Gaussian mixtures or distributions with learned covariance structures, could further improve performance.

  4. The reported PSNR and SSIM are based on best-of-100 predictions. Then why not report single-sample performance and additional temporal consistency evaluations to better assess generation quality without relying on best-sample selection?

3 comments

We thank the reviewer insightful comments. We address the concerns below.

[W1-1, Q1] Why can the inverse source deviate from a standard Gaussian and why does flow reversal provide useful supervision?

We agree that an ideal conditional FM model should recover the standard Gaussian through flow reversal. However, practical pretrained models may exhibit imperfect conditional transport because of approximation error and numerical error. For an invertible transport map Tθ,cT_{\theta,c}, the inverse source distribution qθ=(Tθ,c−1)♯pdata(⋅∣c)q_\theta=(T_{\theta,c}^{-1}) \sharp p_{\mathrm{data}}(\cdot\mid c) satisfies (Tθ,c)♯qθ=pdata(⋅∣c)(T_{\theta,c}) \sharp q_\theta=p_{\mathrm{data}}(\cdot\mid c). Thus, qθq_\theta equals the standard Gaussian if and only if the pretrained flow perfectly matches the target distribution.

The key insight is that Flow Reversal provides explicit supervision of source locations that are already aligned with real targets through the pretrained FM, rather than requiring the model to discover suitable sources from scratch. Unlike source-alignment or regularization objectives, which indirectly encourage favorable source distributions, Flow Reversal directly identifies source samples that map back to real targets. These samples provide a more informative learning signal, transforming an indirect source-search problem into supervised distribution fitting. While existing objectives may also learn suitable distributions, Flow Reversal offers a more direct learning signal that can facilitate optimization. Our experiments support this advantage, with CNP-Flow achieving lower FVD (44.76) than direct regression (46.38) and source alignment (46.25).

[W1-1] The theoretical analysis does not fully explain why learning a condition-dependent source distribution improves generation quality.

To explain why learning a conditional source distribution can improve generation, we establish a theoretical connection between source approximation and generation discrepancy.

Theorem. Let Tθ,cT_{\theta,c} denote the invertible pretrained FM transport map, qθ=(Tθ,c−1)♯pdata(⋅∣c)q_\theta=(T_{\theta,c}^{-1}) \sharp p_{\mathrm{data}}(\cdot\mid c) the inverse source distribution, and pϕ=N(μϕ(c),σϕ2(c)I)p_\phi=\mathcal{N}(\mu_\phi(c),\sigma_\phi^2(c)I) the learned conditional source. Assuming Tθ,cT_{\theta,c} is LcL_c-Lipschitz, 0<σϕ(c)≤10 < \sigma_\phi(c) \leq 1, and finite KL divergence, the Lipschitz stability of Wasserstein distance yields

W2((Tθ,c)♯pϕ,pdata(⋅∣c))≤Lc2DKL(qθ∥pϕ).W_2\left((T_{\theta,c}) \sharp p_\phi, p_{\mathrm{data}}(\cdot\mid c) \right) \leq L_c \sqrt{2D_{\mathrm{KL}}(q_\theta\|p_\phi)}.

Since minimizing Gaussian NLL is equivalent to minimizing DKL(qθ∥pϕ)D_{\mathrm{KL}}(q_\theta\|p_\phi) up to a constant independent of ϕ\phi, this establishes a theoretical connection between our source learning objective and an upper bound on generation discrepancy under the fixed pretrained flow.

Proof Sketch. Since qθ=(Tθ,c−1)♯pdata(⋅∣c)q_\theta=(T_{\theta,c}^{-1}) \sharp p_{\mathrm{data}}(\cdot\mid c), we have (Tθ,c)♯qθ=pdata(⋅∣c)(T_{\theta,c}) \sharp q_\theta=p_{\mathrm{data}}(\cdot\mid c). By the LcL_c-Lipschitz continuity of Tθ,cT_{\theta,c}, the Wasserstein distance between two distributions under the same pushforward mapping is bounded by

(T_{\theta,c}) \sharp p_\phi, p_{\mathrm{data}}(\cdot\mid c) \right) = W_2\left( (T_{\theta,c}) \sharp p_\phi, (T_{\theta,c}) \sharp q_\theta \right) \leq L_c W_2(p_\phi,q_\theta).$$ Since $p_\phi=\mathcal N(\mu_\phi(c),\sigma_\phi^2(c)I)$ is an isotropic Gaussian, by Talagrand's inequality with $0<\sigma_\phi^2(c) \leq 1$, $W_2(p_\phi,q_\theta) \leq \sigma_\phi(c) \sqrt{2D_{\mathrm{KL}}(q_\theta||p_\phi)} \leq \sqrt{2D_{\mathrm{KL}}(q_\theta||p_\phi)}.$ Therefore, \$W_2\left( (T_{\theta,c}) \sharp p_\phi, p_{\mathrm{data}}(\cdot\mid c) \right) \leq L_c \sqrt{2D_{\mathrm{KL}}(q_\theta || p_\phi)}.\$

[W1-2, Q2] Are the improvements come from conditional source learning or solely from additional training and optimization budget?

To show the benefits of conditional source learning from additional optimization, we compared our method with fixed-source FM baseline under the same total computational budget on BAIR 1 -> 15 and KTH 10→30.

  • BAIR 1→15
SourcePSNR ↑SSIM ↑FVD ↓
Fixed-source FM (Equal-budget)20.130.83563.16
CNP-Flow (Ours)20.150.83961.17
  • KTH 10→30
SourcePSNR ↑SSIM ↑FVD ↓
Fixed-source FM (Equal-budget)33.090.93154.01
CNP-Flow (Ours)34.250.94344.76

CNP-Flow outperforms the standard Gaussian baseline on both datasets. The results suggest that the improvements cannot be explained solely by additional fine-tuning, but by the conditional source learning by flow reversal.

[Q3] Is an isotropic Gaussian sufficient to express source distribution? Could more complex source distributions improve generation quality?

We adopt an isotropic Gaussian because for its simplicity and computational efficiency. To examine whether more expressive source distributions would improve generation quality, we compare diagonal Gaussian, full-rank covariance, and a four-component Gaussian mixture (GMM) on BAIR64 1->15. All variants use the same training budgets. We also report LPIPS diversity to evaluate whether more expressive distributions better capture multiple plausible futures.

Source DistributionPSNR ↑SSIM ↑FVD ↓LPIPS Diversity ↑
Isotropic Gaussian20.150.83961.170.0686
Diagonal Gaussian20.190.84061.190.0686
Covariance20.130.83960.370.0676
GMM20.190.84061.880.0681

All variants achieve comparable generation quality, with only minor differences across PSNR, SSIM, and FVD. Although full-rank covariance slightly improves FVD, it requires approximately 71% additional Stage-2 training time. Moreover, all the other more complex Gaussian variants do not improve LPIPS diversity over our isotropic Gaussian (0.0681 vs. 0.0686). These results suggest that the isotropic Gaussian is sufficient to provide source expressiveness and maintain a balance between generation quality and computational efficiency.

[Q4] Why not report single-sample performance but best-sample selection?

We follow the established best-of-100 evaluation protocol in SRVP [1] to ensure fair comparisons with previously published baselines. Since stochastic video prediction admits multiple plausible futures, best-of-100 PSNR/SSIM evaluates whether the generated samples include a prediction close to the ground truth. As our baseline results are taken from the original paper, single-sample performance and additional temporal consistency metrics are not consistently available for direct comparison. However, we also report FVD, which evaluates the generated video distribution using spatiotemporal features, providing a complementary assessment beyond best-of-100 selection.

[1] Franceschi et al. Stochastic latent residual video prediction. ICML 2020.

Thank you for the authors' great efforts! I have no further questions. Best of luck!

Comments on the paper

1 comment

Summary of Rebuttal and Additional Experiments

We thank the reviewers for their constructive feedback. During the rebuttal, we provided additional theoretical analysis and experiments to address the major concerns:

  1. Theoretical Justification: We clarified why imperfect pretrained Flow Matching models can induce condition-dependent source distributions and established a theoretical connection between source approximation and generation discrepancy through a Wasserstein-distance bound.

  2. Effectiveness of Flow Reversal: We compared CNP-Flow against direct regression and source-alignment baselines under matched training budgets. CNP-Flow achieved the best FVD (44.76 vs. 46.38/46.25), supporting the benefit of explicit flow-reversal supervision.

  3. Fair Computational Comparison: Under equal total computational budgets, CNP-Flow consistently outperformed fixed-source FM on both KTH and BAIR, confirming that the improvements are not solely attributable to additional training.

  4. Robustness to Pretrained Model Quality: Across different pretrained FM checkpoints, CNP-Flow consistently improved FVD over fixed-Gaussian baselines, with relative improvements of 25.2%, 9.1%, and 7.7%.

  5. Source Distribution Analysis: Additional comparisons with diagonal Gaussian, full-covariance Gaussian, and GMM demonstrated comparable generation quality and diversity, supporting the efficiency-performance tradeoff of our isotropic Gaussian design.

Reviewer Feedback: Reviewer TShe and ixz1 explicitly indicated that their questions had been resolved or that no further questions remained. Reviewer phLs maintained their original score, and we have invited further clarification regarding any unresolved concerns.

We hope these additional analyses and experiments help clarify the theoretical motivation, methodological contributions, and empirical effectiveness of CNP-Flow. We respectfully ask the AC to consider these rebuttal results in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.