acceptodds
Withdrawn from ICLR 2027

Useful Before It Generates Well: Model-Aligned Sparse Sinkhorn for Flow Retraining

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Partially trained flow models are usually judged by sample quality, yet they may already carry enough coarse geometric structure to improve later training. We ask whether such early-stage models can be useful before they generate well, by serving as coupling constructors for flow retraining. The available coupling choices fail differently. Independent coupling is simple but high variance. Rectified-Flow-style model-generated coupling reduces variance but inherits teacher and solver drift. Optimal-transport couplings are principled but too expensive at dataset scale. MASS (Model-Aligned Sparse Sinkhorn) uses model-generated endpoints from an early-stage flow only to define candidate neighborhoods in the real dataset, then applies semi-relaxed Sinkhorn reweighting on the resulting sparse graph to produce a real-data-anchored transport plan at cost rather than . The plan is built once before retraining and reused across all training steps, unlike minibatch OT methods that re-solve an OT problem at every batch, and it composes with any flow-matching objective. We characterize sufficient conditions on the neighborhood size under which the MASS empirical target marginal provably improves over the pretrained model's output distribution. On a partially trained CIFAR-10 teacher, the induced-target FID drops from ( NN) to ( MASS), and CIFAR-10 retraining FID improves from (iMF) and (Dec-iMF) to (MASS-ReiMF) at matched compute. On 2D moons, MASS achieves transport cost within of full global OT.

Primary area:
generative models
Keywords:
flow matching, optimal transport, Sinkhorn, coupling, rectified flow, mean flow, generative models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BG5vOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

MASS uses endpoint from the partially trained flow model to identify nearby real training examples, then solve a semi-relaxed entropic optimal transport problem on the resulting sparse graph. The fixed plan supplies training pairs during continuation. The paper proposes a threshold-kk for empirical target-marginal quality and evaluates coupling diagnostics, CIFAR-10 retraining, synthetic transport, and single-cell interpolation.

Critical strengths
  1. In Table 3, shuffling the alignment leaves induced-target FID almost unchanged, 0.63 versus 0.64, but worsens retrained FID from 17.46 to 20.30. This supports the importance of the source-to-target correspondence beyond matching target statistics.

  2. Figure 4 reports an actual continuation benefit. The five-seed single-cell experiment also reaches W1=0.764±0.010W_1=0.764\pm0.010, comparable to exact global OT at 0.768±0.0190.768\pm0.019 (Section 4.3). The paper explicitly acknowledges its limited benefit with an already strong teacher.

Critical weaknesses
  1. Let r=f^/fr=\widehat f/f and Δ=DKL(p^1∥p1)>0\Delta=D_{\mathrm{KL}}(\widehat p_1\|p_1)>0, as defined in Section 3.4. The proof of Theorem A.9 establishes ∥r−1∥∞2≤C32Δ1/(d+1),\|r-1\|_\infty^2\le C_3^2\Delta^{1/(d+1)}, and Corollary A.10 sets a3=3C32a_3=3C_3^2. However, \le \chi^2(\widehat p_1\|p_1) \le \|r-1\|_\infty^2 \le C_3^2\Delta^{1/(d+1)}.$$ The first inequality follows from $\log r\le r-1$ and normalization of both densities. Consequently, $$\Delta_\star =\Delta-a_3\Delta^{1/(d+1)} \le -2\Delta<0.$$

This leads to, the required premise Δ⋆>0\Delta_\star>0 in Theorems 3.3/A.12 can't hold with the constants supplied by the proof. Naturally, all anlaysis in the paper simply unreliable. Appendix A.6 checks graph feasibility directly and does not demonstrate a nonvacuous instance of A.12. This

  1. SD-FM[1] already precomputes a dataset-wide coupling potential, including for MeanFlow. COT-FM [2] uses a pretrained flow to construct reusable OT maps to real data. These methods are absent from the positioning and experiments. Including these previous works completely voids the proposed novely in thi spaper.
  • Eq-14: Lipschitz densities give an O(rd+1)O(r^{d+1}) small-ball remainder, rather than the stated O(rd+2)O(r^{d+2}). The weaker remainder appears compatible with the subsequent cross-mass rate.

  • Appendix B: Algorithm 3 relaxes the source marginal and fixes the target, whereas Equation (6) and Algorithm 4 fix the source and relax the target. Explain this change of orientation.

  • Table 2 assigns practical MAC an O(B2)O(B^2) cost, but its Section 3.2 scores BB sampled candidate pairs [3]. Section 5 also incorrectly states that Spar-Sink necessarily shifts both marginals, its balanced version enforces both prescribed marginals when feasible and converged [4].

[1] Mousavi-Hosseini et al. Flow Matching with Semidiscrete Couplings, 2025. https://arxiv.org/abs/2509.25519

[2] Chiang et al. COT-FM: Cluster-wise Optimal Transport Flow Matching, 2026. https://arxiv.org/abs/2603.13395

[3] Lin et al. Beyond Optimal Transport: Model-Aligned Coupling for Flow Matching, 2025. https://arxiv.org/abs/2505.23346

[4] Li et al. Importance Sparsification for Sinkhorn Algorithm, JMLR, 2023. https://www.jmlr.org/papers/v24/22-1311.html

Questions for authors
  1. Please address the inequality in Weakness 1 explicitly. Can you provide a nonvacuous replacement for Theorem A.12.

  2. Can you extend the minibatch-OT and random-support controls to the common 400k horizon and compare against a directly relevant global method. eg: SD-FM or COT-FM.

  3. Which endpoint budget was used for Figure 4 and Table 6: m=10m=10 from Section 4.2 or m=50m=50 from Appendix C.1?

Please state whether MASS and reflow use identical teacher endpoints and separate the costs of endpoint generation, IVF search, Sinkhorn iterations, and continuation. Clarify which costs the advertised O(kN)O(kN) complexity includes.

Reviewer_ZZUTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Model-Aligned Sparse Sinkhorn (MASS), a coupling-construction method for retraining flow models. It utilizes a partially trained flow model with established geometry for semi-relaxed Sinkhorn to construct a sparse coupling between source noise and real targets, thus improving sample quality & reduce sampling curvature.

Critical strengths
  1. MASS is a plug-and-play module that can be combined with any model & different flow trajectory training objectives, and computing the plan once avoids solving a new transport problem at every training iteration.
  2. The motivation is compelling, and the construction is practical. Using early model predictions to locate real-data candidates provides a sensible way to exploit partially learned geometry.
Critical weaknesses

My main concerns lie in both the theorems proposed and the experiment results supporting the method's effectiveness.

  1. The positive-gap condition in Theorem 3.3 appears impossible under the stated population-KL baseline. Section 3.4 defines $ [ \Delta_{\mathrm{KL}}=D_{\mathrm{KL}}(\hat p_1|p_1). ] $ Writing (r=f^/fr=\hat f/f), the proof of Theorem A.9 establishes $ [ |r-1|\infty \le C_3\Delta{\mathrm{KL}}^{1/[2(d+1)]}. ] $ However, the standard KL–chi-square inequality gives $[ \Delta_{\mathrm{KL}} \le \mathbb E_{p_1}[(r-1)^2] \le |r-1|\infty^2 \le C_3^2\Delta{\mathrm{KL}}^{1/(d+1)}. ]$ Since Corollary A.10 sets (a_3=3C_3^2), this implies $[ \Delta_\star =\Delta_{\mathrm{KL}} -a_3\Delta_{\mathrm{KL}}^{1/(d+1)} \le -2\Delta_{\mathrm{KL}}\le0, ] $ contradicting the required assumption (Δ⋆>0)(\Delta_\star>0).
  2. Practical value of the method remains unclear. Random support with Sinkhorn reaches FID 18.14 versus 17.46 for MASS after the short continuation in Table 3. In Table 6, minibatch OT with batch size 256 outperforms MASS at 150k steps and trails it only slightly at 200k: 16.73 versus 16.31.
  3. Unfair baseline comparison. Section 4.2 describes generating the ReFM training pairs with 10 NFE, whereas Appendix C.1 uses 50 NFE to construct MASS. Also the FID numbers from the baseline methods seems a bit off (21.43 for iMF and 15.53 for Dec-iMF), given MF (Geng et al., 2025) already reaches 2.92 with 800k training steps.
Questions for authors
  1. Could the authors clarify how can the condition (Δ⋆>0\Delta_\star>0) hold given the density-ratio bound and the definition (a3=3C32a_3=3C_3^2)?
  2. What happens when ReFM and MASS use exactly the same teacher-generated endpoints and endpoint NFE? Does the reported construction time include the full endpoint-generation pass?
  3. Could the authors clarify the training setup of the baseline methods? Without a matching training & evaluation setup, it is difficult to assess how much of the reported improvement reflects the proposed coupling versus weaknesses in the baseline configurations.
Reviewer_4cZ1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose MASS, a novel data coupling algorithm, that re-weights the sampling distribution of instances from the target set, according to their similarity with outputs from a (partially) trained teacher model and a sampled instance from the source distribution following the Semi-Relaxed Robust Sinkhorn algorithm. The authors provide theoretical guarantees that MASS yields a more representative target distribution compared to that obtained directly from the outputs of a teacher, and validate their approach on the CIFAR-10 dataset. They should that MASS can yield better retraining over flow matching and iMF over time when a partially trained teacher is utilized, but not necessarily when using a fully trained teacher.

Critical strengths
  • The authors propose a novel data-coupling algorithm that reduces a data-anchored transport plan to O(kN) from O(N^2). The authors show how employing MASS with partially trained teachers improves FM training substantially on CIFAR-10 across different settings, and further theoretically demonstrate that MASS yields a better aligned sampling distribution than the relevant baseline.
  • MASS can be performed once at the beginning of training, or per-epoch, decreasing its coupling impact on compute.
Critical weaknesses
  • The case for using a partially trained model as a teacher is unconvincing. The authors state several times throughout the submission that a partially trained teacher has already learned sufficient structure to be a useful guide for training a new model, yet they do not explain why one would use a partially trained teacher over a fully trained one. The closest argument to avoiding this scenario that I could find in the submission is that utilizing the teachers outputs directly as targets can cause drift; yet it does not mean that utilizing them with MASS would yield poor performance. This point appears corroborated by Lines 403-405, where the authors find ReFM outperforming MASS ReFM with a single NFE compared to 4 NFE.
    • Stating the last sentence as a main weakness: A better trained teacher outperforms MASS with a partially trained teacher on CIFAR-10.
  • MASS appears to bias the target distribution. From what I understand, MASS reweights the likelihood of sampling instances from the target distribution, according to the output generations of a teacher. This can cause the resulting target distribution to be biased relative to the actual distribution. I wonder if the perceived benefits from MASS occur because of this bias, rather than necessarily target reconstruction. Could the authors please show the precision / recall metrics for all MASS models and their baselines on the CIFAR-10 dataset?
  • CIFAR-10 is not sufficient to understand the utility of MASS. ImageNet-1K (preferably 256x256) is an established benchmark in image generation, and it would be very helpful to understand the benefits of MASS under this setting.
  • How do MASS trained models fare against baselines under CFG? In this case, in CIFAR-10?
  • Is retraining necessary? How does MASS perform on CIFAR-10 when students are trained from scratch? Does MASS require the student model to be partially trained for stability?
Questions for authors

Clarification requests:

  • Table 1 does not support the statement in line 355 that k plateaus around 100 to 200. There is still substantial improvements at K=500, suggesting the real plateau is past that mark. I wonder if the authors could better argue the case for K=100-200 through a compute lens instead.
  • How robust is MASS to different NFE from the teacher?
  • How much faster does MASS denoise over other FM variants in the partially trained setting? (e.g., ReFM, iMF, FM)
  • It is not clear why demonstrating that MASS is better than the output distribution of a teacher is meaningful, when the ideal is matching the target distribution itself (without utilizing MASS or the outputs of the teacher).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.