acceptodds
Under review as a conference paper at ICLR 2027

Prediction-Loss Alignment for Sampler-Robust Flow Matching Training

Abstract

Recent work has popularized a practical recipe in diffusion and flow matching: predict the clean signal , convert it to a velocity, and train through a velocity-space loss. The conversion contains a singular endpoint amplification and therefore appears prone to unstable optimization, yet recent systems obtain strong empirical results with this recipe. We investigate this tension through the integrability of the pre-optimizer stochastic-gradient second moment. Under stated initialization conditions, the moment diverges under Uniform sampling; boundary-suppressing sampling can restore integrability under an additional upper-growth condition. We then show that prediction-loss alignment eliminates this conversion-induced source of non-integrability. Under a uniform moment bound, alignment yields a finite second moment for every timestep density, including Uniform sampling. Controlled experiments across continuous and binary settings reproduce the predicted sampler-dependent instability and show that aligned objectives remain trainable across the tested samplers. These results reconcile pointwise amplification with sampler-dependent empirical success and support alignment as a principled route to more robust flow-matching training.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
flow matching, diffusion models, prediction-loss alignment, timestep sampling, training stability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FiVKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper discusses the singularity issue caused by the x-pred+v-MSE loss during the flow-matching training. By considering the second moment of the aligned loss gradient and the mismatched loss gradient, it is rigorously shown that the aligned version has finite second moment, while the mismatched version has infinite second moment, under the usual uniform sampling of training timesteps. The paper then proposed a novel way of sampling the timesteps for training using Logit-Normal endpoint suppression, successfully bounding the second moment of the mismatched version gradient with a finite number. Experiment results on the U-Net and JiT-B/4 model structures across different loss types further validates the theoretical results.

Critical strengths

The motivation of the paper is clear and attractive, which comes from the singularity caused by the derived velocity-prediction loss. Rigorous theoretical derivations are conducted inside of the paper, and the experiments show significant differences in both loss and generation quality between the Uniform and Logit-Normal samplings during the training, in the x-pred+v-MSE case. Different model backbones are used for the experiments, so that the results are more convincing across different model structures.

Critical weaknesses

Some weakness points are listed below:

  1. For the experiments, it might be helpful if more than 3 different seeds are used to calculate the mean and standard deviation in for example Figure 3.
  2. Some concepts are not defined clear enough inside of the main paper, for example the aligned loss and the mismatched losses. Worth mentioning that the "aligned loss" corresponds to the x-pred+x-MSE, and the "mismatched loss" corresponds to the x-pred+v-MSE, perhaps in a table.
  3. I noticed that the ODE samplers/solvers during the inference are different across experiments. Specifically, the U-Net Binary-MNIST experiments used Euler's method and the JiT-B/4 experiments used Heun's method. It is better to use consistent ODE solvers to exclude any possible effects ODE solvers bring during the inference rather than what's in the training.
Questions for authors

Here are several questions:

  1. I saw inside of the appendix that for U∼N(m,s2)U\sim\mathcal{N}(m, s^2) in the Logit-Normal endpoint suppression, the experiments chose m=−0.8,s=0.8m=-0.8, s=0.8; intuitively, since in the mismatched singularity happens around t=1, we would like to avoid that region, so T=σ(U)T=\sigma(U) should be away from 11 and UU should in expectation be negative, which is fine. However, this also indicates that when we choose m>0m>0 (eg. m=0.8m=0.8) and some s>0s>0 (and think about the case when ss is small), then TT will likely to fall into the regime that's close to 11 so that singularity continues to dominate; then
    \quad(1) Can you clarify if my understanding and intuitions are correct?
    \quad(2) If what I mentioned are correct, then the Logit-Normal endpoint suppression can only conditionally solve the singularity issue, while Proposition 4.3 does not reflect the distinction about the sign of mm; some more experiments/theoretical explanations about this part would be very much appreciated.
  2. The usage of CC inside of the "Endpoint weighting and generation quality" paragraph is not so clear to me; can you elaborate more on what that is and why is that important? A more clear explanation in the later versions of the main paper will be benificial.

I will raise my score to 3 if both questions are addressed. Edit: The authors addressed my questions thoroughly. I've raised my rating to 3.

7 comments
AuthorsOpenReview ↗

W1 — Ten-Seed Results and Run-to-Run Variability

Thank you for suggesting additional training seeds. We have expanded all eight Binary-MNIST objective–sampler configurations to ten independent training seeds (42–51). The updated FID-50K results are:

Prediction / supervisionUniformLogit-Normal
xx-prediction / vv-MSE (mismatched)1306.61±276.481306.61\pm276.4812.58±3.9512.58\pm3.95
xx-prediction / xx-MSE (aligned)16.50±6.6916.50\pm6.6911.67±2.8111.67\pm2.81
vv-prediction / vv-MSE (aligned)21.09±5.6121.09\pm5.6116.75±4.6616.75\pm4.66
Signal-space BCE (aligned)13.31±3.4813.31\pm3.4811.10±2.6011.10\pm2.60

Values are means ± sample standard deviations across ten independently trained generators. Every run uses the final 100-epoch checkpoint and the same evaluation protocol, with generation randomness held fixed. Thus, the reported variability reflects training-seed variation.

The strong sampler dependence of mismatched training is reproduced across training seeds: Uniform sampling yields severe generation degradation, whereas Logit-Normal supports effective generation. The aligned objectives support effective generation under both tested samplers. These results support the robustness comparison without implying identical FID across samplers or a universally optimal objective.

Figure 3 now reports ten-seed FID statistics. Its training curves and sample grids retain the explicitly identified seed-42 visualization. All individual seed scores are provided in Appendix B.3, Table 9, and the aggregate statistics appear in Appendix B.8, Table 12.

We also completed ten paired training seeds for the Binary-MNIST fixed-objective control (Appendix B.8, Table 13) and three training seeds for each of the six JiT xx-prediction configurations, using a common 400-epoch evaluation protocol (Appendix B.9, Tables 14–15). These additional results quantify variability beyond the Binary-MNIST main comparison.

AuthorsOpenReview ↗

W2 — Definitions of Aligned and Mismatched Supervision

Thank you for suggesting a clearer presentation. We have clarified the definitions in §3.2 and the notation guide in Appendix A. Prediction–loss alignment means that the directly predicted variable and the training supervision occupy the same space. Mismatch occurs when the prediction is converted to another space before applying the training loss.

The main MSE combinations are:

Direct network predictionTraining supervisionClassification
Clean signal xxSignal-space xx-MSEAligned
Velocity vvVelocity-space vv-MSEAligned
Clean signal xxVelocity-space vv-MSE after conversionMismatched

In our xx-prediction comparisons, “aligned” means xx-prediction + xx-MSE, and “mismatched” means xx-prediction + converted vv-MSE. More generally, direct velocity regression also satisfies the alignment principle.

For the same xx-predictor, exact conversion without a denominator floor gives

ℓv(t)=ℓx(t)(1−t)2.\ell_v(t)=\frac{\ell_x(t)}{(1-t)^2}.

The additional endpoint factor introduced by this conversion is the distinction analyzed in the paper (§3.2, Equation (5)). Signal-space BCE provides another form of direct supervision for binary targets through logits.

The revised manuscript uses these terms consistently and includes a notation table for the associated quantities (Appendix A, Table 5).

AuthorsOpenReview ↗

W3 — Euler-50 versus Heun-50 at Fixed Checkpoints

We have added a fixed-checkpoint solver comparison for all six JiT xx-prediction configurations. For training seed 0, we evaluate each 400-epoch checkpoint using both Euler-50 and Heun-50. The checkpoint, EMA, generation seed, guidance, generation denominator floor, and FID reference set are held fixed. Each evaluation generates 50,000 images.

FID-50K results (Appendix B.10, Table 16):

Training configurationTraining samplerHeun-50Euler-50
Mismatched vv-MSE, 10−610^{-6} floorUniform402.37404.58
Mismatched vv-MSE, 10−610^{-6} floorLogit-Normal15.1414.01
Mismatched vv-MSE, 0.050.05 floorUniform14.6914.35
Mismatched vv-MSE, 0.050.05 floorLogit-Normal15.0414.01
Aligned xx-MSEUniform16.8016.29
Aligned xx-MSELogit-Normal24.0323.04

Both solvers preserve the severe generation failure of mismatch+Uniform under weak endpoint protection and effective generation for the other configurations. Thus, the principal qualitative contrast persists when the inference solver is changed.

At the standard 0.050.05 training floor, the small sampler ranking reverses: Uniform has the lower FID with Heun, whereas Logit-Normal has the lower FID with Euler. Solver choice can therefore affect fine FID rankings without removing the observed endpoint contrast. These are paired evaluations of the same seed-0 checkpoints, rather than multi-seed solver statistics.

Both schemes use 50 integration steps. Our Heun implementation performs 49 Heun updates followed by one Euler update, requiring 99 network evaluations; Euler-50 requires 50. The comparison therefore matches integration steps rather than computational cost.

The original objective–sampler comparisons already used a consistent solver within each task. The additional Euler-50 results also provide a JiT evaluation using the same solver and step count as Binary-MNIST, directly addressing the requested consistency check.

Reviewer_YPfPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies endpoint gradient amplification when a clean-data predictor is trained with a velocity-space loss. It derives initialization-time conditions for divergence of the gradient second moment and explains how timestep sampling affects integrability.

Critical strengths
  • The distinction between pointwise gradient amplification and its integrated second moment is useful. It explains how an apparently singular recipe can still work with endpoint-suppressing sampling.
  • The paired gradient diagnostics and objective–sampler comparisons support the proposed mechanism. The endpoint-cap sweep also shows that bounded weights can still seriously hurt training when they concentrate heavily near the endpoint.
Critical weaknesses
  • The main theoretical distinction concerns gradient moments at initialization, not training convergence. It therefore does not establish that the identified divergence causes optimizer failure.
  • Alignment changes the expected objective as well as gradient variability. The experiments do not isolate which change explains the training improvements.
  • The main JiT comparison uses a 10−610^{-6} denominator floor rather than the published 0.05 safeguard. It does not establish that the proposed remedy is needed under the standard recipe.
  • The aligned-objective guarantee assumes a uniformly bounded gradient second moment. Under that assumption, integrability for any timestep density follows immediately; the difficult question is whether the bound holds during training.
  • The practical advantage is sampler tolerance rather than better generation quality: mismatch with Logit-Normal sampling achieves the best reported JiT FID. With one run per JiT configuration, even this tradeoff is not well quantified.
Questions for authors
  • Does alignment retain a meaningful robustness advantage with the published 0.05 denominator floor and comparable optimizer tuning?
  • Can you isolate the effect of gradient variability while holding the expected training objective fixed?
5 comments
AuthorsOpenReview ↗

W1 — Scope of the Moment Analysis and Optimization Relevance

This concern involves two distinct questions: whether the mechanism extends beyond initialization, and what it establishes about optimization. The absence of a full training-convergence theorem does not imply the absence of optimization relevance.

1. The mechanism is not restricted to initialization.

Initialization provides a setting in which endpoint gradient lower bounds can be checked explicitly from the output-head structure. To address trained states, §4.1 now includes two additional results. Both concern exact conversion without a denominator floor, at fixed finite parameter states satisfying the stated regularity conditions, with an unscaled, trainable additive output bias SbSb, where S≠0S\neq0 is fixed.

  • Nonzero endpoint residual. If rθ(t)→rθ(1)r_\theta(t)\to r_\theta(1) in L2L^2 and E∥S⊤rθ(1)∥2>0\mathbb E\|S^\top r_\theta(1)\|^2>0, the mismatched gradient second moment under Uniform sampling diverges. The trainable bias transfers the projected residual into an endpoint gradient lower bound. This condition can hold at trained states, independently of initialization (§4.1, “Nonzero endpoint residual”; Appendix A.3, Corollary A.2 and its first instance).

  • Gaussian signals. For unconditional, nondegenerate Gaussian signals and independent standard Gaussian noise, a predictor observing only (Zt,t)(Z_t,t) has an infinite mismatched gradient second moment under Uniform sampling at every fixed state satisfying the common regularity and output-bias conditions, including states realizing the Bayes predictor. No nonzero endpoint residual or endpoint L2L^2 convergence is required. Irreducible posterior uncertainty supplies an order-(1−t)2(1-t)^2 lower bound on the projected residual second moment, yielding an order-(1−t)−2(1-t)^{-2} gradient-second-moment lower bound after conversion. Thus, even exact endpoint reconstruction does not remove the divergence. The result also applies conditionally on training history when the next data, noise, and timestep samples are independent of that history (§4.1, “Gaussian signals”; Appendix A.3, Proposition A.3, Equations (43)–(46), and “Training states and scope”).

2. The gradient-moment result has a direct optimization interpretation, complemented by actual continuation experiments.

For single-sample plain SGD at a fixed state and fixed learning rate η>0\eta>0, Equation (17) gives

Δθ=−ηg^.\Delta\theta=-\eta\hat g.

=\eta^2\mathbb E\|\hat g\|^2.$$ Therefore, **gradient-second-moment divergence directly implies parameter-update-second-moment divergence**. This is a property of stochastic optimization updates, without requiring a theorem about convergence of the entire trajectory. The identity alone does not establish numerical optimizer failure; actual training degradation is examined through paired continuations from trained checkpoints. **The continuation results show amplification and degradation after training has already progressed:** | Continuation experiment | First-minibatch gradient ratio: mismatch / aligned | Aligned outcome | Mismatched outcome | |---|---:|---|---| | Binary-MNIST: 3 trained checkpoints, 2,000 steps | **$306$–$2766\times$** | FID-50K: **$30.08\pm19.13$** | FID-50K: **$1554.13\pm612.71$** | | JiT: 1 trained checkpoint, 1,000 steps | **Approximately $19{,}896\times$** | Reference mean signal MSE | **43% higher mean signal MSE** | Binary-MNIST continuations retain optimizer states; FID values are means ± sample standard deviations across the three checkpoint continuations. JiT uses a fresh AdamW optimizer for each branch, and both branches remain numerically finite. **Switching to mismatched supervision at trained states produces substantial gradient amplification together with degraded generation or signal fitting** (§5.1; Appendix B.6, Figure 9, for frozen-checkpoint diagnostics; Appendix B.6.1 for paired continuations). The additional AdamW/SGD update diagnostics also show reproducible differences in actual parameter-update fluctuations under the standard $0.05$ safeguard; we report these in our response to W3 (§6, Figure 5; Appendix B.11).
AuthorsOpenReview ↗

W2 and Question 2 — Fixed-Objective Control of Stochastic-Gradient Behavior

We appreciate the distinction between the expected objective and its stochastic estimator, and have added fixed-objective controls in direct response to Question 2. These controls preserve the aligned-Uniform expected objective while retaining pointwise mismatched supervision.

The objective reweighting highlighted in W2 is itself the source of conversion-induced gradient amplification: the same timestep-dependent factor scales both the pointwise loss and its parameter gradient. This connection is explicitly analyzed in §3.2, Equations (5)–(9), and §6, Equation (26). Our analysis concerns this joint effect; it does not attribute final FID differences solely to gradient variance.

For the fixed-objective control, let wϵ(t)=max⁡{1−t,ϵ}−2w_\epsilon(t)=\max\{1-t,\epsilon\}^{-2} and define

Aϵ=∫01wϵ(t)−1 dt.A_\epsilon=\int_0^1 w_\epsilon(t)^{-1}\,dt.

qϵ(t)=1Aϵwϵ(t).q_\epsilon(t)=\frac{1}{A_\epsilon w_\epsilon(t)}.

We sample T∼qϵT\sim q_\epsilon and optimize Aϵℓv,ϵ(T)A_\epsilon\ell_{v,\epsilon}(T), where ℓv,ϵ(t)=wϵ(t)ℓx(t)\ell_{v,\epsilon}(t)=w_\epsilon(t)\ell_x(t). Prediction and target use the same conversion denominator. Consequently,

&\mathbb E_{T\sim q_\epsilon}[A_\epsilon\ell_{v,\epsilon}(T)]\\ &\qquad=\int_0^1\mathbb E[\ell_x(t)]\,dt. \end{aligned}$$ Expectations also include data and noise. **The expected objective—and, when integrable, the expected gradient at the same parameter state—is identical to aligned+Uniform** (§5.2, Equation (25); Appendix A.7, Equations (61)–(65)). **Equal expected gradients do not imply identical stochastic-gradient distributions.** Aligned+Uniform samples timesteps uniformly and uses the signal-space gradient directly. Compensation changes how frequently each timestep is sampled and multiplies each sampled signal-space gradient by $A_\epsilon w_\epsilon(t)$. These changes cancel in expectation, but generally produce different gradient distributions and second moments. Thus, the control fixes the expected objective while testing the stochastic estimator as a whole, rather than variance alone. **Differences in stochastic optimization provide a potential explanation for different generation results despite the shared expected objective.** **Fixed-objective FID-50K results:** | Dataset / model | Aligned + Uniform | Compensated mismatch | Training seeds | |---|---:|---:|---| | Binary-MNIST / U-Net | **$18.69\pm5.35$** | **$14.66\pm2.39$** | 10 paired seeds | | Tiny-ImageNet / JiT | **$17.12\pm0.41$** | **$16.47\pm0.18$** | 3 per arm; independently trained, unpaired | **Both approaches support effective generation while targeting the same expected objective.** All values are means ± sample standard deviations. For Binary-MNIST, the paired difference (compensated minus aligned) has a 95% confidence interval of $[-8.78,0.73]$. The JiT comparison uses three training seeds per arm, with 400 training epochs and the same evaluation protocol; the two arms are independently trained and unpaired (§5.2, Table 4(b); Appendix B.8, Table 13). At common checkpoints, we additionally compare empirical gradient-covariance traces while holding the parameter state and expected objective fixed. **The covariance ordering varies with checkpoint**, consistent with the absence of a universal variance ordering between the two estimators (Appendix A.7, Equation (65); Appendix B.8; probe protocol in Appendix B.9, Equation (80)). **Exact compensation and Logit-Normal both regulate endpoint exposure through sampling and support effective mismatched training in our experiments.** Exact compensation additionally normalizes the loss scale to preserve the expected objective. These results support the paper’s central point that conversion and sampling must be analyzed jointly: alignment removes the conversion factor, while appropriately chosen sampling can support effective training even when pointwise mismatch is retained.
AuthorsOpenReview ↗

W3 and Question 1 — Standard Safeguards and Comparable Optimization Settings

Our added experiments show that the standard 0.050.05 safeguard supports effective training, while alignment still reduces measured update fluctuations under the tested optimization settings. These are compatible findings: the safeguard limits conversion-induced amplification, whereas alignment removes the conversion factor itself.

Standard-safeguard generation results. The 0.050.05 denominator floor caps the conversion weight at 400400 without changing the sampled timestep tt. Under the common 400-epoch training and FID-50K evaluation protocol at the nominal learning rate, we obtain:

Training objectiveUniform FID-50KLogit-Normal FID-50KTraining seeds
Mismatched vv-MSE with 0.050.05 safeguard14.78±0.0914.78\pm0.0915.39±0.5215.39\pm0.523 per sampler

Values are means ± sample standard deviations. Effective Uniform training after restricting endpoint amplification is consistent with the mechanism identified in the paper (§6; Appendix A.6, Equations (59)–(60); Appendix B.9, Tables 14–15).

To address optimizer comparability in Question 1, we used an equal-budget, shared learning-rate grid within each optimizer. For both AdamW and SGD, all four objective–sampler combinations were evaluated at 32×32\times, 64×64\times, and 128×128\times the base learning rate of 5×10−55\times10^{-5}, with three seeds and the same short-training budget of 26 epochs. This gives 72 runs. All 72 remained numerically finite, but their actual parameter-update behavior differed.

Following Equation (27), we measure local update excursions using

Δθk=θk+1−θk.\Delta\theta_k=\theta_{k+1}-\theta_k.

dk=∥Δθk∥2ηk.d_k=\frac{\|\Delta\theta_k\|_2}{\eta_k}.

Rk=dkmedian⁡(dk−100,…,dk−1).R_k=\frac{d_k}{\operatorname{median}(d_{k-100},\ldots,d_{k-1})}.

Actual parameter changes include weight decay, and every update is normalized by its own logged learning rate. The local ratio cancels any fixed positive scaling of update norms, so these measurements capture relative excursions within a run.

Actual-update diagnostics under the standard safeguard:

DiagnosticMismatch + UniformCorresponding aligned runs
SGD, 64×64\times: post-warmup maximum RkR_k8.398.39–12.0212.02Below 22 in every seed
SGD, 128×128\times: post-warmup maximum RkR_k7.107.10–7.367.36Below 22 in every seed
AdamW, 32×32\times/64×64\times/128×128\times: update-norm interquartile range2.032.03–4.314.31 times the aligned valueReference

SGD ranges span three seeds at each learning rate, with maxima restricted to k>5500k>5500. AdamW interquartile-range ratios are computed separately for each learning rate and seed using the same 46 recorded steps. The SGD update excursions recur across all three seeds at both learning rates, while alignment consistently reduces these measured fluctuations (§6, Equation (27), Figure 5; Appendix B.9, Equation (81); Appendix B.11, Figures 12–16).

This shared-grid comparison establishes an advantage in measured update fluctuations under comparable tested settings. It does not equate the effective update scales of AdamW and SGD or establish each recipe’s independently optimized final FID.

The original endpoint-cap intervention already showed that explicitly limiting the conversion weight mitigates training degradation (§5.1, Equation (24), Table 2; Appendix B.5). We therefore used untruncated conversion in Binary-MNIST and a 10−610^{-6} floor in JiT to minimize intervention from endpoint protection and examine the conversion–sampling interaction. These settings test the theoretical mechanism.

Effective safeguarded training and additional optimization benefits from alignment can coexist. Our central question concerns the joint action of conversion and sampling. We do not question the effectiveness of the published JiT or ELF recipes; attributing a complete recipe’s success to a single factor lies outside that research scope. The revised manuscript clarifies this distinction (§6).

Reviewer_AXRAOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates the optimization instability caused by training a clean-data predictor with a velocity-space loss in flow matching. The authors show that the prediction-to-velocity conversion introduces an endpoint-dependent amplification factor, which may lead to an unbounded stochastic-gradient second moment under Uniform timestep sampling. They further analyze how Logit-Normal sampling mitigates this issue and advocate prediction–loss alignment, i.e., directly supervising the predicted clean data. Experiments on synthetic Gaussian data, Binary MNIST, Tiny-ImageNet, and a supplementary MIMO detection task are provided.

Critical strengths

The paper identifies a potentially relevant optimization issue in clean-data parameterizations of flow matching. The experiments also cover multiple model architectures, and the appendix provides relatively detailed implementation protocols.

Critical weaknesses
  1. Limited theoretical novelty and rather straightforward theoretical results.

My primary concern is that the theoretical contributions are considerably weaker than their presentation suggests. The central observation is the elementary identity in Eq. (3), which directly introduces the amplification factor (1−t)−2(1-t)^{-2}. Most subsequent theoretical results follow directly from integrating this factor against the timestep sampling density.

In particular, Theorem 4.1 essentially reduces to checking the integrability of a power-law function near the endpoint under an assumed lower bound on the underlying gradient moment. Proposition 4.2 follows immediately from the assumed uniform boundedness of the aligned gradient moment, while Proposition 4.3 largely relies on the existence of exponential moments of a Gaussian random variable.

Although these derivations appear mathematically reasonable under the stated assumptions, I find their technical depth and novelty quite limited. The paper introduces substantial mathematical machinery to establish consequences that are already largely apparent from the initial loss-conversion identity. More importantly, these results provide little insight into the actual optimization dynamics beyond the endpoint behavior of raw gradient moments.

  1. The experimental setup does not adequately reflect practical implementations of the motivating methods.

The paper motivates its investigation using JiT and ELF. However, Appendix B.6 explicitly acknowledges that both methods employ denominator safeguards with teps=0.05t_{\mathrm{eps}}=0.05. In contrast, the main JiT experiments use a much smaller denominator floor of 10−610^{-6}, which substantially increases the maximum loss amplification. This raises a fundamental concern about the practical relevance of the experimental findings. The severe instability demonstrated in the paper may largely reflect a deliberately weakly implementation.

  1. The effects of optimization variance and changes to the training objective are not sufficiently disentangled.

The paper interprets the benefits of prediction–loss alignment primarily through improved gradient-moment behavior. However, replacing velocity-space supervision with signal-space supervision also changes the effective weighting across timesteps. Specifically, the mismatched objective weights signal prediction errors by (1−t)−2(1-t)^{-2}, whereas the aligned objective does not. Therefore, the observed performance differences cannot be attributed solely to improved gradient conditioning or variance reduction. They may also reflect changes in the relative emphasis placed on different timestep regions.

  1. Limited empirical gains and insufficient evidence of practical superiority.

The empirical results do not convincingly establish prediction–loss alignment as a better practical training strategy.Notably, in the JiT-B/4 Tiny-ImageNet experiments, the original mismatched objective with Logit-Normal sampling achieves an FID of 14.56, outperforming the aligned objective with the same sampling strategy, which obtains an FID of 24.32.

  1. Unnecessarily complicated presentation and unclear notation.

The paper is considerably more difficult to follow than the underlying mathematical ideas warrant. The notation is also not particularly reader-friendly. Several symbols are reused for different purposes, and the distinctions among pointwise loss amplification, conditional gradient moments, integrated moments, and practical optimization behavior are obscured by the presentation. I strongly encourage the authors to simplify the theoretical derivations.

Questions for authors

Same to the weaknesses

10 comments
AuthorsOpenReview ↗

W1 – Theoretical Contribution and Optimization Relevance

The contribution of a mechanism analysis cannot be assessed solely by the length of its proofs. Equation (3) gives a pointwise amplification factor; it does not, by itself, determine whether the timestep-integrated gradient second moment is finite. That conclusion depends jointly on the endpoint residual, the predictor’s parameter Jacobian, and the sampling density. Our analysis identifies sufficient endpoint criteria and checkable output-head conditions, and distinguishes suppressing endpoint exposure through sampling from removing the conversion factor through alignment (§§3.3–4.2; Equation (14); Appendix A.2).

To address applicability beyond initialization, §4.1 now includes two fixed-state results. Both concern exact conversion without a denominator floor, under the stated regularity conditions and an unscaled, trainable additive output bias SbSb, with fixed S≠0S\neq0:

  • Nonzero endpoint residual. If rθ(t)→rθ(1)r_\theta(t)\to r_\theta(1) in L2L^2 and E∥S⊤rθ(1)∥2>0\mathbb E\|S^\top r_\theta(1)\|^2>0, the gradient second moment under Uniform sampling diverges. The bias block transfers the projected residual into a gradient lower bound (Appendix A.3, Corollary A.2 and its first instance).

  • Gaussian signals. For unconditional, nondegenerate Gaussian signals and independent standard Gaussian noise, a predictor observing only (Zt,t)(Z_t,t) has an infinite mismatched gradient second moment under Uniform sampling at every fixed state satisfying the common regularity and output-bias conditions. This includes states realizing the Bayes predictor and does not require the preceding residual-convergence or nonzero-endpoint-residual conditions. The irreducible posterior error supplies an order-(1−t)2(1-t)^2 lower bound, yielding an order-(1−t)−2(1-t)^{-2} lower bound on the mismatched gradient second moment after conversion. The result also applies conditionally on training history when fresh samples are drawn independently (Appendix A.3, Proposition A.3, Equations (43)–(46)).

The moment analysis also has a direct optimization interpretation. For single-sample plain SGD at a fixed state and fixed η>0\eta>0, Equation (17) gives

\mathbb E\|\Delta\theta\|^2 =\eta^2\mathbb E\|\hat g\|^2.$$ Thus, **gradient-second-moment divergence directly implies update-second-moment divergence.** This establishes a property of stochastic optimization updates without requiring a theorem about convergence of the entire training trajectory. The original trained-checkpoint continuations provide complementary empirical evidence. Across three Binary-MNIST checkpoints, switching to mismatched supervision increased the first paired-minibatch gradient norm by 306–2766 times. After 2,000 continuation steps with optimizer states retained, aligned and mismatched FID-50K were $30.08\pm19.13$ and $1554.13\pm612.71$, respectively. A fresh-AdamW restart from a trained JiT checkpoint produced approximately 19,896-fold first-minibatch amplification and 43% higher mean signal MSE under mismatch over 1,000 steps; both branches remained numerically finite (§5.1; Appendix B.6.1). **These results show that the observed amplification and degradation after changing supervision are not confined to initialization.**
AuthorsOpenReview ↗

W2 – Practical Relevance and the Role of Endpoint Safeguards

The original paper already showed that explicitly limiting the conversion weight mitigates training degradation. Its endpoint-cap intervention demonstrates that endpoint protection affects how the identified mechanism manifests in practice (§5.1, Equation (24), Table 2; Appendix B.5).

To directly address practical relevance, we added experiments with the standard 0.050.05 safeguard. This caps the conversion weight at 400400 without changing the sampled timestep tt. After 400 epochs, the three-seed FID-50K results (mean ± sample standard deviation) are 14.78±0.0914.78\pm0.09 for Uniform and 15.39±0.5215.39\pm0.52 for Logit-Normal under mismatched supervision. Thus, Uniform training is effective when endpoint amplification is sufficiently restricted. This is consistent with the endpoint mechanism identified in the paper and does not invalidate it (§6; Appendix A.6, Equations (59)–(60); Appendix B.9, Tables 14–15).

Nevertheless, the standard safeguard does not eliminate the differences in optimization fluctuations. In SGD experiments at 64×64\times and 128×128\times the base learning rate, mismatch+Uniform exhibits local spikes in actual parameter updates across all three seeds at each learning rate. Following Equation (27), we measure

Δθk=θk+1−θk.\Delta\theta_k=\theta_{k+1}-\theta_k.

dk=∥Δθk∥2ηk.d_k=\frac{\|\Delta\theta_k\|_2}{\eta_k}.

Rk=dkmedian⁡(dk−100,…,dk−1).R_k=\frac{d_k}{\operatorname{median}(d_{k-100},\ldots,d_{k-1})}.

The actual updates include weight decay; each update is normalized by its own learning rate. Dividing by the recent median cancels any fixed positive scaling of the update norms. After warmup (k>5500k>5500), the maximum local ratios are 7.107.10–12.0212.02 for mismatch+Uniform, whereas the corresponding aligned runs all remain below 22. All 72 safeguarded AdamW/SGD short runs remain numerically finite. These results show that alignment reduces measured update fluctuations even when the safeguard already supports numerically stable training over the tested horizon (§6, Figure 5; Appendix B.9, Equation (81); Appendix B.11, Figures 14–15).

The purpose of using an untruncated conversion in Binary-MNIST and a 10−610^{-6} denominator floor in JiT was to minimize intervention from endpoint protection and examine the conversion–sampling interaction predicted by the theory. The original cap experiment and the added standard-safeguard results make this experimental rationale explicit.

Effective safeguarded training and the additional optimization benefits of alignment can coexist. Alignment removes the conversion-induced endpoint amplification itself and, under the stated conditions, reduces this source of sampler sensitivity. Our central research question concerns the interaction between conversion and sampling. We do not question the effectiveness of the published JiT or ELF recipes; attributing the success of a complete recipe to a single factor lies outside that research scope. The revised manuscript clarifies this distinction (§6).

AuthorsOpenReview ↗

W3 – Objective Reweighting and the Fixed-Objective Control

The objective reweighting highlighted in the review is precisely the source of conversion-induced gradient amplification: the same timestep-dependent factor scales both the pointwise loss and its parameter gradient. This connection is the subject of our analysis. The original manuscript already discussed effective timestep weighting and proposed aligned signal space as a common reference for analyzing conversion and sampling (original §§3.2 and 6; revised §3.2, Equations (5)–(9), and §6, Equation (26)). A change in objective weighting therefore does not, by itself, contradict the endpoint mechanism. We do not attribute final FID differences solely to gradient variance.

We appreciate the reviewer’s emphasis on distinguishing objective weighting from stochastic-gradient behavior. To examine this distinction directly, we added a control that preserves the aligned-Uniform expected objective while retaining pointwise mismatched supervision. For wϵ(t)=max⁡{1−t,ϵ}−2w_\epsilon(t)=\max\{1-t,\epsilon\}^{-2}, define

Aϵ=∫01wϵ(t)−1 dt.A_\epsilon=\int_0^1 w_\epsilon(t)^{-1}\,dt.

qϵ(t)=1Aϵwϵ(t).q_\epsilon(t)=\frac{1}{A_\epsilon w_\epsilon(t)}.

We sample T∼qϵT\sim q_\epsilon and optimize Aϵℓv,ϵ(T)A_\epsilon\ell_{v,\epsilon}(T), where ℓv,ϵ(t)=wϵ(t)ℓx(t)\ell_{v,\epsilon}(t)=w_\epsilon(t)\ell_x(t). Prediction and target use the same conversion denominator. Consequently,

&\mathbb E_{T\sim q_\epsilon}[A_\epsilon\ell_{v,\epsilon}(T)]\\ &\qquad=\int_0^1\mathbb E[\ell_x(t)]\,dt. \end{aligned}$$ Expectations also include data and noise. **The expected objective—and, when integrable, the expected gradient at the same parameter state—is identical to aligned+Uniform** (§5.2, Equation (25); Appendix A.7, Equations (61)–(65)). **Equal expected gradients do not imply identical stochastic-gradient distributions.** Aligned+Uniform samples timesteps uniformly and uses the signal-space gradient directly. Compensation changes how frequently each timestep is sampled and multiplies each sampled signal-space gradient by $A_\epsilon w_\epsilon(t)$. These changes cancel in expectation, but generally produce different gradient distributions and second moments. **The resulting differences in stochastic optimization provide a potential explanation for different generation results despite the shared expected objective.** **Fixed-objective FID-50K results:** | Dataset / model | Aligned + Uniform | Compensated mismatch | Training seeds | |---|---:|---:|---| | Binary-MNIST / U-Net | **$18.69\pm5.35$** | **$14.66\pm2.39$** | 10 paired seeds | | Tiny-ImageNet / JiT | **$17.09$** | **$16.64$** | 1 per arm; independently trained | **Both approaches support effective generation while targeting the same expected objective.** Binary-MNIST values are means ± sample standard deviations; the paired difference (compensated minus aligned) has a 95% confidence interval of $[-8.78,0.73]$. Both JiT models were trained for 400 epochs and evaluated using the same protocol; the aligned comparator is an independently trained historical baseline. Probes at common checkpoints further show that the ordering of gradient covariance varies with parameter state (§5.2, Table 4(b); Appendix B.8, Table 13; Appendix B.9, Equation (80)). **Exact compensation and Logit-Normal both regulate endpoint exposure through sampling and support effective mismatched training in our experiments.** Exact compensation additionally normalizes the loss scale to preserve the expected objective. These findings support the paper’s central point that conversion and sampling must be analyzed jointly: alignment removes the conversion factor, while appropriately chosen sampling can support effective training even when pointwise mismatch is retained.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions and Additional Evidence

This paper presents a theoretical and empirical analysis of prediction–loss conversion and timestep sampling, motivated by successful systems such as JiT and ELF. We do not question the effectiveness of their published training recipes. Some wording in the original manuscript may have blurred the distinction between analyzing the conversion-induced endpoint mechanism and explaining the performance of a complete training recipe. We have therefore narrowed and clarified the framing to make the intended scope explicit, while retaining the paper’s core mechanism and principal results.

The main additions address the following concerns:

  1. Theoretical scope — AXRA and YPfP. We add conditional fixed-state extensions, including a Gaussian-signal result (§4.1; Appendix A.3), and connect gradient and SGD update second moments through Equation (17). The original trained-checkpoint diagnostics and paired continuations are now presented in the main text (§5.1; Appendix B.6.1).

  2. Objective versus stochastic estimator — AXRA and YPfP. Exact sampling-compensation controls preserve the expected objective while changing the stochastic gradient estimator (§5.2, Equation (25), Table 4(b); Appendices A.7 and B.8).

  3. Practical relevance of endpoint protection — AXRA and YPfP. We add standard 0.050.05-safeguard experiments and AdamW/SGD learning-rate comparisons, reporting effective safeguarded training alongside reproducible differences in update fluctuations (§6, Figure 5; Appendix B.9, Tables 14–15; Appendix B.11).

  4. Repeatability and clarity. We report ten-seed Binary-MNIST and three-seed JiT xx-prediction results, plus fixed-checkpoint solver comparisons (Tables 12, 14–16). We also unify notation and clarify the Logit-Normal parameters and endpoint cap (Appendix A, Table 5; Appendix A.5; §5.1, Equation (24)).

To provide timely responses, we are sharing this revision while additional-seed runs for a small number of experiments are still in progress. All experimental configurations reported here already have results from at least one seed, with the available seed counts explicitly indicated. We will complete the remaining multi-seed evaluations and update the corresponding results before the end of the author–reviewer discussion period.

AuthorsOpenReview ↗

Update: Completed Multi-Seed Experiments and Revised Manuscript (Second Version of Revision)

All previously pending multi-seed experiments and evaluations are now complete, and the final results have been incorporated into the revised manuscript.

The table below reports FID-50K mean ± sample standard deviation over three training seeds, after 400 epochs with Heun-50 evaluation.

ExperimentConfigurationFID-50K
Direct vv-prediction + vv-MSEUniform24.82±0.8124.82\pm0.81
Direct vv-prediction + vv-MSELogit-Normal30.35±0.4230.35\pm0.42
Fixed-objective controlAligned + Uniform17.12±0.4117.12\pm0.41
Fixed-objective controlCompensated mismatch16.47±0.1816.47\pm0.18

Direct-vv results are reported in Figure 4 and Tables 14–15 (§5.2; Appendix B.9). Fixed-objective results appear in Table 4(b) and Table 13 (§5.2; Appendix B.8); these two arms are independently trained and unpaired.

The ten-seed Binary-MNIST comparison suggested by reviewer FiVK, testing m=−0.8m=-0.8 versus m=+0.8m=+0.8 at fixed s=0.8s=0.8, is now included in Appendix B.12, with a reference in §4.2. This experiment demonstrates that larger mm can substantially impair generation quality despite the theoretical moment-finiteness guarantee under the stated assumptions.

All previously existing section, equation, figure, and table numbers are preserved, so earlier response references remain valid.

For clarity, the Binary-MNIST multi-seed experiments use 100 training epochs and Euler-50; the reported 99/50 solver evaluation counts refer to drift evaluations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.