- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper investigates perceptual quality degradation during reinforcement learning post-training of flow-matching models. The authors identify an increase in velocity norm relative to the pretrained reference and propose NormGuard, a one-sided regularizer that penalizes excess velocity magnitude. Experiments across two base models, three post-training algorithms, and two reward models suggest that the method improves perceptual quality while largely preserving reward gains.
Critical strengths
-
The paper identifies consistent velocity-norm inflation following RL post-training across several algorithms. This provides a concrete, measurable signature of post-training drift and a useful starting point for understanding quality degradation.
-
NormGuard achieves substantial improvements in MLLM-assessed quality across several configurations, while improving forensic realism in most settings and largely retaining reward gains. These results suggest that a simple norm constraint can be practically effective.
Critical weaknesses
-
The origin of velocity-norm inflation remains unexplained. The paper documents that increases after RL post-training, but provides little theoretical analysis of why reward-driven optimization systematically produces this behavior. The energy argument addresses the potential consequences of norm inflation rather than its underlying cause, and the sensitivity analysis does not explain why training updates favor increasing velocity magnitude. An analysis of the radial component of the optimization updates, and its relationship to the reward objective, would substantially strengthen the paper’s central motivation.
-
The focus on velocity norm is insufficiently justified relative to alternative statistics. is only one statistic of the model’s predictions. Do other quantities, such as , , and the correlation between and , also change systematically during RL post-training? Since these quantities are algebraically related under the flow-matching parameterization, velocity-norm inflation may reflect broader changes in prediction statistics rather than an independently privileged failure mechanism. Would directly regularizing these alternative quantities achieve comparable or better results? Relatedly, CFG++ [1] motivates manifold-aware guidance through the denoised estimate, highlighting the relevance of constraints involving the predicted clean sample. This is conceptually different from a simple norm penalty, but motivates examining alternatives to velocity-space regularization. Without such comparisons, it remains unclear why velocity norm is the most appropriate intervention target.
[1] CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models.
Questions for authors
Both AWM and DiffusionNFT show that RL implicitly performs CFG distillation. Could this explain the increase in ||v||?
4 comments
Follow-up
Dear Reviewer AEtH,
Thank you again for asking the potential cause that drives velocity-norm inflation and other alternative regularization targets. In the additional experiments, we observed less straight sampling trajectories after RL, and the -penalty as a less-effective target than the velocity-norm penalty. We also appreciate your suggestion about implicit CFG distillation and will discuss it as a possible contributor in the revision.
We would be happy to clarify any remaining questions about the analysis and the new results. Thank you for your time and feedback.
Best regards,
The Authors
Thanks for you reply on the theory part and discussion on the CFG. I believe that these insightful information would strengthen your paper. Now I would like to increase my score.
Thank you very much for your positive feedback and willingness to increase your score! We sincerely appreciate your recognition of our additional analysis and your constructive suggestions.
Thank you for your review. We address the concerns below:
W1. Theoretical analysis of why training updates favor increasing velocity magnitude
Thank you for raising this question.
We hypothesize that in flow-matching RL, as the policy drifts across denoising steps, the trajectory can increasingly deviate from the straight-line paths used in Rectified Flow, since this prior is not explicitly supervised during training. When the start-to-end displacement stays roughly the same but the trajectory takes a longer route, the model must travel farther in the same sampling time, which may help explain the observed velocity-norm inflation.
To validate this hypothesis, we use the straightness measure from Rectified Flow [1] (Eq. 3) to compare sampling trajectories before and after RL. For a trajectory with displacement d=xend−xstart, a straight path traversed at constant speed would have velocity d at every step. We measure how far the actual velocities ui depart from this reference:
=\frac{\sum_i \Delta t_i |u_i-d|^2} {\sum_i \Delta t_i |u_i|^2}.$$ On SD3.5M, across 100 rollouts, mean $S_{\mathrm{norm}}$ increased $+11.45\\%$ ($0.04822 \to 0.05374$) after RL on PickScore, which supports the hypothesis that the policy is taking less straight paths and thus increasing the velocity norm. [1] Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. ***W2. Velocity norm vs. alternative regularization targets*** Thank you for the suggestion. We conduct an additional experiment using predicted clean endpoint norm regularization ($x_0$-Norm), where we replace the velocity norm in NormGuard with $\\|{x}_{0}\\|_2$: $$\begin{aligned} \hat{x}_{0,\theta} &= x_t-\sigma_t v_\theta(x_t,t,c), \\\\ \hat{x}_{0,\mathrm{ref}} &= x_t-\sigma_t v_{\mathrm{ref}}(x_t,t,c); \\\\ \mathcal{R}_{\text{norm}}(\theta;x_t,t,c) &= \lambda_{x_0}\max\\!\left\\{ 0,\\, \frac{ \\|\hat{x}_{0,\theta}\\|_2^2-\\|\hat{x}_{0,\mathrm{ref}}\\|_2^2 }{ \\|\hat{x}_{0,\mathrm{ref}}\\|_2^2 } \right\\}. \end{aligned}$$ We test this $x_0$-Norm regularization on SD3.5M + PickScore + NFT. The results are summarized below: | Method | PickScore ↑ | MLLM Score ↑ | RealScore ↑ | |---|---|---|---| | $x_0$-Norm | **0.871** | 5.93 | 0.274 | | $v$-Norm (Ours) | 0.868 | **5.99** | **0.297** | We also examined when each penalty was active during the training process. For $x_0$-Norm, the mean policy-to-reference norm ratio $\\|\hat{x}_{0,\theta}\\|_2 / \\|\hat{x}_{0,\mathrm{ref}}\\|_2$ was 0.937, and the hinge activated on 1.14% of predictions on average ($v$-Norm is 85.5% across steps, however). These results support using velocity norm as the target for a one-sided penalty in this RL setting, where $\\|x_0\\|$ rarely exceeded its reference. ***Q1. Can implicit CFG explain velocity-norm inflation?*** Thank you for suggesting this insightful connection. Consistent with DiffusionNFT, we also observe obvious better prompt alignment after RL at CFG = 1.0 compared with the pretrained model at the same or higher CFG scale. This improvement is present both with and without NormGuard. Implicit CFG distillation is therefore a plausible contributor to the observed velocity-norm inflation. We will discuss this possibility in the revised manuscript.