acceptodds
Under review as a conference paper at ICLR 2027

NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learning

Abstract

Reinforcement learning (RL) post-training improves the reward alignment of flow-based generators, but often degrades perceptual quality in ways that are not captured by the reward proxy. We identify a simple structural signature of this drift: across three post-training methods (NFT, AWM, DPO), RL fine-tuning inflates the per-step velocity norm by to relative to the reference. This inflation is perceptually consequential: because the generated sample is the displacement accumulated by the velocity field, a norm inflation raises the second moment of the sample by roughly , which surfaces in the decoded image as deviations in luminance and chrominance, i.e., boosted contrast, over-sharpened edges, and unnatural white balance. The obvious remedy, rescaling back to at inference time as done for classifier-free guidance, does not transfer to RL: it neither improves reward nor fixes the quality degradation, because the inflation is co-adapted into the model weights. We further show that velocity magnitude rescaling carries no coherent reward signal at the batch level, indicating that suppressing norm inflation is unlikely to remove a consistently reward-carrying component. Both findings point to NormGuard, a simple hinge penalty that activates only when exceeds and composes additively with any velocity-local base loss. Across two base models, three post-training methods, and two reward proxies, NormGuard consistently improves MLLM-judged image quality, usually improves forensic realism, and largely preserves reward, with gains that amplify under few-step inference and are not explained by early stopping.

Primary area:
generative models
Keywords:
Image Generation, Diffusion Models, Reward Hacking, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_AEtHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates perceptual quality degradation during reinforcement learning post-training of flow-matching models. The authors identify an increase in velocity norm relative to the pretrained reference and propose NormGuard, a one-sided regularizer that penalizes excess velocity magnitude. Experiments across two base models, three post-training algorithms, and two reward models suggest that the method improves perceptual quality while largely preserving reward gains.

Critical strengths
  1. The paper identifies consistent velocity-norm inflation following RL post-training across several algorithms. This provides a concrete, measurable signature of post-training drift and a useful starting point for understanding quality degradation.

  2. NormGuard achieves substantial improvements in MLLM-assessed quality across several configurations, while improving forensic realism in most settings and largely retaining reward gains. These results suggest that a simple norm constraint can be practically effective.

Critical weaknesses
  1. The origin of velocity-norm inflation remains unexplained. The paper documents that �0��0� increases after RL post-training, but provides little theoretical analysis of why reward-driven optimization systematically produces this behavior. The energy argument addresses the potential consequences of norm inflation rather than its underlying cause, and the sensitivity analysis does not explain why training updates favor increasing velocity magnitude. An analysis of the radial component of the optimization updates, and its relationship to the reward objective, would substantially strengthen the paper’s central motivation.

  2. The focus on velocity norm is insufficiently justified relative to alternative statistics. ∥v∥\|v\| is only one statistic of the model’s predictions. Do other quantities, such as ∥xpred∥\|x_{\mathrm{pred}}\|, ∥ϵpred∥\|\epsilon_{\mathrm{pred}}\|, and the correlation between xpredx_{\mathrm{pred}} and ϵpred\epsilon_{\mathrm{pred}}, also change systematically during RL post-training? Since these quantities are algebraically related under the flow-matching parameterization, velocity-norm inflation may reflect broader changes in prediction statistics rather than an independently privileged failure mechanism. Would directly regularizing these alternative quantities achieve comparable or better results? Relatedly, CFG++ [1] motivates manifold-aware guidance through the denoised estimate, highlighting the relevance of constraints involving the predicted clean sample. This is conceptually different from a simple norm penalty, but motivates examining alternatives to velocity-space regularization. Without such comparisons, it remains unclear why velocity norm is the most appropriate intervention target.

[1] CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models.

Questions for authors

Both AWM and DiffusionNFT show that RL implicitly performs CFG distillation. Could this explain the increase in ||v||?

4 comments

Thank you for your review. We address the concerns below:

W1. Theoretical analysis of why training updates favor increasing velocity magnitude

Thank you for raising this question.

We hypothesize that in flow-matching RL, as the policy drifts across denoising steps, the trajectory can increasingly deviate from the straight-line paths used in Rectified Flow, since this prior is not explicitly supervised during training. When the start-to-end displacement stays roughly the same but the trajectory takes a longer route, the model must travel farther in the same sampling time, which may help explain the observed velocity-norm inflation.

To validate this hypothesis, we use the straightness measure from Rectified Flow [1] (Eq. 3) to compare sampling trajectories before and after RL. For a trajectory with displacement d=xend−xstartd=x_{\mathrm{end}}-x_{\mathrm{start}}, a straight path traversed at constant speed would have velocity dd at every step. We measure how far the actual velocities uiu_i depart from this reference:

=\frac{\sum_i \Delta t_i |u_i-d|^2} {\sum_i \Delta t_i |u_i|^2}.$$ On SD3.5M, across 100 rollouts, mean $S_{\mathrm{norm}}$ increased $+11.45\\%$ ($0.04822 \to 0.05374$) after RL on PickScore, which supports the hypothesis that the policy is taking less straight paths and thus increasing the velocity norm. [1] Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. ***W2. Velocity norm vs. alternative regularization targets*** Thank you for the suggestion. We conduct an additional experiment using predicted clean endpoint norm regularization ($x_0$-Norm), where we replace the velocity norm in NormGuard with $\\|{x}_{0}\\|_2$: $$\begin{aligned} \hat{x}_{0,\theta} &= x_t-\sigma_t v_\theta(x_t,t,c), \\\\ \hat{x}_{0,\mathrm{ref}} &= x_t-\sigma_t v_{\mathrm{ref}}(x_t,t,c); \\\\ \mathcal{R}_{\text{norm}}(\theta;x_t,t,c) &= \lambda_{x_0}\max\\!\left\\{ 0,\\, \frac{ \\|\hat{x}_{0,\theta}\\|_2^2-\\|\hat{x}_{0,\mathrm{ref}}\\|_2^2 }{ \\|\hat{x}_{0,\mathrm{ref}}\\|_2^2 } \right\\}. \end{aligned}$$ We test this $x_0$-Norm regularization on SD3.5M + PickScore + NFT. The results are summarized below: | Method | PickScore ↑ | MLLM Score ↑ | RealScore ↑ | |---|---|---|---| | $x_0$-Norm | **0.871** | 5.93 | 0.274 | | $v$-Norm (Ours) | 0.868 | **5.99** | **0.297** | We also examined when each penalty was active during the training process. For $x_0$-Norm, the mean policy-to-reference norm ratio $\\|\hat{x}_{0,\theta}\\|_2 / \\|\hat{x}_{0,\mathrm{ref}}\\|_2$ was 0.937, and the hinge activated on 1.14% of predictions on average ($v$-Norm is 85.5% across steps, however). These results support using velocity norm as the target for a one-sided penalty in this RL setting, where $\\|x_0\\|$ rarely exceeded its reference. ***Q1. Can implicit CFG explain velocity-norm inflation?*** Thank you for suggesting this insightful connection. Consistent with DiffusionNFT, we also observe obvious better prompt alignment after RL at CFG = 1.0 compared with the pretrained model at the same or higher CFG scale. This improvement is present both with and without NormGuard. Implicit CFG distillation is therefore a plausible contributor to the observed velocity-norm inflation. We will discuss this possibility in the revised manuscript.

Dear Reviewer AEtH,

Thank you again for asking the potential cause that drives velocity-norm inflation and other alternative regularization targets. In the additional experiments, we observed less straight sampling trajectories after RL, and the x0x_0-penalty as a less-effective target than the velocity-norm penalty. We also appreciate your suggestion about implicit CFG distillation and will discuss it as a possible contributor in the revision.

We would be happy to clarify any remaining questions about the analysis and the new results. Thank you for your time and feedback.

Best regards,

The Authors

Thanks for you reply on the theory part and discussion on the CFG. I believe that these insightful information would strengthen your paper. Now I would like to increase my score.

Thank you very much for your positive feedback and willingness to increase your score! We sincerely appreciate your recognition of our additional analysis and your constructive suggestions.

Reviewer_7zDKOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies that RL post-training of flow-matching generative models consistently inflates the per-step velocity norm by 5%–15%, which mathematically increases latent energy and manifests visually as reward-hacking artifacts such as over-sharpening and color oversaturation. To address this issue, the authors propose NORMGUARD, a simple training-time hinge penalty that constrains excess velocity norm beyond the reference model. Extensive experiments across two base models, three RL algorithms, and two reward proxies demonstrate that NORMGUARD significantly improves MLLM-judged image quality and forensic realism while preserving reward gains.

Critical strengths
  1. The paper provides a clear mechanistic diagnosis of reward over-optimization by linking RL-induced velocity norm inflation to excess image energy and visual artifacts like over-sharpening and oversaturation. Crucially, the authors demonstrate that inference-time renormalization fails and use adjoint sensitivity analysis to prove that suppressing velocity magnitude does not harm batch-level reward signals, which provides a solid theoretical and empirical justification for why a training-time radial constraint is both necessary and safe.

  2. The proposed NORMGUARD is a simple, plug-and-play hinge penalty that requires no extra model forward passes and only a single hyperparameter. Its strong empirical effectiveness across multiple backbones, RL algorithms, and reward proxies, along with ablations showing it is orthogonal to KL regularization and early stopping, makes it an immediately practical and impactful tool for flow-matching post-training.

Critical weaknesses
  1. The theoretical formulation and empirical validation of NORMGUARD are currently scoped to velocity-local RL objectives (NFT, AWM, and DPO), leaving out trajectory-level policy gradient methods like Flow-GRPO. While the authors transparently explain this boundary in Appendix A.2.4, it remains unclear whether a similar norm-inflation pathology occurs in Flow-GRPO and whether applying NORMGUARD heuristically would still be effective there.

  2. The experiments exclusively evaluate aesthetic and human-preference reward proxies (PickScore and HPSv2). It remains unexplored whether RL post-training with rule-based or compositional rewards (such as GenEval or OCR accuracy) also induces the same 5%–15% velocity norm inflation and benefits similarly from NORMGUARD.

Questions for authors
  1. While the method only introduces a single hyperparameter λ\lambda, its optimal scale varies across different RL algorithms (e.g., λ=1.0\lambda=1.0 works universally for NFT and DPO, whereas AWM requires λ=0.01\lambda=0.01 in Table 7). A brief explanation of why AWM requires a much smaller regularization weight would improve the practical usability of the method.
4 comments

W1. Effectiveness of NormGuard on Flow-GRPO.

Thank you for the suggestion.

We further investigate velocity norm changes under SD3.5 + PickScore + Flow-GRPO after 200 training steps. In detail, we divide the 40 denoising steps into three intervals and compare the average velocity norms before and after RL training. We find only a marginal velocity norm increase in the early three steps, while the remaining steps exhibit decreased trends. The results are summarized below:

Denoising StepsBaseFlow-GRPO
1–3847.0850.3
4–20766.3751.6
21–40732.2714.0

We will further incorporate this observation into our existing discussion and theoretical analysis of Flow-GRPO's distinctive behavior.

W2. Norm inflation in rule-based/compositional rewards

Thank you for the suggestion.

We conduct an additional experiment on SD3.5M + GenEval + NFT. We measure the average velocity norm at different training steps and compare it against the pretrained reference model (Ref.). The results are summarized below:

MetricRef.Step 40Step 120Step 200
Avg. ∥v∥\lVert v \rVert755.2735890.1472922.8827918.1692

We observe a consistent increase in velocity norms compared to the reference model, suggesting that norm inflation also occurs with rule-based compositional rewards. To further evaluate the effectiveness of NormGuard in this setting, we compare NFT training w/ and w/o NormGuard at Step 200. The results are summarized below:

MethodGenEval ↑MLLM Score ↑RealScore ↑
w/o NormGuard0.8485.710.268
w/ NormGuard0.8425.870.277

Plus, we observe that reward hacking under GenEval often manifests as oversimplified, nearly uniform-color backgrounds, which differs from the white balance and illumination distortions that NormGuard more effectively mitigates. This difference may explain NormGuard's relatively limited visual benefits under rule-based rewards.

Q1. Why AWM needs smaller regularization scale?

Thank you for raising this question.

The appropriate regularization strength depends on the relative scales of the base optimization loss and the norm regularization term. Given our total objective,

L=Lbase+λRnorm,\mathcal{L} = \mathcal{L}_{\mathrm{base}} + \lambda \mathcal{R}_{\mathrm{norm}},

we observe that AWM typically has a much smaller Lbase\mathcal{L}_{\mathrm{base}} magnitude, fluctuating around zero (~10−310^{-3} for SD3.5-M and ~10−210^{-2} for FLUX.2-Klein). Consequently, larger regularization coefficients can make the norm penalty overly dominant, excessively constraining policy updates and preventing meaningful reward gain. Therefore, we use a smaller λ\lambda for AWM to balance optimization and regularization.

Dear Reviewer 7zDK,

Thank you again for suggesting that we test Flow-GRPO and a compositional reward. In our Flow-GRPO experiment, velocity norms increased slightly in the first three denoising steps and decreased thereafter. In the GenEval experiment, we also observed norm inflation, and NormGuard improved both MLLM-judged quality and RealScore in GenEval.

We would be happy to discuss these new results. Thank you for your time and feedback.

Best regards,

The Authors

Thank you for the detailed response and the additional clarifications. The authors have successfully addressed all of my concerns. I remain positive about this paper and will keep my original score.

Thank you very much for your positive feedback and continued support! We sincerely appreciate your recognition of our work and are glad that our responses have addressed all your concerns.

Reviewer_5a9UOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Decompose the velocity drift induced by RL post-training into a radial (magnitude) and an angular (directional) component, show that the radial component is perceptually harmful but carries no reward, and therefore suppress only the radial part while leaving the angular part free. Three Contributions: Identifies and explains a measurable reward-hacking fingerprint; Two diagnostics that pin down where to intervene and whether it is safe; A one-sided hinge with a single scalar hyperparameter, composing additively with any velocity-local loss.

Critical strengths

The paper does not stop at correlational observations of the kind "RL post-training produces artifacts." Instead, it lays out a complete, verifiable causal chain: a δ-level velocity-norm inflation → an increase of roughly 2δ in the second moment of the generated sample (derived, relying on E⟨x0,ϵ⟩≈0\mathbb{E}\langle x_0,\epsilon\rangle\approx 0) → surplus energy in the non-DC frequencies of the decoded image → over-sharpening / over-saturation / unnatural white balance. The chain begins inside the model (∣vθ∣|v_\theta|) and ends in image space (ElumE_{\text{lum}}, EchromaE_{\text{chroma}}, cleanly defined via Parseval's identity), so both ends can be measured independently; the measured luminance energy rises by 24%–46%, matching the prediction in both direction and magnitude.

Critical weaknesses
  1. The related work section explicitly lists six works — GRPO-Guard, RSA-FT, RealGen, RewardDance, Pref-GRPO, and GARDO — and states for each that it targets the same problem (reward over-optimization in text-to-image RL). Yet the experiments compare against only three things: the unregularized baseline, early stopping, and KL regularization — all of which are either "do nothing" or the most naive options. we have no idea where it stands relative to the solutions the field already has.

2."Mitigates quality degradation" is evidenced only relative to the baseline, with no reference-model anchor。In the main results table (MLLM win rate, RealScore) and the early-stopping ablation table (MLLM absolute score), not a single row corresponds to the pretrained reference model. Every quality conclusion takes the form "better than the hacked baseline."

Questions for authors

1.Could the authors measure the per-step velocity-norm trajectory of a checkpoint trained with a competing method (GARDO or RSA-FT)? If those methods already suppress radial inflation, this paper reads as an explanation of why they work rather than a new failure mode; if they don't, radial inflation is a genuinely independent channel. I can't tell which from the current manuscript, and this single measurement would settle it.

2.Please add pretrained-reference rows to the main results and early-stopping tables (RealScore and absolute MLLM score). Both are absolute metrics and the reference model is already used to compute vrefv_{\text{ref}}, so no retraining is required. Without this anchor, every quality claim is "better than the unregularized baseline," which leaves the key question open: what fraction of the RL-induced degradation is actually recovered?

2 comments

Thank you for your review. We address your concerns below.

W1. Comparison with prior methods for mitigating reward hacking

Thank you for suggesting a comparison with prior work.

GRPO-Guard and Pref-GRPO are built on Flow-GRPO, which NormGuard does not support. GARDO supports DiffusionNFT, so we can compare fairly. At the same training steps, we report their RealScore and MLLM Score as follows:

MethodPickScore ↑RealScore ↑MLLM Score ↑
DiffusionNFT + GARDO0.8600.3115.86
DiffusionNFT + NormGuard0.8680.2975.99

At the same number of training steps, GARDO achieves a higher RealScore possibly because it constrains the policy more strongly toward its reference. NormGuard achieves a higher PickScore and MLLM Score in nearly half the training time.

We also examine the velocity norm changes under DiffusionNFT + GARDO (Q1), where we also find overall velocity-norm inflation relative to the base model, with the increase concentrated in the first three denoising steps (+14.37%).

Denoising StepsBase v-normGARDO
1–3847.0968.7 (+14.37%)
4–20766.3779.4 (+1.71%)
21–40732.2728.1 (−0.56%)

W2. Comparison with the pretrained reference model

Thank you for your suggestion.

We include the pretrained reference in the two tables below. Across all settings, NormGuard consistently outperforms the pretrained reference in MLLM evaluations, while also improving the optimized reward.

Meanwhile, we acknowledge that the pretrained reference still achieves higher RealScore than the RL-finetuned ones, indicating that NormGuard mitigates realism degradation but does not fully eliminate it.

Table 2. Main Results

RewardModelMethodReward Ref.Reward Bas.Reward OursMLLM Win Rate (Ours vs Bas.)MLLM Win Rate (Ours vs Ref.)RealScore Ref.RealScore Bas.RealScore Ours
PickScoreSD3.5-MNFT0.8410.8690.86664% (1% tie)60% (0% tie)0.5830.2480.310
PickScoreSD3.5-MDPO0.8410.8490.85168% (4% tie)71% (2% tie)0.5830.4680.472
PickScoreSD3.5-MAWM0.8410.8690.88072% (0% tie)68% (0% tie)0.5830.2830.269
PickScoreFLUX.2-4BNFT0.8400.9050.91047% (15% tie)55% (1% tie)0.4890.2390.274
PickScoreFLUX.2-4BAWM0.8400.8860.88659% (0% tie)57% (1% tie)0.4890.2770.320
HPSv2SD3.5-MNFT0.2860.3080.30958% (0% tie)66% (0% tie)0.5830.2180.299
HPSv2FLUX.2-4BNFT0.2850.3150.31163% (1% tie)70% (0% tie)0.4890.2400.274

Table 4. Early stopping ablation

CheckpointPickScore ↑RealScore ↑MLLM Score ↑
Pretrained reference0.84010.4895.69
NFT, iter 1600.89780.2596.08
NFT, iter 1800.90100.2525.99
NFT, iter 2000.90530.2406.05
NFT + NormGuard, iter 2000.91030.2746.16

Q1. & Q.2

Please refer to W1 and W2 above.

Dear Reviewer 5a9U,

Thank you again for suggesting that we compare NormGuard with GARDO and include the pretrained reference. The new experiments showed that GARDO also exhibits velocity-norm inflation in the first few denoising steps. The reference comparison helped us clarify that NormGuard improves MLLM-judged quality over the pretrained model across the tested settings.

We would be happy to clarify any remaining questions about these comparisons. Thank you for your time and consideration.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Experiments and Clarifications.

We thank all reviewers for their thoughtful feedback. Our paper introduces NormGuard, a one-sided penalty on excess velocity norm that improves perceptual quality while largely preserving reward gains. We summarize the additional findings from our individual responses below.

  • Baselines and pretrained reference (5a9U). At matched training steps, NormGuard achieves higher PickScore and MLLM scores than GARDO, which also shows early-step norm inflation. Against the pretrained reference, NormGuard improves reward in all seven settings and achieves 55–71% MLLM win rates.
  • Potential causes and regularization targets (AEtH). We finder greater deviation from constant-speed straight-line trajectories after RL, consistent with the hypothesis that changes in trajectory geometry contribute to norm inflation. Plus, the vv-norm penalty activates more often and yields higher MLLM and RealScore scores than the x0x_0-norm penalty.
  • Flow-GRPO, GenEval, and AWM (7zDK). Flow-GRPO shows a slight norm increase over the first three denoising steps and decreases after. NormGuard also applies for rule-based reward like GenEval. AWM’s smaller base-loss scale motivates a smaller penalty weight.

We appreciate the reviewers’ suggestions and would be happy to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.