acceptodds
Under review as a conference paper at ICLR 2027

-Controlled GRPO: Turning Flow-Matching Ratio Instability into a Budgeted Resource

Abstract

Reinforcement learning is increasingly used to align image generators with reward signals, and Flow-GRPO recently extended this paradigm to flow-matching models by treating the denoising sampler as a stochastic policy that can be optimized from reward feedback. Training in this setting is unstable in a way specific to multi-step denoising: the policy update changes systematically across denoising steps, with importance ratios drifting below one, becoming increasingly dispersed, clipping at different rates, and leaving fewer usable samples late in training. Prior work treats these effects as separate failure modes and addresses each with a hand-tuned stabilizer. We show instead that they arise from a single per-step quantity, which we call path variance. This quantity is determined exactly by the sampler's Gaussian transition kernel and can be estimated cheaply during training. This reframes instability as a resource that can be measured and budgeted rather than a collection of symptoms to repair. Our method, -Controlled GRPO, calibrates importance-ratio behavior from this predicted law rather than from noisy empirical statistics, and allocates gradient effort across denoising steps according to their predicted cost. The two scales governing the update are fixed by standard policy choices rather than introduced as free tuning parameters. On a text-to-image model under two reward settings, rendering difficult target text scored by optical character recognition and matching human preferences scored by a preference model, -Controlled GRPO improves both text accuracy and preference reward over the strongest empirical stabilizer. It also keeps late-step path variance within its intended budget, precisely where the baseline systematically overshoots. The result is a Flow-GRPO update calibrated by its own transition law rather than stabilized after instability appears.

Primary area:
reinforcement learning
Keywords:
GRPO, Flow-matching, Image Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oqFuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies timestep-dependent importance-ratio instability in Flow-GRPO. Starting from equal-covariance Gaussian transition kernels, it characterizes the conditional log-ratio distribution through a scalar policy displacement, termed path variance. It distinguishes the centering and variance scales under mean-reduced log probabilities and proposes LambdaNorm-T calibration combined with damp-only timestep weighting. Experiments include tiny-SD3 diagnostics and reward optimization on SD3.5 and FLUX.

Critical strengths

S1: The paper addresses a relevant optimization problem. Connecting timestep-dependent ratio behavior to quantities available from the transition kernel could provide useful diagnostics.

S2: The distinction between full and mean-reduced log probabilities is practically important. The corresponding Gaussian moment calculations are sound for a fixed pair of policies evaluated on old-policy transitions.

Critical weaknesses

W1:Novelty and budget justification remain unclear. GRPO-Guard’s Eqs. (7)–(8) already establish the Gaussian expansion and analytic centering. The additional calibration and damping may contribute, but the ESS interpretation of the proposed budget lacks a derivation accounting for latent-dimension scaling.

W2: Missing component ablations make it difficult to isolate the effects of calibration and damping. Table 6 does not demonstrate consistent budget satisfaction, and the OCR/CER confidence intervals in Table 2 require clarification.

Questions for authors

Q1: What precisely is new beyond GRPO-Guard’s existing analysis and normalization?

Q2: How does the budget imply retention target (q), including dimension factors, and how is Table 6’s overspending explained?

Q3: Can the authors provide matched component ablations and clarify the OCR/CER definitions and bootstrap intervals in Table 2?

Reviewer_koTKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors derive the finite-grid importance-ratio law and its mean-reduced corollary, obtaining an exact per-step prediction for the log-ratio statistics from the path variance, and give the online estimator of the quantity from samples. They propose the method, named λ-Controlled GRPO, which allocates gradient effort according to predicted path-variance cost using analytic calibration instead of empirical ratio normalization. The experimental validation verifies the effectiveness and theoretical correctness.

Critical strengths
  1. This paper addresses the training instability of Flow-GRPO, which primarily manifests as pathological behaviors in policy importance ratios, such as drifting below one, becoming highly dispersed, clipping asymmetrically, and losing effective samples in late steps. Theoretically, this paper derives a finite-grid importance-ratio law demonstrating that the step log-ratio follows a Gaussian distribution with mean −λk/2-\lambda_k/2 and variance λk\lambda_k, meaning that this single scalar predicts drift, dispersion, clipping imbalance, and sample loss. The theoretical derivation is detailed and well-founded.
  2. Second, the paper proposes a novel method, λ-Controlled GRPO. It replaces empirical ratio normalization with analytic calibration (LambdaNorm-T) and allocates gradient effort according to the predicted path-variance cost. The proposed method is relatively simple and easy to reproduce, and experiments demonstrate its effectiveness in stabilizing training.
Critical weaknesses

1.The performance improvements are inconsistent across different tasks. While the method demonstrates clear mechanistic benefits (e.g., reduced clipping) on the FLUX backbone, task-level gains are not uniform. For instance, Hard-OCR improvements remain statistically inconclusive under single-seed settings, and GenEval performance is statistically indistinguishable from baselines. Could you explain the reasons behind these discrepancies? 1.I am familiar with this field, but I lack a comprehensive understanding of the current landscape regarding the problem this paper aims to solve. Are there any existing state-of-the-art (SOTA) solutions for reference, such as GRPO-Guard? Furthermore, since GRPO has been widely adopted in large language models (LLMs), do any of the conclusions in this paper draw upon related work in language models, such as "DAPO: An Open-Source LLM Reinforcement Learning System at Scale"?

Questions for authors

Please answer the questions in the weaknesses.

Reviewer_q7vpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper resolves training instabilities in RL-aligned flow-matching image models (Flow-GRPO) by tracing issues like drifting importance ratios to a single root cause: path variance.

Critical strengths

Hyperparameter Reduction: Eliminates the need for hand-tuned stabilizers. The update scales are mathematically fixed by standard policy choices rather than trial-and-error tuning.

Proactive Calibration: Predicts and budgets variance computationally from the transition kernel, preventing late-stage instability before it occurs, rather than reacting to noisy empirical statistics.

Critical weaknesses
  • experiment is not solid. many popular RL algorithm like Diffusion NFT seem have better performance.
Questions for authors

na

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.