acceptodds
Under review as a conference paper at ICLR 2027

LC-GRPO: Bridging Train-Inference Gap for Flow-Based GRPO with Langevin Correction

Abstract

Flow-based generative models are typically sampled by solving a deterministic ordinary differential equation (ODE), whereas reinforcement learning (RL) requires stochastic rollouts for exploration and optimization. Existing online RL methods for flow models therefore replace the inference-time ODE with a stochastic differential equation (SDE) during training. Although the ODE and SDE share the same marginal distribution in continuous time, they can behave differently after finite-step discretization. In particular, SDE rollouts often become blurry as exploration noise increases, creating a mismatch between training rollouts and inference-time ODE samples. We introduce LC-GRPO, a flow-based GRPO framework with Langevin correction. Each rollout transition takes an inference-aligned ODE Euler step followed by a stochastic Langevin correction targeting the marginal at the resulting timestep. The required score is recovered directly from the flow velocity, requiring no additional score model, while the transition remains an isotropic Gaussian with tractable likelihood. We theoretically show that, under suitable conditions, a single Langevin correction step reduces the error of an imperfect ODE Euler step and, under matched randomness, yields a more accurate transition than standard Euler–Maruyama discretization of the reverse SDE. Additionally, experiments on SD3.5-Medium, FLUX.1-Dev, and HunyuanVideo show that LC-GRPO consistently improves reward optimization across text-to-image and text-to-video tasks, preserves generation quality, and substantially narrows the gap between stochastic training rollouts and deterministic ODE inference.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Flow-Based GRPO, Train-Inference Gap

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qoLmOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the rollout problem in GRPO for flow-based models. To address the large discretization errors of SDE-based rollouts with few sampling steps, the authors propose a method that combines ODE sampling with Langevin correction. Theoretically, Langevin correction can approximate the target rollout distribution more accurately than few-step SDE sampling under certain conditions. Experiments on both image and video generation show that the proposed method outperforms the Flow-GRPO and DanceGRPO baselines.

Critical strengths

This paper addresses an important issue in Flow-GRPO and proposes a seemingly reasonable solution. The proposed algorithm achieves good experimental results, especially in reducing the training–inference mismatch. The paper also provides some theoretical analysis to support the performance advantages of the proposed method.

Critical weaknesses
  1. I think the baselines considered in this paper are generally reasonable. However, I am curious about another possible baseline. Since ODE + Langevin requires two NFEs per step, would it be possible to compare it with a baseline that alternates between ODE and SDE steps under the same NFE budget, where stochasticity is introduced only through the SDE steps? This seems to be a more closely related alternative to the proposed ODE + Langevin approach.
  2. I would also like to understand how sensitive LC-GRPO is to the noise parameter η\eta. Appendix B shows that different η\eta values are used for LC-GRPO across different tasks. How much does this parameter affect performance? If possible, I would also suggest comparing the sensitivity of LC-GRPO, Flow-GRPO, and CPS to η\eta, to see whether LC-GRPO is more robust to different noise levels.
  3. I noticed that the theoretical analysis relies on some assumptions that do not seem particularly straightforward, such as the condition in Eq. (13). I think it would be helpful to quantitatively investigate when this condition holds. For example, a numerical study using a multivariate Gaussian distribution could help illustrate the conditions under which the theoretical results apply.
Questions for authors

I noticed that the Langevin step size is chosen differently from the noise schedule in SDE sampling. Although the footnote states that the total injected noise variance is larger for Langevin under the same η\eta, this does not necessarily hold at every individual timestep. Noise injected at different timesteps may have different effects on the final generated samples, I wonder whether comparing the sum of noise variances is sufficient to support this claim. Could the authors provide further clarification or analysis?

4 comments
AuthorsOpenReview ↗

Response to Reviewer qoLm (1/2)

Thank you for your thoughtful and valuable feedback! We address your concerns below.

Q1 (Weakness 1): I think the baselines considered in this paper are generally reasonable. ... compare it with a baseline that alternates between ODE and SDE steps under the same NFE budget ...?

A1: We thank the reviewer for suggesting Alternating ODE/SDE (a MixGRPO-style method) as an additional baseline. Following this suggestion, we conducted experiments with alternating ODE and SDE sampling on SD3.5-M using 20 sampling steps, matching the NFEs of our method to ensure a fair comparison. This is the same experiment as the MixGRPO baseline in our response to Reviewer B4Rv. The results are presented below.

Table: Comparison with the Alternating ODE/SDE baseline on SD3.5-M under matched NFEs.

MethodHPS-v2.1 (Task)AestheticCLIPImgRwdHPS-v2.1 (Quality)PickScore
Alternating ODE/SDE0.3816.120.2761.320.35522.94
Flow-GRPO0.3816.300.2671.410.35722.89
CPS0.3716.120.2701.380.34422.62
LC-GRPO0.3936.110.2801.440.36723.03

As shown in the table, LC-GRPO outperforms the newly added Alternating ODE/SDE baseline under matched NFEs, further demonstrating the effectiveness of our approach.

Q2 (Weakness 2): I would also like to understand how sensitive LC-GRPO is to the noise parameter η\eta. ... comparing the sensitivity of LC-GRPO, Flow-GRPO, and CPS to η\eta ...

A2: We conducted additional experiments using η=0.7\eta=0.7 for SD3.5-M and η=0.9\eta=0.9 for FLUX.1-Dev, fixed across tasks within each model. LC-GRPO retains its reward advantage across the evaluated image tasks, showing that the improvements do not depend on task-specific noise selection.

These results are summarized in our common response and reported in Appendix B.1. In particular, in the two settings where the fixed η\eta differs from our original choice, LC-GRPO's task reward changes only slightly and remains the best: SD3.5-M OCR goes from 0.960 (η=0.8\eta=0.8) to 0.946 (η=0.7\eta=0.7), compared with 0.914 for Flow-GRPO and 0.935 for CPS, and FLUX.1-Dev HPS-v2.1 goes from 0.384 (η=0.8\eta=0.8) to 0.379 (η=0.9\eta=0.9), compared with 0.378 for Flow-GRPO and 0.373 for CPS.

Q3 (Weakness 3): I noticed that the theoretical analysis relies on some assumptions ... such as the condition in Eq. (13). ... a numerical study using a multivariate Gaussian distribution could help ...

A3: We thank the reviewer for this valuable suggestion. Please refer to our common response.

AuthorsOpenReview ↗

Response to Reviewer qoLm (2/2)

Q4 (Question 1): I noticed that the Langevin step size is chosen differently ... I wonder whether comparing the sum of noise variances is sufficient to support this claim.

A4: Thank you for the question. We would like to clarify that the claim in Figure 2 is primarily an empirical observation: at comparable levels of injected noise, ODE + Langevin sampling produces samples with better visual clarity than SDE sampling.

To quantify the injected noise levels, we report the noise standard deviation at each timestep for both methods (10 steps):

TimestepOurs (t−Δt)(t - \Delta t)SDE (t1−t⋅Δt)(\sqrt{\frac{t}{1-t}\cdot \Delta t})
1.000.961.00
0.960.911.06
0.910.860.77
0.860.790.64
0.790.710.56
0.710.600.50
0.600.460.46
0.460.280.40
0.280.010.32
0.010.000.01
Sum of variances4.274.16

We would like to highlight two points:

  1. Noise levels do not match exactly at each timestep under the respective sampling schedules. Our Langevin step size is derived from the predictor-corrector formulation, whereas SDE sampling follows a different noise schedule. Ideally, we would control the exploration level to isolate differences in sample quality, but exact per-timestep matching is not inherent to these two formulations.
  2. The summed noise variance provides a reasonable quantitative characterization of trajectory-level exploration. The similar sums of injected noise variances (4.27 vs. 4.16) indicate comparable overall noise-injection budgets. While this metric does not fully establish equivalent exploration, as the timing of noise injection and its interaction with sampling dynamics can influence the resulting trajectories, it provides a meaningful quantitative basis for comparison. Together with the empirical results in Figure 2, this supports our observation that ODE + Langevin sampling achieves better sample clarity at comparable noise levels.

We thank the reviewer for raising this point. Developing a principled quantitative characterization of exploration over the entire sampling trajectory remains an interesting open research question. In this work, the summed noise variance provides a reasonable and informative measure of the overall noise-injection budget, offering quantitative support for our empirical observations.

Thank you for your detailed responses. I have decided to maintain my original score.

We are glad that our responses were helpful, and we sincerely appreciate your continued positive assessment of our work.

Reviewer_CY9BOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors consider text to image generation with GRPO. Flow based models require additional noise injection to give rise to stochastic policies suitable for GRPO fine tuning. Perhaps the key contribution in the paper is a simple alternative to SDE: ODE step followed by a single Langevin correction. The authors provide theoretical arguments in favor of this small change. However, the conditions/assumptions in the theorems are not checked nor likely to hold in practice. Empirical results show consistent improvements though differences are small and hyper-parameters change per problem/method.

Critical strengths

The authors provide a simple and clean predictor - corrector variant to realize stochastic policies suitable for use with GRPO.

Critical weaknesses

The conditions in the theorems are not verified. E.g., log-concavity in theorem 1 is unlikely to hold and the extra condition (eq 13) in theorem 2 is never checked.

Hyper-parameter tuning (noise level) changes by problem and method. Since differences are small, it is challenging to interpret whether the ordering would change with hyper-parameter choices.

Some comparison to previous theoretical results would be helpful such as Xu et al. "Restart Sampling for Improving Generative Processes" where contraction arguments are used to characterize ODE approximation errors relative to SDE.

Questions for authors

How were the hyper-parameters chosen per method/problem?

While obtaining actual error bars would be expensive, rerunning some of the experiments with different random initializations or different hyper-parameters would offer some indication of whether the small observed differences are real.

5 comments
AuthorsOpenReview ↗

Response to Reviewer CY9B

Thank you for your thoughtful and valuable feedback! We address your concerns below.

Q1 (Weakness 1): The conditions in the theorems are not verified. ... log-concavity in theorem 1 is unlikely to hold and ... (eq 13) in theorem 2 is never checked.

A1: We thank the reviewer for raising these points. We acknowledge that α\alpha-strong log-concavity is a restrictive assumption that may not hold in practical settings. However, strong log-concavity is a well-established assumption in theoretical analyses of Langevin sampling and diffusion models [1,2,3], commonly used to establish Wasserstein convergence guarantees. Our Theorem 1 follows this theoretical framework to characterize the effect of Langevin correction under controlled assumptions.

Regarding the condition in Eq. 13 of Theorem 2, please refer to our common response.

Q2 (Weakness 2): Hyper-parameter tuning (noise level) changes by problem and method. ... it is challenging to interpret whether the ordering would change with hyper-parameter choices.

A2: Please see our common response on hyperparameter tuning. With η\eta fixed across tasks within each base model, LC-GRPO retains its reward advantage across the evaluated image tasks (Appendix B.1).

Q3 (Weakness 3): Some comparison to previous theoretical results would be helpful such as Xu et al. “Restart Sampling for Improving Generative Processes”

A3: We thank the reviewer for pointing out Xu et al. (2023), "Restart Sampling for Improving Generative Processes". We agree that their contraction-based analysis is relevant to our theoretical results and have included a citation to their work in our revised manuscript.

Xu et al. demonstrate that stochastic dynamics can contract accumulated sampling errors, while deterministic ODE solvers generally exhibit smaller discretization errors. Although both works investigate stochastic error contraction, our work focuses on a distinct theoretical setting. Specifically, we establish a single-step Wasserstein-2 contraction guarantee for Langevin correction applied after ODE integration, explicitly characterizing the trade-off between error contraction and discretization bias. Furthermore, we theoretically show that LC can achieve smaller one-step approximation error than standard SDE sampling under matched exploration noise.

Despite this theoretical connection, the two methods differ in their objectives and implementations. Unlike Restart, which improves inference-time generation through repeated noise injection and denoising, LC-GRPO targets the training–inference gap in flow-based reinforcement learning while preserving tractable transition likelihoods required for GRPO optimization.

Q4 (Question 1): How were the hyper-parameters chosen per method/problem?

A4: Please refer to the common response on how we choose the hyperparameters. More specifically, we follow the setting from the previous Flow-GRPO and then tune the parameters based on the training curve.

Q5 (Question 2): While obtaining actual error bars would be expensive, rerunning ... with different random initializations ... would offer some indication of whether the small observed differences are real.

A5: We thank the reviewer for the suggestion. We have conducted additional experiments with multiple random seeds. Please refer to our common response for details.

[1] On diffusion-based generative models and their error bounds: The log-concave case with full convergence estimates, Stefano Bruno et al., Transactions on Machine Learning Research (TMLR), 2025

[2] User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient, Arnak S. Dalalyan, Avetik G. Karagulyan, Stochastic Processes and their Applications, 2019

[3] The query complexity of sampling from strongly log-concave distributions in one dimension, Sinho Chewi et al., Conference on Learning Theory (COLT), 2022

Thank you for the clarifications. I am more positive about the paper after the careful response though still feel that the overall contribution is somewhat borderline due to a small methodological step. For example, Langevin corrections have appeared many times before with the goal of correcting the sampler to follow a prescribed marginal probability path. See, e.g.,

Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC http://arxiv.org/abs/2302.11552

Thank you for carefully considering our response and for your positive update. We agree that Langevin correction itself is not new. Both predictor–corrector sampling (Song et al.), as discussed in our paper, and the MCMC sampler (Du et al.), as you pointed out, use stochastic correction steps to improve sampling accuracy. We acknowledge this line of work in Section 4 and do not claim the correction itself as our contribution.

Our contribution lies in developing a GRPO-compatible Langevin correction mechanism that addresses the train–inference gap in flow-based RL fine-tuning. We highlight three key distinctions:

  1. A new role: enabling RL exploration while preserving sample quality. Prior work (Song et al.; Du et al.) primarily uses stochastic correction to improve generation quality. In contrast, LC-GRPO uses Langevin correction as the exploration mechanism for RL rollouts. Each transition first follows the same ODE Euler step used at inference, then applies a Langevin step to introduce stochasticity while keeping the distribution close to the target marginal. Experiments across multiple models and reward optimization tasks validate the effectiveness of this design, with LC-GRPO consistently outperforming existing baselines, including SDE-based sampling methods.

  2. A GRPO-compatible design with tractable policy likelihoods. GRPO requires evaluating the transition density pθ(xt−Δt∣xt,c)p_\theta(x_{t-\Delta t}\mid x_t,c) to compute policy ratios. Directly adopting multi-step MCMC or Metropolis-adjusted samplers is not straightforward, as their marginal transition densities generally lack the simple closed form required by our GRPO formulation. We address this by using a single unadjusted Langevin step after the deterministic ODE update, yielding an isotropic Gaussian transition with a closed-form likelihood. The required score is recovered directly from the flow velocity via Tweedie's formula, without an additional score or energy model. This enables Langevin correction to be integrated directly into GRPO optimization.

  3. Theoretical justification for RL rollouts. Theorem 2 establishes conditions under which Langevin correction achieves lower discretization error than Euler–Maruyama sampling of the reverse SDE, at matched randomness levels. This provides theoretical support for the advantage of our exploration mechanism over conventional SDE-based rollouts.

Together, these contributions demonstrate how LC-GRPO jointly addresses the challenges of train–inference alignment, policy tractability, and rollout accuracy. Supported by both theoretical analysis and empirical results, our method represents a meaningful methodological advance beyond existing Langevin-based sampling approaches.

Thank you again for engaging with our response. We hope these distinctions clarify the scope of our contribution and would be grateful if you could take them into account in your final assessment.

Reviewer_B4RvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Flow-based GRPO methods swap the inference-time ODE for an SDE to get stochastic rollouts, but with few steps the SDE rollouts become blurry and no longer match the ODE samples used at test time. The paper proposes LC-GRPO, where each rollout transition takes the same ODE Euler step used at inference, then one Langevin step at time t−Δtt - \Delta t that uses a score computed from the velocity via Tweedie's formula, so the transition stays an isotropic Gaussian with a closed-form likelihood.

Critical strengths
  1. The method is simple and easy to plug in. It changes only the rollout sampler, needs no extra score model, keeps the GRPO objective, and the last step is a plain ODE step. The idea of separating transport (ODE step) from exploration (Langevin step) is clear.
  2. The paper gives a theoretical reason for the design. Under strong log-concavity, one Langevin step provably reduces the W2W_2 error of an imperfect Euler step, and a leading-order analysis gives an explicit condition under which the corrected step beats the Euler–Maruyama step at matched noise.
  3. The evaluation covers several models and tasks: SD3.5-M and FLUX.1-Dev for OCR, HPS-v2.1, and HPS-v2.1 plus CLIP, and HunyuanVideo with VideoAlign. The compute setup is careful, since each Langevin step costs two NFEs and the baselines are run at both the same and twice the number of rollout steps. The train versus eval reward curves are a useful way to show the gap the paper targets.
Critical weaknesses
  1. Theorem 1 needs pt−Δtp_{t - \Delta t} to be α\alpha-strongly log-concave, which does not hold for image or video distributions, and it needs ϵt\epsilon_t to scale like α2ε2\alpha^2 \varepsilon^2 divided by L2dL^2 d, which is tiny for a high-dimensional latent. The theory also uses the true score, while the method uses a score derived from the learned velocity, which divides by tt and can be unstable near t=0t = 0. Theorem 2 needs ϵ<=Λh\epsilon <= \Lambda h and holds only "for all sufficiently small Δt\Delta t", but the practical rule 2ϵt=η(t−Δt)\sqrt{2\epsilon_t} = \eta(t - \Delta t) does not shrink with Δt\Delta t, and training uses only 6 to 10 steps. The key condition (Eq. 13) is assumed and never checked, even roughly. Also, the experiments do not use the matched step ϵt=0.5σt2Δt\epsilon_t = 0.5 \sigma_t^2 \Delta t: the footnote says that at the same η\eta the total noise variance is larger for Langevin than for the SDE, and each method gets its own η\eta. So the theorems motivate the method but do not explain the reported results. The claim that one Langevin step "leaves the marginal pt−Δtp_{t-\Delta t} invariant" is only true in continuous time; the paper's own Eq. 12 shows a bias term GG for a finite step.
  2. On FLUX.1-Dev, HPS-v2.1 goes from 0.378 (Flow-GRPO) to 0.384, and in the multi-reward case from 0.371 to 0.376. No seeds, variance, or error bars are given, so it is hard to tell if these gaps are real. Hyperparameters differ by method: for OCR, β=0.04\beta = 0.04 for Flow-GRPO, 10−410^{-4} for CPS, and 4x10−34 x 10^{-3} for LC-GRPO, and η\eta also changes per method and task, so part of the gain may come from tuning. For each baseline the better of the two step counts is reported, but which one won is not stated. The text says LC-GRPO "consistently outperforms all baselines" and preserves quality, but Aesthetic is lower than Flow-GRPO in four of five image settings (for example 6.11 vs 6.30 on SD3.5-M HPS-v2.1). On video, LC-GRPO lowers Text Alignment from 1.247 (base) to 1.151 (DanceGRPO gets 1.312), Semantic Score from 69.96 to 67.83, and Dynamic Degree from 52.8 to 45.8. Wall-clock training time is not reported.
  3. MixGRPO and SAGE-GRPO are discussed but not compared, and DiffusionNFT is cited but not used as a baseline. The ablations cover only gradient flow through the Langevin step and CFG in that step. There is no study of η\eta, of the step-size rule, of more than one Langevin step, or of Langevin versus SDE at matched total noise. The rollout-quality claim rests on Fig. 2, which shows two prompts and no numbers; rollout reward versus η\eta for both samplers would be more convincing. The train versus eval gap curve is shown only for FLUX.1-Dev with HPS-v2.1. A smaller gap alone also does not show better optimization, since a sampler that adds little noise would also have a small gap. The paper should discuss prior work on adding noise and then correcting in diffusion and flow samplers, and on timestep-aware flow GRPO.
Questions for authors
  • For each baseline, which step count (for example 10 or 20 on SD3.5-M) gave the reported result, and what is the wall-clock training time per method?
  • Can you check the condition in Eq. 13, even roughly, on a trained model, and explain how Theorem 2 relates to the practical rule 2ϵt=η(t−Δt)\sqrt{2\epsilon_t} = \eta(t - \Delta t), which does not shrink with Δt\Delta t?
2 comments
AuthorsOpenReview ↗

Response to Reviewer B4Rv (1/2)

Thank you for your thoughtful and valuable feedback! We address your concerns below.

Q1 (Weakness 1): Theorem 1 needs pt−Δtp_{t-\Delta t} to be α\alpha-strongly log-concave ... the theorems motivate the method but do not explain the reported results.

A1: We thank the reviewer for raising these points. We acknowledge that α\alpha-strong log-concavity is a restrictive assumption that may not hold in practical settings. However, strong log-concavity is a well-established assumption in theoretical analyses of Langevin sampling and diffusion models [1,2,3], commonly used to establish Wasserstein convergence guarantees. Our Theorem 1 follows this theoretical framework to characterize the effect of Langevin correction under controlled assumptions.

Theorem 2 more directly supports our main claim regarding the advantage of Langevin correction over SDE sampling in RL algorithms requiring exploration during rollouts. While its theoretical guarantees rely on specific assumptions, our empirical results demonstrate the effectiveness of the proposed method under practical sampling settings. We provide additional numerical analysis of the condition in Eq. 13 and clarify the relationship between theoretical and practical step-size choices in our response to Q5 below.

We appreciate the reviewer's observations and will revise the presentation to clarify the assumptions, limitations, and respective roles of the two theorems.

Q2 (Weakness 2): On FLUX.1-Dev, HPS-v2.1 goes from 0.378 ... to 0.384 ... No seeds, variance, or error bars are given ... part of the gain may come from tuning.

A2: Following the Flow-GRPO setting, we conducted additional experiments with η=0.7\eta=0.7 for SD3.5-M and η=0.9\eta=0.9 for FLUX.1-Dev, fixed across tasks within each model. LC-GRPO retains its reward advantage across the evaluated image tasks, as reported in Appendix B.1 and summarized in our common response.

Please refer to our common response for the multi-seed comparison results. Our results show that our method performs better than Flow-GRPO under the multi-seed experimental setting.

For the KL coefficient, the OCR settings are β=0.04\beta=0.04, 10−410^{-4}, and 4×10−34\times10^{-3} for Flow-GRPO, CPS, and LC-GRPO, respectively. These coefficients act on different transition distributions, so identical numerical values need not impose identical regularization. In the human-preference image experiments, including both FLUX settings, all three methods use β=0\beta=0; the smaller FLUX reward differences therefore cannot be attributed to different KL coefficients.

The quality results involve trade-offs. LC-GRPO does not improve every metric: the aesthetic-score decreases and the reductions in video Text Alignment, Semantic Score, and Dynamic Degree should be stated explicitly. For example, its Dynamic Degree of 45.8 is below the base model’s 52.8, although it exceeds DanceGRPO’s 12.5 and CPS’s 19.4. We will narrow the quality claim and distinguish stochastic rollout clarity from the quality metrics of the fine-tuned models.

The selected baseline step counts and wall-clock times are provided in Q4 below.

[1] On diffusion-based generative models and their error bounds: The log-concave case with full convergence estimates, Stefano Bruno et al., Transactions on Machine Learning Research (TMLR), 2025

[2] User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient, Arnak S. Dalalyan, Avetik G. Karagulyan, Stochastic Processes and their Applications, 2019

[3] The query complexity of sampling from strongly log-concave distributions in one dimension, Sinho Chewi et al., Conference on Learning Theory (COLT), 2022

AuthorsOpenReview ↗

Response to Reviewer B4Rv (2/2)

Q3 (Weakness 3): MixGRPO and SAGE-GRPO are discussed but not compared ... There is no study of η\eta, of the step-size rule, of more than one Langevin step ...

A3: Thanks for the suggestion. We added the baseline result for MixGRPO (more specifically, alternating ODE/SDE with 20 sampling steps, sharing the same rollout cost as our method, but without the sliding window, so that the optimized timesteps stay fixed during training). This is the same experiment as the Alternating ODE/SDE baseline in our response to Reviewer qoLm. From the results we can see that our method performs better than MixGRPO.

Table: Comparison with the MixGRPO baseline on SD3.5-M under matched NFEs.

MethodHPS-v2.1 (Task)AestheticCLIPImgRwdHPS-v2.1 (Quality)PickScore
MixGRPO0.3816.120.2761.320.35522.94
LC-GRPO0.3936.110.2801.440.36723.03

On the other hand, SAGE-GRPO changes both the rollout and the GRPO optimization procedure, while DiffusionNFT uses a different forward-process fine-tuning objective rather than a GRPO rollout objective. We therefore focused this ablation on MixGRPO, which directly tests the alternative mixed ODE/SDE rollout design. Thus we discussed SAGE-GRPO and DiffusionNFT as related methods, but did not perform the experimental comparison with them here.

Q4 (Question 1): For each baseline, which step count (for example 10 or 20 on SD3.5-M) gave the reported result, and what is the wall-clock training time per method?

A4: We thank the reviewer for their attention to the experimental details. To clarify the settings used for the reported results, we provide the number of sampling steps during training for each baseline in Table 1, along with the corresponding wall-clock training times in Table 2.

Table 1: Number of sampling steps during training.

MethodSD3.5-M HPSSD3.5-M HPS+CLIPSD3.5-M OCRFLUX.1-Dev HPSFLUX.1-Dev HPS+CLIPHunyuanVideo VideoAlign
Flow-GRPO / DanceGRPO (video)101010121216
CPS1010106616
LC-GRPO101010668

Table 2: Wall-clock training time (hours:minutes).

MethodSD3.5-M HPSSD3.5-M HPS+CLIPSD3.5-M OCRFLUX.1-Dev HPSFLUX.1-Dev HPS+CLIPHunyuanVideo VideoAlign
Flow-GRPO / DanceGRPO (video)23:5415:20N/A (checkpoint)27:1830:0614:31
CPS19:4310:1708:4814:1614:5214:26
LC-GRPO22:1923:4323:1719:0522:3313:27

Note: The reported wall-clock training time is measured up to the checkpoint at which each method converges. "N/A (checkpoint)" indicates that we use the checkpoint released by the authors of the original paper.

Q5 (Question 2): Can you check the condition in Eq. 13 ... on a trained model, and explain how Theorem 2 relates to the practical rule ... which does not shrink with Δt\Delta t?

A5: We thank the reviewer for raising these two important questions.

Regarding the condition in Eq. 13, please refer to our common response.

Regarding the practical step-size rule, Theorem 2 aims to establish the accuracy advantage of Langevin correction over SDE sampling under matched noise levels. By controlling the injected noise to be equal across both methods, we isolate the effect of sampling dynamics on accuracy. This is a controlled theoretical comparison, rather than a requirement for practical step-size selection. In practice, our Langevin step size follows the predictor-corrector formulation and is not constrained by the noise-matching condition in Theorem 2. Thus, the theorem provides theoretical support for the advantage of Langevin correction, while our empirical results (Figure 2) further demonstrate its effectiveness under the practical noise schedule.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Responses on Hyperparameter Tuning, Theoretical Conditions and Additional Random-Seed Experiments

We appreciate all reviewers for their valuable feedback! We first answer common questions then respond to each reviewer individually.

Hyperparameter Tuning

We conducted additional experiments using a fixed noise level across all tasks within each image model: η=0.7\eta=0.7 for SD3.5-M and η=0.9\eta=0.9 for FLUX.1-Dev, matching the corresponding Flow-GRPO settings.

With these fixed values, LC-GRPO continues to outperform the baselines in reward optimization across the evaluated image tasks, as reported in Appendix B.1 of the revised manuscript. These results show that the improvements do not depend on selecting a separate η\eta for each task.

In the original experiments, we selected LC-GRPO's η\eta and β\beta based on the training curve. Suitable noise levels can differ across base models and sampling schemes. The additional fixed-η\eta experiments test whether the advantage remains without task-specific noise tuning.

Verification of Eq.(13) in Theorem 2

We thank the reviewers for this valuable suggestion. To examine when Eq. (13) holds, we consider a multivariate Gaussian setting with data distribution x0∼N(μ,λId)x_0 \sim \mathcal{N}(\mu, \lambda I_d), with the time-dependent SDE noise schedule σt=ηt/(1−t)\sigma_t = \eta \sqrt{t/(1-t)}. We derive that Eq. (13) holds if and only if

\left(\lambda-\lambda_*(t,\eta)\right)>0,$$ where \$\lambda_*(t,\eta)= \frac{\eta^2 t^2\left[\eta^2+2(1-t)\right]} {2(1-t)^2(2+\eta^2 t)}\$. We provide the complete derivation in Appendix A.3 of the [revised manuscript](https://openreview.net/pdf?id=JoQmrxygxH). To further illustrate this condition, we visualize the regions where Eq. (13) holds across different covariance scales $\lambda$, timesteps $t$, and noise levels $\eta$, as shown in [condition visualization](https://anonymous.4open.science/api/repo/ICLR27_3384/file/Figures/condition_eq13_full_figure.pdf?v=e5e08aa5). The results reveal two key insights: 1. **The condition holds over most timesteps.** The visualization shows that the condition is satisfied over substantial portions of the sampling trajectory across a broad range of Gaussian settings, providing theoretical support for the improved sampling accuracy of Langevin correction compared to SDE sampling. 2. **Coverage increases with noise level $\eta$.** As $\eta$ increases, the condition holds over a larger portion of the sampling trajectory. This suggests that Langevin correction becomes more effective at preserving sample quality under stronger noise injection and exploration, consistent with our empirical observations in Figure 2. ### **Additional Random-Seed Experiments** We thank the reviewers for suggesting experiments with multiple random seeds to report error bars. Accordingly, we conduct additional experiments using three random seeds (42, 43, and 44). Due to computational resource constraints, we limit these experiments to optimizing SD3.5-M with the HPS-v2.1 reward. The results are presented below. **Table: Results across multiple random seeds on SD3.5-M with the HPS-v2.1 reward.** | Method | HPS-v2.1 (Task) | Aesthetic | CLIP | ImgRwd | HPS-v2.1 (Quality) | PickScore | |:-------|:--------------:|:---------:|:----:|:------:|:------------------:|:---------:| |Flow-GRPO|0.383 ± 0.001| 6.18 ± 0.08 | 0.269 ± 0.003 | 1.41 ± 0.02 | 0.356 ± 0.001 | 22.84 ± 0.04 | |LC-GRPO|0.390 ± 0.002 | 6.03 ± 0.06 | 0.278 ± 0.002 | 1.41 ± 0.02 | 0.364 ± 0.002 | 22.94 ± 0.08|

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.