- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies timestep weighting in ELBO-based RL for flow matching models. Existing methods differ in prediction targets (FPO: noise, FPO++: velocity, V-GRPO: clean sample); in a common coordinate, these targets assign different weights to errors across noise levels (, , ), so target choice implicitly specifies a timestep weighting. This choice has a large impact on performance, yet remains under-researched and is often inherited from pretrain configs.
Through controlled CIFAR image generation, complemented by robotics, the authors show that effective weighting depends on both the reward landscape and the stage of learning. Gradient analysis reveals distinct patterns of cross-noise coordination across tasks and their evolution during training, motivating a behavioral-gap hypothesis: useful weighting depends on the gap between the policy's current behavior and the behavior favored by the reward.
Guided by this analysis, the authors study static softmax weighting, budgeted profile selection (successive halving), and dynamic schedules, which improve performance beyond conventional target choices; notably, a high-to-low-noise annealing schedule outperforms the best fixed weightings on three rewards.
Critical strengths
1. Lots of experiments. Many weighting profiles, six rewards, multiple seeds, a separate grid for shape versus scale, and a repeat on robotics with FPO and FPO++. On this testbed the effect is real: pick the wrong profile and some rewards barely train.
2. Complete derivations. The appendix proves everything the main text uses, from how prediction targets turn into timestep weights to why weighting and timestep sampling are the same thing. Nothing has to be taken on faith.
3. Honest reporting. The authors say outright that their online re-weighting failed, that successive halving picked wrong on one task, and that the gradient-cosine analysis gives no consistent signal. The appendix also shows exactly which runs and selection rules are behind each number.
Why it matters: the work is reproducible and not oversold. This is the main reason I do not recommend clear rejection.
Critical weaknesses
1. The methodological novelty is limited, as the core training update is essentially the same as AWM.
Under the paper's CIFAR training setup (one update per rollout), PPO clipping is inactive, so the update reduces to advantage-weighted flow matching—the core idea of AWM, which is itself also a single-step, clip-free method. Although the two differ in a few details (KL term, advantage normalization, timestep sampling), the paper cites AWM without directly comparing against it.
The main new contribution is therefore the study of which noise levels should receive more weight during training. AWM already explores similar choices, though without a controlled ablation, and TempFlow-GRPO (not cited) explicitly reweights per-timestep losses by noise level. The paper should include AWM as a baseline and clearly demonstrate the additional benefit of its timestep-weighting method.
2. For text-to-image flow-matching RL the paper's central question has already been studied and ablated at scale, so the "under-researched hidden key" framing does not hold; the evidence for the proposed strategies is not robust to selection effects.
Setting mismatch. FPO and FPO++ are robot-control methods and V-GRPO is validated on FLUX / SD 3.5, whereas the paper's main evidence is a 32×32 CIFAR-10 U-Net with six hand-constructed rewards and no KL regularizer (G.1). The only large-model setting, Appendix F, uses one seed per arm and shows almost no difference across α (PickScore 0.910–0.918).
Prior work at scale. In the paper's own forward-process formulation, AWM compares three choices of w(t) and finds that weighting high-noise steps works best and the ELBO-derived weight worst—the paper's Fig. 1 finding; DiffusionNFT replaces a hand-chosen w(t) with adaptive weighting; V-GRPO ablates loss weighting and timestep sampling. The denoising-MDP route used by most T2I work addresses the same object: by AWM's Eq. 10, at θ = θ_old (the paper's CIFAR operating point) reweighting MDP step t is a change of w(t). There, TempFlow-GRPO up-weights high-noise steps, MixGRPO slides a window from high to low noise (a schedule), E-GRPO and Chunk-GRPO concentrate updates on high-noise steps, and GRPO-Guard and UniGRPO normalize the implicit timestep weights. The MDP route regresses noisy targets and couples weighting with exploration noise and clipping, so effect sizes need not transfer, but the direction does: the paper's two conclusions—global rewards prefer high-noise emphasis; high-to-low annealing works best—coincide with these results. The paper cites AWM and builds on V-GRPO but compares with neither, and cites none of the MDP-route works.
Why it matters: for the T2I/T2V community the paper adds no new information—its question has already been answered on SD3.5/FLUX, and the paper reproduces that answer on CIFAR.
Questions for authors
- (W2) Report at least one metric that is not the training reward: a sample-quality or diversity measure (FID to CIFAR-10, or drift from the pretrained model) and a held-out score, together with final-step reward rather than the peak. GRPO-Guard and Chunk-GRPO show that timestep weighting barely moves the trained reward but changes held-out scores substantially; the same split is needed here to separate real improvement from reward hacking.
- (W1/W2) Compare against the weighting choices already in the literature: AWM with and , and DiffusionNFT's self-normalized weighting.
- For Fig. 17, show generated samples for every α arm at the checkpoint used to report its score (same prompts and seeds across arms), the pre-RL base model's samples and score, and full training curves for all four rewards. Without curves and samples it cannot be told which reported peaks are stable and whether OCR's late α = −1 gain costs image quality.
2 comments
Rebuttal (2) - Responses to your questions
Q1:
We have added final-checkpoint test reward and FID, independent scores, and paired samples. We report reward curves and quality measurements separately (Appendices D.4, D.6 and E.5). The results specify evaluation protocols and distinguish final from selected checkpoints.
In the local AWM comparison, the alpha = −2 variant lowers the endpoint automobile-class FID512 from 184.45 to 82.89 among configurations reaching mean reward at least 0.5. Regularization settings and selection are given in Appendix D.4.
Q2:
We have added AWM and DiffusionNFT adaptations and complete self-normalization on/off curves (Appendices D.3–D.4).
We omitted self-normalization from the main sweep because its error-dependent rescaling interacts with timestep weighting. In our ablations, self-normalization generally reduces the reward of otherwise strong weighting choices rather than improving them. The best seed-averaged alpha on the tested grid is unchanged for both rewards. These results support studying weighting explicitly, including with additional normalization components.
Q3:
Thank you for these suggestions. Appendix E.5 provides training curves for all four rewards, independent endpoint scores, and paired OCR base/alpha samples with common prompts and noise. The sample panel states the evaluated checkpoints and prompt set. We distinguish final-checkpoint scores from the original figure’s peak/checkpoint rules so that curves, scores, and samples can be read together.
Rebuttal (1)
Thank you for the detailed review and for recognizing the breadth of the experiments and the value of the derivations. We really wish to express our gratitude for your feedback, which helped us significantly strengthen our presentation and results, especially by helping us identify where our contribution and its relation to existing work needed to be clearer. We address the novelty concerns and generation quality concerns together here, then each question in the second comment.
Regarding the novelty concerns
We're sincerely grateful for this important and constructive question. It shows that our original presentation did not sufficiently position our contribution against the relevant literature. We have clarified both points and strengthened the discussion of denoising-MDP methods, including the weighting and scheduling work identified in the review (Introduction; Appendix A). Our contribution is the analysis of weighting within ELBO-based flow RL, rather than the introduction of nonuniform timestep weights in flow RL. We highlight three distinctions:
First, weighting conclusions from Gaussian denoising-MDP formulations do not automatically transfer to ELBO-based RL. Both allocate learning across noise levels, but their update signals, sampling distributions, and timestep-dependent signal-to-noise structures differ. A weighting rule that works well in one therefore need not work equally well in the other. Our analysis (Appendix B) identifies the assumptions required for an expected-gradient correspondence and explains why it does not establish equivalence between the implemented procedures. Controlled allocation-transfer experiments also show a performance cost when an allocation selected for one formulation is used in the other. We also implemented MixGRPO’s scheduling design in our scale-controlled ELBO setting, where it did not consistently improve over fixed and continuous-weighting controls (Appendix D.5). These observations motivate analyzing weighting within ELBO-based RL itself as an isolated research topic.
Second, proposing a weighting choice and systematically studying its consequences are different contributions. AWM examines several weighting choices, whereas we map weighting-response curves across rewards and training stages and analyze the associated optimization dynamics. These insights can also inform existing algorithms: in our local AWM comparison, the alternative weighting alpha = −2 improves endpoint quality. Configurations and results are in Appendix D.4.
Indeed, our single-epoch CIFAR protocol yields AWM-like updates. We chose this setting to isolate timestep weighting from rollout reuse, clipping, and regularization. Our E4/E8 and fixed-base regularization experiments show that these factors can change the response curve and preferred alpha, while weighting remains consequential and its effects depend on the reward and optimization setting. Our conclusion is thus not tied to one optimal alpha or to the single-update regime. The controlled E1 study is a starting point; interactions with off-policy robustness and regularization motivate further analysis. Our robotics experiments already use multiple PPO epochs with active clipping, extending the evidence beyond the single-update case. These results are now in the revised manuscript (Section 2.3; Appendices D.1–D.2 and E.1).
Third, the contribution is broader than “high-noise weighting works well for images.” Our study spans image generation and robotics. For images, we examine reward landscapes, weighting-response curves, and how cross-noise gradient relationships evolve during training. This breadth requires systematic sweeps and repeated stage-specific measurements, which cost far more than evaluating a few configurations. We therefore use smaller, controlled testbeds. This choice serves the scientific question: identifying weighting-dependent optimization behavior does not require the largest available model.
Regarding Generation Quality
Following your suggestion, we have added generation-quality analyses, including test FID, independent scores, and generated samples as in the next comment. However, our primary objective is to understand optimization, not to achieve the best image quality. Image quality also depends on reward design, reward-model limitations, and how the score relates to desired visual properties. These questions are important but distinct from how timestep weighting changes the optimization of a given reward.