acceptodds
Under review as a conference paper at ICLR 2027

Timestep Weighting: A Hidden Key to Effective ELBO-Based Flow-Matching RL

Abstract

ELBO-based reinforcement learning offers a sampler-agnostic approach to fine-tuning flow matching models with reward feedback. Timestep weighting in ELBO-based RL has large impact on performance, and it also provides a unified view (as we show in this work) to understand prediction losses heuristically chosen in prior work, yet it remains under-researched and is often chosen to inherit pretrain configs. We investigate impacts and dynamics of timestep weighting in ELBO-based RL. We show that effective weighting depends on both the reward landscape and stage of learning. (1) Through experiments on controlled CIFAR image generation, complemented by robotics, we investigate how weighting impacts reward-driven updates across noise levels. (2) Through gradient analysis, we reveal distinct patterns of cross-noise coordination across tasks and their evolution during training. These findings motivate the hypothesis that useful weighting depends on the gap between the policy's current behavior and the behavior favored by the reward. (3) Guided by this analysis, we study simple static weighting, budgeted profile selection, and dynamic schedules that improve performance beyond conventional target choices. Our results establish timestep weighting as an important design choice for flow-matching RL and motivate further research into methods that choose and adapt it throughout learning.

Primary area:
generative models
Keywords:
Flow Matching, Reinforcement Learning, Optimization Dynamics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_u7owOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies timestep weighting in ELBO-based RL for flow matching models. Existing methods differ in prediction targets (FPO: noise, FPO++: velocity, V-GRPO: clean sample); in a common coordinate, these targets assign different weights to errors across noise levels (wv=1w_v=1, wx0=(1−t)2w_{x_0}=(1-t)^2, wϵ=t2w_\epsilon=t^2), so target choice implicitly specifies a timestep weighting. This choice has a large impact on performance, yet remains under-researched and is often inherited from pretrain configs.

Through controlled CIFAR image generation, complemented by robotics, the authors show that effective weighting depends on both the reward landscape and the stage of learning. Gradient analysis reveals distinct patterns of cross-noise coordination across tasks and their evolution during training, motivating a behavioral-gap hypothesis: useful weighting depends on the gap between the policy's current behavior and the behavior favored by the reward.

Guided by this analysis, the authors study static softmax weighting, budgeted profile selection (successive halving), and dynamic schedules, which improve performance beyond conventional target choices; notably, a high-to-low-noise annealing schedule outperforms the best fixed weightings on three rewards.

Critical strengths

1. Lots of experiments. Many weighting profiles, six rewards, multiple seeds, a separate grid for shape versus scale, and a repeat on robotics with FPO and FPO++. On this testbed the effect is real: pick the wrong profile and some rewards barely train.

2. Complete derivations. The appendix proves everything the main text uses, from how prediction targets turn into timestep weights to why weighting and timestep sampling are the same thing. Nothing has to be taken on faith.

3. Honest reporting. The authors say outright that their online re-weighting failed, that successive halving picked wrong on one task, and that the gradient-cosine analysis gives no consistent signal. The appendix also shows exactly which runs and selection rules are behind each number.

Why it matters: the work is reproducible and not oversold. This is the main reason I do not recommend clear rejection.

Critical weaknesses

1. The methodological novelty is limited, as the core training update is essentially the same as AWM.

Under the paper's CIFAR training setup (one update per rollout), PPO clipping is inactive, so the update reduces to advantage-weighted flow matching—the core idea of AWM, which is itself also a single-step, clip-free method. Although the two differ in a few details (KL term, advantage normalization, timestep sampling), the paper cites AWM without directly comparing against it.

The main new contribution is therefore the study of which noise levels should receive more weight during training. AWM already explores similar choices, though without a controlled ablation, and TempFlow-GRPO (not cited) explicitly reweights per-timestep losses by noise level. The paper should include AWM as a baseline and clearly demonstrate the additional benefit of its timestep-weighting method.

2. For text-to-image flow-matching RL the paper's central question has already been studied and ablated at scale, so the "under-researched hidden key" framing does not hold; the evidence for the proposed strategies is not robust to selection effects.

Setting mismatch. FPO and FPO++ are robot-control methods and V-GRPO is validated on FLUX / SD 3.5, whereas the paper's main evidence is a 32×32 CIFAR-10 U-Net with six hand-constructed rewards and no KL regularizer (G.1). The only large-model setting, Appendix F, uses one seed per arm and shows almost no difference across α (PickScore 0.910–0.918).

Prior work at scale. In the paper's own forward-process formulation, AWM compares three choices of w(t) and finds that weighting high-noise steps works best and the ELBO-derived weight worst—the paper's Fig. 1 finding; DiffusionNFT replaces a hand-chosen w(t) with adaptive weighting; V-GRPO ablates loss weighting and timestep sampling. The denoising-MDP route used by most T2I work addresses the same object: by AWM's Eq. 10, at θ = θ_old (the paper's CIFAR operating point) reweighting MDP step t is a change of w(t). There, TempFlow-GRPO up-weights high-noise steps, MixGRPO slides a window from high to low noise (a schedule), E-GRPO and Chunk-GRPO concentrate updates on high-noise steps, and GRPO-Guard and UniGRPO normalize the implicit timestep weights. The MDP route regresses noisy targets and couples weighting with exploration noise and clipping, so effect sizes need not transfer, but the direction does: the paper's two conclusions—global rewards prefer high-noise emphasis; high-to-low annealing works best—coincide with these results. The paper cites AWM and builds on V-GRPO but compares with neither, and cites none of the MDP-route works.

Why it matters: for the T2I/T2V community the paper adds no new information—its question has already been answered on SD3.5/FLUX, and the paper reproduces that answer on CIFAR.

Questions for authors
  1. (W2) Report at least one metric that is not the training reward: a sample-quality or diversity measure (FID to CIFAR-10, or drift from the pretrained model) and a held-out score, together with final-step reward rather than the peak. GRPO-Guard and Chunk-GRPO show that timestep weighting barely moves the trained reward but changes held-out scores substantially; the same split is needed here to separate real improvement from reward hacking.
  2. (W1/W2) Compare against the weighting choices already in the literature: AWM with w(t)=1w(t) = 1 and w(t)=tw(t) = t, and DiffusionNFT's self-normalized x0x_0 weighting.
  3. For Fig. 17, show generated samples for every α arm at the checkpoint used to report its score (same prompts and seeds across arms), the pre-RL base model's samples and score, and full training curves for all four rewards. Without curves and samples it cannot be told which reported peaks are stable and whether OCR's late α = −1 gain costs image quality.
2 comments
AuthorsOpenReview ↗

Rebuttal (1)

Thank you for the detailed review and for recognizing the breadth of the experiments and the value of the derivations. We really wish to express our gratitude for your feedback, which helped us significantly strengthen our presentation and results, especially by helping us identify where our contribution and its relation to existing work needed to be clearer. We address the novelty concerns and generation quality concerns together here, then each question in the second comment.

Regarding the novelty concerns

We're sincerely grateful for this important and constructive question. It shows that our original presentation did not sufficiently position our contribution against the relevant literature. We have clarified both points and strengthened the discussion of denoising-MDP methods, including the weighting and scheduling work identified in the review (Introduction; Appendix A). Our contribution is the analysis of weighting within ELBO-based flow RL, rather than the introduction of nonuniform timestep weights in flow RL. We highlight three distinctions:

First, weighting conclusions from Gaussian denoising-MDP formulations do not automatically transfer to ELBO-based RL. Both allocate learning across noise levels, but their update signals, sampling distributions, and timestep-dependent signal-to-noise structures differ. A weighting rule that works well in one therefore need not work equally well in the other. Our analysis (Appendix B) identifies the assumptions required for an expected-gradient correspondence and explains why it does not establish equivalence between the implemented procedures. Controlled allocation-transfer experiments also show a performance cost when an allocation selected for one formulation is used in the other. We also implemented MixGRPO’s scheduling design in our scale-controlled ELBO setting, where it did not consistently improve over fixed and continuous-weighting controls (Appendix D.5). These observations motivate analyzing weighting within ELBO-based RL itself as an isolated research topic.

Second, proposing a weighting choice and systematically studying its consequences are different contributions. AWM examines several weighting choices, whereas we map weighting-response curves across rewards and training stages and analyze the associated optimization dynamics. These insights can also inform existing algorithms: in our local AWM comparison, the alternative weighting alpha = −2 improves endpoint quality. Configurations and results are in Appendix D.4.

Indeed, our single-epoch CIFAR protocol yields AWM-like updates. We chose this setting to isolate timestep weighting from rollout reuse, clipping, and regularization. Our E4/E8 and fixed-base regularization experiments show that these factors can change the response curve and preferred alpha, while weighting remains consequential and its effects depend on the reward and optimization setting. Our conclusion is thus not tied to one optimal alpha or to the single-update regime. The controlled E1 study is a starting point; interactions with off-policy robustness and regularization motivate further analysis. Our robotics experiments already use multiple PPO epochs with active clipping, extending the evidence beyond the single-update case. These results are now in the revised manuscript (Section 2.3; Appendices D.1–D.2 and E.1).

Third, the contribution is broader than “high-noise weighting works well for images.” Our study spans image generation and robotics. For images, we examine reward landscapes, weighting-response curves, and how cross-noise gradient relationships evolve during training. This breadth requires systematic sweeps and repeated stage-specific measurements, which cost far more than evaluating a few configurations. We therefore use smaller, controlled testbeds. This choice serves the scientific question: identifying weighting-dependent optimization behavior does not require the largest available model.

Regarding Generation Quality

Following your suggestion, we have added generation-quality analyses, including test FID, independent scores, and generated samples as in the next comment. However, our primary objective is to understand optimization, not to achieve the best image quality. Image quality also depends on reward design, reward-model limitations, and how the score relates to desired visual properties. These questions are important but distinct from how timestep weighting changes the optimization of a given reward.

AuthorsOpenReview ↗

Rebuttal (2) - Responses to your questions

Q1:

We have added final-checkpoint test reward and FID, independent scores, and paired samples. We report reward curves and quality measurements separately (Appendices D.4, D.6 and E.5). The results specify evaluation protocols and distinguish final from selected checkpoints.

In the local AWM comparison, the alpha = −2 variant lowers the endpoint automobile-class FID512 from 184.45 to 82.89 among configurations reaching mean reward at least 0.5. Regularization settings and selection are given in Appendix D.4.

Q2:

We have added AWM and DiffusionNFT adaptations and complete self-normalization on/off curves (Appendices D.3–D.4).

We omitted self-normalization from the main sweep because its error-dependent rescaling interacts with timestep weighting. In our ablations, self-normalization generally reduces the reward of otherwise strong weighting choices rather than improving them. The best seed-averaged alpha on the tested grid is unchanged for both rewards. These results support studying weighting explicitly, including with additional normalization components.

Q3:

Thank you for these suggestions. Appendix E.5 provides training curves for all four rewards, independent endpoint scores, and paired OCR base/alpha samples with common prompts and noise. The sample panel states the evaluated checkpoints and prompt set. We distinguish final-checkpoint scores from the original figure’s peak/checkpoint rules so that curves, scores, and samples can be read together.

Reviewer_mh9eOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes timestep weighting in ELBO-based flow-matching RL. It expresses prediction-target choices in common velocity coordinates, separates allocation shape from a calibrated loss scale, and studies how the preferred weighting varies across rewards and training stages. Empirically, it evaluates fixed profiles, coherence-based static profiles, successive-halving selection, and prescribed schedules on controlled CIFAR10 image generation and robot control, with additional V-GRPO observations. The work gives a systematic empirical test of such a design.

Critical strengths

This work is a carefully organized investigation of an implementation choice. The common-coordinate identities and shape–scale studies connect apparently different prediction targets to timestep allocation, while recognizing that expectations do not imply identical clipped optimization dynamics.

Critical weaknesses

The proposed mechanism is not fully isolated from optimization effects (main shape–scale analysis, etc.). Matching a frozen-bank loss statistic does not match gradient magnitude, clipping frequency, optimizer adaptation, or estimator variance. Coherence and the behavioral-gap interpretation are therefore plausible diagnostics rather than an established causal explanation. More prominent qualification, and existing gradient/clip statistics if available, would improve the interpretation.

Questions for authors

Can Figure 6's comparisons be labeled by estimator/calibration, cohort, and executed versus replayed training cost, including whether evaluation and profile-construction costs are included?

1 comment

Thank you for your assessment and for recognizing the common-coordinate formulation and controlled shape–scale study. We appreciate your suggestions on calibration, comparison protocols, and compute accounting. We address related concerns below.

What does scale calibration control?

It is indeed a good question. Whole-loss calibration separates allocation shape from one predefined measure of overall loss scale, estimated on a frozen bank. This provides a principled and reproducible control for studying weighting shape. One scalar cannot also equalize gradient norms, clipping, and optimizer adaptation. We now state this in Section 2.2 and report gradient-norm and clipping statistics separately (Appendices D.3, E.1 and E.2). These results clarify how weighting shape, loss scale, and update dynamics interact. Coherence and the behavioral gap serve as diagnostic hypotheses for these dynamics.

Fig. 6 and compute accounting

Thank you for these suggestions. We have revised the Fig. 6 caption to state the estimator and calibration settings, cohorts, and whether training cost comes from executed runs or retrospective replay; Appendix E.4 gives a fully labeled version. The caption also specifies the budget units and whether evaluation and profile-construction costs are included.

To make the overhead of successive halving explicit, we measured the cost of its validation evaluations and report it separately from training cost. Appendix E.4 gives the details.

Reviewer_RuUTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies timestep weighting in RL for flow-matching policies. It expresses noise, velocity, and clean-sample prediction losses in common velocity coordinates, showing how target choice induces different timestep weightings. These observations motivate a behavioral-gap hypothesis and three practical strategies: static coherence-based weighting, successive halving over fixed profiles, and prescribed schedules.

Critical strengths
  • It identifies a consequential practical design choice. The fixed-profile experiments provide convincing evidence that inheriting a prediction target or weighting profile can substantially affect reward optimization. Making this choice explicit is useful for understanding differences between existing flow-policy implementations.

  • Some empirical results are substantial. Comparisons between baselines and ablation variants provide meaningful evidence that the proposed components can matter.

Critical weaknesses
  • The central mechanistic explanation is not convincingly established. The post-training versus from-scratch comparison uses substantially different diagnostic batch sizes and objectives (Appendices D.3–D.5). Since cross-batch cosine reflects both directional agreement and estimation noise, these differences confound the proposed behavioral-gap interpretation. This matters because explaining why weighting preferences change is a central contribution beyond demonstrating hyperparameter sensitivity.

  • The headline strategy gains lack sufficiently controlled confirmation. Figure 6 compares successive halving with different cohorts. Moreover, Appendix H.7 reports a later Acrobot cohort where scheduling underperforms its corresponding fixed baseline. Combined with configuration selection and reporting on the same validation banks, these issues prevent a reliable conclusion that the proposed strategies outperform well-tuned fixed weighting.

  • The main CIFAR protocol limits the optimization claims it can support. Algorithm 1 and Appendix G.1 specify identical old/current evaluation inputs and one optimizer step per freshly copied old policy. Under this setup, the ratio is 11 when gradients are evaluated, so clipping is inactive, and the update reduces to advantage-weighted matching. Without experiments involving multiple updates per frozen rollout, the main controlled study provides limited evidence about weighting behavior under the nonlinear ratio-and-clipping dynamics discussed in the paper.

Questions for authors
  • Does the main CIFAR implementation indeed evaluate every update at ratio r=1r=1? If not, which operation creates a nonzero old/current loss difference before the single optimizer step?
2 comments

Thank you for your careful review, for recognizing the importance of explicit weighting choices, and for the suggestion on diagnostic batch size for gradient analysis. We address this point, the strategies, and the single-update protocol below.

Diagnostic draw count and gradient heatmaps

We are sincerely grateful for your critical suggestion. We repeated the diagnostics with larger per-bin samples and independent measurement batches, obtaining clearer and more repeatable heatmaps. The results remain consistent with our analysis of stage-dependent cross-noise coordination, and we will replace the original CIFAR heatmaps with them in our camera-ready version. Now the heatmaps and repeat statistics are in Appendix E.3, referenced from Section 4.

CIFAR and robotics differ in objectives and training protocols, but our analysis also includes a within-CIFAR comparison: Digit7 is an out-of-distribution reward for the CIFAR-pretrained model, trained under the same framework as the other image rewards. This complements the cross-domain observations in a more controlled setting. We view the behavioral gap as an explanatory hypothesis motivated by these observations, and present it as such (Section 4). Establishing a causal account would require further targeted interventions.

Selection and scheduling strategies

We appreciate your comments on strategy comparisons. Our intention is to show that insights into timestep weighting can translate into performance gains through simple selection and scheduling heuristics. We do not claim that successive halving or a prescribed schedule universally outperforms well-tuned fixed weighting. Their effectiveness depends on the task and training stage, as the Acrobot example illustrates.

These experiments show the practical value of the analysis and motivate more broadly applicable weighting strategies. We have clarified this positioning in Section 5. We also revised the Fig. 6 caption to distinguish cohorts and selection protocols; Appendix E.4 gives the compute accounting, including selection-evaluation cost.

Ratio-one updates and multiple PPO epochs

Yes, in the original CIFAR protocol, the old policy is freshly copied and both policies are evaluated on identical inputs before a single optimizer step. The ratio therefore equals one at the gradient evaluation, and clipping is inactive. We now state this explicitly in Section 2.3.

We chose this E1 setting to study timestep weighting under a controlled protocol without rollout reuse. Our new experiments compare E1, E4, and E8 and separately examine fixed-base regularization (Appendices D.1–D.2). More rollout reuse substantially reduces performance for some weighting choices and can change the response curve, while weighting remains consequential and setting-dependent.

These observations motivate further analysis of how weighting interacts with off-policy updates, regularization, and clipping. E1 provides a controlled starting point, and the new results extend the study to multiple-update regimes. Our robotics experiments already use multiple PPO epochs with active clipping (Appendix E.1), so the evidence extends beyond the single-update case.

Thanks to the authors for the additional experiments and clarification. After carefully reading the rebuttal and other reviews, I have updated my score accordingly.

Comments on the paper

4 comments

We sincerely thank all reviewers for their careful reading, constructive questions, and recognition of our extensive experiments and of the importance of timestep weighting. The reviews helped us improve both the presentation and the evidence. Individual responses follow; the revised manuscript marks all changes in blue.

Contribution and relation to prior work. Our contribution is a systematic analysis of timestep weighting in ELBO-based flow-policy RL: how it interacts with the reward, task, and optimization stage. Existing methods motivate this question through their different objectives and weighting choices; we make these choices explicit and examine their consequences through controlled sweeps and gradient diagnostics. We have clarified this positioning in the Introduction and strengthened the related-work discussion (Appendix A). Simple selection and scheduling heuristics demonstrate how these insights can translate into improved RL performance. They illustrate the practical value of studying weighting and motivate more broadly applicable weighting and scheduling methods.

Additional evidence and clearer presentation. We have added experiments with one, four, and eight PPO epochs per rollout batch (E1, E4, E8 below) at a fixed number of Adam updates. We also add AWM-style KL regularization, comparisons with existing weighting and scheduling designs, and larger-sample gradient diagnostics. For image generation, we also report final-checkpoint FID and generated samples to complement our focus on optimization dynamics. Together, these additions clarify how weighting behavior depends on the setting and optimization regime. We appreciate the reviewers’ suggestions, which sharpened the presentation of our analysis and its practical implications. Each response below points to the corresponding revision.

AuthorsOpenReview ↗

Looking forward to further discussion

Dear reviewers, thank you again for reviewing our paper! As it comes to the end of reviewing period, could you please kindly provide further feedbacks to our updates? We are open to further discussion, and we sincerely hope you could reconsider the score based on our feedback if we addressed your concerns. Thank you so much

Dear reviewers,

The discussion period closes in a little less than 24 hours. Hence, we wish to reach out again and ask whether our updated rebuttal and paper address your initial concerns.

We believe our rebuttal has clarified and addressed almost all concerns. If you have further comments or questions, we sincerely wish to hear them during the rebuttal period, so that we can better understand the scores or provide our feedback.

Especially, we'd like to clarify that, our contribution is a full analysis and experiments on timestep weighting's impact for ELBO-based flow matching RL, especially how it interacts with different rewards and training stages. Previous work uses non-uniform weighting, but lacks analysis and is mostly chosen at will or unnoticed (for instance, directly borrowing the weighting used while pretraining the model). We hope this clarification can give you a better understanding of the contribution and experimental design of our paper.

Thanks again for the reviewers' constructive suggestions.

AuthorsOpenReview ↗

Author comment before discussion phase ends

We'd like to thank all reviewers for their time and efforts. These reviews have recognized our strengths and helped improve our work. In detail,

  • Reviewer RuUT recognized our practical design intuitions convincing and experiments substantial. The reviewer's comments further motivate a diagnostic draw count for a more reliable gradient heatmap (Appendix E.3) and more convincingly established comparison within Cifar, further validating the behavioral gap explanation we presented (Section 4).

    We also conduct more experiments on effective PPO ratio updates (Section 2.3, Appendix D), and include more details about proposed selection and scheduling strategies to show further insight (Section 5). We are glad the reviewer has become positive (score 3) about our paper, after checking other reviews and our updates with detailed consideration.

  • Reviewer mh9e found the analysis on common coordinate timestep weighting to connect apparently different prediction targets, and recognized our analysis to be carefully organized. The reviewer's comments further motivates better presentation, especially about timestep weighting scale calibration (Section 2.2, Appendix D and E) and about better illustration of computes presented in Figure 6. Reviewer mh9e has remained positive (score 3) throughout discussion phase.

  • Reviewer u7ow found our experiments thorough, our derivations complete and recognized upright reportings on potentially negative results of the intuitive proposed timestep weighting methods. Reviewer u7ow's main concern is that non-uniform timestep weighting has appeared in previous work using flow models for image (especially in Gaussian denoising-MDP formulations). As claimed in our rebuttal and mostly recognized by reviewers, our contribution is a detailed, careful analysis on timestep weighting for ELBO-based flow model RL with substantial experiments, which (1) does not transfer directly from timestep weighting discussions about Gaussian-MDP based flow model RL, (2) brings focused detailed analysis, while previous ELBO-based RL work mainly chooses timestep weighting mostly at will or unnoticed (e.g. inheriting from pretraining settings). We have improved presentations to clarify and underline this (Section 1).

    Also, we have presented more ablations and experiments, about metrics (FID) other than reward (Appendix D); we compare AWM and DiffusionNFT adaptations to show that timestep weighting intuitively chosen without careful consideration can be sub-optimal (Appendix D); and we provide further training curves for Cifar experiments (Appendix E).

    The reviewer's detailed initial comments (score 2) have motivated us to improve our presentation (especially about contributions), and helped to consolidate our conclusions.

Overall, we are grateful for all reviewers' constructive comments, which have helped to improve our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.