acceptodds
Under review as a conference paper at ICLR 2027

Tilted Generator Matching: One RL Objective For Every Diffusion and Flow Model

Abstract

Diffusion and flow models are a flexible framework for generative modeling: they can corrupt data in almost any way, live on almost any state space, and generate with almost any sampler. This flexibility, however, makes them hard to post-train. Reinforcement learning (RL) methods for diffusions and flows have been built one at a time, and none applies across this zoo of models. To fix this, we introduce Tilted Generator Matching (TGM), a single RL objective for every model expressible in the generator matching framework. This encapsulates nearly all standard generative models, such as flow matching, score-based diffusion, and discrete diffusion. Our key insight is that RL reduces to weighted pretraining: each step of KL-regularized RL is solved exactly by the model's own pretraining loss, tilted by exponentiated reward. TGM is therefore likelihood-free, works with any sampler, and on-policy. We study TGM on diffusion and flow language models (DLMs), which span discrete and continuous state spaces and masked, uniform, and Gaussian noise. TGM is the first RL method demonstrated on uniform-noise DLMs and the first designed for Gaussian DLMs, as well as for few-step flow-map language models. On Sudoku, GSM8K, and sentiment tasks, a single configuration improves every DLM family and matches or beats family-specific methods, in one case more than doubling accuracy ( on GSM8K for uniform DLMs). On flow maps, TGM-Map lifts the entire accuracy–NFE frontier, matching base accuracy with up to fewer sampling steps, and learns from rollouts that are up to cheaper to generate, at no cost in final accuracy.

Primary area:
generative models
Keywords:
diffusion language models, discrete diffusion, continuous diffusion, flow matching, flow maps, reinforcement learning, RL post-training, reward fine-tuning, KL-regularized RL, forward-process RL, tilt matching, GRPO, masked diffusion, uniform diffusion, few-step generation, language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bVdyOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors introduce Tilted Generator Matching, single RL objective for every model expressible in the generator matching framework, such as flow matching, score-based diffusion, and discrete diffusion. The authors use the idea of reward tilted distribution: KL-regularized RL is solved exactly by the model’s own pretraining loss, tilted by exponentiated reward. They study TGM on diffusion and flow language models (DLMs), including masked, uniform-noise diffusion discrete diffusion language models, and continuous Gaussian DLMs.

Critical strengths
  1. The paper is overall well written and easy to follow.

  2. It first conduct RL experiments on uniform, Gaussian, and flow-map language models. The experiments on Sudoku and GSM8k shows the effectiveness of the method. Table 3 and Table 4 provides a comprehensive survey on diffusion/flow related models and their RL post-training methods.

Critical weaknesses
  1. My main concern is that the paper offers less novelty than its presentation suggests. The optimal solution of KL regularized RL and learning by weighted maximum likelihood can be traced back to reward weighted regression, the annealed tilt can be traced back to sequential Monte Carlo. The authors state that the one contribution is to generalize tilted matching to generator matching framework. This can be further generalized to generative models that satisfying two properties: a) Fisher consistency, which is with sufficient expressive capacity, the model is capable of recovering data distribution; b) the objective function is linear to data distribution. However, such generalization offers little new insights to us. In addition, the experiments only cover masked, uniform, Gaussian, and flow-map diffusion language models. If the paper were positioned as the first RL method shown to work on these model families, I think that would be a fair and useful contribution. For example, an autoregressive model can be viewed as masked diffusion with a fixed left-to-right order, it also fits into the generator matching framework. In that case, TGM reduces to reward-weighted next-token cross-entropy, i.e., RWR/AWR, however, the paper does not provide the experiments on autoregressive models.

  2. Small writing problem: Immediately after Proposition 2, the paper states that TGM recovers TM and DTM with c=0, without introducing the control-variate coefficient c.

Questions for authors

I have no other questions, please see the weakness.

3 comments

We thank Reviewer bVdy for the direct and constructive review. We agree with repositioning the paper as you suggest, and have run the experiments you suggest. If the repositioning and new experiments address your concern, we would be grateful if you would consider raising your score.

1. Lineage and positioning: we agree

The tilted optimum and its solution by weighted maximum likelihood are classical (RWR [1], AWR [2]), and the annealed tilt goes back to sequential Monte Carlo [11]. Your generalization is correct: Prop. 2 holds for any Fisher-consistent objective that is linear in the data distribution, and we will state this in a remark after Prop. 2. We will also revise the abstract and introduction so that the identity is presented as classical, and the contribution is positioned as you propose: RL for diffusion and flow models, which generate by iterative refinement and have no tractable likelihood, with the first results on uniform-noise and flow-map language models.

MethodWeight on a rolloutNeeds a likelihood?Control variate
RWR [1] / AWR [2]exp(reward) / exp(advantage)Yes: it weights the log-likelihoodNo
GRPO [3], diffu-GRPO [4], Flow-GRPO [5]policy gradientYes: exact, one-step proxy, or SDE transitionsn/a
RAFT [6]1 for the best rollout of a group, else 0NoNo
RWR on the denoising loss [7, 8]exp(reward)NoNo
TM [9] / DTM [10]exp(h·reward), small stepsNoYes
TGM (ours)exp(h·advantage), small stepsNoYes

RAFT is the zero-temperature limit of RWR: RWR weights each rollout in a group by exp(reward/β), and as β → 0 all of the weight goes to the highest-reward rollout, which is exactly the one RAFT trains on. It is therefore the closest baseline to TGM, and Table 1 already includes it for every family. TM and DTM introduced the control variate for Gaussian flows and masked diffusion. TGM states it for any generator-matching loss, adds advantage centering, and extends it to flow maps (Prop. 3, which lies outside your conditions because the consistency loss depends on the model).

2. Is RWR enough on a diffusion model? A new ablation

A ladder on Duo GSM8K, with the same rollouts, budget and decoder at every rung (full test set):

RungMethodDuo GSM8K (%)
0Duo (base)17.0
1RAFT20.5
2RWR-style: exp(h·reward) weights on the pretraining loss17.7
3AWR-style: exp(h·advantage) weights14.7
4+ control variate (TGM, ours)37.2
  • Likelihood-free RWR and AWR do not work here. Rungs 2 and 3 never rise above the base during training. Flow maps behave the same: without the control variate TGM-Map ends near 5% on GSM8K (base 31.5%, with it 43.7%). This agrees with reports on image models, where RWR on the denoising loss trails policy gradients [7] and collapses without an added regularizer [8], and with DTM's own ablation [10].
  • The entire gain appears with the control variate. A diffusion sampler is not an exact sampler of its own model, so refitting the model to its rollouts moves it even with no reward; the control variate removes that reward-free part of the loss. An AR model samples exactly from itself, so the same term has zero expected gradient there.

So the identity is RWR, but on these models the identity alone is not enough. We will add the ladder to the paper.

3. The undefined c. Thank you. c is the control-variate coefficient, introduced only in Section 3.2. We now remove it from the text.

[1] Peters & Schaal. Reward-weighted regression. ICML 2007. [2] Peng et al. Advantage-weighted regression. arXiv:1910.00177. [3] Shao et al. DeepSeekMath (GRPO). arXiv:2402.03300. [4] Zhao et al. d1 (diffu-GRPO). arXiv:2504.12216. [5] Liu et al. Flow-GRPO. NeurIPS 2025. [6] Dong et al. RAFT. TMLR 2023. [7] Black et al. DDPO. ICLR 2024. [8] Fan et al. Online reward-weighted flow matching (ORW-CFM-W2). ICLR 2025. [9] Potaptchik et al. Tilt matching. ICML 2026. [10] Chen et al. Discrete tilt matching. arXiv:2604.18739. [11] Del Moral et al. Sequential Monte Carlo samplers. JRSS-B 2006.

The authors address most of my concerns, so I raise my score to 3. However, to isolate TGM’s contribution beyond TM/DTM, it would help to add a rung with exp(h·reward) weights plus the control variate (i.e., the TM/DTM setting), so the effect of advantage centering is visible separately.

Thank you for raising your score and for suggesting this ablation. We ran the TM/DTM rung you proposed: exp(h·reward) weights plus the control variate, with no advantage centering. Rollouts, update budget, and decoder are the same as every other rung. Results on Duo GSM8K (full 1,319-problem test set):

RungMethodCenteredControl variateDuo GSM8K (%)Change from base
0Duo (base)––17.0–
1RAFT––20.5+3.5
2RWR-style: exp(h·reward)nono17.7+0.7
3AWR-style: exp(h·advantage)yesno14.7−2.3
4TM/DTM setting: exp(h·reward) + control variate (new)noyes23.7+6.7
5TGM (ours): exp(h·advantage) + control variateyesyes37.2+20.2

The control variate is necessary, and centering roughly triples the gain. Neither component works alone: without the control variate, both weightings end at or near the base (rungs 2–3). With it, uncentered weights give +6.7 points and centered weights give +20.2.

Training dynamics (in-loop evaluation on the first 512 test problems):

Phase0102030405060
TM/DTM setting (%)18.614.416.021.520.923.424.2
TGM (%)18.822.321.728.932.835.738.3

The TM/DTM setting first drops below the base, recovers to it only around phase 22, and then gains about 6 points by phase 60. TGM improves from phase 6 onward and plateaus at 37–39%. Both runs are stable throughout. The TM/DTM run was still improving slightly at phase 60, so a longer run might narrow the gap somewhat, but not plausibly close a 14-point difference.

Mechanism. With uncentered weights, every rollout that solves the problem gets a large positive weight and none gets a negative one. After the control-variate shift, the weights average about 2.3, so the update mostly reinforces what the model already produces. With centering, the weights average about 0.25 and below-average rollouts receive negative weight. The model is pushed toward the better rollouts within each prompt, not just toward any correct one.

We will add this rung to the ladder in the paper and use it to state TGM's contribution over TM/DTM precisely: the generator-matching formulation, advantage centering, and the flow-map extension.

Reviewer_36SoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Tilted Generator Matching (TGM), a unified reinforcement learning objective for post-training diffusion and flow models. The core idea is to reweight samples generated by the current model according to their rewards and then update the model using its original pretraining loss, without explicitly computing generation likelihoods. The authors further introduce TGM-Map, extending the approach to few-step flow-map models. Starting from KL-regularized reward maximization, the authors prove that, under the stated assumptions, incremental reward tilting can be achieved through a reward-weighted pretraining objective. Experiments cover masked, uniform-noise, and Gaussian-noise models, evaluated on Sudoku, GSM8K, and sentiment generation. For example, Duo's GSM8K accuracy increases from 17.0% to 37.2%. TGM-Map also achieves promising results when the number of sampling steps used to generate training rollouts is reduced.

Critical strengths
  1. Proposition 2 establishes that, under the stated assumptions, incremental reward tilting can be implemented through a reward-weighted pretraining loss without explicitly computing generation likelihoods. Extending this idea to different noise kernels and state spaces, and further to flow-map models, is the paper's main contribution.
  2. Table 1 shows that TGM outperforms the corresponding baselines in most evaluated settings. For example, Duo's GSM8K accuracy increases from 17.0% to 37.2%, while MDLM's accuracy increases from 12.9% to 20.7%. These results suggest that the method has practical potential across different model families, although improvements are not observed in every setting.
Critical weaknesses
  1. Section 5 discusses the relationship between TGM and prior reward-weighting approaches, while Appendix C.6 explicitly relates TGM to Tilt Matching (TM) and Discrete Tilt Matching (DTM). The main contribution of this paper is to unify these ideas within the generator matching framework and extend them to a broader range of model types. The authors could more precisely explain which theoretical results are new and which models or generative processes become amenable to post-training approaches that existing methods cannot readily handle.
  2. In Table 2, training with rollout budgets of 8 NFE and 32 NFE yields the same GSM8K accuracy of 41.6%, while throughput increases from 9.9 to 39.0 samples per second. However, Appendix E.6 states that this ablation uses a fixed budget of 800 optimization steps, whereas different configurations require different numbers of steps to reach their best performance. The results therefore demonstrate higher throughput for low-NFE rollouts under a fixed training budget, but do not establish that the configurations achieve comparable performance after sufficient training.
Questions for authors
  1. Given that Appendix C.6 discusses the relationship between TGM and TM/DTM, what are the main theoretical contributions of TGM beyond these existing methods? By introducing the generator matching formulation, which models or generative processes become amenable to post-training approaches that existing methods cannot readily support?
  2. Appendix C.3 notes that the control variate may introduce bias when the current model does not accurately represent its target. Have the authors evaluated the effect of this bias on the empirical results? Is this issue more pronounced in TGM-Map when using few-step samplers?
4 comments

We thank Reviewer 36So for the careful reading of the appendix.

W1 / Q1: What is new beyond TM and DTM, and which models become amenable?

TM and DTM each derive the tilt identity for one family: TM for Gaussian flows with velocity regression, DTM for masked diffusion with cross-entropy. Our results are:

ResultPrior workNew in this paper
Tilt identity (Prop. 2)TM: Gaussian flows. DTM: masked CTMC.Stated at the level of generator matching: exact for any noising kernel, process, state space and Bregman loss, and for rollouts from any sampler.
Control variate (Prop. 6, App. C.6)TM's implicit tilt matching; DTM's c-DTMOne family of targets that contains both as special cases, with a closed form for the bias when the current model is inexact (Eq. 19).
Centering (App. C.4)DTM tilts by raw non-negative rewards, so no rollout is pushed away.Advantage centering rescales the increment without moving the minimizer, so below-average rollouts can repel.
Flow maps (Prop. 3)TM suggests the construction.Proof for the cross-entropy two-time head, and the first demonstration on language.

Models that gain an RL method (Table 4): uniform-noise DLMs, where no RL method had been demonstrated; Gaussian DLMs, where existing methods require a stochastic sampler and stored trajectories; and flow-map language models, which had none. We will add this table to Section 5.

W2: Table 2 does not establish comparable accuracy after sufficient training

We agree: Table 2 was a fixed-budget comparison at 800 updates and reported rollout throughput, not total training time. We reran the study to 10,000 updates (12.5× longer) for each rollout NFE, using the recipe of our main flow-map experiments, with one seed. Accuracy is on the full 1,319-problem GSM8K test set at evaluation NFE 128 (standard error ≈ 1.4 points).

Rollout NFEAccuracy (%)Rollout (h)Optimization (h)Total (h)
FMLM+ (base)31.46———
12845.1926.111.938.1
6443.4413.012.825.9
3244.736.512.819.4
1646.023.312.115.4
843.751.612.814.5
440.64 (diverges)0.812.913.8

After sufficient training, rollouts at 8 to 64 NFE finish within 1.8 points of 128-NFE rollouts (43.4–46.0 vs. 45.2) with no trend in NFE, which is within evaluation noise. Rollout time falls 16× from 128 to 8 NFE; because optimization is a fixed ≈12.8 h, total training time falls 2.6×. Below 8 NFE training is unstable: the 4-, 2- and 1-NFE runs diverge after 4,400–6,000 updates.

Total training time to reach a given accuracy (training curves evaluated every 400 updates on a fixed 512-problem subset of the test set, to save compute; 3-evaluation rolling mean; standard error ≈ 2.2 points):

Rollout NFEHours to 42%Hours to 44%
12819.431.9
649.413.5
327.89.3
169.812.3
88.19.9

We will replace Table 2 with this study and revise the abstract and Section 4.2 to state the claim it supports: TGM-Map learns from rollouts that are up to 16× cheaper to generate (8 vs. 128 NFE) with final accuracy within noise, which reduces total training time by 2.6×; rollouts below 8 NFE are unstable.

Q2: Effect of the control-variate bias, and is it more pronounced in TGM-Map?

We had not evaluated it in the submission. We have now measured the net effect of the control variate, bias included: TGM-Map on GSM8K with and without it, at three rollout NFEs, for 10,000 updates each.

Rollout NFEBaseControl variate onControl variate offDifference
431.4640.644.85+35.8
831.4643.755.38+38.4
3231.4644.734.62+40.1

(Full test set. "Off" is the final checkpoint; none is above the base.)

Two findings:

  1. The control variate is necessary, and its bias is negligible compared to the collapse it prevents. Without it, accuracy falls below the base within 400 updates and ends near 5% at every rollout NFE. The same holds on Duo GSM8K: 14.7% without it (base 17.0%, TGM 37.2%). Without the control variate, the model is refit to its own rollouts, which copies the sampler's error into the model.
  2. Few-step rollouts. Your intuition is consistent with what we see. With the control variate on, runs at 8 NFE and above are stable for all 10,000 updates, while runs at 4, 2 and 1 NFE improve and then diverge, which is where Eq. (19) predicts the largest bias. This ablation does not isolate the bias term, so we cannot attribute the divergence to it; we will state this as a limitation.

All Table 1 numbers are trained with the control variate on at the default rollout NFE (128 on GSM8K), in the stable regime. We will add this table and discussion to Appendix C.3.

If this addresses your concerns, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Follow up on the rebuttal

Hi,

Please let us know if our response answered the questions. If that addresses your concerns, we would be grateful if you would consider raising your score

Thanks for the additional results. Most of my concerns have been solved. I will keep my positive score as is.

We will include these additional experiments in our final version and thank the reviewer for their feedback.

Reviewer_yuk7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper gives a common formulation of reward-weighted fine-tuning for diffusion and flow models. It generates answers, scores them, and trains on corrupted versions of those answers using reward-dependent weights. The practical loss uses a frozen model snapshot refreshed each phase, and the flow-map extension retains its consistency loss. Experiments show substantial gains across several language-model families and sampling budgets. The results also support cheaper rollout generation, although equal final accuracy and the full training-cost benefit remain unresolved.

Critical strengths

1.) The common formulation is useful. It connects reward-weighted training across discrete and continuous models while requiring neither sequence likelihoods nor stored generation trajectories. The demonstrations on uniform-noise language models and few-step flow maps give this generality practical value.

2.) The accuracy gains are substantial. In Table 1, Duo improves from 17.0% to 37.2% on GSM8K, compared with 20.5% for RAFT. Figure 3 also shows improved flow-map accuracy across sampling budgets. These results support the usefulness of the method even where the source of its advantage remains uncertain.

Critical weaknesses

1.) The advantage over GRPO is not fully isolated from the comparison settings. Appendix E.3 reports a penalty toward the original pretrained model for the GRPO baselines. TGM uses an anchor to a snapshot refreshed each phase and disables its optional pretrained-model anchor. On FLM Sudoku, TGM also receives 100 updates per phase versus 40 for Flow-GRPO. These differences could affect the gap. Their effect is not isolated in the reported comparisons, so I would be cautious about attributing the full advantage to the proposed update.

2.) The final quality–cost tradeoff remains unresolved. Table 2 shows that eight- and 32-call rollouts both reach 41.6% accuracy after 800 updates. Appendix E.6 explicitly identifies this as a fixed-budget comparison and reports that the 32-call run later reaches 50.8%. This leaves the abstract’s claim of no cost in final accuracy unestablished. The reported 16× speedup measures rollout throughput relative to 128 calls. Total training time to a given accuracy is not reported.

Questions for authors

1.) On one representative task, how does TGM compare with GRPO at matched rollout and update budgets, with and without GRPO’s penalty toward the original pretrained model?

2.) For eight- and 32-call rollouts, how does accuracy change with total training time beyond 800 updates, including generation, scoring, and optimization?

4 comments

We thank Reviewer yuk7 for the careful review and ran new experiments for both concerns.

W1 / Q1: Is the advantage over GRPO isolated from the comparison settings?

Update budget. You are right that on FLM Sudoku, TGM received 100 optimizer steps per phase against 40 for Flow-GRPO. We reran FLM + TGM and FLM + RAFT at Flow-GRPO's budget (40 steps/phase × 30 phases = 1,200 steps). All methods now use the same rollout budget (G = 16, 256 prompts/phase), the same evaluation sampler, and the same number of updates.

Penalty toward the pretrained model. The baselines keep the penalty because it is part of the published methods: GRPO and Flow-GRPO both include a KL term to the frozen pretrained model, and Flow-GRPO's own experiments use β = 0.04, the value we use. TGM uses no such anchor (η = 0). To test whether the penalty holds the baseline back, we reran Flow-GRPO with β = 0 and everything else unchanged.

FLM Sudoku (accuracy %, 2,000 test puzzles; original 100 steps/phase numbers in parentheses):

FLM, Sudoku (1,200 steps)HardExtremeEvil
FLM (base)49.4014.658.40
+ RAFT52.85 (53.45)15.10 (15.45)8.80 (8.45)
+ Flow-GRPO, β = 0.04 (original submission)51.7014.858.40
+ Flow-GRPO, β = 049.3515.459.00
+ TGM60.45 (58.85)18.00 (18.30)12.50 (11.75)
  • Budget. At the matched budget TGM keeps its lead on all three tiers, moving by at most 1.6 points, so the extra updates were not the source of the gap.
  • Penalty. Removing the penalty does not help Flow-GRPO: it changes by −2.35 / +0.60 / +0.60 points and stays within 2.3 points of the base. TGM leads the better of the two Flow-GRPO variants by 8.8 / 2.6 / 3.5 points.

We also removed the penalty from AR + GRPO, where GRPO is strongest:

AR + GRPOSudoku hardSudoku extremeSudoku evilGSM8K
AR (base)0.350.100.0061.49
β = 0.04 (original submission)0.350.100.0068.46
β = 00.600.100.0070.66

Here the penalty costs GRPO 2.2 points on GSM8K and at most 0.25 points on Sudoku, where the AR model solves almost no puzzles. On the AR model itself GRPO remains ahead of TGM (65.9 on GSM8K; see our response to Reviewer bVdy). We will update Table 1 with the matched-budget numbers and add the β = 0 baselines.

W2 / Q2: Final quality–cost tradeoff of few-step rollouts

We agree: Table 2 was a fixed-budget comparison at 800 updates and reported rollout throughput, not total training time. We reran the study to 10,000 updates (12.5× longer) for each rollout NFE, using the recipe of our main flow-map experiments, on one GPU with one seed. Accuracy is on the full 1,319-problem GSM8K test set at evaluation NFE 128 (standard error ≈ 1.4 points). Reward scoring takes under 0.1 h in every run.

Rollout NFEAccuracy (%)Rollout (h)Optimization (h)Total (h)Total speedup
FMLM+ (base)31.46————
12845.1926.111.938.11.0×
6443.4413.012.825.91.5×
3244.736.512.819.42.0×
1646.023.312.115.42.5×
843.751.612.814.52.6×
440.64 (diverges)0.812.913.8—

To save compute, training curves are evaluated every 400 updates on a fixed subset of the test set (the first 512 problems, identical for every run and checkpoint; standard error ≈ 2.2 points). This subset is used only for curves and checkpoint selection. Total training time to reach a given accuracy on this subset (3-evaluation rolling mean):

Rollout NFEHours to 42%Hours to 44%
12819.431.9
649.413.5
327.89.3
169.812.3
88.19.9

Three conclusions:

  1. Final accuracy. Rollouts at 8 to 64 NFE finish within 1.8 points of 128-NFE rollouts (43.4–46.0 vs. 45.2) with no trend in NFE, which is within evaluation noise.
  2. Cost. Rollout time falls 16× from 128 to 8 NFE. Optimization is a fixed ≈12.8 h, so total training time falls 2.6×, and time to 44% accuracy falls 3.2× (31.9 h to 9.9 h).
  3. Limit. Below 8 NFE training is unstable: the 4-, 2- and 1-NFE runs diverge after 4,400–6,000 updates.

On the 50.8% in Appendix E.6: that number came from an earlier recipe (h = 2, G = 8, a larger prompt pool) and was read on the 512-problem subset at evaluation NFE 32, at that arm's own peak. It is not comparable with the numbers above. We will replace Table 2 and Appendix E.6 with this study so the paper uses one recipe throughout.

We will revise the abstract and Section 4.2 to state the claim these results support: TGM-Map learns from rollouts that are up to 16× cheaper to generate (8 vs. 128 NFE) with final accuracy within noise, which reduces total training time by 2.6×; rollouts below 8 NFE are unstable.

If this addresses your concerns, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Follow up on the rebuttal

Hi,

Please let us know if our response answered the questions. If that addresses your concerns, we would be grateful if you would consider raising your score.

Thanks for running these additional experiments. The matched-budget comparisons and longer runs with total training times address my main concerns. Please include these results and the more qualified accuracy claim in the revision. This strengthens my support for acceptance. I have no further questions.

We will include these additional experiments in our final version and thank the reviewer for their feedback.

Comments on the paper

2 comments

Dear Reviewers,

The author-reviewer discussion phase is ending soon. If you have not done so yet, please read the author rebuttals and respond.

Please recall that, as a reviewer, you have agreed to engage in discussion with the authors to clarify and improve the paper. Failure to do so may result in penalties on your own submissions, as per the official ICLR policy. Given the high volume of submissions, your participation is crucial.

Thank you for your time and help.

Best regards,

Your AC

AuthorsOpenReview ↗

Summary of contributions

We thank all reviewers for their constructive feedback. All three reviewers now support acceptance. Below we summarize the new experiments and the revisions we will make.

1. Positioning (bVdy, 36So). The reward-tilted optimum and its solution by weighted maximum likelihood are classical (RWR/AWR), and we will present them as such. We now position the contribution as RL for diffusion and flow models, which generate by iterative refinement and lack tractable likelihoods. This includes the first RL results on uniform-noise DLMs and flow-map language models, and the first method designed for Gaussian DLMs. We will add a table separating what is new beyond TM/DTM: the identity stated at the level of generator matching, a unified control-variate family with a closed-form bias, advantage centering, and the flow-map extension.

2. Ablation ladder: what makes TGM work (bVdy, 36So). On Duo GSM8K, with the same rollouts, budget, and decoder at every rung:

MethodCenteredControl variateAccuracy (%)
Duo (base)––17.0
RAFT––20.5
RWR-stylenono17.7
AWR-styleyesno14.7
TM/DTM settingnoyes23.7
TGM (ours)yesyes37.2

The control variate prevents collapse. Without it, accuracy ends at or below the base, and TGM-Map falls to about 5% on GSM8K. Advantage centering then roughly triples the gain over the TM/DTM setting (+20.2 vs. +6.7 points).

3. Fair comparison with GRPO (yuk7). At matched update budgets (1,200 steps), TGM keeps its lead over Flow-GRPO on all FLM Sudoku tiers. Removing GRPO's KL penalty to the pretrained model (β = 0) does not close the gap: TGM leads the better Flow-GRPO variant by 8.8 / 2.6 / 3.5 points. We will update Table 1 with matched-budget numbers and the β = 0 baselines.

4. Few-step rollouts after full training (yuk7, 36So). We reran the rollout-NFE study for 10,000 updates (12.5× longer). Rollouts at 8–64 NFE reach final accuracy within evaluation noise of 128-NFE rollouts (43.4–46.0% vs. 45.2%). Total training time falls 2.6×, and time to 44% accuracy falls 3.2×. Rollouts below 8 NFE are unstable. We will replace Table 2 and revise the abstract to state this qualified claim.

5. Control-variate bias (36So). The net effect of the control variate, bias included, is strongly positive at every rollout NFE (+36 to +40 points). Runs below 8 NFE diverge, which is consistent with where Eq. (19) predicts the largest bias. We will note this as a limitation.

6. Writing (bVdy). We fixed the undefined control-variate coefficient c after Proposition 2.

All new results will be included in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.