- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
In this paper, the authors introduce Tilted Generator Matching, single RL objective for every model expressible in the generator matching framework, such as flow matching, score-based diffusion, and discrete diffusion. The authors use the idea of reward tilted distribution: KL-regularized RL is solved exactly by the model’s own pretraining loss, tilted by exponentiated reward. They study TGM on diffusion and flow language models (DLMs), including masked, uniform-noise diffusion discrete diffusion language models, and continuous Gaussian DLMs.
Critical strengths
-
The paper is overall well written and easy to follow.
-
It first conduct RL experiments on uniform, Gaussian, and flow-map language models. The experiments on Sudoku and GSM8k shows the effectiveness of the method. Table 3 and Table 4 provides a comprehensive survey on diffusion/flow related models and their RL post-training methods.
Critical weaknesses
-
My main concern is that the paper offers less novelty than its presentation suggests. The optimal solution of KL regularized RL and learning by weighted maximum likelihood can be traced back to reward weighted regression, the annealed tilt can be traced back to sequential Monte Carlo. The authors state that the one contribution is to generalize tilted matching to generator matching framework. This can be further generalized to generative models that satisfying two properties: a) Fisher consistency, which is with sufficient expressive capacity, the model is capable of recovering data distribution; b) the objective function is linear to data distribution. However, such generalization offers little new insights to us. In addition, the experiments only cover masked, uniform, Gaussian, and flow-map diffusion language models. If the paper were positioned as the first RL method shown to work on these model families, I think that would be a fair and useful contribution. For example, an autoregressive model can be viewed as masked diffusion with a fixed left-to-right order, it also fits into the generator matching framework. In that case, TGM reduces to reward-weighted next-token cross-entropy, i.e., RWR/AWR, however, the paper does not provide the experiments on autoregressive models.
-
Small writing problem: Immediately after Proposition 2, the paper states that TGM recovers TM and DTM with c=0, without introducing the control-variate coefficient c.
Questions for authors
I have no other questions, please see the weakness.
3 comments
The authors address most of my concerns, so I raise my score to 3. However, to isolate TGM’s contribution beyond TM/DTM, it would help to add a rung with exp(h·reward) weights plus the control variate (i.e., the TM/DTM setting), so the effect of advantage centering is visible separately.
Thank you for raising your score and for suggesting this ablation. We ran the TM/DTM rung you proposed: exp(h·reward) weights plus the control variate, with no advantage centering. Rollouts, update budget, and decoder are the same as every other rung. Results on Duo GSM8K (full 1,319-problem test set):
| Rung | Method | Centered | Control variate | Duo GSM8K (%) | Change from base |
|---|---|---|---|---|---|
| 0 | Duo (base) | – | – | 17.0 | – |
| 1 | RAFT | – | – | 20.5 | +3.5 |
| 2 | RWR-style: exp(h·reward) | no | no | 17.7 | +0.7 |
| 3 | AWR-style: exp(h·advantage) | yes | no | 14.7 | −2.3 |
| 4 | TM/DTM setting: exp(h·reward) + control variate (new) | no | yes | 23.7 | +6.7 |
| 5 | TGM (ours): exp(h·advantage) + control variate | yes | yes | 37.2 | +20.2 |
The control variate is necessary, and centering roughly triples the gain. Neither component works alone: without the control variate, both weightings end at or near the base (rungs 2–3). With it, uncentered weights give +6.7 points and centered weights give +20.2.
Training dynamics (in-loop evaluation on the first 512 test problems):
| Phase | 0 | 10 | 20 | 30 | 40 | 50 | 60 |
|---|---|---|---|---|---|---|---|
| TM/DTM setting (%) | 18.6 | 14.4 | 16.0 | 21.5 | 20.9 | 23.4 | 24.2 |
| TGM (%) | 18.8 | 22.3 | 21.7 | 28.9 | 32.8 | 35.7 | 38.3 |
The TM/DTM setting first drops below the base, recovers to it only around phase 22, and then gains about 6 points by phase 60. TGM improves from phase 6 onward and plateaus at 37–39%. Both runs are stable throughout. The TM/DTM run was still improving slightly at phase 60, so a longer run might narrow the gap somewhat, but not plausibly close a 14-point difference.
Mechanism. With uncentered weights, every rollout that solves the problem gets a large positive weight and none gets a negative one. After the control-variate shift, the weights average about 2.3, so the update mostly reinforces what the model already produces. With centering, the weights average about 0.25 and below-average rollouts receive negative weight. The model is pushed toward the better rollouts within each prompt, not just toward any correct one.
We will add this rung to the ladder in the paper and use it to state TGM's contribution over TM/DTM precisely: the generator-matching formulation, advantage centering, and the flow-map extension.
We thank Reviewer bVdy for the direct and constructive review. We agree with repositioning the paper as you suggest, and have run the experiments you suggest. If the repositioning and new experiments address your concern, we would be grateful if you would consider raising your score.
1. Lineage and positioning: we agree
The tilted optimum and its solution by weighted maximum likelihood are classical (RWR [1], AWR [2]), and the annealed tilt goes back to sequential Monte Carlo [11]. Your generalization is correct: Prop. 2 holds for any Fisher-consistent objective that is linear in the data distribution, and we will state this in a remark after Prop. 2. We will also revise the abstract and introduction so that the identity is presented as classical, and the contribution is positioned as you propose: RL for diffusion and flow models, which generate by iterative refinement and have no tractable likelihood, with the first results on uniform-noise and flow-map language models.
RAFT is the zero-temperature limit of RWR: RWR weights each rollout in a group by exp(reward/β), and as β → 0 all of the weight goes to the highest-reward rollout, which is exactly the one RAFT trains on. It is therefore the closest baseline to TGM, and Table 1 already includes it for every family. TM and DTM introduced the control variate for Gaussian flows and masked diffusion. TGM states it for any generator-matching loss, adds advantage centering, and extends it to flow maps (Prop. 3, which lies outside your conditions because the consistency loss depends on the model).
2. Is RWR enough on a diffusion model? A new ablation
A ladder on Duo GSM8K, with the same rollouts, budget and decoder at every rung (full test set):
So the identity is RWR, but on these models the identity alone is not enough. We will add the ladder to the paper.
3. The undefined c. Thank you. c is the control-variate coefficient, introduced only in Section 3.2. We now remove it from the text.
[1] Peters & Schaal. Reward-weighted regression. ICML 2007. [2] Peng et al. Advantage-weighted regression. arXiv:1910.00177. [3] Shao et al. DeepSeekMath (GRPO). arXiv:2402.03300. [4] Zhao et al. d1 (diffu-GRPO). arXiv:2504.12216. [5] Liu et al. Flow-GRPO. NeurIPS 2025. [6] Dong et al. RAFT. TMLR 2023. [7] Black et al. DDPO. ICLR 2024. [8] Fan et al. Online reward-weighted flow matching (ORW-CFM-W2). ICLR 2025. [9] Potaptchik et al. Tilt matching. ICML 2026. [10] Chen et al. Discrete tilt matching. arXiv:2604.18739. [11] Del Moral et al. Sequential Monte Carlo samplers. JRSS-B 2006.