- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper addresses an important and timely problem: enabling reinforcement-learning post-training for deterministic few-step flow-map generators. The proposed anchored stochastic composition provides a practical way to construct stochastic policies for pretrained MeanFlow and consistency models without changing their underlying parameterizations. The mathematical motivation is interesting, and the substantial improvements in OCR, PickScore, and GenEval demonstrate the method's practical effectiveness. I lean toward a positive evaluation, but I have several questions about the theoretical and methodological justification.
Critical strengths
-
This paper solves an important challenge for flow-map model-based RL. The deterministic flow map lacks the explicit random term found in an SDE, which hinders direct RL implementation. This paper proposes a way to convert a deterministic flow map back into a stochastic path, thereby enabling RL.
-
The experimental results clearly demonstrate the validity of the proposed method.
Critical weaknesses
Weaknesses:
-
Marginal preservation for learned long-range transitions. The theoretical guarantees rely on exact flow maps and exact conditional resampling, whereas practical implementation uses approximate learned maps and, for the local anchor, a discretized reverse transition. Can the authors quantify how well the constructed stochastic policies preserve the intended probability marginals, especially for large transport intervals and after RL adaptation?
-
Interpretation of normalized likelihood ratios. Appendix A.3 states that the implementation averages log probabilities over latent dimensions, yielding a normalized ratio equal to the -th root of the joint policy likelihood ratio. This differs from the importance ratio used in the theoretical GRPO objective. Could the authors clarify the optimization interpretation of this normalization and investigate its influence on training stability and performance?
-
Comparison with adapted velocity-based RL. Although the comparisons with SFT and DiffusionNFT are useful, an adapted Flow-GRPO baseline using the pretrained MeanFlow model's equal-time velocity predictions would better establish the advantage of directly stochasticizing long-range flow maps. Reporting matched sampling costs and training overhead would further strengthen the practical contribution.
Questions for authors
Please refer to the weaknesses.
To sum up, I think this paper is interesting and tends to be a weak accept. During the rebuttal, the author should clarify the weaknesses.
2 comments
Response to Reviewer ma2p (2/2)
W2: Interpretation of the normalized likelihood ratio and its influence.
We follow Flow-GRPO (Liu et al., 2025) here, whose public implementation uses the same normalization. It averages the log-probability over the latent dimensions, so the ratio is , the -th root of the joint ratio , which is the geometric mean of the per-dimension ratios. This normalization has two effects.
- The policy gradient keeps its direction. At the rollout policy, for every sample, so the gradient of the normalized objective equals the gradient of the joint-likelihood objective scaled by . Each round takes two gradient updates on the same rollouts. In the second update, an unclipped sample contributes : its importance weight is instead of , which stays close to 1.
- One clipping threshold can be used for any latent size. Clipping to bounds the joint log-ratio by about , that is, per dimension on average. With , the joint bound is ±6.6 nats for FLUX latents () and ±0.41 nats for ImageNet (). We train both models with this , although their latents differ 16-fold in size.
Influence on training. To measure the effect, we retrained the ImageNet model with the joint ratio in place of the normalized ratio , with everything else unchanged:
| Ratio | ε | FID at round 100 | Final FID |
|---|---|---|---|
| Normalized (ours) | 18.45 | 16.72 | |
| Joint | 23.09 | 17.93 | |
| Joint | 0.4 (≈ dε) | 18.44 | 18.24 |
With the joint ratio and the same ε, the clipping is about times tighter. In the second update 99.7% of the samples are clipped, and convergence is much slower. Enlarging ε by about gives the same clipped fraction as the normalized ratio (19%) and the same early progress, as the clipping bound above predicts. Its importance weights are still rather than , however, and its final FID is higher. The normalized ratio therefore converges at least as fast as the joint ratio and reaches the best FID, and its ε does not have to be retuned when the latent size changes.
We have extended App. A.3 with this derivation and comparison.
W3: An adapted Flow-GRPO baseline on the equal-time velocity, with matched sampling cost and training overhead.
Thank you for proposing this baseline. We ran Flow-GRPO on the equal-time velocity of the same FLUX.1-lite MeanFlow network, using Flow-GRPO's ten-step SDE rollouts, the same LoRA configuration and the same rollout budget (1,152 rollouts per round, 400 rounds). We also compare with MeanFlowNFT, an RL method designed for few-step flow maps. It is concurrent work that appeared on arXiv after our method was publicly released, so the original submission did not include it. We have now added it with its official hyperparameters and the same rollout budget. All methods are evaluated with EMA weights and the same flow-map sampler at K=4, so inference cost is matched. Training cost is reported as wall-clock time per round on 8×H100, which covers rollouts, reward computation and updates:
| Method | Eval K | Time per round | OCR |
|---|---|---|---|
| Base | 4 | — | 0.349 |
| Flow-GRPO (equal-time velocity) | 4 | 441 s | 0.408 |
| MeanFlowNFT | 4 | 309 s | 0.713 |
| Flow-Map GRPO (training K=2) | 4 | 154 s | 0.762 |
| Flow-Map GRPO (training K=3) | 4 | 221 s | 0.770 |
| Flow-Map GRPO (training K=4) | 4 | 247 s | 0.850 |
At every training K, Flow-Map GRPO reaches a higher OCR than both baselines in less time per round. Flow-GRPO defines its RL objective on the instantaneous velocity, while the K=4 sampler uses the long-range map. Flow-Map GRPO optimizes this map directly.
We have also uploaded a revised version of the paper, with all changes highlighted in red.
Response to Reviewer ma2p (1/2)
We sincerely thank you for your positive assessment and your questions. In summary:
W1: Can the authors quantify how well the stochastic policies preserve the marginals with learned maps, for large intervals and after RL?
Directly measuring the intermediate marginals of a learned high-dimensional model is impractical, so we measure their effect on the generated distribution: if a stochastic policy preserves the marginals, its rollouts should match deterministic sampling of the same network. We have added this comparison to App. C.6 (ImageNet, 5K images, K=4 stochastic transitions per rollout):
Before RL, both anchors reach an FID within 0.8 of deterministic sampling. After RL, local-anchor rollouts reach an FID about 1.5 lower than deterministic sampling, with matching precision and higher recall. This is consistent with RL optimizing the stochastic policy itself.
The table above is the case of an accurate flow map, where the choice of anchor has little effect on the final FID: after RL, the endpoint, midpoint and local anchors reach nearly the same FID (16.68 ± 0.32, 16.51 ± 0.18 and 16.65 ± 0.12 over 3 seeds). The effect of a large transport interval appears when the learned map is less accurate over long ranges, as for FLUX.1-lite MeanFlow.
With a learned map, the quality of the stochastic policy thus depends on how far the anchor's prediction has to reach. The local anchor uses the equal-time velocity at the next state instead of a further long-range prediction, which is why we use it.