- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces a fine-tuning strategy for pretrained few-step generative flow maps. It combines direct optimization of differentiable rewards with integrated KL or Fisher divergence regularization. By using a single flow-map model to represent both the generator and intermediate-time scores, the authors aim to preserve any-step generation while avoiding an additional score network.
Critical strengths
-
The proposed framework is mathematically motivated and combines few-step reward optimization with distributional regularization.
-
The experiments cover multiple image-generation settings and include useful ablations of different regularization schemes.
-
The single-model formulation is conceptually appealing and reduces memory consumption relative to the corresponding two-model approach.
Critical weaknesses
My main concerns are the following:
-
Limited methodological novelty: The main components of the proposed approach, including direct reward differentiation, integrated KL/Fisher regularization, and flow-map consistency objectives, have been introduced in previous work. The main contribution appears to be their combination into a single trainable flow map. However, the advantages of this combination are not convincingly established. The related-work discussion is largely relegated to the appendix, making it difficult to identify the precise novelty.
-
Limited improvements over baselines: The proposed methods are competitive but do not consistently outperform existing approaches in Figures 3 and 4. In particular, FAV achieves better diversity, while the two-model approach obtains comparable reward transfer. The single-model implementation also incurs greater training time than its two-model counterpart, weakening the computational efficiency argument.
Furthermore, I have the following reservations:
-
The baseline implementations differ in training budgets, batch sizes, and other important settings, making direct comparisons challenging. More carefully controlled evaluations would strengthen the empirical claims.
-
The authors repeatedly present plateauing reward curves as desirable evidence of successful regularization. While this can indicate convergence, it does not establish that the achieved reward is optimal or that the model retains sufficient diversity. A more systematic characterization of the reward–regularization tradeoff is needed.
-
The approach requires differentiable reward functions, limiting its applicability to common settings involving black-box simulators, experimental feedback, or non-differentiable evaluation metrics.
-
Although protein generation is presented as an application, its experimental results are entirely deferred to the appendix.
Questions for authors
-
What is the central advantage of the proposed single-model formulation over existing two-model regularization approaches, given the similar generative quality and increased training time?
-
How does the proposed approach compare against baselines under matched computational budgets and at comparable reward and diversity levels?
I would also reconsider my score if the authors can demonstrate and discuss the clear distinction of this work to prior literature better.
7 comments
Response to Reviewer tb3o
Performance relative to baselines
The reviewer is correct that our results show that our approach is 'competitive' but does not conclusively outperform some of the methods we compare against. We would like to emphasise that we do not claim to beat all the methods. Rather, we aim to demonstrate competitive performance with strong existing methods while also obtaining a flow map, which the other strongly-performing methods (the two-model version, and FAV) do not do.
Generally, most of the methods performed well, and it's difficult to conclusively claim that one is 'the best'. For a practitioner, the best choice would come down to many factors that are generally determined by the task at hand. Of the methods we looked at, here would be our summary of the pros and cons of each of the few-step regularisations (whether these are in fact a pro or con will depend on the particular use-case):
| Method | Pros | Cons |
|---|---|---|
| Direct optimisation | • Prone to reward hacking • Doesn't give a flow map | |
| VFM | • (Mainly designed for inverse problems) | • Prone to reward hacking |
| FAV | • Effective regularisation • Performed best on diversity | • Requires a choice of kernel in a suitable latent space • Doesn't give a flow map |
| Two-model version | • Effective regularisation | • Requires two sets of optimised parameters • Doesn't give a flow map |
| Ours | • Effective regularisation • Gives a flow map | • Slightly slower to train than the two-model variant |
Flow-map property: As discussed in the above sections, one of the key benefits of our approach is that it provides a fine-tuned flow map. To quantitatively demonstrate how our approach obtains a flow-map while the other approaches do not, we have added new results in Figure 4 and along the bottom rows of Figures 14-16. Here, we plot the MSE between the 1-step prediction and the final K-step generation. For a perfect flow map, these generations would be the same. We see that for our method, this error remains pretty much constant as the level of reward alignment increases, and remains similar to that of the base model. In comparison, for the other methods there is a large increase. This indicates that the flow-map property of the model is being much better preserved by our method while it is lost for the other methods
Response to Reviewer tb3o
Characterisation of reward-regularisation trade-off
We respectfully disagree with the reviewer's comment that 'a more systematic characterization of the reward–regularization tradeoff is needed'. Based on the reviewer's comments, we believe they may have misunderstood the purpose of our plots and the results that they show. We believe that we already provide a strong characterization of the reward–regularization tradeoff that in fact goes beyond many existing analyses in the surrounding literature.
The reviewer's comments refer to the plateauing reward curves not conveying the reward-regularization trade-off. The reviewer is correct that the plateauing reward curves do not say much about the behaviour of the optimised model, and we would like to clarify that this is not the aim of these reward curve plots. Rather, the aim of the plateuaing reward curves is to convey that our regularisation mechanism is effective at preventing movement away from the base model and can therefore avoid reward hacking - we note that this is not always the case for all reward-alignment approaches, for example VFM (one of the methods to which we compare) does not plateau and thus will reward hack if trained for longer.
The main characterisation of the reward-regularisation trade-off is in our reward-transfer plots (such as Figures 3 and 4) rather than the plateauing reward curves. As we discuss in the appendix, in these plots we show how the values of the other reward functions change as the the optimized reward value increases (which corresponds to increasing the reward strength). This reporting convention is chosen because it helps to assess generation quaility in a manner independent of the optimised reward - a ‘good’ sample should score highly in the other rewards too, whereas ‘reward-hacked’ images are more likely to score poorly for the alternate reward functions. In our opinion this is a strong way of characterising the reward-regularisation trade-off; indeed, this matches the reporting convention advocated for in [1], and was mentioned as a strength of our paper by Reviewer 1Kp7.
Diversity: With regards to diversity, we already provide a quantitative assessment of how the diversity of the generations changes as the attained reward increases (due to using higher reward strength). We use the DreamSim diversity metric, which is widely used in the literature as a quantitative metric of diversity (see for example [1] and [2]).
[1] Zhang et al 2024, Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases
[1] Domingo-Enrich et al 2024, Adjoint Matching
[2] Potaptchik et al 2026, Tilt Matching for Scalable Sampling and Fine-Tuning
Response to Reviewer tb3o
We thank the reviewer for their comments, and we address their points below. We have uploaded an updated paper with subsequent changes highlighted in red.
Relation to prior literature
We first would like to clarify the positioning and motivation of our work relative to prior literature, as this appears to be one of the reviewer's main concerns.
Few-step models: Recently there has been a large amount of works looking at reward alignment of few-step generative models, which have proposed a variety of different mechanisms for doing so. For example, several (like FAV) use regularisation mechanisms that compare representations in a suitable latent space, and other learn separate value functions, to name a few.
Of these, the most related to our proposed approach are works like [1] and [2], which similarly use integrated divergences along the probability path. Indeed, these works have strongly inspired our approach. We remark that they train two different models: one for learning the score of the current endpoint distribution, and one for the reward-aligned generator itself. They also do not learn a flow map which is the main aim of our work, as we now explain.
Flow maps: Despite the large number of few-step reward-alignment methods that are now being developed, there are very few methods that look at fine-tuning a flow map specifically. Obtaining a reward-aligned flow map has several benefits. For example, if optimised well then a flow map can be used for any-step generation. Secondly, flow maps are often required for certain downstream tasks, in particular for inference-time steering (such as in [3], [4], [5]). They are becoming an increasingly popular paradigm in the generative modelling and scientific literatures, including recent applications for protein and text generation. Findings ways to reward-align flow maps in particular is therefore an important task, and it is this that we aim to target in our work.
Our Contribution: The main contribution of our approach is to recognise that the existing integrated regularisation mechanisms are an excellent fit for reward-aligning flow maps. This is because a flow map provides both the few-step generator and the 'diagonal' flow matching drift all in the same model, which is all that is required for the integrated regularisation mechanism. Despite its simplicity, this has so far been overlooked as a mechanism for reward-aligning flow maps, which is what we attempt to rectify in this work.
Our work therefore lies at the intersection of the two literatures that we discuss above. Compared to other methods that give a reward-aligned flow map, we use the integrated regularisation mechanism that is already proven to work well in practice, and is in our opinion a simple and very natural approach (it uses just the output of the flow map, without any additional machinery e.g. requiring learning separate value functions or anything like that). Compared to the existing integrated regularisation reward-alignment works like [1] and [2], using a flow map means we require only a single model, and moreover we specifically obtain a flow-map at the end.
To make our contribution more clear, we have adjusted the paper's title to 'Reward-Aligning Flow Maps with Integrated Regularizers' (we initially thought that using 'Flow Maps' in the title might be too niche a term, but given their increasing prevalence in the literature we think this is no longer a concern and better represents our contribution).
Novelty of results: The reviewer is correct that our proposed approach is mostly based on existing tools in the literature (we want to emphasise that we don't claim novelty regarding these results, which is hopefully clear from the paper). As noted above, we actually view this as one of the key strengths of our approach, as it relies on well-established and reliable regularisation mechanisms that have been proven over several years to work well in practice. For this reason and also because of its simplicity, we feel that reward-aligning flow maps in this way is very natural.
This was mainly due to space constraints in the original submission. We have added a paragraph to the updated version in the Discussion section explaining the main related works, which we hope helps to place our work more clearly in the context of existing approaches.
[1] Luo 2025, Diff-instruct++: Training one-step text-to-image generator model to align with human preferences
[2] Luo et al 2025, David and goliath: Small one-step model beats large diffusion with score post-training
[3] Sabour et al 2025, Test-time scaling of diffusions with flow maps
[4] Potaptchik et al 2026, Meta Flow Maps enable scalable reward alignment
[5] Holderrieth et al 2026, Diamond Maps: Efficient Reward Alignment via Stochastic Flow Maps