acceptodds
Under review as a conference paper at ICLR 2027

Reward-Aligning Few-Step Flow Models with Integrated Regularizers

Abstract

Flow-based generative models are increasingly adapted to downstream rewards, preferences, and design objectives. Most existing approaches adapt the time-dependent drift of the underlying continuous flow. These continuous-time dynamics enable effective and probabilistically-principled reward alignment, but incur the cost of expensive simulations and require the challenging propagation of terminal reward gradients backwards through the trajectories. Recent accelerated one- and few-step samplers raise the prospect of improved fine-tuning efficiency, however such approaches side-step the continuous dynamics that are useful for effective and stable regularization. In this work, we propose a method for fine-tuning a flow map that preserves the advantages of both regimes, while also retaining the ability to perform any-step generation. Our approach directly optimizes the few-step generations using gradients of the reward function, while regularizing using a family of integrated divergences along the continuous-time probability flow induced by the model. Experiments on image and protein generation show that the proposed approach combines the efficiency of few-step generation with the stability of continuous-time regularization along the probability flow path.

Primary area:
generative models
Keywords:
Flow maps, Reward fine-tuning, Flow-based generative models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tb3oOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a fine-tuning strategy for pretrained few-step generative flow maps. It combines direct optimization of differentiable rewards with integrated KL or Fisher divergence regularization. By using a single flow-map model to represent both the generator and intermediate-time scores, the authors aim to preserve any-step generation while avoiding an additional score network.

Critical strengths
  • The proposed framework is mathematically motivated and combines few-step reward optimization with distributional regularization.

  • The experiments cover multiple image-generation settings and include useful ablations of different regularization schemes.

  • The single-model formulation is conceptually appealing and reduces memory consumption relative to the corresponding two-model approach.

Critical weaknesses

My main concerns are the following:

  • Limited methodological novelty: The main components of the proposed approach, including direct reward differentiation, integrated KL/Fisher regularization, and flow-map consistency objectives, have been introduced in previous work. The main contribution appears to be their combination into a single trainable flow map. However, the advantages of this combination are not convincingly established. The related-work discussion is largely relegated to the appendix, making it difficult to identify the precise novelty.

  • Limited improvements over baselines: The proposed methods are competitive but do not consistently outperform existing approaches in Figures 3 and 4. In particular, FAV achieves better diversity, while the two-model approach obtains comparable reward transfer. The single-model implementation also incurs greater training time than its two-model counterpart, weakening the computational efficiency argument.

Furthermore, I have the following reservations:

  • The baseline implementations differ in training budgets, batch sizes, and other important settings, making direct comparisons challenging. More carefully controlled evaluations would strengthen the empirical claims.

  • The authors repeatedly present plateauing reward curves as desirable evidence of successful regularization. While this can indicate convergence, it does not establish that the achieved reward is optimal or that the model retains sufficient diversity. A more systematic characterization of the reward–regularization tradeoff is needed.

  • The approach requires differentiable reward functions, limiting its applicability to common settings involving black-box simulators, experimental feedback, or non-differentiable evaluation metrics.

  • Although protein generation is presented as an application, its experimental results are entirely deferred to the appendix.

Questions for authors
  • What is the central advantage of the proposed single-model formulation over existing two-model regularization approaches, given the similar generative quality and increased training time?

  • How does the proposed approach compare against baselines under matched computational budgets and at comparable reward and diversity levels?

I would also reconsider my score if the authors can demonstrate and discuss the clear distinction of this work to prior literature better.

7 comments
AuthorsOpenReview ↗

Response to Reviewer tb3o

We thank the reviewer for their comments, and we address their points below. We have uploaded an updated paper with subsequent changes highlighted in red.

Relation to prior literature

We first would like to clarify the positioning and motivation of our work relative to prior literature, as this appears to be one of the reviewer's main concerns.

Few-step models: Recently there has been a large amount of works looking at reward alignment of few-step generative models, which have proposed a variety of different mechanisms for doing so. For example, several (like FAV) use regularisation mechanisms that compare representations in a suitable latent space, and other learn separate value functions, to name a few.

Of these, the most related to our proposed approach are works like [1] and [2], which similarly use integrated divergences along the probability path. Indeed, these works have strongly inspired our approach. We remark that they train two different models: one for learning the score of the current endpoint distribution, and one for the reward-aligned generator itself. They also do not learn a flow map which is the main aim of our work, as we now explain.

Flow maps: Despite the large number of few-step reward-alignment methods that are now being developed, there are very few methods that look at fine-tuning a flow map specifically. Obtaining a reward-aligned flow map has several benefits. For example, if optimised well then a flow map can be used for any-step generation. Secondly, flow maps are often required for certain downstream tasks, in particular for inference-time steering (such as in [3], [4], [5]). They are becoming an increasingly popular paradigm in the generative modelling and scientific literatures, including recent applications for protein and text generation. Findings ways to reward-align flow maps in particular is therefore an important task, and it is this that we aim to target in our work.

Our Contribution: The main contribution of our approach is to recognise that the existing integrated regularisation mechanisms are an excellent fit for reward-aligning flow maps. This is because a flow map provides both the few-step generator and the 'diagonal' flow matching drift all in the same model, which is all that is required for the integrated regularisation mechanism. Despite its simplicity, this has so far been overlooked as a mechanism for reward-aligning flow maps, which is what we attempt to rectify in this work.

Our work therefore lies at the intersection of the two literatures that we discuss above. Compared to other methods that give a reward-aligned flow map, we use the integrated regularisation mechanism that is already proven to work well in practice, and is in our opinion a simple and very natural approach (it uses just the output of the flow map, without any additional machinery e.g. requiring learning separate value functions or anything like that). Compared to the existing integrated regularisation reward-alignment works like [1] and [2], using a flow map means we require only a single model, and moreover we specifically obtain a flow-map at the end.

To make our contribution more clear, we have adjusted the paper's title to 'Reward-Aligning Flow Maps with Integrated Regularizers' (we initially thought that using 'Flow Maps' in the title might be too niche a term, but given their increasing prevalence in the literature we think this is no longer a concern and better represents our contribution).

Novelty of results: The reviewer is correct that our proposed approach is mostly based on existing tools in the literature (we want to emphasise that we don't claim novelty regarding these results, which is hopefully clear from the paper). As noted above, we actually view this as one of the key strengths of our approach, as it relies on well-established and reliable regularisation mechanisms that have been proven over several years to work well in practice. For this reason and also because of its simplicity, we feel that reward-aligning flow maps in this way is very natural.

related work in the appendix

This was mainly due to space constraints in the original submission. We have added a paragraph to the updated version in the Discussion section explaining the main related works, which we hope helps to place our work more clearly in the context of existing approaches.

[1] Luo 2025, Diff-instruct++: Training one-step text-to-image generator model to align with human preferences

[2] Luo et al 2025, David and goliath: Small one-step model beats large diffusion with score post-training

[3] Sabour et al 2025, Test-time scaling of diffusions with flow maps

[4] Potaptchik et al 2026, Meta Flow Maps enable scalable reward alignment

[5] Holderrieth et al 2026, Diamond Maps: Efficient Reward Alignment via Stochastic Flow Maps

AuthorsOpenReview ↗

Response to Reviewer tb3o

Performance relative to baselines

The reviewer is correct that our results show that our approach is 'competitive' but does not conclusively outperform some of the methods we compare against. We would like to emphasise that we do not claim to beat all the methods. Rather, we aim to demonstrate competitive performance with strong existing methods while also obtaining a flow map, which the other strongly-performing methods (the two-model version, and FAV) do not do.

Generally, most of the methods performed well, and it's difficult to conclusively claim that one is 'the best'. For a practitioner, the best choice would come down to many factors that are generally determined by the task at hand. Of the methods we looked at, here would be our summary of the pros and cons of each of the few-step regularisations (whether these are in fact a pro or con will depend on the particular use-case):

MethodProsCons
Direct optimisation• Prone to reward hacking • Doesn't give a flow map
VFM• (Mainly designed for inverse problems)• Prone to reward hacking
FAV• Effective regularisation • Performed best on diversity• Requires a choice of kernel in a suitable latent space • Doesn't give a flow map
Two-model version• Effective regularisation• Requires two sets of optimised parameters • Doesn't give a flow map
Ours• Effective regularisation • Gives a flow map• Slightly slower to train than the two-model variant

Flow-map property: As discussed in the above sections, one of the key benefits of our approach is that it provides a fine-tuned flow map. To quantitatively demonstrate how our approach obtains a flow-map while the other approaches do not, we have added new results in Figure 4 and along the bottom rows of Figures 14-16. Here, we plot the MSE between the 1-step prediction and the final K-step generation. For a perfect flow map, these generations would be the same. We see that for our method, this error remains pretty much constant as the level of reward alignment increases, and remains similar to that of the base model. In comparison, for the other methods there is a large increase. This indicates that the flow-map property of the model is being much better preserved by our method while it is lost for the other methods

AuthorsOpenReview ↗

Response to Reviewer tb3o

Characterisation of reward-regularisation trade-off

We respectfully disagree with the reviewer's comment that 'a more systematic characterization of the reward–regularization tradeoff is needed'. Based on the reviewer's comments, we believe they may have misunderstood the purpose of our plots and the results that they show. We believe that we already provide a strong characterization of the reward–regularization tradeoff that in fact goes beyond many existing analyses in the surrounding literature.

The reviewer's comments refer to the plateauing reward curves not conveying the reward-regularization trade-off. The reviewer is correct that the plateauing reward curves do not say much about the behaviour of the optimised model, and we would like to clarify that this is not the aim of these reward curve plots. Rather, the aim of the plateuaing reward curves is to convey that our regularisation mechanism is effective at preventing movement away from the base model and can therefore avoid reward hacking - we note that this is not always the case for all reward-alignment approaches, for example VFM (one of the methods to which we compare) does not plateau and thus will reward hack if trained for longer.

The main characterisation of the reward-regularisation trade-off is in our reward-transfer plots (such as Figures 3 and 4) rather than the plateauing reward curves. As we discuss in the appendix, in these plots we show how the values of the other reward functions change as the the optimized reward value increases (which corresponds to increasing the reward strength). This reporting convention is chosen because it helps to assess generation quaility in a manner independent of the optimised reward - a ‘good’ sample should score highly in the other rewards too, whereas ‘reward-hacked’ images are more likely to score poorly for the alternate reward functions. In our opinion this is a strong way of characterising the reward-regularisation trade-off; indeed, this matches the reporting convention advocated for in [1], and was mentioned as a strength of our paper by Reviewer 1Kp7.

Diversity: With regards to diversity, we already provide a quantitative assessment of how the diversity of the generations changes as the attained reward increases (due to using higher reward strength). We use the DreamSim diversity metric, which is widely used in the literature as a quantitative metric of diversity (see for example [1] and [2]).

[1] Zhang et al 2024, Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases

[1] Domingo-Enrich et al 2024, Adjoint Matching

[2] Potaptchik et al 2026, Tilt Matching for Scalable Sampling and Fine-Tuning

Reviewer_brEWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies reward alignment of accelerated flow maps. It directly differentiates a terminal reward through one- or few-step samples, while regularizing the current model against the base model through integrated KL or Fisher divergences along a self-induced continuous probability path. A single flow map is jointly trained with self-induced flow-matching and semigroup-consistency losses, allowing its diagonal velocity to provide the intermediate score used by the regularizer. Experiments on ImageNet, a distilled FLUX model, and protein generation suggest that path-integrated regularization improves reward-transfer trade-offs relative to reward-only and several prior alignment baselines.

Critical strengths

S1: The paper identifies a meaningful practical gap between cheap direct reward differentiation for few-step samplers and the pathwise regularization available to continuous-time methods. The flow-map formulation is a natural way to connect these regimes.

S2: The single-model construction is technically appealing. Reusing the diagonal velocity of a coherent flow map to estimate the self-induced score avoids the auxiliary score model used in related integrated-divergence recipes, while retaining potential any-step functionality.

S3: The paper presents a useful breadth of evidence: ImageNet reward-transfer curves, a FLUX experiment, multiple divergence variants and path weights, and an extension to Riemannian protein flow maps. The limitations concerning differentiable rewards and score lag are stated candidly.

Critical weaknesses

W1: The empirical novelty over the closest literature is insufficiently isolated. Integrated KL/Fisher regularization of self-induced paths and few-step reward alignment are already present in the cited distillation/alignment work. The central claimed increment is the single coherent flow-map parameterization, yet the main comparisons do not cleanly control for (i) the same integrated regularizer with a separately trained score model and (ii) the closest flow-map reward-guidance method under matched rewards, steps, and compute.

W2: The evidence relies almost entirely on learned automatic rewards (HPSv2, ImageReward, PickScore, and GenEval). Since direct optimization can exploit reward models, transfer among correlated automated evaluators is not sufficient evidence of perceptual quality or preservation of the base distribution. No human preference study, adversarial reward-overoptimization check, or broad quality/diversity metric accompanies the headline gains.

W3: The paper makes stability/efficiency claims without an adequately controlled outcome-level comparison. It gives training-step, memory, and partial wall-clock information, but the extra self-flow-matching and semigroup losses have nontrivial cost. A matched quality/reward-versus-wall-clock or versus total FLOPs comparison with Adjoint Matching, FAV, and the two-model integrated baseline is needed, particularly for the large FLUX experiment.

Questions for authors

Q1: Can the authors compare against the same integrated KL/Fisher objective with a separately trained intermediate score model, and against the closest flow-map reward-guidance baseline, using matched reward scales, sampling steps, and tuning budget?

Q2: Can the authors add human preference evaluation or a reward over-optimization stress test, and quantify terminal image quality/diversity independently of the optimized reward models?

Q3: Can the authors report a diagnostic of self-induced score/flow-map consistency throughout finetuning, and explain what safeguards prevent score lag from invalidating the integrated regularizer?

Q4: For each principal method, what are total training FLOPs/wall-clock time and final reward-quality trade-offs at matched compute? Does the claimed efficiency remain after including the flow-matching and semigroup losses?

4 comments
AuthorsOpenReview ↗

Response to Reviewer brEW

We thank the reviewer for their positive review, and we address their comments below. We have uploaded an updated paper with subsequent changes highlighted in red.

W1: Empirical novelty over the closest literature

Comparison with separately-trained score network: The reviewer comments that we 'do not cleanly control for (i) the same integrated regularizer with a separetely trained score model'. We would like to clarify that we do precisely this in the comparison with the 'two-model' version that we compare extensively to. We used the exact matched training setups, and the only change is using two models rather than one. We believe that this comparison is already what you are requesting here so we hope this helps to allay this concern.

Matched experimental settings: We would like to clarify that where possible in the current experiments, we already chose the hyperparameters to make the methods comparable as possible. We appreciate this may not have been sufficiently clear in the initial submission, so we have added additional explanations to Appendix B.1.1 to give more details, and we hope this alleviates the concerns regarding the chosen experimental setups. In particular, the strongest performing methods (ours, the two-model version, and FAV) all use comparable experimental setups.

We have also added an additional compute-matched analysis of training speed between these methods at the end of Appendix B.1.3 (see Figure 21). Here, we see how that our method is slightly slower than the two-model version (which agrees with the results we already provide in Table 2), and is very similar to FAV. Overall, these results show that these methods perform similarly at matched compute.

W2: Choice of rewards

Reward over-optimisation: We agree that direct optimisation can exploit the specific reward model, hence our decision to evaluate in our experiments specifically how the improvements in the reward being optimised transfers to other reward functions. In our opinion, this already goes substantially beyond many other analyses in similar works in attempting to disentangle the effect of reward exploitation. For this reason, we believe that this already does a good job of testing reward over-optimisation by assessing independently of the reward model being optimised, and indeed this was noted as a strength of our work by reviewer 1Kp7

We appreciate that the concern that these different rewards could potentially have some degree of correlation, but we have not seen this established as a concern in the literature before. We note that our reported results follow the cross-reward convention advocated for in [1] (which is designed to mitigate the problem you refer to). We therefore feel that our existing analysis is already a very strong way of addressing the reward exploitation concern of the reviewer. We also note the choices of reward functions that we currently use are very in-keeping with those used in the literature (for example in [2], [3]).

[1] Zhang et al 2024, Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases

[2] Potaptchik et al 2025, Tilt Matching for Scalable Sampling and Fine-Tuning

[3] Mammadov et al 2026, Variational Flow Maps

'human preference study': We agree that this would be useful but this is unfortunately beyond the scope of our resources.

'diversity': Regarding diversity, we already report diversity metrics in Appendix B.1.3 (Figure 16), which indicates that our diversity decays fairly in-line with other methods as the level of reward optimisation increases (we note that FAV was the strongest, but this relies on a very different regularisation done in the latent space of a reward which may give an advantage here).

AuthorsOpenReview ↗

Response to Reviewer brEW

W3: stability and efficiency claims

We believe that the reviewer may have slightly misunderstood the efficiency and stability claims that we make, so we clarify them below.

Efficiency: We would like to clarify that the efficiency benefits that we refer to in the paper are about the efficiency gains of reward-aligning few-step models in comparison to continuous flow models. This is a benefit shared by all few-step reward-alignment methods and is not specific to our method. We would like to emphasise that we do not make any specific efficiency claim relative to other similar few-step reward alignment methods, as they all inherit this benefit (and will in many cases be largely similar).

We agree that the extra semigroup losses (or whichever flow map consistency loss is used) adds additional network evaluations compared to the equivalent two step model, which we have discussed in Appendix B.1.3.

Stability: We would also like to clarify that the stability claim that we make in the paper is not about performance relative to other methods, but is rather the observation that using regularisation integrated over the (possibly weighted) continuous time is more stable than using regularisation defined on the endpoint distribution only. This is evidenced by the weighting function w(t)=1−ttw(t) = \tfrac{1-t}{t} being much more stable than the constant schedule w(t)=1w(t)=1, and we provide a more in-depth analysis of this in the Appendix (Figure 18). This is a known benefit of integrated regularisation versus endpoint-only regularisation (e.g. from the distillation literature [1], and also two-model reward-alignment methods [2]), and we are demonstrating that it transfers to the setting considered here too. When we refer to good 'stability' of our approach (e.g. in the abstract, and in the contributions), it is this effect that we are referring too.

Nevertheless, we do also observe stability benefits relative to some of the methods we compare to. This is because the integrated regularisation mechanism gives effective regularisation that restricts movement away from the base model, thus meaning we can train arbitrarily long without reward hacking. This is not the case for all methods we tried. Indeed, we found VFM to often collapse due to reward hacking if trained for longer, meaning it requires careful parameter tuning (and direct reward optimisation obviously also does the same). We would therefore claim a stability benefit over these approaches. We do not claim any stability benefit over the two-model version, or FAV - these methods trained stably too. Note that the two-model version also benefits from the same stability of the integrated regularisation mechinism, whereas FAV is able to make endpoint-only regularisation more stable by working in a suitably-chosen latent space rather than the space where the generations take place.

[1] Luo et al 2025, One-step diffusion distillation through score implicit matching

[2] Luo 2025, Diff-instruct++: Training one-step text-to-image generator model to align with human preferences

Key benefit - preserving the flow map property: Of the methods we tried, we would say that the two model version and FAV are also compelling options, and the best choice will likely depend on the exact task at hand. Please see our response to Reviewer 1Kp7, for a more in-depth discussino of this point, including a summary of the pros and cons of the different approaches.

The main benefit of our approach over these others methods is that ours retains the flow-map property in the fine-tuned model, which is not the case otherwise. Indeed, in the updated submission we have added extra rows to the multi-step generation plots showing the MSE between the 1- and K-step generations. We see that this error remains approximately the same as for the base model for our method, whereas for the others it increases as reward-alignment increases. This indicates that the flow-map property of the model is being much better preserved by our method while it is lost for the other methods. The flow map property may be desirable for many applications, for example for inference-time steering approaches like [1], [2], [3]).

[1] Sabour et al 2025, Test-time scaling of diffusions with flow maps

[2] Potaptchik et al 2026, Meta Flow Maps enable scalable reward alignment

[3] Holderrieth et al 2026, Diamond Maps: Efficient Reward Alignment via Stochastic Flow Maps

AuthorsOpenReview ↗

Response to Reviewer brEW

larger comparison on FLUX experiment

We agree that this would be great but this is unfortunately far beyond the scope of our fairly limited compute resources. This is why we have focused our more extensive comparisons and ablations on the ImageNet experiments which are much more manageable in scale. We hope that future work can build on the method proposed here to scale the comparisons to more computationally-demanding settings.

Questions

Q1, Q2, Q4: Please see our comments above regarding these questions.

Q3: Training lag during optimisation This is an interesting point, and we thank the reviewer for raising it.

In our experiments, we generally did not observe any issue arising from possible 'diagonal lag'. We note that this is also a potential concern for the two-model version which is already widely used, and we are not aware of any safeguards being needed in those cases.

We do note however that the reviewer's concern is well founded - during development of our approach, we did one time do a training run using EMA parameters to evaluate the diagonal terms in the regularisation, but this caused training to diverge; we would attribute this to the 'diagonal lag' effect that you inquire about, where the flow matching 'diagonal' does not keep pace with the current endpoint.

As we mention, we never observed any issue caused by this during the remainder of our runs. Based on the reviewer's concern, we ran some training runs in which we decreased the weight of the diagonal term, using coefficients 1 (as in the submission), 0.1, 0.01, and 0.001. The first three all trained stably without any issues, but the 0.001 run did collapse due to the diagonal term not sufficiently tracking the current endpoint distribution. This suggests that the window for stable training is large, and in our experience we would not aniticipate this being a concern for the stability of training. In the event that there ever was any issue, a simple solution would be to just train the diagonal more, for example with more training samples or simply increasing its weight in the loss.

The reviewer raises an interesting point regarding possible diagnostics for the self-induced diagonal lag and flow-map consistency during training. In Figures 14-16, we have added a quantitative metric for the flow-map consistency (at the end of training), and we saw similar values of this metric for the intermediate training checkpoints too. We have not had chance to investigate a direct diagnostic for 'diagonal lag' in the short review period, but we think it is an excellent question and we will add such an analysis if accepted. Based on the experiment in the above paragraph, we do not anticipate this being a concern in practice.

We would like to thank the reviewer again for their comments, and we think that our resulting changes have strengthened our paper. We hope our response has helped to address your concerns. If you have any further questions, please let us know!

Dear Reviewer,

As we approach the end of the discussion period, we would be grateful to hear whether our rebuttal has addressed your questions and concerns.

Best,

The authors

Reviewer_1Kp7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a new method for reward fine-tuning of flow maps. The reward is directly optimized on the flow map’s one-step (or few-step) generations, and regularize toward the base model with a choice of divergence along the path between noise and the model’s own samples. Prior distillation-style approaches train a separate score network for this regularizer. Here, the fine-tuned model’s score comes from the flow map’s own diagonal velocity, so only a single model is trained. To keep this score accurate and the model a valid flow map, they add a self-induced flow matching loss and a consistency loss. The authors show show that the integrated Fisher divergence with a specific weighting recovers the exponentially reward-tilted distribution. The method is evaluated on ImageNet 256 with a one-step flow map, optimizing HPSv2, ImageReward, or PickScore and measuring transfer to the other rewards. The comparison is against a two-model version, VFM, FAV, Adjoint Matching, and a reward-only baseline. Additionally, the method is benchmarked on reward-aligned T2I generation with a FLUX flow map, evaluated with GenEval, and a protein flow map in the appendix.

Critical strengths
  • New reward fine-tuning method designed for the full flow map for reward-aligned few- and multi-step generation (in contrast to just fine-tuning for one-step generation).
  • Meaningful competing methods, including Adjoint matching with 20 steps, reward-only backprop, which shows larger reward hacking behavior, and ablations including different weighting schedules.
  • Meaningful evaluation, always including disentangled rewards/benchmarks from the training reward.
Critical weaknesses
  • The results are hard to read. Almost all results are presented in figures where the competing methods are always in a different panel. This makes directly comparing them a bit difficult, as one needs to eyeball two different panels.
  • This leaves it a bit unclear how good the empirical performance of the method actually is. It seems like for multiple settings, other methods are at least on par. For the T2I results, no competing method is evaluated. The strongest baseline in Figure 3/4 is VFM, which, at least for PickScore transfer, seems to outperform the proposed method. Since VFM is run under a different setup (EMA weights, larger batch size, and I think real data), I understand it’s hard to conclude much from this comparison. One option is to run a noise-space baseline that can be run under the same data-free setup, like Noise Hypernetworks [1], which also targets the reward-tilted distribution for few-step generators.
  • The main motivation of the paper in my view is fine-tuning a full flow map for few and multi-step reward-aligned generation. This is only supported through Appendix B.1.3. with 4 inference steps. Further analysis of the generalization to more steps could significantly strengthen the paper. I also think this deserves a spot in the main text of the paper as it is a central part of the method and the one big distinguishing factor from other methods. Fig. 15 even shows that 4-step generation works without the consistency loss, and larger weights hurt. So it’s unclear whether the method maintains the flow map, or whether this is inherited from the pretrained model over a short fine-tuning run. This is my main weakness, and a more convincing step-count analysis across models, steps, and methods could significantly strengthen the paper.
  • (minor) The theory mostly restates known results from distillation (Props. 3.1–3.3). The new result, Prop. 3.4, applies to iFI with the memoryless weighting, while iKL performs best in practice.

[1] Eyring et al. "Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models". NeurIPS 2025.

Questions for authors
  • How does the base and reward fine-tuned flow map perform on 8/16/24 inference steps? This would be interesting to also see for the T2I FLUX experiments.
  • How does the two-model version perform with 4 or more steps? How about the reward-finetuned Adjoint Matching/VFM/FAV ckpt?
  • As far as I understand, VFM requires real data samples? What dataset was used for the VFM comparisons?
  • The reward-only baseline seems (at least to me) surprisingly strong in some of the experiments. Did you observe convergence during fine-tuning or would the reward hacking behavior be more pronounced when fine-tuning for longer? Or is this because of the lower learning rate reported in B.1.1? With higher learning rate does the reward-only baseline reach higher training rewards and lower eval rewards?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 1Kp7

We thank the reviewer for their very insightful and helpful review, and we address their comments below. We have uploaded an updated paper with subsequent changes highlighted in red.

Flow-map motivation

We agree that the main motivation of our paper is that we obtain a fine-tuned model that preserves the flow map structure. From the feedback in the reviews, we have adjusted the title of the paper and added some extra discussion throughout, which we believe makes it more clear that this is the main distinguishing factor from the surrounding literature. We also agree that this property was somewhat under-investigated in the paper, so we have added the following analysis which we hope helps to address this.

Multi-step generation ability: As the reviewer notes, in our original submission we performed fine-tuning using single-step generation, and in Appendix B.1.3 performed an evaluation using 4-step generation to validate if the fine-tuned behaviour transfers to multi-step generation. As requested, we have since also added additional plots (Figures 14 and 15) showing the same results for 8-step and 16-step generation. Unsurprisingly, the few step performance of adjoint matching is not as strong because it is trained for continuous dynamics. For VFM, the reward-aligned generation appears to not transfer to few-step generation, and the quality is either similar or worse than the base model. The other methods largely show good transfer to multi-step generation, and all show significantly better reward-alignment than the base multi-step generated models. This is unsurprising for our model as we deliberately enforce the flow map property (though we note that by 16 steps we do begin to degrade slightly in the transfer to PickScore - we anticipate that this could be reduced by training with a mixture of multiple steps). We find this quite surprising for the two-model and FAV methods though, as this property is presumably only inherited from the inductive bias of the architecture. Again, this relates to our comment above that such observations are worthy of investigation in future works.

Flow map property: The big advantage of our approach is visible in the new third row of these same Figure 14, 15 and 16. Here, we plot the MSE between the 1-step prediction and the final K-step generation. For a perfect flow map, these generations would be the same. We see that for our method, this error remains pretty much constant as the level of reward alignment increases, and remains similar to that of the base model. In comparison, for the other methods there s a large increase. This indicates that the flow-map property of the model is being much better preserved by our method while it is lost for the other methods (despite in some cases their multi-step generation ability remaining strong, as discussed above). This illustrates the key benefit of our approach over the other methods, as in many cases having a flow map may be desirable (for example if used for inference-time steering approaches like [1], [2], [3]).

[1] Sabour et al 2025, Test-time scaling of diffusions with flow maps

[2] Potaptchik et al 2026, Meta Flow Maps enable scalable reward alignment

[3] Holderrieth et al 2026, Diamond Maps: Efficient Reward Alignment via Stochastic Flow Maps

AuthorsOpenReview ↗

Response to Reviewer 1Kp7

Visualisation of results

Thanks for raising that is can be difficult to compare results across the plots, and yes we agree - this is unfortunately something of a consequence of having many variations of our approach, and needing to visualising all of them, but we certainly agree that we could make it easier to visualise.

We have added the results for the iKL method with the memoryless weighting to the rightmost comparison plot as well, as this was generally the strongest and most stable variation of our approach. It also matches the two-model version already displayed in the plot.

Unclear how good the empirical performance is

This is a certainly a fair comment. Generally, most of these methods perform well, and it's actually quite difficult to conclusively claim that one is 'the best'. In our opinion, the best choice would come down to many factors that are generally determined by the task at hand. Of the methods we looked at, here would be our summary of the pros and cons of each of the few-step regularisations (whether these are in fact a pro or con will depend on the particular use-case):

MethodProsCons
Direct optimisation• Prone to reward hacking • Doesn't give a flow map
VFM• (Mainly designed for inverse problems)• Prone to reward hacking
FAV• Effective regularisation • Performed best on diversity• Requires a choice of kernel in a suitable latent space • Doesn't give a flow map
Two-model version• Effective regularisation• Requires two sets of optimised parameters • Doesn't give a flow map
Ours• Effective regularisation • Gives a flow map• Slightly slower to train than the two-model variant

In particular, we would suggest that our approach is most suitable when a practitioner would like the fine-tuned model to retain the flow map structure, which indeed agrees with your assessment in your review.

Noise hypernetworks: We thank the reviewer for this suggestion, and we have tried out the noise hypernetworks approach during the rebuttal period. However, unfortunately we have not been able to get good performance in our ImageNet experiment. We anticipate that this is because noise hypernetworks can only generate samples that were possible under the base model (as the base model is frozen), and thus struggles in this setting as the base model generations are quite poor. Indeed, looking at the reward-aligned samples we see that they are significantly better than that which the base model can produce. We would anticipate the noise hypernetworks approach to be more suitable in settings where the base model is already good visually (as in the experiments shown in their paper). We have added the citation to our related works section, as we agree that this is an alternative approach for reward-aligning a flow map.

AuthorsOpenReview ↗

Response to Reviewer 1Kp7

Theory using known results from distillation

Yes, you are correct that we mostly use existing theoretical results from the distillation literature (we want to emphasise that we don't claim novelty regarding these results, which is hopefully clear from the paper - we did try to write out the proofs in a self-contained manner in the paper to make the presentation pedagogical, which we hope can be useful to readers).

Regarding this point, we in fact actually consider this one of the main strengths of our approach. There has been really quite a lot of interest recently in reward-alignment of few-step generative models, including for flow-maps specifically. These approaches have generally proposed new mechanisms for trying to regularise such reward optimisation due to the new few-step nature of the generation.

In our view, one of the key strengths of our approach is that it relies on well-established and reliable regularisation mechanisms that have been proven over several years to work well in practice. The novelty is in recognising that these existing and very successful integrated regularisation mechanisms are in fact very well suited for flow-map fine-tuning, giving a simple and intuitive method that does not require adding additional machinery.

Questions:

Additional steps: In addition to the extra figures we mentioned above, we have also added samples and rewards for the fine-tuned FLUX models for 8, 16, and 24 steps as you suggest, and we see that the reward-alignment has successfully transferred across these step counts (please see the additions to Appendix B.2, and Table 3 and Figure 29 for details).

VFM data samples Yes, you are correct that VFM requires real data samples. We used the same ImageNet dataset that was used to train the base model.

Reward-only baseline: Yes, as you suggest, this is because we made the learning rate very small in order to reach the same reward level over the same training period (as we report in B.1.1 - we note that this is maybe a bit kind to this baseline). The reward increased pretty much linearly, which continued if trained for longer until it reward hacked (which makes other rewards decrease, and causes diversity to collapse).

We would like to thank the reviewer again for their thoughtful review, and we believe that our subsequent additions have strengthened our paper. If we have addressed your concerns, we would be grateful if you would reconsider your assessment of our work. If you have any further questions, please let us know!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.