acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning via Fixed-point Flow Matching

Abstract

In maximum entropy and behavior-constrained reinforcement learning (RL), the optimal policy is given by a Boltzmann distribution. Generating samples from this distribution can be challenging, as it is often complex and multimodal. While flow policies trained with flow matching provide a powerful framework for representing such distributions, they traditionally rely on access to training data, which is generally unavailable in online RL. To address this limitation, we propose fixed-point flow matching (FFM), a data-free procedure for training and fine-tuning flow policies in RL. FFM leverages the target flow identity to formulate the actor update as a fixed-point iteration, utilizing a flow-matching-style regression objective where the critic's action gradients serve as regression targets. We show that the unique fixed point of this iteration corresponds to the optimal policy. Through extensive experiments, we demonstrate that FFM effectively fine-tunes pretrained flow policies in the offline-to-online setting and enables efficient and accurate training of flow policies from scratch in online RL. These results establish fixed-point flow matching as a principled approach for learning expressive flow-based policies without requiring training data.

Primary area:
reinforcement learning
Keywords:
online RL, offline to online RL, flow policies

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_H3hvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Fixed-Point Flow Matching (FFM), a data-free method for training flow policies toward a Boltzmann target. It rewrites the target velocity field as a conditional expectation of critic gradients and evaluates it with samples from the current policy, giving a fixed-point iteration with π⋆\pi^\star as a fixed point. The resulting regression needs no ODE backpropagation, importance weights, or adjoints. FFM is evaluated on 50 offline-to-online and 24 online tasks against recent baselines.

Critical strengths
  • The basic idea is clean. Writing the velocity field of the Boltzmann policy as a conditional expectation of critic gradients at clean actions, and then just regressing onto that, is a natural thing to do and avoids the usual pain of backpropagating through the sampler or dealing with importance weights. The experiments are also more thorough than what I usually see in this area: 50 offline-to-online tasks and 24 online tasks, 8 to 10 seeds, IQM with bootstrap intervals, and the baselines are run under the same protocol.
  • Getting the entropy from the divergence of the velocity field, instead of fitting a GMM or using a bound as other max-entropy methods do, is a nice consequence of using a deterministic flow and is done properly here. The same goes for extracting the reference score from the pretrained flow through the velocity–score bridge; it is a small observation, but it fits this setting well.
Critical weaknesses
  • My main concern is that the paper never really shows that the FFM target itself is what makes things work. Theorem 3 says π⋆\pi^\star is a fixed point of the iteration, but that is all it says. Nothing is shown about whether the iteration actually moves toward π⋆\pi^\star from somewhere else, and the update away from the fixed point is biased in a way the paper does not discuss. On the experimental side, I could not find any ablation where only the regression target is swapped out, e.g., for a QSM-style target at the noisy action, a DIPO-style target, or importance-weighted regression, with everything else held fixed. There is also no comparison between the mixed target that is actually used (Eq. 34) and the pure target of Eq. 11. Given how much else is going on in the full method (trust region, deterministic evaluation, the per-domain recipe), I cannot tell how much of the performance is due to the target and how much is due to the rest. The efficiency argument against ODE backprop and adjoint methods is also made in words only, with no memory or runtime numbers.
  • The paper repeatedly describes FFM as conceptually simple and easy to implement, but the method that is actually run is quite far from the single regression presented in Section 4. Reading Appendices D.3 and E.3, one finds a mixed target with a Fisher normalizer, a velocity-space trust region with a dual variable, reference-score extraction at t=0.9t = 0.9, softened denominators, Hutchinson entropy estimates, squashing corrections, deterministic evaluation, and OU-correlated exploration, several of which are not mentioned in the main text. On top of that, in the offline-to-online setting τQ\tau_Q, the normalizer, the trust region, and ϵTR\epsilon_{TR} all change between locomotion and manipulation (Table 4), and Fig. 18 shows that the wrong combination gives near zero. ϵTR\epsilon_{TR} and τOU\tau_{OU} also behave differently on different tasks (Figs. 15, 20). I am not sure how one would choose these settings on a new domain, and I think the paper should be more upfront about this.
Questions for authors
  • Theorem 3 only tells us that π⋆\pi^\star stays put if we are already there. Do you have any evidence that the iteration actually converges to π⋆\pi^\star from a different starting point? Even a 1-D toy example would help, ideally with a case where it fails.
  • Could you run an ablation where only the regression target is changed, keeping everything else in the code the same?
  • Eq. 13 is 0/0 at t=1t = 1. In practice, you evaluate it at t=0.9t = 0.9 (Appendix E.3.4). So the score being used is that of a smoothed version of prefp_{\text{ref}} rather than prefp_{\text{ref}} itself, and dividing by κ(0.9)≈0.11\kappa(0.9) \approx 0.11 amplifies any error in vrefv_{\text{ref}} by roughly an order of magnitude. How sensitive are the results to δref\delta_{\text{ref}}, and did you observe any instability from this amplification?
  • In the online max-entropy setting, does FFM still work without the trust region?
  • The paper argues that FFM is cheaper than ODE backprop and adjoint methods, but there are no numbers. Could you report memory and wall-clock time?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer H3hv (1/3)

We thank the reviewer for their careful reading and constructive feedback. Following the review, we have added the requested ablations and extended the discussion of the method. Below, we address each point in turn and indicate the corresponding changes in the revised manuscript.


My main concern is that the paper never really shows that the FFM target itself is what makes things work. […] I could not find any ablation where only the regression target is swapped out, e.g., for a QSM-style target at the noisy action, a DIPO-style target, or importance-weighted regression, with everything else held fixed. […] Could you run an ablation where only the regression target is changed, keeping everything else in the code the same?

We ran this ablation on all 24 online tasks with 10 seeds each (App. D.4, Fig. 8). Only the actor's regression target changes. Critic, rollouts, entropy estimate and temperature adaptation, trust region, schedule, network, optimizer and deterministic evaluation are identical. The final IQM with 95% confidence intervals is reported below.

Regression targetDMC (20 tasks)Humanoid (4 tasks)
FFM, mixed (as in the paper; now Eq. 33)0.879 [0.857, 0.897]35.4 [35.2, 35.5]
FFM, pure (Eq. 11)0.896 [0.879, 0.910]35.1 [34.8, 35.3]
Importance-weighted regression0.873 [0.859, 0.886]−0.7 [−1.4, 0.0]
DIPO-style0.798 [0.779, 0.815]0.3 [0.0, 0.5]
QSM-style (score at the noisy action)0.614 [0.572, 0.652]−0.7 [−1.0, −0.3]

On DMC, importance-weighted regression is close to FFM, and the DIPO- and QSM-style targets are clearly worse. On the high-dimensional humanoid tasks only the FFM targets learn. With everything else fixed, the FFM target is thus the only one of the compared targets that trains reliably on both suites.


There is also no comparison between the mixed target that is actually used (Eq. 34) and the pure target of Eq. 11.

The first two rows of the table are this comparison. The two targets perform on par on both suites (0.879 against 0.896 on DMC, 35.4 against 35.1 on the humanoid tasks). The control variate of the mixed target reduces the variance of the target but is not the source of the performance.


Given how much else is going on in the full method (trust region, deterministic evaluation, the per-domain recipe), I cannot tell how much of the performance is due to the target and how much is due to the rest. […] In the online max-entropy setting, does FFM still work without the trust region?

We thank the reviewer for the question. We added an ablation of FFM without the trust region in Fig. 9 ("FFM w/o TR", App. D.4). Without it, FFM still learns on most DMC tasks, with a moderately lower aggregate return, but it does not learn on the high-dimensional humanoid tasks. The trust region is therefore essential in high dimensions, which agrees with the role of trust regions in on-policy methods in general.

Regarding the deterministic evaluation: Fig. 16 to 18 compare it with stochastic evaluation. The two are close on DMC, and the deterministic evaluation leads to slighly better performance on the humanoid tasks. It changes only how the trained policy is evaluated, not how it is trained.


The efficiency argument against ODE backprop and adjoint methods is also made in words only, with no memory or runtime numbers. […] Could you report memory and wall-clock time?

We added both (Sec. 6, Fig. 4). We compare one update of FFM with the adjoint method, as used by TRQAM, in an implementation that matches FFM's memory, and with backpropagation through the unrolled flow (BPTT), which minimizes the reverse KL as DIME, TRUDI and FLAC do, on identical networks and as a function of the number of integration steps and the actor size. FFM regresses on a fixed number of interpolant samples and generates its actions without gradients, so its memory is constant in the number of steps and its time grows only with the gradient-free forward generation. The adjoint method has to solve the adjoint equation backward through the flow, one vector-Jacobian product per step, so its time grows linearly with the number of steps even with matched memory; BPTT stores the activations of every step, so its time and its memory grow linearly with the steps and with the actor size. At 80 steps the alternatives take two to three times longer than FFM and BPTT needs three times the memory, and the gap widens with every further step and parameter. This is what lets FFM scale to larger actors and more integration steps without a corresponding growth in cost.

AuthorsOpenReview ↗

Reply to Reviewer H3hv (2/3)

Theorem 3 says π⋆ is a fixed point of the iteration, but that is all it says. Nothing is shown about whether the iteration actually moves toward π⋆ from somewhere else, and the update away from the fixed point is biased in a way the paper does not discuss. […] Do you have any evidence that the iteration actually converges to π⋆ from a different starting point?

We agree that Theorem 3 is only a fixed-point statement. We added the closed form of the population update for an arbitrary current policy (App. B.2, Eq. 37):

 s)}{\pi_{\bar\theta}(X_1\mid s)} \Big|\ X_t = x\Big]$$ Here the first term is the velocity field that reproduces the current policy, and $c \equiv 1$ gives the pure target. The update is the current policy's velocity plus a correction: the score of the density ratio between the Boltzmann policy and the current policy, averaged over the clean actions of the current policy (this correction vanishes at the fixed point since the density ratio is 1). We do not have a convergence proof from an arbitrary initialization, and the revision states this in the limitations (Sec. 7); the same question is open for related fixed-point matching objectives such as iDEM [1], adjoint sampling [2], flow and the bridge matching sampler [3]. Empirically, the iteration converges from a randomly initialized policy on all 24 online tasks for both the pure and the mixed target, while the alternative targets in the same training loop do not (Fig. 8). --- > Even a 1-D toy example would help, ideally with a case where it fails. We added the one-dimensional example the reviewer asks for (App. B.2, Fig. 5). For a Gaussian target it has a closed form where every iteration is shrinking the offset between the policy mean and the target mean by a constant factor, which depends on the step size and on the mixing schedule. For a bimodal target with overlapping modes, the mixed target converges to the Boltzmann policy from four different initial policies, up to a small error from the discretization and the softening. The example also contains two failure cases: - Without damping ($\eta = 1$) the update overshoots and oscillates. - For well-separated modes the iteration recovers the shape of each mode but not their weights, and a policy that starts on one mode stays on it.  Both cases are described with the example (App. B.2). The limitations (Sec. 7) now state that a theoretical analysis of the convergence is left for future work. --- > The paper repeatedly describes FFM as conceptually simple and easy to implement, but the method that is actually run is quite far from the single regression presented in Section 4. […] in the offline-to-online setting τQ, the normalizer, the trust region, and ϵTR all change between locomotion and manipulation (Table 4) [...] I am not sure how one would choose these settings on a new domain, and I think the paper should be more upfront about this. We agree that the paper should be more upfront, and we revised it accordingly. The Limitations paragraph now states that the offline-to-online experiments need two different settings, one for locomotion and one for manipulation, and names the components that the practical instantiations add to the regression of Sec. 4.  We would like to add two remarks. First, most of these components are not specific to FFM. Trust regions, action squashing and distributional critics are shared with the on-policy baselines. With all of them fixed, the target ablation shows that it is the regression of Sec. 4 that matters (Fig. 8). Second, Table 4 is one two-valued choice per domain family, made once for all locomotion and all manipulation tasks, without per-task tuning. 
AuthorsOpenReview ↗

Reply to Reviewer H3hv (3/3)

Eq. 13 is 0/0 at t=1. In practice, you evaluate it at t=0.9 (Appendix E.3.4). So the score being used is that of a smoothed version of pref rather than pref itself, and dividing by κ(0.9)≈0.11 amplifies any error in vref by roughly an order of magnitude. How sensitive are the results to δref, and did you observe any instability from this amplification?

We did not observe any instability from this amplification. To quantify the sensitivity, we added a sweep over δref\delta_{\mathrm{ref}} on antmaze-large to the revision (App. E.5, Table 7), with the full protocol (1M offline and 500K online steps, three seeds, success averaged over the last 150K steps of each phase):

δref\delta_{\mathrm{ref}}0.020.050.1 (paper)0.20.4
antmaze-large, offline0.95 ± 0.010.95 ± 0.010.94 ± 0.000.94 ± 0.010.93 ± 0.03
antmaze-large, online1.00 ± 0.001.00 ± 0.001.00 ± 0.001.00 ± 0.001.00 ± 0.00

On this task the success does not depend on δref\delta_{\mathrm{ref}} over the whole range, in either phase. Reading the score slightly before the endpoint is common practice for numerical stability [4,5,6]; the value t=0.9t = 0.9 itself was an ad-hoc choice that we did not tune.


References

[1] Akhound-Sadegh et al. Iterated Denoising Energy Matching for Sampling from Boltzmann Densities. ICML 2024.

[2] Havens et al. Adjoint Sampling: Highly Scalable Diffusion Samplers via Adjoint Matching. ICML 2025.

[3] Blessing et al. Bridge Matching Sampler: Scalable Sampling via Generalized Fixed-Point Diffusion Matching. ICML 2026.

[4] Poole et al. DreamFusion: Text-to-3D using 2D Diffusion. ICLR 2023.

[5] Song et al. Consistency Models. ICML 2023.

[6] Yin et al. One-step Diffusion with Distribution Matching Distillation. CVPR 2024.


We hope these clarifications and additional experiments address the reviewer's concerns. If they do, we would be grateful if the reviewer would consider raising their score. We would like to thank the reviewer again for the detailed review, which has helped us improve our manuscript substantially. We are happy to answer any further questions during the rebuttal period.

Thanks for the detailed response and the new experiments. They resolve most of my concerns, so I have raised my score.

That said, in my view, the method actually used is not as simple as the paper suggests, since it includes several components beyond the core regression, such as the Fisher normalizer, the softened denominators, and the reference score at t=0.9t = 0.9. I believe explaining why they are needed and what each contributes would make it a stronger paper.

Reviewer_G6LUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes fixed-point flow matching (FFM) for training flow policies in maximum-entropy and behavior-constrained reinforcement learning. It derives a velocity regression target from critic action gradients using the target-score identity, then replaces unavailable optimal-policy samples with samples from the current policy. This avoids differentiating through the sampling trajectory. Theorem 3 establishes that the desired Boltzmann policy is a fixed point of the ideal population update, but does not establish convergence from arbitrary initialization.

The paper develops online and offline-to-online implementations with trust-region regularization, stabilized regression targets, and optional correlated exploration. Experiments cover 50 OGBench offline-to-online tasks and 24 online control tasks, with multiple seeds and extensive ablations. Results show strong humanoid performance and the best reported aggregate offline-to-online success, with the largest improvement concentrated in HumanoidMaze-Large.

Critical strengths
  1. FFM provides a simple, well-motivated way to train expressive flow policies through regression on critic gradients, without backpropagating through the sampling trajectory or simulating adjoints. The ideal target-flow identity and fixed-point interpretation give the update a clear mathematical motivation.

  2. The empirical evaluation is substantial: 74 tasks across two RL settings, multiple seeds, and numerous ablations. The reported gains on HumanoidMaze-Large and high-dimensional humanoid control provide meaningful evidence of practical utility.

Critical weaknesses
  1. The on-policy justification for correlated exploration is incorrect as stated. Stationary OU noise can remain marginally Gaussian while its distribution conditional on the current state changes, as that state depends on previous actions and correlated noise. Therefore, the claim that the executed conditional policy, likelihood calculation, and entropy-regularized backups remain unchanged does not follow. The authors should clarify how this dependence is handled in training or revise the interpretation of the extension.

  2. Important evaluation and efficiency claims need clearer support. Deterministic evaluation substantially improves FFM’s humanoid performance, but corresponding evaluation conventions for the baselines are unspecified, and TRUDI’s hyperparameters are missing from Table 1. The FFM+OU aggregate also combines task-specific variants, leaving its selection procedure and comparative tuning budget unclear. These details could materially affect the interpretation of the reported advantages. Additionally, the paper motivates FFM through reduced optimization complexity but provides no wall-clock or peak-memory comparisons.

Questions for authors
  1. Does the implemented critic target include the current-action term −λlog⁡π(at∣st)-\lambda\log\pi(a_t\mid s_t)? Please provide the exact return calculation and reconcile Equation (45) and Algorithm 1 with Equation (42).

  2. Which fixed-point guarantee applies to the mixed, softened regression used in experiments? What evidence quantifies the bias or sensitivity introduced by denominator softening and evaluating the reference score at t=0.9t=0.9?

  3. How are actor targets, likelihoods, and soft critic returns handled when OU noise is correlated with the current state? Are fresh independent policy samples used, or should FFM+OU be interpreted as an exploration behavior process requiring a different justification?

  4. Please clarify how actions are selected during evaluation for each method, including whether evaluation is deterministic or stochastic and how these conventions were chosen. Since deterministic evaluation substantially improves FFM’s humanoid performance (Figures 12–14), were comparable evaluation choices considered for the baselines? For the FFM+OU aggregate in Figure 3, please also specify the per-task correlation settings, their selection procedure, and the tuning budget relative to the baselines. Also, please provide wall-clock and memory consumption comparisons.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer G6LU (1/2)

We thank the reviewer for their careful reading and constructive feedback. Following the review, we have corrected the manuscript and added experiments and discussions. Below, we address each point in turn and indicate the corresponding changes in the revised manuscript.


The on-policy justification for correlated exploration is incorrect as stated.

We thank the reviewer for this careful analysis and agree with their assessment. We have therefore removed correlated exploration from the paper for now. We will work out a principled treatment before adding it back. Correlated exploration was a minor, optional extension and not the focus of the paper. It was used only for the dashed FFM + OU curve on the DMC tasks. All other results, including every comparison on the humanoid tasks and all offline-to-online results, use independent prior draws and are not affected.


Deterministic evaluation substantially improves FFM's humanoid performance, but corresponding evaluation conventions for the baselines are unspecified […] Please clarify how actions are selected during evaluation for each method, including whether evaluation is deterministic or stochastic and how these conventions were chosen. […] were comparable evaluation choices considered for the baselines?

We thank the reviewer for this question. Every method in our comparison is evaluated with its own inference-time routine, and several of these routines are deterministic as well. We now state them in App. D.3:

  • Gaussian policies (REPPO, PPO, SPO) execute the mean action.
  • TRUDI integrates the probability-flow ODE together with the evaluation heuristics described in the TRUDI paper.
  • AMDP starts its sampling process deterministically at zero by construction.
  • DIME, DPPO and FPO use their standard inference procedures.
  • FLAC and FLAG, which we added at the request of Reviewer gYhk, use the same deterministic initialization as FFM.

For reference, Fig. 16 to 18 also report FFM under stochastic evaluation: the two conventions are close on DMC, and the deterministic one is better on the humanoid tasks.


TRUDI's hyperparameters are missing from Table 1.

We thank the reviewer for noticing this and added a TRUDI column to Table 1.


The FFM+OU aggregate also combines task-specific variants, leaving its selection procedure and comparative tuning budget unclear. […] please also specify the per-task correlation settings, their selection procedure, and the tuning budget relative to the baselines.

Despite correlated exploration not being part of the paper anymore, but we answer this question for completeness. In the submission, the FFM + OU curve used correlated exploration on eight of the 20 DMC tasks and the plain FFM curve on the other twelve. For these eight tasks the correlation time was selected per task, by its final return, from a sweep over \tau_{\mathrm{OU}} \in \\{1, 2, 4, 8, 16, 32\\\} with ten seeds per value (Fig. 15 of the submission). The selected values were printed in the corresponding panels of Fig. 4 of the submission. All results that remain in the paper use one FFM configuration on all 24 online tasks.


Additionally, the paper motivates FFM through reduced optimization complexity but provides no wall-clock or peak-memory comparisons.

We added both (Sec. 6, Fig. 4). We compare one update of FFM with the adjoint method, as used by TRQAM, in an implementation that matches FFM's memory, and with backpropagation through the unrolled flow (BPTT), which minimizes the reverse KL as DIME, TRUDI and FLAC do, on identical networks and as a function of the number of integration steps and the actor size. FFM regresses on a fixed number of interpolant samples and generates its actions without gradients, so its memory is constant in the number of steps and its time grows only with the gradient-free forward generation. The adjoint method has to solve the adjoint equation backward through the flow, one vector-Jacobian product per step, so its time grows linearly with the number of steps even with matched memory; BPTT stores the activations of every step, so its time and its memory grow linearly with the steps and with the actor size. At 80 steps the alternatives take two to three times longer than FFM and BPTT needs three times the memory, and the gap widens with every further step and parameter. This is what lets FFM scale to larger actors and more integration steps without a corresponding growth in cost.

AuthorsOpenReview ↗

Reply to Reviewer G6LU (2/2)

Does the implemented critic target include the current-action term −λ log π(a_t | s_t)? Please provide the exact return calculation and reconcile Equation (45) and Algorithm 1 with Equation (42).

No. The target for the pair (sk,xk)(s_k, x_k) contains no entropy term of the current action. The inconsistency was an error in the text of the submission, not in the implementation, and we corrected the soft Q-function (now Eq. 39), the n-step return (now Eq. 43) and Algorithm 1.


Which fixed-point guarantee applies to the mixed, softened regression used in experiments? What evidence quantifies the bias or sensitivity introduced by denominator softening and evaluating the reference score at t=0.9?

Mixed target. Theorem 3 holds for every mixing schedule c(t)∈[0,1]c(t) \in [0, 1] and the mixed target therefore has the same fixed point. We included a new ablation study which confirms that mixing does not change the outcome: the mixed and the pure target reach 0.879 and 0.896 on DMC and 35.4 and 35.1 on the humanoid tasks (Fig. 8).

Softened denominators. Theorem 3 is stated for the regression without softening. The softening is a numerical safeguard: the two denominators of the regression vanish at the endpoints of the path, and adding a small constant δ\delta keeps the loss finite there. Safeguards of this kind are common for objectives whose coefficients diverge at the endpoints. Score-based diffusion models restrict training to t≥ϵt \ge \epsilon [1] and flow matching keeps a constant σmin⁡\sigma_{\min} in the denominator of its conditional velocity [2]. To quantify the sensitivity, we added a sweep over δ\delta to the revision (App. D.4, Fig. 19). We reran FFM with δ∈0.01,0.025,0.1,0.2\delta \in \\{0.01, 0.025, 0.1, 0.2\\} on all 24 online tasks with five seeds per value, and compare with the runs of the paper (δ=0.05\delta = 0.05, ten seeds). The table reports the final IQM with 95% confidence intervals:

δ\delta0.010.0250.05 (paper)0.10.2
DMC (20 tasks)0.888 [0.860, 0.908]0.842 [0.807, 0.874]0.879 [0.857, 0.897]0.882 [0.855, 0.905]0.894 [0.868, 0.912]
Humanoid (4 tasks)35.1 [34.8, 35.5]35.5 [35.3, 35.8]35.4 [35.2, 35.5]35.3 [34.9, 35.6]35.5 [35.2, 35.9]

The results do not depend on δ\delta over this range of a factor of 20.

Reference score at t=0.9t = 0.9. The velocity–score bridge divides by κ(t)\kappa(t), which vanishes at t=1t = 1. We therefore read the reference score slightly before the endpoint, at t=0.9t = 0.9. This is done for numerical stability and is common practice when a pretrained diffusion or flow model is queried. E.g. score distillation samples the time away from the endpoints [3,4]. A sweep over δref=1−t\delta_{\mathrm{ref}} = 1 - t on antmaze-large (App. E.5, Table 7; three seeds, success averaged over the last 150K steps of each phase) shows no dependence of the success on it:

δref\delta_{\mathrm{ref}}0.020.050.1 (paper)0.20.4
antmaze-large, offline0.95 ± 0.010.95 ± 0.010.94 ± 0.000.94 ± 0.010.93 ± 0.03
antmaze-large, online1.00 ± 0.001.00 ± 0.001.00 ± 0.001.00 ± 0.001.00 ± 0.00

References

[1] Song et al. Score-Based Generative Modeling through Stochastic Differential Equations. ICLR 2021.

[2] Lipman et al. Flow Matching for Generative Modeling. ICLR 2023.

[3] Poole et al. DreamFusion: Text-to-3D using 2D Diffusion. ICLR 2023.

[4] Yin et al. One-step Diffusion with Distribution Matching Distillation. CVPR 2024.


We hope these clarifications and additional experiments address the reviewer's concerns. If they do, we would be grateful if the reviewer would consider raising their score. We would like to thank the reviewer again for the insightful review, which has helped us improve our manuscript. We are happy to answer any further questions during the rebuttal period.

Thanks for the detailed reply. I increased my score.

Reviewer_gYhkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Fixed-Point Flow Matching (FFM). FFM trains flow policies using critic-gradient regression. The method is evaluated on both online maximum-entropy RL and offline-to-online fine-tuning.

Critical strengths

The proposed actor update avoids backpropagation through the sampling trajectory and expensive adjoint simulation.

The method establishes a connection between target score identities and flow-based policy optimization.

Extensive experiments across diverse tasks demonstrate competitive performance, particularly in offline-to-online fine-tuning.

Critical weaknesses

Limited methodological distinction. The core idea closely resembles QSM, QAM, and endpoint-based Q-guidance methods. The specific advantages of FFM remain unclear.

Theorem 3 establishes a basic fixed-point consistency property, this is common to well-designed policy improvement operators.

MaxEnt from of FFM and efficiency concerns. Eq. (45) appears inconsistent with the soft Bellman equation in Eq. (42). Computing log-probabilities through ODE divergence estimation is also expensive, motivating comparisons with methods such as FLAC/FLAG.

Questions for authors

refer weakness

2 comments
AuthorsOpenReview ↗

Reply to Reviewer gYhk

We thank the reviewer for their careful reading and constructive feedback. F Below, we address each point in turn and indicate the corresponding changes in the revised manuscript.


Limited methodological distinction. The core idea closely resembles QSM, QAM, and endpoint-based Q-guidance methods. The specific advantages of FFM remain unclear.

We thank the reviewer for this comment and agree that the distinction should be stated more sharply. The methods differ in where the critic gradient is evaluated and in what the regression target is.

QSM regresses the score at the noisy action XtX_t onto the critic gradient evaluated at that same noisy action. This equals the score of the noised Boltzmann policy only at the clean end t=1t = 1. FFM instead uses the target flow identity (Proposition 1), which is exact at every tt. DIPO regresses onto improved actions and does not account for the noise level of the interpolant. QAM and TRQAM obtain an exact target as well, but through adjoint matching, which converts the flow to a memoryless SDE and solves an adjoint ODE backwards along stored trajectories which is not required for FFM.

We added two experiments that quantify these differences. First, an ablation that swaps only the regression target and keeps everything else fixed, which Reviewer H3hv also asked for (App. D.4, Fig. 8): on DMC the FFM target reaches a final IQM of 0.879 to 0.896, a DIPO-style target 0.798 and a QSM-style target 0.614; on the humanoid tasks only the FFM target learns. Second, a cost comparison with the adjoint method, as used by TRQAM, and with BPTT, as used by DIME, TRUDI and FLAC, on identical networks (Sec. 6, Fig. 4). The results show that the cost of an FFM update is independent of the number of integration steps apart from the gradient-free forward generation, whereas the cost of the alternatives grows linearly with the steps and with the actor size, so the gap widens with every further step and parameter.

To make sure that we address this point in full, could the reviewer kindly let us know which endpoint-based Q-guidance methods they have in mind? We would be glad to add a discussion of them, and a comparison where this is feasible within the discussion period.


Theorem 3 establishes a basic fixed-point consistency property, this is common to well-designed policy improvement operators.

We agree that Theorem 3 is a consistency statement and not a convergence result, and the limitations (Sec. 7) now state that a convergence analysis is left for future work. We would like to add two remarks. First, the property is not automatic for critic-gradient regressions of flow policies: with a QSM-style target, the velocity field of the Boltzmann policy is not the minimizer at intermediate noise levels, because the critic gradient at a noisy action is not the score of the noised policy. Second, we extended the analysis beyond the fixed point. The revision gives the population update in closed form for an arbitrary current policy (App. B.2, Eq. 37): it equals the velocity field of the current policy plus a correction, the score of the density ratio between the Boltzmann policy and the current policy, averaged over the clean actions consistent with XtX_t. This shows in which direction an update moves away from the fixed point.


MaxEnt from of FFM and efficiency concerns. Eq. (45) appears inconsistent with the soft Bellman equation in Eq. (42).

We thank the reviewer for catching this. It was an error in the text, and we fixed it: the soft Q-function (now Eq. 39), the n-step return (now Eq. 43) and Algorithm 1 now agree with the soft Bellman equation. The implementation used the correct form, so no result changes.


Computing log-probabilities through ODE divergence estimation is also expensive, motivating comparisons with methods such as FLAC/FLAG.

We would first like to clarify that FFM does not compute the exact divergence. It uses Hutchinson's trace estimator with four random probes per integration step. Each probe is one Jacobian-vector product, so the cost does not grow with the action dimension, and no backpropagation through the ODE is needed.

We also added both methods as baselines, run in FFM's training setup with 10 seeds per task (Fig. 3, 6 and 7, App. D.3). Final IQM with 95% confidence intervals:

MethodDMC (20 tasks)Humanoid (4 tasks)
FFM0.879 [0.857, 0.897]35.4 [35.2, 35.5]
FLAG0.914 [0.909, 0.918]22.5 [22.1, 22.8]
FLAC0.734 [0.703, 0.765]12.4 [10.7, 14.0]

FLAG is ahead of FFM on DMC. Its lead comes mainly from a few hard-exploration tasks such as HopperStand and PendulumSwingup (Fig. 6). On the high-dimensional humanoid tasks FFM is clearly ahead of both methods.


We hope these clarifications and additional experiments address the reviewer's concerns. We are happy to answer any further questions the reviewer might have.

Thank you for the clarifications and additional experiments. The prior endpoint-based methods I referred to include VINE and QGF (although QGF operates at test time). FFM’s core idea is similar to theirs, and I see little additional insight. The DMC evaluation focuses on relatively simple tasks, and estimating log⁡π\log \pi in the maximum-entropy setting still incurs the overhead of four JVPs per integration step. Given the limited discussion time after the November 17 rebuttal, it would be unreasonable to request further experiments or revisions. I maintain my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their time and their constructive and detailed feedback, which has helped us improve the paper substantially. Following the reviewers' feedback, we have added experiments, ablations, and discussions. The major changes are marked in blue\textcolor{blue}{\text{blue}} in the revised manuscript. Below we summarize the main additions to the paper.

  • FLAC and FLAG baselines (Fig. 3). We run both methods in FFM's training setup with 10 seeds per task. On DMC, FLAG reaches 0.914 and FLAC 0.734, against 0.879 for FFM. On the humanoid tasks FFM reaches 35.4, against 22.5 for FLAG and 12.4 for FLAC.
  • Wall-clock and memory (Fig. 4). We compare one update of FFM with the adjoint method, as used by TRQAM, and with backpropagation through the flow (BPTT), as used by DIME, TRUDI and FLAC, as a function of the number of integration steps and the actor size. FFM's memory is constant in the number of steps and its time grows only with the gradient-free forward generation, whereas the time of the adjoint method and both the time and the memory of BPTT grow linearly with the steps and with the actor size. At 80 steps the alternatives take two to three times longer and BPTT needs three times the memory, and the gap widens with every further step and parameter, which lets FFM scale to larger actors and finer integration.
  • Regression-target ablation (Fig. 8). We swap only the actor's regression target and keep everything else fixed, on all 24 online tasks with 10 seeds each. On the DMC tasks the pure and the mixed FFM target reach a final IQM of 0.896 and 0.879, importance-weighted regression 0.873, a DIPO-style target 0.798, and a QSM-style target 0.614. On the humanoid tasks only the two FFM targets learn (35.1 and 35.4); all alternatives end below a return of 1.
  • FFM without trust region (Fig. 9). Without the trust region the final IQM drops from 0.879 to 0.788 on DMC and from 35.4 to 3.7 on the humanoid tasks.

We hope these changes address the reviewers' concerns and are happy to answer any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.