- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes Fixed-Point Flow Matching (FFM), a data-free method for training flow policies toward a Boltzmann target. It rewrites the target velocity field as a conditional expectation of critic gradients and evaluates it with samples from the current policy, giving a fixed-point iteration with as a fixed point. The resulting regression needs no ODE backpropagation, importance weights, or adjoints. FFM is evaluated on 50 offline-to-online and 24 online tasks against recent baselines.
Critical strengths
- The basic idea is clean. Writing the velocity field of the Boltzmann policy as a conditional expectation of critic gradients at clean actions, and then just regressing onto that, is a natural thing to do and avoids the usual pain of backpropagating through the sampler or dealing with importance weights. The experiments are also more thorough than what I usually see in this area: 50 offline-to-online tasks and 24 online tasks, 8 to 10 seeds, IQM with bootstrap intervals, and the baselines are run under the same protocol.
- Getting the entropy from the divergence of the velocity field, instead of fitting a GMM or using a bound as other max-entropy methods do, is a nice consequence of using a deterministic flow and is done properly here. The same goes for extracting the reference score from the pretrained flow through the velocity–score bridge; it is a small observation, but it fits this setting well.
Critical weaknesses
- My main concern is that the paper never really shows that the FFM target itself is what makes things work. Theorem 3 says is a fixed point of the iteration, but that is all it says. Nothing is shown about whether the iteration actually moves toward from somewhere else, and the update away from the fixed point is biased in a way the paper does not discuss. On the experimental side, I could not find any ablation where only the regression target is swapped out, e.g., for a QSM-style target at the noisy action, a DIPO-style target, or importance-weighted regression, with everything else held fixed. There is also no comparison between the mixed target that is actually used (Eq. 34) and the pure target of Eq. 11. Given how much else is going on in the full method (trust region, deterministic evaluation, the per-domain recipe), I cannot tell how much of the performance is due to the target and how much is due to the rest. The efficiency argument against ODE backprop and adjoint methods is also made in words only, with no memory or runtime numbers.
- The paper repeatedly describes FFM as conceptually simple and easy to implement, but the method that is actually run is quite far from the single regression presented in Section 4. Reading Appendices D.3 and E.3, one finds a mixed target with a Fisher normalizer, a velocity-space trust region with a dual variable, reference-score extraction at , softened denominators, Hutchinson entropy estimates, squashing corrections, deterministic evaluation, and OU-correlated exploration, several of which are not mentioned in the main text. On top of that, in the offline-to-online setting , the normalizer, the trust region, and all change between locomotion and manipulation (Table 4), and Fig. 18 shows that the wrong combination gives near zero. and also behave differently on different tasks (Figs. 15, 20). I am not sure how one would choose these settings on a new domain, and I think the paper should be more upfront about this.
Questions for authors
- Theorem 3 only tells us that stays put if we are already there. Do you have any evidence that the iteration actually converges to from a different starting point? Even a 1-D toy example would help, ideally with a case where it fails.
- Could you run an ablation where only the regression target is changed, keeping everything else in the code the same?
- Eq. 13 is 0/0 at . In practice, you evaluate it at (Appendix E.3.4). So the score being used is that of a smoothed version of rather than itself, and dividing by amplifies any error in by roughly an order of magnitude. How sensitive are the results to , and did you observe any instability from this amplification?
- In the online max-entropy setting, does FFM still work without the trust region?
- The paper argues that FFM is cheaper than ODE backprop and adjoint methods, but there are no numbers. Could you report memory and wall-clock time?
4 comments
Reply to Reviewer H3hv (2/3)
Theorem 3 says π⋆ is a fixed point of the iteration, but that is all it says. Nothing is shown about whether the iteration actually moves toward π⋆ from somewhere else, and the update away from the fixed point is biased in a way the paper does not discuss. […] Do you have any evidence that the iteration actually converges to π⋆ from a different starting point?
We agree that Theorem 3 is only a fixed-point statement. We added the closed form of the population update for an arbitrary current policy (App. B.2, Eq. 37):
s)}{\pi_{\bar\theta}(X_1\mid s)} \Big|\ X_t = x\Big]$$ Here the first term is the velocity field that reproduces the current policy, and $c \equiv 1$ gives the pure target. The update is the current policy's velocity plus a correction: the score of the density ratio between the Boltzmann policy and the current policy, averaged over the clean actions of the current policy (this correction vanishes at the fixed point since the density ratio is 1). We do not have a convergence proof from an arbitrary initialization, and the revision states this in the limitations (Sec. 7); the same question is open for related fixed-point matching objectives such as iDEM [1], adjoint sampling [2], flow and the bridge matching sampler [3]. Empirically, the iteration converges from a randomly initialized policy on all 24 online tasks for both the pure and the mixed target, while the alternative targets in the same training loop do not (Fig. 8). --- > Even a 1-D toy example would help, ideally with a case where it fails. We added the one-dimensional example the reviewer asks for (App. B.2, Fig. 5). For a Gaussian target it has a closed form where every iteration is shrinking the offset between the policy mean and the target mean by a constant factor, which depends on the step size and on the mixing schedule. For a bimodal target with overlapping modes, the mixed target converges to the Boltzmann policy from four different initial policies, up to a small error from the discretization and the softening. The example also contains two failure cases: - Without damping ($\eta = 1$) the update overshoots and oscillates. - For well-separated modes the iteration recovers the shape of each mode but not their weights, and a policy that starts on one mode stays on it.  Both cases are described with the example (App. B.2). The limitations (Sec. 7) now state that a theoretical analysis of the convergence is left for future work. --- > The paper repeatedly describes FFM as conceptually simple and easy to implement, but the method that is actually run is quite far from the single regression presented in Section 4. […] in the offline-to-online setting τQ, the normalizer, the trust region, and ϵTR all change between locomotion and manipulation (Table 4) [...] I am not sure how one would choose these settings on a new domain, and I think the paper should be more upfront about this. We agree that the paper should be more upfront, and we revised it accordingly. The Limitations paragraph now states that the offline-to-online experiments need two different settings, one for locomotion and one for manipulation, and names the components that the practical instantiations add to the regression of Sec. 4.  We would like to add two remarks. First, most of these components are not specific to FFM. Trust regions, action squashing and distributional critics are shared with the on-policy baselines. With all of them fixed, the target ablation shows that it is the regression of Sec. 4 that matters (Fig. 8). Second, Table 4 is one two-valued choice per domain family, made once for all locomotion and all manipulation tasks, without per-task tuning. Reply to Reviewer H3hv (3/3)
Eq. 13 is 0/0 at t=1. In practice, you evaluate it at t=0.9 (Appendix E.3.4). So the score being used is that of a smoothed version of pref rather than pref itself, and dividing by κ(0.9)≈0.11 amplifies any error in vref by roughly an order of magnitude. How sensitive are the results to δref, and did you observe any instability from this amplification?
We did not observe any instability from this amplification. To quantify the sensitivity, we added a sweep over on antmaze-large to the revision (App. E.5, Table 7), with the full protocol (1M offline and 500K online steps, three seeds, success averaged over the last 150K steps of each phase):
| 0.02 | 0.05 | 0.1 (paper) | 0.2 | 0.4 | |
|---|---|---|---|---|---|
| antmaze-large, offline | 0.95 ± 0.01 | 0.95 ± 0.01 | 0.94 ± 0.00 | 0.94 ± 0.01 | 0.93 ± 0.03 |
| antmaze-large, online | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 |
On this task the success does not depend on over the whole range, in either phase. Reading the score slightly before the endpoint is common practice for numerical stability [4,5,6]; the value itself was an ad-hoc choice that we did not tune.
References
[1] Akhound-Sadegh et al. Iterated Denoising Energy Matching for Sampling from Boltzmann Densities. ICML 2024.
[2] Havens et al. Adjoint Sampling: Highly Scalable Diffusion Samplers via Adjoint Matching. ICML 2025.
[3] Blessing et al. Bridge Matching Sampler: Scalable Sampling via Generalized Fixed-Point Diffusion Matching. ICML 2026.
[4] Poole et al. DreamFusion: Text-to-3D using 2D Diffusion. ICLR 2023.
[5] Song et al. Consistency Models. ICML 2023.
[6] Yin et al. One-step Diffusion with Distribution Matching Distillation. CVPR 2024.
We hope these clarifications and additional experiments address the reviewer's concerns. If they do, we would be grateful if the reviewer would consider raising their score. We would like to thank the reviewer again for the detailed review, which has helped us improve our manuscript substantially. We are happy to answer any further questions during the rebuttal period.
Thanks for the detailed response and the new experiments. They resolve most of my concerns, so I have raised my score.
That said, in my view, the method actually used is not as simple as the paper suggests, since it includes several components beyond the core regression, such as the Fisher normalizer, the softened denominators, and the reference score at . I believe explaining why they are needed and what each contributes would make it a stronger paper.
Reply to Reviewer H3hv (1/3)
We thank the reviewer for their careful reading and constructive feedback. Following the review, we have added the requested ablations and extended the discussion of the method. Below, we address each point in turn and indicate the corresponding changes in the revised manuscript.
We ran this ablation on all 24 online tasks with 10 seeds each (App. D.4, Fig. 8). Only the actor's regression target changes. Critic, rollouts, entropy estimate and temperature adaptation, trust region, schedule, network, optimizer and deterministic evaluation are identical. The final IQM with 95% confidence intervals is reported below.
On DMC, importance-weighted regression is close to FFM, and the DIPO- and QSM-style targets are clearly worse. On the high-dimensional humanoid tasks only the FFM targets learn. With everything else fixed, the FFM target is thus the only one of the compared targets that trains reliably on both suites.
The first two rows of the table are this comparison. The two targets perform on par on both suites (0.879 against 0.896 on DMC, 35.4 against 35.1 on the humanoid tasks). The control variate of the mixed target reduces the variance of the target but is not the source of the performance.
We thank the reviewer for the question. We added an ablation of FFM without the trust region in Fig. 9 ("FFM w/o TR", App. D.4). Without it, FFM still learns on most DMC tasks, with a moderately lower aggregate return, but it does not learn on the high-dimensional humanoid tasks. The trust region is therefore essential in high dimensions, which agrees with the role of trust regions in on-policy methods in general.
Regarding the deterministic evaluation: Fig. 16 to 18 compare it with stochastic evaluation. The two are close on DMC, and the deterministic evaluation leads to slighly better performance on the humanoid tasks. It changes only how the trained policy is evaluated, not how it is trained.
We added both (Sec. 6, Fig. 4). We compare one update of FFM with the adjoint method, as used by TRQAM, in an implementation that matches FFM's memory, and with backpropagation through the unrolled flow (BPTT), which minimizes the reverse KL as DIME, TRUDI and FLAC do, on identical networks and as a function of the number of integration steps and the actor size. FFM regresses on a fixed number of interpolant samples and generates its actions without gradients, so its memory is constant in the number of steps and its time grows only with the gradient-free forward generation. The adjoint method has to solve the adjoint equation backward through the flow, one vector-Jacobian product per step, so its time grows linearly with the number of steps even with matched memory; BPTT stores the activations of every step, so its time and its memory grow linearly with the steps and with the actor size. At 80 steps the alternatives take two to three times longer than FFM and BPTT needs three times the memory, and the gap widens with every further step and parameter. This is what lets FFM scale to larger actors and more integration steps without a corresponding growth in cost.