- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Soft-Value Flow Matching (SVFM), an offline reinforcement learning method that improves multi-step flow policies without backpropagating through sampling trajectories. SVFM estimates a conditional soft-value function by evaluating critic values at the endpoints of stochastic continuations from a fixed behavior policy, reducing exposure to out-of-distribution critic errors. The learned soft-value gradient guides actor training through a flow-matching objective, while an adaptive temperature adjusts guidance strength according to critic-value variation. Experiments demonstrate improved average success rates over existing baselines on OGBench, LIBERO, and LIBERO-Plus, along with promising results in real-world robotic manipulation and VLA fine-tuning.
Critical strengths
- The value learning approach is novel. The theory of Stochastic Optimal Control (SOC) is well-used, with the proposed critic processing intermediate actions and time steps.
- Mitigates the need for backward vector—Jacobian product (VJP) propagation of adjoint matching.
- Results on OGBench cube-quadruple are outstanding, and the results with VLAs are also promising.
Critical weaknesses
- It is hard to find a clear insight into why the authors came up with the Doob h-transform and time-step-conditioned critic. Strictly speaking, SVFM is solving the same mathematical problem as QAM in terms of policy learning, but more inefficiently.
- Computational inefficiency is of concern since SVFM requires twice the training runtime and slower inference than Q-Flow and QAM. I believe this is attributable to the number of continuations () and additional engineering details, e.g., adaptive temperature.
- It is hard to observe how much the formulation and adaptive temperature affect the final task performance. These two components can also be applied to other methods if they use multiple terminal action samples per training step. More analyses are needed on whether SVFM’s high final performance comes from adaptive temperature or from the soft-value network.
Questions for authors
Please answer the following questions using the default cube-quadruple task in OGBench, under the original QAM experimental setting.
- Q.1. How efficient is SVFM with ? Following the settings in Table 24, please compare (1) QAM, (2) Q-Flow, (3) SVFM with , and (4) SVFM with in terms of final task performance and training/inference runtime.
- Q.2. How does SVFM perform without and adaptive temperature? Specifically, I am curious how SVFM performs when we set and tune directly without using .
- Q.3. How do and adaptive temperature work on QAM? Please compare SVFM with two QAM variants that tune for and with : (1) use action samples originated from the same noise (prior) sample, (2) use action samples originated from different noise (prior) samples.
- Q.4. Under the setting of Q.2 and Q.3, please compare the following four cases: (1) QAM with current policy sampling for value learning, (2) QAM with behavior policy sampling for value learning, (3) SVFM with current policy sampling for value learning, and (4) SVFM with behavior policy sampling for value learning (proposed by the authors). Do not use adaptive temperature for any of the four cases.
6 comments
Response #2
[W2, Q1] Results with and computational cost.
| Method | Hyperparameters | Success rate (%) | ms / update | hours / 1M updates | Inference (ms) |
|---|---|---|---|---|---|
| QAM | 0.0 | 5.82 | 1.62 | 0.87 | |
| Q-Flow | 9.3 | 5.72 | 1.59 | 0.84 | |
| SVFM () | , | 64.2 | 6.94 | 1.93 | 0.87 |
| SVFM () | , | 74.0 | 10.89 | 3.02 | 0.85 |
Table a. Performance and computational cost of SVFM, QAM and Q-Flow. Cube-quadruple, default task (advanced setting). Success rates averaged over 4 seeds. Training and inference times measured on one RTX 3090.
We compare SVFM with and , holding and fixed, and report task success, training cost, and inference latency alongside QAM and Q-Flow. Reducing from 8 to 2 lowers the training cost from 10.89 to 6.94 ms/update and from 3.02 to 1.93 hours per 1M updates, a reduction of approximately 36%. The success rate decreases from 74.0% to 64.2%, demonstrating a practical trade-off between training cost and task performance. Compared with the baselines, SVFM with still substantially outperforms QAM (0.0%) and Q-Flow (9.3%), while both training-cost metrics are only approximately those of QAM.
We would also like to clarify the concern about slower inference. The continuation samples are used only at training time, not at inference time. SVFM incorporates soft-value guidance into the actor during training. At inference, only this trained actor is used, with no continuation sampling or evaluations of the critic or soft-value network. Consequently, SVFM adds no inference-time overhead; all four methods take 0.84–0.87 ms per policy call. The additional continuation-sampling cost therefore affects training, rather than deployment speed. We have added this clarification to Section 3, under “Policy improvement via the probability-flow ODE.”
[W3, Q2] Effects of and adaptive temperature.
| Method | Hyperparameters | Temperature | Success rate (%) |
|---|---|---|---|
| SVFM, tuned | , | Adaptive () | 74.0 |
| SVFM, tuned | , | Constant () | 25.8 |
| SVFM, | , | Constant () | 17.7 |
Table b. Adaptive versus constant temperature in SVFM with tuned or fixed .
has two roles in SVFM: (1) it controls continuation stochasticity; and (2) together with and , it determines the adaptive temperature . To examine their contributions to the performance gains, we additionally evaluate a constant-temperature variant with tuned, alongside the reviewer-requested variant with and constant temperature.
Adaptive temperature makes a substantial practical difference: comparing the best configurations found for each variant, SVFM achieves 74.0% success with adaptive temperature versus 25.8% with constant temperature. Importantly, even the reviewer-requested variant with and constant temperature achieves 17.7%, exceeding Q-Flow’s 9.3%. Section 4.3 (Table 3) also reports higher average success with adaptive temperature under the standard setting.
| Method | Hyperparameters | Guidance coefficient | Success rate (%) |
|---|---|---|---|
| Q-Flow | Constant () | 9.3 | |
| Q-Flow + adaptive guidance | , | Adaptive () | 19.8 |
Table c. Applying the adaptive rule to Q-Flow’s guidance coefficient.
We additionally apply the adaptive rule used for SVFM’s temperature to Q-Flow’s guidance coefficient to test whether its benefit extends beyond SVFM. The best configuration found in this sweep achieves 19.8% success, compared with 9.3% for the original Q-Flow baseline. This indicates that adapting the guidance coefficient can benefit Q-Flow as well.
Q-Flow + adaptive guidance replaces its fixed guidance coefficient with . Here, is estimated as in SVFM, using continuations of the -scaled SDE from noised dataset actions; these continuations use Q-Flow’s current policy because Q-Flow has no separate behavior flow.
Response #3
[W3, Q3] Applying and adaptive temperature to QAM.
| Method | Prior noise | Hyperparameters | Success rate (%) |
|---|---|---|---|
| QAM | – | 0.0 | |
| QAM + adaptive temperature | same | , | 0.0 |
| QAM + adaptive temperature | different | , | 0.0 |
| QAM + adaptive temperature + SDE | same | , | 1.3 |
| QAM + adaptive temperature + SDE | different | , | 1.0 |
| SVFM | - | , | 74.0 |
Table d. Applying adaptive temperature and -scaling to QAM.
We augment QAM with adaptive temperature and, additionally, -scaling of its adjoint-matching SDE, evaluating both shared and independent prior-noise samples for temperature estimation. In QAM + adaptive temperature, is used only for temperature estimation; QAM’s adjoint-matching SDE is unchanged. The aim is to test whether transferring these components to QAM can account for SVFM’s performance advantage.
On cube-quadruple, adaptive temperature alone leaves QAM at 0.0% success, while additionally applying -scaling to its adjoint-matching SDE yields only 1.0–1.3%, compared with 74.0% for SVFM. Estimating the temperature using the same noised-data-anchor continuations as SVFM also yields only 0.2% for QAM (, ). Thus, transferring adaptive temperature and -scaling to QAM does not close the performance gap.
Although QAM and SVFM target the same optimal control for a fixed critic, temperature, and reference process, they construct actor supervision differently. QAM samples trajectories using the current actor and evaluates critic gradients at their endpoints. It then propagates these gradients backward along the sampled trajectories using the behavior-flow Jacobians to construct actor-supervision targets. SVFM instead learns a soft value from critic values at the endpoints of fixed behavior continuations and uses its input gradient to supervise the actor. This makes its guidance-target construction independent of the current actor. These results indicate that SVFM’s empirical advantage cannot be explained by adaptive temperature or -scaling alone.
[Q4] Behavior-policy versus current-policy sampling.
| Case | Method | Value sampling | Hyperparameters | Success rate (%) |
|---|---|---|---|---|
| 1 | QAM + SDE | Current policy | , | 2.2 |
| 2 | QAM + SDE | Behavior policy | , | 0.3 |
| 3 | SVFM | Current policy | , | 17.7 |
| 4 | SVFM | Behavior policy | , | 25.8 |
Table e. Behavior-policy versus current-policy sampling for value learning in QAM and SVFM.
We compare current-policy and behavior-policy sampling for value learning in both QAM and SVFM, using constant temperature. The aim is to assess the role of the sampling source in each method and whether behavior-policy sampling alone closes the performance gap between QAM and SVFM. Here “value learning” refers to the value that guides the actor: the soft value estimated from continuations in SVFM, and, in QAM, the value implicit in adjoint matching, whose lean adjoint is propagated along trajectories sampled from the current actor or from the behavior flow. In all four cases, the TD critic uses current-policy next actions.
For SVFM, behavior-policy continuations improve success from 17.7% to 25.8% with and held fixed. QAM achieves only 2.2% with current-policy sampling and 0.3% with behavior-policy sampling under the selected configurations. These results support the benefit of behavior-policy continuations within SVFM, while showing that adopting behavior-policy sampling alone does not make QAM competitive with SVFM.
Response #1
Thank you for recognizing the novelty of our value-learning approach and the promising results on cube-quadruple and VLA fine-tuning. Below, we address your concerns and report the requested experimental comparisons.
We evaluated the selected configurations using four seeds (10001, 20002, 30003, and 40004). Tables report mean success rates across these four evaluation seeds. We tuned κ and g within the search ranges specified in our paper: κ∈{0.2,0.4,0.6,0.8,1.0} and g∈{0.25,0.5}. Constant λ was tuned over the range used by QAM [2], {1/30,0.1,1/3,1,10/3,10}. For the original baselines, we used Q-Flow’s guidance coefficient λ=0.5 [3] and QAM’s temperature λ=1 [2], as reported in their original papers.
[W1] Motivation for introducing the Doob h-transform and time-conditioned soft-value network.
Motivation. SVFM and QAM [2] share the same KL-regularized policy-improvement objective, but differ in how they estimate optimal control. In QAM, actor updates also move the critic-query distribution used for actor supervision. QAM queries terminal critic action gradients on current-actor trajectories and propagates a lean adjoint backward through Jacobians of the base/behavior drift. In offline RL, these queries may move outside the data, where critic values may be overestimated. Its regression target matches optimal control in expectation at the optimal-actor fixed point.
Decoupled target construction. SVFM addresses this dependence by sampling continuations from a fixed behavior process and querying terminal critic values. For a fixed critic, temperature, and reference process, actor updates do not directly shift this conditional query distribution. The TD backup remains the same. This decouples target construction from the trained actor, but does not eliminate critic error, reference coverage limitations, or the finite-K Monte Carlo soft-value bias analyzed in Appendix A.3.
Why the h-transform and time conditioning. The h-transform expresses optimal control for the shared KL-regularized policy-improvement problem through the input gradient of the conditional soft value defined under behavior continuations in our Eq. (7). Our soft value and the corresponding optimal control are the same objects characterized in Eqs. (16)–(17) of Adjoint Matching [1]. This identity motivates learning the scalar Vψ(s,xt,t), whose input gradient supplies actor supervision with the prescribed scaling, instead of constructing targets by backward adjoint propagation. Time conditioning is necessary because the conditional terminal law depends on the intermediate action and remaining generation time.
Supporting evidence. Appendix M.2, Table 21 compares behavior and current-actor continuations within the same SVFM framework on the default task of each of ten domains in the standard setting, with all other components and hyperparameters unchanged (3 seeds): mean success is 56.3% versus 51.3%, respectively. This supports choosing behavior continuations within SVFM; it does not isolate or fully explain the overall performance gap between QAM and SVFM.
Efficiency. With K=2, SVFM (64.2%) outperforms QAM (0.0%) and Q-Flow (9.3%) at 1.19× QAM’s training cost, with no additional inference-time overhead. See our response to [W2, Q1] below for details.
We have added this explanation of our motivation and the difference from QAM to Section 3, under “Difference from QAM.”