acceptodds
Under review as a conference paper at ICLR 2027

Soft-Value Flow Matching for Offline Reinforcement Learning

Abstract

Flow-matching policies represent multimodal actions, but improving them with a learned critic in offline reinforcement learning faces two difficulties: backpropagating through multi-step sampling is costly and unstable, and critic queries on actor-generated actions can expose the policy to out-of-distribution value errors. We introduce Soft-Value Flow Matching (SVFM), which trains a multi-step actor with the gradient of a conditional soft value for KL-regularized policy improvement. Starting from noised data actions, SVFM samples stochastic continuations of a fixed behavior process and evaluates the critic at their endpoints, keeping the queries used for guidance independent of the current actor. A log-mean-exp of these values provides a Monte Carlo target for a soft-value network, whose gradient guides actor updates without trajectory backpropagation. We further propose an adaptive temperature that scales with the spread of critic values across continuations. SVFM achieves higher average success rates than the compared baselines on OGBench, LIBERO, and LIBERO-Plus. It also improves real-world manipulation with both task-specific flow policies and a pretrained VLA model.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, robotics, flow matching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jFujOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Soft-Value Flow Matching (SVFM), an offline reinforcement learning method that improves multi-step flow policies without backpropagating through sampling trajectories. SVFM estimates a conditional soft-value function by evaluating critic values at the endpoints of stochastic continuations from a fixed behavior policy, reducing exposure to out-of-distribution critic errors. The learned soft-value gradient guides actor training through a flow-matching objective, while an adaptive temperature adjusts guidance strength according to critic-value variation. Experiments demonstrate improved average success rates over existing baselines on OGBench, LIBERO, and LIBERO-Plus, along with promising results in real-world robotic manipulation and VLA fine-tuning.

Critical strengths
  • The value learning approach is novel. The theory of Stochastic Optimal Control (SOC) is well-used, with the proposed critic processing intermediate actions and time steps.
  • Mitigates the need for backward vector—Jacobian product (VJP) propagation of adjoint matching.
  • Results on OGBench cube-quadruple are outstanding, and the results with VLAs are also promising.
Critical weaknesses
  • It is hard to find a clear insight into why the authors came up with the Doob h-transform and time-step-conditioned critic. Strictly speaking, SVFM is solving the same mathematical problem as QAM in terms of policy learning, but more inefficiently.
  • Computational inefficiency is of concern since SVFM requires twice the training runtime and slower inference than Q-Flow and QAM. I believe this is attributable to the number of continuations (KK) and additional engineering details, e.g., adaptive temperature.
  • It is hard to observe how much the κ\kappa formulation and adaptive temperature affect the final task performance. These two components can also be applied to other methods if they use multiple terminal action samples per training step. More analyses are needed on whether SVFM’s high final performance comes from adaptive temperature or from the soft-value network.
Questions for authors

Please answer the following questions using the default cube-quadruple task in OGBench, under the original QAM experimental setting.

  • Q.1. How efficient is SVFM with K=2K=2? Following the settings in Table 24, please compare (1) QAM, (2) Q-Flow, (3) SVFM with K=2K=2, and (4) SVFM with K=8K=8 in terms of final task performance and training/inference runtime.
  • Q.2. How does SVFM perform without κ\kappa and adaptive temperature? Specifically, I am curious how SVFM performs when we set κ=1\kappa=1 and tune λ\lambda directly without using sQs_Q.
  • Q.3. How do κ\kappa and adaptive temperature work on QAM? Please compare SVFM with two QAM variants that tune for κ\kappa and gg with K=8K=8: (1) use KK action samples originated from the same noise (prior) sample, (2) use KK action samples originated from different noise (prior) samples.
  • Q.4. Under the setting of Q.2 and Q.3, please compare the following four cases: (1) QAM with current policy sampling for value learning, (2) QAM with behavior policy sampling for value learning, (3) SVFM with current policy sampling for value learning, and (4) SVFM with behavior policy sampling for value learning (proposed by the authors). Do not use adaptive temperature for any of the four cases.
6 comments
AuthorsOpenReview ↗

Response #1

Thank you for recognizing the novelty of our value-learning approach and the promising results on cube-quadruple and VLA fine-tuning. Below, we address your concerns and report the requested experimental comparisons.

We evaluated the selected configurations using four seeds (10001, 20002, 30003, and 40004). Tables report mean success rates across these four evaluation seeds. We tuned κ\kappa and gg within the search ranges specified in our paper: κ∈{0.2,0.4,0.6,0.8,1.0}\kappa \in \lbrace 0.2, 0.4, 0.6, 0.8, 1.0\rbrace and g∈{0.25,0.5}g \in \lbrace 0.25, 0.5\rbrace. Constant λ\lambda was tuned over the range used by QAM [2], {1/30,0.1,1/3,1,10/3,10}\lbrace 1/30, 0.1, 1/3, 1, 10/3, 10\rbrace. For the original baselines, we used Q-Flow’s guidance coefficient λ=0.5\lambda=0.5 [3] and QAM’s temperature λ=1\lambda=1 [2], as reported in their original papers.


[W1] Motivation for introducing the Doob hh-transform and time-conditioned soft-value network.

Motivation. SVFM and QAM [2] share the same KL-regularized policy-improvement objective, but differ in how they estimate optimal control. In QAM, actor updates also move the critic-query distribution used for actor supervision. QAM queries terminal critic action gradients on current-actor trajectories and propagates a lean adjoint backward through Jacobians of the base/behavior drift. In offline RL, these queries may move outside the data, where critic values may be overestimated. Its regression target matches optimal control in expectation at the optimal-actor fixed point.

Decoupled target construction. SVFM addresses this dependence by sampling continuations from a fixed behavior process and querying terminal critic values. For a fixed critic, temperature, and reference process, actor updates do not directly shift this conditional query distribution. The TD backup remains the same. This decouples target construction from the trained actor, but does not eliminate critic error, reference coverage limitations, or the finite-KK Monte Carlo soft-value bias analyzed in Appendix A.3.

Why the hh-transform and time conditioning. The hh-transform expresses optimal control for the shared KL-regularized policy-improvement problem through the input gradient of the conditional soft value defined under behavior continuations in our Eq. (7). Our soft value and the corresponding optimal control are the same objects characterized in Eqs. (16)–(17) of Adjoint Matching [1]. This identity motivates learning the scalar Vψ(s,xt,t)V_\psi(s,x_t,t), whose input gradient supplies actor supervision with the prescribed scaling, instead of constructing targets by backward adjoint propagation. Time conditioning is necessary because the conditional terminal law depends on the intermediate action and remaining generation time.

Supporting evidence. Appendix M.2, Table 21 compares behavior and current-actor continuations within the same SVFM framework on the default task of each of ten domains in the standard setting, with all other components and hyperparameters unchanged (3 seeds): mean success is 56.3% versus 51.3%, respectively. This supports choosing behavior continuations within SVFM; it does not isolate or fully explain the overall performance gap between QAM and SVFM.

Efficiency. With K=2K=2, SVFM (64.2%) outperforms QAM (0.0%) and Q-Flow (9.3%) at 1.19×1.19\times QAM’s training cost, with no additional inference-time overhead. See our response to [W2, Q1] below for details.

We have added this explanation of our motivation and the difference from QAM to Section 3, under “Difference from QAM.”

AuthorsOpenReview ↗

Response #2

[W2, Q1] Results with K=2K=2 and computational cost.

MethodHyperparametersSuccess rate (%) ↑\uparrowms / update ↓\downarrowhours / 1M updates ↓\downarrowInference (ms) ↓\downarrow
QAMλ=1\lambda = 10.05.821.620.87
Q-Flowλ=0.5\lambda = 0.59.35.721.590.84
SVFM (K=2K=2)κ=0.2\kappa = 0.2, g=0.25g = 0.2564.26.941.930.87
SVFM (K=8K=8)κ=0.2\kappa = 0.2, g=0.25g = 0.2574.010.893.020.85

Table a. Performance and computational cost of SVFM, QAM and Q-Flow. Cube-quadruple, default task (advanced setting). Success rates averaged over 4 seeds. Training and inference times measured on one RTX 3090.

We compare SVFM with K=2K=2 and K=8K=8, holding κ\kappa and gg fixed, and report task success, training cost, and inference latency alongside QAM and Q-Flow. Reducing KK from 8 to 2 lowers the training cost from 10.89 to 6.94 ms/update and from 3.02 to 1.93 hours per 1M updates, a reduction of approximately 36%. The success rate decreases from 74.0% to 64.2%, demonstrating a practical trade-off between training cost and task performance. Compared with the baselines, SVFM with K=2K=2 still substantially outperforms QAM (0.0%) and Q-Flow (9.3%), while both training-cost metrics are only approximately 1.19×1.19\times those of QAM.

We would also like to clarify the concern about slower inference. The KK continuation samples are used only at training time, not at inference time. SVFM incorporates soft-value guidance into the actor during training. At inference, only this trained actor is used, with no continuation sampling or evaluations of the critic or soft-value network. Consequently, SVFM adds no inference-time overhead; all four methods take 0.84–0.87 ms per policy call. The additional continuation-sampling cost therefore affects training, rather than deployment speed. We have added this clarification to Section 3, under “Policy improvement via the probability-flow ODE.”


[W3, Q2] Effects of κ\kappa and adaptive temperature.

MethodHyperparametersTemperatureSuccess rate (%)
SVFM, tuned κ\kappaκ=0.2\kappa=0.2, g=0.25g=0.25Adaptive (κ2sQ/g\kappa^2s_Q/g)74.0
SVFM, tuned κ\kappaκ=0.6\kappa=0.6, λ=1/3\lambda=1/3Constant (λ\lambda)25.8
SVFM, κ=1\kappa=1κ=1\kappa=1, λ=1/3\lambda=1/3Constant (λ\lambda)17.7

Table b. Adaptive versus constant temperature in SVFM with tuned or fixed κ\kappa.

κ\kappa has two roles in SVFM: (1) it controls continuation stochasticity; and (2) together with sQs_Q and gg, it determines the adaptive temperature λ=κ2sQ/g\lambda=\kappa^2 s_Q/g. To examine their contributions to the performance gains, we additionally evaluate a constant-temperature variant with κ\kappa tuned, alongside the reviewer-requested variant with κ=1\kappa=1 and constant temperature.

Adaptive temperature makes a substantial practical difference: comparing the best configurations found for each variant, SVFM achieves 74.0% success with adaptive temperature versus 25.8% with constant temperature. Importantly, even the reviewer-requested variant with κ=1\kappa=1 and constant temperature achieves 17.7%, exceeding Q-Flow’s 9.3%. Section 4.3 (Table 3) also reports higher average success with adaptive temperature under the standard setting.

MethodHyperparametersGuidance coefficientSuccess rate (%)
Q-Flowλ=0.5\lambda=0.5Constant (λ\lambda)9.3
Q-Flow + adaptive guidanceκ=0.6\kappa=0.6, g=0.25g=0.25Adaptive (κ2sQ/g\kappa^2s_Q/g)19.8

Table c. Applying the adaptive rule to Q-Flow’s guidance coefficient.

We additionally apply the adaptive rule used for SVFM’s temperature to Q-Flow’s guidance coefficient to test whether its benefit extends beyond SVFM. The best configuration found in this sweep achieves 19.8% success, compared with 9.3% for the original Q-Flow baseline. This indicates that adapting the guidance coefficient can benefit Q-Flow as well.

Q-Flow + adaptive guidance replaces its fixed guidance coefficient λ\lambda with λ=κ2sQ/g\lambda=\kappa^2 s_Q/g. Here, sQs_Q is estimated as in SVFM, using K=8K=8 continuations of the κ\kappa-scaled SDE from noised dataset actions; these continuations use Q-Flow’s current policy because Q-Flow has no separate behavior flow.

AuthorsOpenReview ↗

Response #3

[W3, Q3] Applying κ\kappa and adaptive temperature to QAM.

MethodPrior noiseHyperparametersSuccess rate (%)
QAM–λ=1\lambda=10.0
QAM + adaptive temperaturesameκ=0.2\kappa=0.2, g=0.25g=0.250.0
QAM + adaptive temperaturedifferentκ=0.2\kappa=0.2, g=0.25g=0.250.0
QAM + adaptive temperature + SDE κ\kappasameκ=0.2\kappa=0.2, g=0.5g=0.51.3
QAM + adaptive temperature + SDE κ\kappadifferentκ=0.2\kappa=0.2, g=0.5g=0.51.0
SVFM-κ=0.2\kappa=0.2, g=0.25g=0.2574.0

Table d. Applying adaptive temperature and κ\kappa-scaling to QAM.

We augment QAM with adaptive temperature and, additionally, κ\kappa-scaling of its adjoint-matching SDE, evaluating both shared and independent prior-noise samples for temperature estimation. In QAM + adaptive temperature, κ\kappa is used only for temperature estimation; QAM’s adjoint-matching SDE is unchanged. The aim is to test whether transferring these components to QAM can account for SVFM’s performance advantage.

On cube-quadruple, adaptive temperature alone leaves QAM at 0.0% success, while additionally applying κ\kappa-scaling to its adjoint-matching SDE yields only 1.0–1.3%, compared with 74.0% for SVFM. Estimating the temperature using the same noised-data-anchor continuations as SVFM also yields only 0.2% for QAM (κ=0.6\kappa=0.6, g=0.5g=0.5). Thus, transferring adaptive temperature and κ\kappa-scaling to QAM does not close the performance gap.

Although QAM and SVFM target the same optimal control for a fixed critic, temperature, and reference process, they construct actor supervision differently. QAM samples trajectories using the current actor and evaluates critic gradients at their endpoints. It then propagates these gradients backward along the sampled trajectories using the behavior-flow Jacobians to construct actor-supervision targets. SVFM instead learns a soft value from critic values at the endpoints of fixed behavior continuations and uses its input gradient to supervise the actor. This makes its guidance-target construction independent of the current actor. These results indicate that SVFM’s empirical advantage cannot be explained by adaptive temperature or κ\kappa-scaling alone.


[Q4] Behavior-policy versus current-policy sampling.

CaseMethodValue samplingHyperparametersSuccess rate (%)
1QAM + SDE κ\kappaCurrent policyκ=0.4\kappa=0.4, λ=0.1\lambda=0.12.2
2QAM + SDE κ\kappaBehavior policyκ=0.6\kappa=0.6, λ=1/3\lambda=1/30.3
3SVFMCurrent policyκ=0.6\kappa=0.6, λ=1/3\lambda=1/317.7
4SVFMBehavior policyκ=0.6\kappa=0.6, λ=1/3\lambda=1/325.8

Table e. Behavior-policy versus current-policy sampling for value learning in QAM and SVFM.

We compare current-policy and behavior-policy sampling for value learning in both QAM and SVFM, using constant temperature. The aim is to assess the role of the sampling source in each method and whether behavior-policy sampling alone closes the performance gap between QAM and SVFM. Here “value learning” refers to the value that guides the actor: the soft value estimated from K=8K=8 continuations in SVFM, and, in QAM, the value implicit in adjoint matching, whose lean adjoint is propagated along trajectories sampled from the current actor or from the behavior flow. In all four cases, the TD critic uses current-policy next actions.

For SVFM, behavior-policy continuations improve success from 17.7% to 25.8% with κ\kappa and λ\lambda held fixed. QAM achieves only 2.2% with current-policy sampling and 0.3% with behavior-policy sampling under the selected configurations. These results support the benefit of behavior-policy continuations within SVFM, while showing that adopting behavior-policy sampling alone does not make QAM competitive with SVFM.

Reviewer_YZxYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Soft-Value Flow Matching (SCFM) for improving flow policies in offline RL. It samples stochastic continuations of a fixed behavior process from noisy data actions and evaluates the critic at their endpoints. These values train a soft-value network, whose gradient guides the actor without backpropagation through the sampling trajectory. An adaptive temperature changes with the spread of critic values. Experiments cover OGBench, LIBERO, LIBERO-Plus, and real-world manipulation.

Critical strengths
  1. The problem is clearly defined, and the approach to solving it is also clear. Using fixed behavior continuations prevents actor updates from directly shifting the critic queries used for guidance. The toy experiment and behavior-versus-actor continuation ablation provide useful evidence for this design.

  2. The mathematical analysis is clearly connected to the objective. The paper derives the conditional soft value, probability-flow correction, and guided regression objective from KL-regularized policy improvement. It also distinguishes its exact population results from practical approximations.

  3. The results are strong. SVFM improves average success from Q-Flow’s 53% to 57% in the standard setting and from 47% to 56% in the advanced setting, using eight and twelve seeds respectively (Tables 1–2).

  4. The paper includes large VLA and real robot experiments, including comparisons with Filtered BC and additional BC training.

Critical weaknesses
  1. The paper explicitly states in the introduction that BPTT is costly, yet subsequent results indicate that SVFM requires multi-sample processing, which may be even more computationally expensive.
Questions for authors
  1. How closely does the learned soft-value gradient match a higher-sample reference during training, especially after substantial changes in the critic or temperature?

  2. Does the SVFM method have any special requirements for the diversity of offline datasets or online actor rollouts?

5 comments
AuthorsOpenReview ↗

Response #1

Thank you for recognizing the clear formulation, the mathematical analysis, and the strong empirical results. Below, we address your concerns and questions.


[W1] The “costly BPTT” wording.

Thank you for pointing out our imprecise wording. We acknowledge that describing BPTT as “costly” was too strong: SVFM additionally samples KK continuations and trains a soft-value network, so avoiding BPTT does not necessarily lower the overall training cost. We have revised the abstract accordingly.

Our motivation was to avoid two requirements of BPTT: (1) propagating gradients backward through every denoising step in sequence, and (2) storing the intermediate states of all steps for the backward pass. SVFM instead integrates KK continuations forward only, as a single batch, without retaining their intermediate states for backpropagation. Actor guidance is obtained from a single input-gradient of the soft-value network at each anchor.


[Q1] Accuracy of the learned soft-value gradient.

StepSoft-value error (↓\downarrow)Gradient cosine, mean / median (↑\uparrow)
100K0.080.57 / 0.67
200K0.070.62 / 0.73
300K0.060.60 / 0.72
400K0.070.62 / 0.70
500K0.060.66 / 0.74
600K0.070.65 / 0.76
700K0.070.69 / 0.78
800K0.060.70 / 0.79
900K0.070.70 / 0.81
1M0.060.69 / 0.81

Table a. Accuracy of SVFM’s soft value and gradient during training. Cube-quadruple, default task (advanced), 4 seeds, K=8K=8, 256 fixed held-out anchors.

We evaluate how accurately the learned soft value and its gradient match a higher-sample reference during training.

Soft-value error: ∣Vψ−Vref∣|V_\psi - V_{\text{ref}}| relative to the QQ scale. Gradient cosine: cosine similarity between ∇xVψ\nabla_x V_\psi and the reference gradient, mean / median over 256 held-out anchors. Averaged over 4 seeds.

At each held-out anchor (s,xt,t)(s, x_t, t), the higher-sample reference is the gradient with respect to xtx_t of the Monte Carlo soft value λlog⁡1K∑m=1Kexp⁡(Q(s,am)/λ)\lambda \log \frac{1}{K}\sum_{m=1}^{K} \exp(Q(s, a_m)/\lambda) computed from K=1024K=1024 continuations per anchor (K=8K=8 in training), with the training's temperature and critic aggregation. Using 2×2\times as many continuations (K=2048K=2048) leaves the reference essentially unchanged (median cosine 0.998).

The learned soft value closely tracks the reference, and its gradient becomes increasingly aligned with the reference during training.

StepSoft-value error (↓\downarrow)Gradient cosine, mean / median (↑\uparrow)Unchanged run: gradient cosine
500K0.0640.66 / 0.740.66 / 0.74
501K0.0770.63 / 0.72–
502K0.0730.64 / 0.72–
505K0.0710.61 / 0.73–
510K0.0670.60 / 0.74–
550K0.0650.62 / 0.740.66 / 0.78
600K0.0740.63 / 0.750.65 / 0.76
800K0.0630.70 / 0.810.70 / 0.79
1M0.0670.68 / 0.780.69 / 0.81

Table b. Accuracy of SVFM’s soft value and gradient after a 4×4\times temperature increase. Cube-quadruple, default task (advanced), 4 seeds, K=8K=8, λ×4\lambda\times4 at 500K updates.

We test how quickly the learned soft value and its gradient adapt to a sudden 4×4\times temperature increase at 500K updates. The soft-value error briefly rose from 0.064 to 0.077 and returned within about 10K updates; the gradient cosine dropped by at most 0.06 and recovered to the level of the unchanged run by 800K.

AuthorsOpenReview ↗

Response #2

[Q2] Diversity requirements for offline datasets and online actor rollouts.

(1) Diversity requirements for offline datasets.

SVFM performs well across different levels of behavioral diversity, from OGBench’s diverse play-style manipulation datasets collected by scripted policies to our task-specific real-world datasets with a narrower range of behaviors.

BC referenceMeanMedian
Cube-quadruple0.0098−0.0135-0.0135
Can-MH0.62900.6957

Table c. Cosine similarities between mean BC actions across states (translation components only).

We additionally evaluate SVFM on Robomimic Can-MH [1] to test whether its gains extend to a setting with lower action diversity. To show that Can-MH has lower action diversity than OGBench cube-quadruple, we compare the across-state variation of mean action directions in their learned Flow BC references (Table c). For each reference, we sample 64 five-step action chunks at each of 1,024 states and average the translation components of the sampled chunks. Cosine similarities between these averages indicate lower action diversity in Can-MH (mean cosine: 0.6290) than in cube-quadruple (0.0098).

MethodCan-MH hyperparametersCan-MH success (%)Cube-quadruple hyperparametersCube-quadruple success (%)
QAMλ=10/3\lambda = 10/377.8λ=1\lambda = 10.0
Q-Flowλ=2\lambda = 283.6λ=0.5\lambda = 0.59.3
SVFMκ=0.6\kappa = 0.6, g=0.5g = 0.589.2κ=0.2\kappa = 0.2, g=0.25g = 0.2574.0

Table d. Mean success rates (%) on Robomimic Can-MH (5 seeds, 500K checkpoint, 100 evaluation episodes per seed) and OGBench cube-quadruple (advanced default task, 4 seeds; Reviewer jFuj, Table a). Hyperparameters are selected separately for each benchmark.

We swept QAM’s λ∈{1/30,0.1,1/3,1,10/3,10}\lambda \in \lbrace 1/30, 0.1, 1/3, 1, 10/3, 10\rbrace [2], Q-Flow’s λ∈{0.2,0.5,1,2,5,10,20}\lambda \in \lbrace 0.2, 0.5, 1, 2, 5, 10, 20\rbrace [3], and SVFM’s κ∈{0.2,0.6,1}\kappa \in \lbrace 0.2, 0.6, 1\rbrace and g∈{0.25,0.5}g \in \lbrace 0.25, 0.5\rbrace.

SVFM achieves the highest mean success rate on Can-MH. Together with the real-world results, this shows that its empirical gains are not confined to the diverse play-style behaviors in OGBench.

(2) Diversity requirements for online actor rollouts.

Because SVFM is based on KL-regularized policy improvement, coverage by the reference policy matters. Online rollouts may visit actions poorly represented by a frozen offline reference, potentially reducing the reliability of behavior continuations and soft-value estimates at anchors constructed from these actions.

We therefore allow the reference to adapt during online fine-tuning. Appendix G.3 describes three options: unfreezing the BC flow and continuing flow-matching training on the replay buffer, periodically refreshing a policy copy, or using the current actor directly. In our reported offline-to-online experiments, we select either the current actor or a periodically refreshed copy per domain and observe further performance improvements during online learning.

These choices allow the reference to reflect changes in the actions visited online. Our results demonstrate their effectiveness in the evaluated settings(Figure 2.).

To further evaluate SVFM with more limited offline data and noisy online rollouts, we conduct an additional offline-to-online experiment on Robomimic Can-MH. We randomly subsample 100 of the original 300 demonstration episodes and add independent zero-mean Gaussian noise with a standard deviation of 0.1 to each action component at every online environment step.

We first pretrain the BC reference for 100K updates, followed by 500K offline training updates and 500K online environment steps. Throughout offline and online training, the reference flow continues to learn from replay-buffer actions through flow matching. We reuse the best κ\kappa and gg values from our offline experiments without retuning them for the online phase.

MethodNoisy env success(%)Clean env success(%)
SVFM(offline 500k)6984.6
SVFM(offline 500k+online 500k)8993.8

Table e. Mean success rates (%) on Robomimic Can-MH (5 seeds, 500K offline & 500k offline+500k online checkpoint, 100 evaluation episodes per seed)

These results show that SVFM can improve through noisy online interaction with a reduced offline demonstration set, achieving higher success rates in both noisy and clean evaluation environments.

AuthorsOpenReview ↗

Response #3

We would be grateful if you could let us know whether our response addresses your concerns. We are happy to discuss any remaining issues. We are prepared to make every effort to address them.

References

[1] Mandlekar et al., What Matters in Learning from Offline Human Demonstrations for Robot Manipulation, https://arxiv.org/abs/2108.03298

[2] Li and Levine, Q-learning with Adjoint Matching, https://arxiv.org/abs/2601.14234

[3] Doo et al., Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policy, https://arxiv.org/abs/2605.13435

Reviewer_54U6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed Soft-Value Flow Matching (SVFM), an approach to train offline flow policy without backpropagation through time. SVFM consists of training a QQ-function using TD-learning, constructing a Monte Carlo of the λ\lambda-scaled log of the Doob hh-transform of the QQ-function which is used to train a soft-value network. Then, the gradient of the soft-value network is plug into the standard flow matching objective to guide the policy improvement. The paper also proposed an adapting strategy for the λ\lambda parameter. SVFM is evaluated on OGBench. The method is also used to finetune pretrained VLA models which are evaluated on LIBERO and LIBERO-Plus.

Critical strengths

The proposed method is evaluated on multiple benchmarks including real work experiments, although the preliminaries can be smoother, the method is easy to understand.

Critical weaknesses
  • The contribution is moderate.
  • Lack of details about computational cost as each soft-value network update require sampling K=8K=8 actions by integrating the SDE, computing the gradients of the soft-vaue network in the policy objective, etc.
  • Missing algorithm's details: The algorithm for the SDE intergration (to sample the KK actions)
  • Introduction of two additionals hyperparameters (κ,g)(κ, g). For VLA finetune missing explanation of how actions chunking fit in the proposed method. Empirical results on OGbench are moderate compare with to baseline.
Questions for authors
  • can you please provide the missing details about the computational costs mentioned above ?
  • how is the action chunking integrated into your proposed method ?
6 comments
AuthorsOpenReview ↗

Response #1

Thank you for recognizing the clarity of our method and our evaluation on multiple benchmarks, including real-world experiments. Below, we respond to your concerns and questions.


[W1, W4] Contribution and empirical gains.

We would like to clarify our contributions and highlight the performance gains across multiple experimental settings. Our contributions are as follows: (1) We propose a soft-value-guided method for KL-regularized RL. By constructing actor supervision from fixed behavior continuations, SVFM prevents actor updates from directly shifting the critic-query distribution used for guidance. (2) We introduce an adaptive temperature that changes during training with the spread of critic values across behavior continuations. (3) Our method achieves the strongest overall performance in both offline and offline-to-online RL on OGBench, with results averaged over 8 or 12 random seeds depending on the setting. (4) We extend the approach to VLA fine-tuning on LIBERO and LIBERO-Plus and to real-world manipulation.

(1) An algorithm for KL-regularized RL.

MethodStates with queries above all behavior samples (%) ↓\downarrow
Q-Flow14.5
SVFM, continuations from the current actor5.6
SVFM1.6

Table a. Critic queries of Q-Flow and two SVFM variants. Standard setting, default task of each of the 10 domains, 3 seeds, averaged over domains. From Appendix I.2, Table 15 and Appendix M.2, Table 21.

The main difference from QAM [3] and Q-Flow [2] is where the critic is queried for actor supervision. QAM and Q-Flow construct actor supervision from current-actor samples, so actor updates directly shift the critic-query distribution used for guidance. As the actor changes during training, these actions may move outside the dataset, where the offline critic can overestimate their values. SVFM queries it at endpoints of continuations of a fixed behavior process started from noised data actions, and the resulting soft-value gradient supervises the actor. With exact components, this recovers the KL-regularized optimal policy (Proposition 1). We check whether the average critic value of the actions used for actor supervision exceeds the highest value among 32 behavior-policy samples. This occurs in 1.6% of states for SVFM, compared with 14.5% for Q-Flow (Appendix I.2, Table 15, default tasks). Continuations from the current actor raise this fraction to 5.6% and lower average success (Appendix M.2: SVFM 56.3% vs. actor continuations 51.3%).

(2) Adaptive temperature.

MethodHyperparametersGuidance coefficientSuccess rate (%)
Q-Flowλ=0.5\lambda=0.5Constant (λ\lambda)9.3
Q-Flow + adaptive guidanceκ=0.6\kappa=0.6, g=0.25g=0.25Adaptive (κ2sQ/g\kappa^2s_Q/g)19.8

Table b. Adaptive guidance in Q-Flow. Cube-quadruple, default task (advanced), 4 seeds. Adaptive variant: K=8K=8, κ=0.6\kappa=0.6, g=0.25g=0.25.

SVFM’s adaptive temperature scales with the spread of critic values across continuations. In Section 4.3 (Table 3), adaptive temperature achieves 56% average success, compared with 51% for the best constant temperature selected separately for each domain based on the evaluation results.

We apply SVFM’s adaptive rule to Q-Flow’s guidance coefficient to test its benefit beyond SVFM. The best configuration found in this sweep achieves 19.8% success, compared with 9.3% for the original Q-Flow baseline (Table b). Thus, adapting the guidance coefficient can benefit Q-Flow as well.

(3) Performance gains in offline and offline-to-online RL.

Advanced settingQAMQAM-EQ-FlowSVFM
Average success rate (%)43444756

Table c. Average success rates of QAM, QAM-E, Q-Flow and SVFM in the advanced setting. From Section 4.1, Table 2 (averaged over 10 domains ×\times 5 tasks, 12 seeds).

In the advanced setting (Section 4.1, Table 2), SVFM improves over the strongest baseline, Q-Flow, by 9 percentage points (Table c). This gain is 3×3\times the 3-point improvement that Q-Flow, the previous best, achieved over QAM-E [3].

The gap is largest on cube-quadruple, a long-horizon manipulation domain (SVFM 50% vs. Q-Flow 10%, best baseline QSM [4] 19%). SVFM also achieves the highest average success rate in the standard setting (Section 4.1, Table 1: SVFM 57% vs. Q-Flow 53%), with a particularly large gain on puzzle-3x3 (SVFM 68% vs. Q-Flow 48%). In offline-to-online RL, SVFM achieves the highest final average success rate across five OGBench domains after 1M online environment steps (SVFM approximately 93% vs. Q-Flow 78%; Figure 2, Section 4.2), with large gains on puzzle-4x4 during online fine-tuning.

(4) Extension to real-world manipulation and VLA fine-tuning. While QAM and Q-Flow were evaluated in simulation in their original papers, SVFM also works on a real robot and extends to VLA fine-tuning, where it improves on behavior cloning (GR00T N1.7: SVFM 99% vs. BC+ 73%).

AuthorsOpenReview ↗

Response #2

[W2, Q1] Computational cost of the proposed method.

MethodHyperparametersSuccess rate (%) ↑\uparrowms / update ↓\downarrowhours / 1M updates ↓\downarrowInference (ms) ↓\downarrow
QAMλ=1\lambda = 10.05.821.620.87
Q-Flowλ=0.5\lambda = 0.59.35.721.590.84
SVFM (K=2K=2)κ=0.2\kappa = 0.2, g=0.25g = 0.2564.26.941.930.87
SVFM (K=8K=8)κ=0.2\kappa = 0.2, g=0.25g = 0.2574.010.893.020.85

Table d. Performance and computational cost of SVFM, QAM and Q-Flow. Cube-quadruple, default task (advanced), 4 seeds for success rates. SVFM: K∈{2,8}K\in \lbrace 2, 8\rbrace, κ=0.2\kappa=0.2, g=0.25g=0.25. Training and inference times measured on one RTX 3090.

Appendix M.5 (Tables 24–26) reports the computational cost in the standard setting. We conducted additional experiments, including SVFM with K=2K=2 (Table d). With K=8K=8, SVFM achieves 74.0% success, substantially outperforming QAM (0.0%) and Q-Flow (9.3%), at approximately 1.9×1.9\times their training cost. SVFM adds no inference-time overhead: it generates actions with the trained actor alone, without the KK continuations or the soft-value network (SVFM 0.85 ms vs. Q-Flow 0.84 ms and QAM 0.87 ms per policy call).

To reduce training overhead, we also evaluate K=2K=2 while keeping κ\kappa and gg fixed. This lowers training cost by approximately 36%, bringing it to about 1.2×1.2\times that of the baselines, while success decreases from 74.0% to 64.2%. Despite this decrease, SVFM with K=2K=2 still outperforms both baselines, demonstrating a practical trade-off between training cost and task performance.

PhaseSVFM (K=2K=2)SVFM (K=8K=8)
Critic TD update1.68 ms (24%)1.75 ms (16%)
Sampling the KK continuations (SDE integration, forward only)0.68 ms (10%)2.16 ms (20%)
Soft-value target and soft-value network regression1.45 ms (21%)3.02 ms (28%)
Policy objective: soft-value gradient ∇xVψ\nabla_x V_\psi and actor regression0.86 ms (12%)0.88 ms (8%)
Adaptive temperature, optimizer and target networks2.28 ms (33%)3.00 ms (28%)
One full update6.64 ms10.47 ms

Table e. Per-phase training time of SVFM. K∈{2,8}K\in \lbrace 2, 8\rbrace. One RTX 3090, median over 9 trials of 300 updates.

To clarify the costs of sampling KK continuations via SDE integration and computing the soft-value gradient in the policy objective, Appendix M.5 (Table 25) breaks one SVFM update into its phases (standard setting, cube-double / humanoidmaze-medium). We repeated this breakdown under the same setting as Table d.

In Table e, sampling the K=8K=8 continuations by integrating the SDE takes 20% of an update. The continuations are integrated forward only, without gradients, and as a single batch. Increasing KK from 2 to 8 raises the sampling time from 0.68 to 2.16 ms, an average increase of about 0.25 ms per additional continuation. Computing the gradient of the soft-value network in the policy objective requires a single input-gradient of that network and, together with the actor regression, takes 8% of an update. Neither cost is incurred at inference.

AuthorsOpenReview ↗

Response #3

[W3] SDE integration details.

The KK actions are sampled as follows. Starting from a data anchor xtx_t, constructed by noising a dataset action, we integrate the SDE in Eq. (10) from tt to 11 using Euler–Maruyama on the grid tk=min⁡(t+k/N,1)t_k=\min(t+k/N,1), k=0,…,Nk=0,\ldots,N, with N=10N=10 on OGBench:

Xtk+1=Xtk+bκ(Xtk,tk)Δk+σκ,tkΔkξ,ξ∼N(0,I),\displaystyle X_{t_{k+1}}=X_{t_k}+b_\kappa(X_{t_k},t_k)\Delta_k+\sigma_{\kappa,t_k}\sqrt{\Delta_k}\xi,\quad \xi\sim\mathcal N(0,I),

where Δk=tk+1−tk\Delta_k=t_{k+1}-t_k, bκ(x,t)=uϕ(x,t)−κ2(x−tuϕ(x,t))/tb_\kappa(x,t)=u_\phi(x,t)-\kappa^2(x-tu_\phi(x,t))/t, σκ,t=κ2(1−t)/t\sigma_{\kappa,t}=\kappa\sqrt{2(1-t)/t}, and uϕu_\phi is the frozen behavior velocity field. The K=8K=8 continuations start from the same xtx_t, differ only in their noise, and are integrated in parallel as one batch. The SDE integration is performed only during training.

To make this easier to follow, we have added pseudocode blocks to Appendix E.1 of the revision: Algorithm 1 for one training update, showing where the integration is used, and Algorithm 2 for the integration itself, including the solver and the number of steps.


[W4] Additional hyperparameters κ\kappa and gg.

Although SVFM introduces one additional hyperparameter compared with QAM and Q-Flow, the total number of configurations in our sweep remains comparable to those of the baselines. SVFM searches κ∈{0.2,0.4,0.6,0.8,1.0}\kappa \in \lbrace 0.2, 0.4, 0.6, 0.8, 1.0\rbrace and g∈{0.25,0.5}g \in \lbrace 0.25, 0.5\rbrace, i.e., 10 configurations per domain.

  • Q-Flow sweeps 7 values of its guidance coefficient, {0.2,0.5,1,2,5,10,20}\lbrace 0.2, 0.5, 1, 2, 5, 10, 20\rbrace ([2], Appendix C.4).
  • QAM sweeps 6 values of its temperature λ\lambda, {1/30,0.1,1/3,1,10/3,10}\lbrace 1/30, 0.1, 1/3, 1, 10/3, 10\rbrace ([3], Table 6).
  • QAM-E [3], which also has two hyperparameters, sweeps all 20 combinations of λ∈{0.1,1/3,1,10/3,10}\lambda \in \lbrace 0.1, 1/3, 1, 10/3, 10\rbrace and σa∈{0,0.1,0.5,0.9}\sigma_a \in \lbrace 0, 0.1, 0.5, 0.9\rbrace.

SVFM's 10 configurations are therefore within the range used by these baselines.

Moreover, although we evaluated five κ\kappa values, the selected configurations in the standard setting (Appendix G.1, Table 9), offline-to-online learning (Appendix G.3, Table 11), and SVFM+FAC [5] experiments (Appendix M.1, Table 18) all use κ∈{0.4,0.6,0.8}\kappa \in \lbrace 0.4, 0.6, 0.8\rbrace. This suggests a smaller initial search grid for practitioners: combining these three κ\kappa values with the two gg values requires only six configurations, rather than ten.


[W4, Q2] Integration with action chunking.

SVFM integrates action chunking without changing its objectives (chunk lengths: OGBench advanced manipulation 5, LIBERO(-Plus) 10, real-world task-specific policies 8, GR00T N1.7 16). All baselines use the same HH and execution protocol as SVFM, and critic-based baselines also use a chunk critic, following Q-chunking[1]. The critic is defined on a chunk of HH actions, Q(st,at:t+H)Q(s_t, a_{t:t+H}), and the chunk is treated as a single action vector a∈RH⋅da \in \mathbb{R}^{H \cdot d} (dd: per-step action dimension), on which the behavior flow, the KK continuations, the soft value VψV_\psi and the actor all operate.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all three reviewers for their careful reading, constructive feedback, and time spent evaluating our work.

We appreciate Reviewer 54U6’s recognition of the clarity of our method and its evaluation across multiple benchmarks, including real-world experiments; Reviewer YZxY’s recognition of the clear formulation, mathematical analysis, and strong empirical results; and Reviewer jFuj’s recognition of our novel approach and promising results on cube-quadruple and VLA tasks.

We have provided point-by-point responses to each reviewer’s concerns and questions below, together with additional experiments and clarifications. If any concern remains unresolved or any question has not been fully answered, please do let us know. We are happy to continue the discussion and will make every effort to address any remaining issues.

Dear AC and Reviewers,

Thank you for the constructive feedback. The reviewers recognized the clear formulation and mathematical analysis of our approach (YZxY), the novelty of our value-learning method (jFuj), and its evaluation across offline RL, VLA fine-tuning, and real-world manipulation (54U6, YZxY, jFuj). They highlighted evidence supporting fixed behavior continuations (YZxY) and the strong results on challenging manipulation tasks (YZxY, jFuj).

We find that Soft-Value Flow Matching (SVFM) improves average OGBench success over Q-Flow from 53% to 57% in the standard setting and from 47% to 56% in the advanced setting, evaluated over eight and twelve seeds, respectively. It also improves pretrained VLA policies in simulation and on a real robot. In the rebuttal, we clarified the methodological contribution and added controlled comparisons addressing the sources of these gains, computational cost, guidance accuracy, and online adaptation. Key findings follow.

Contribution and distinction from QAM (54U6, jFuj). SVFM and QAM share the same KL-regularized policy-improvement objective but estimate optimal control differently. Rather than propagating terminal critic gradients backward along current-actor trajectories, SVFM learns a conditional soft value from terminal critic values under fixed behavior continuations. Its input gradient then supervises the actor. With the behavior reference held fixed, actor updates do not directly change the critic-query distribution used for guidance. We clarified this distinction in Section 3.

Computational trade-offs and deployment cost (54U6, YZxY, jFuj). We clarified that avoiding trajectory BPTT does not necessarily reduce total training cost. SVFM replaces trajectory backpropagation with forward-only continuation sampling and soft-value learning. On the default cube-quadruple task under the advanced setting, reducing the number of continuations from eight to two lowers training time by approximately 36%, while retaining 64.2% success versus 74.0% with eight continuations and 9.3% for Q-Flow. The two-continuation variant requires approximately 1.2× the training time of QAM and Q-Flow. Importantly, deployment uses only the trained actor: SVFM, QAM, and Q-Flow all take 0.84–0.87 ms per policy call.

Separating the contributions of sampling and adaptive temperature (jFuj). On this task, we conducted controlled comparisons. With κ and the constant temperature held identical, switching from actor continuations to behavior continuations improves SVFM from 17.7% to 25.8%. Adaptive temperature is also important in practice: the best configurations found for the constant- and adaptive-temperature variants achieve 25.8% and 74.0%, respectively. However, transferring adaptive temperature and κ to QAM yields only 1.0–1.3%, while adaptive guidance improves Q-Flow from 9.3% to 19.8%. Thus, these adaptive components help but do not alone reproduce SVFM’s performance in this setting.

Reliability of learned soft-value guidance (YZxY). New diagnostics compare the learned soft value and its gradient against a reference computed using 1,024 continuations at held-out anchors. During training, the Q-scale-normalized soft-value error remains approximately 0.06–0.08, while median gradient cosine similarity increases from 0.67 to 0.81. A separate fourfold temperature-increase test shows that the soft-value error returns near its previous level within approximately 10K updates, while gradient alignment recovers more gradually. These results directly assess learned guidance quality and tracking, not just downstream success.

Dataset diversity, additional Robomimic experiments, and sources of online improvement (YZxY). Additional Robomimic Can-MH experiments extend the evidence beyond OGBench’s diverse play-style data: SVFM achieves 89.2% success versus 83.6% for Q-Flow and 77.8% for QAM over five seeds. We also conducted a humanoidmaze-large ablation with a matched additional RL-update budget. From a 15.0% offline checkpoint, continued offline updates reach 28.5%; adding online transitions with a fixed reference reaches 35.3%; and updating the reference further improves success to 44.8–54.0%, with replay-buffer BC refitting achieving the highest success. These comparisons distinguish the benefits of continued learning, new interaction data, and reference adaptation in this setting.

Beyond OGBench, SVFM achieves 86% on LIBERO and 73% on LIBERO-Plus, compared with 82% and 69% for Filtered BC (Section 4.4). On real-world GR00T N1.7 fine-tuning, it achieves 99% average success versus 73% for matched-budget BC+ (Section 4.5). We also acknowledge the remaining trade-off: finite-sample soft-value estimation introduces approximation error and additional training cost.

We appreciate the reviewers’ feedback, which strengthened the evidence and sharpened our claims. We ask that these results and clarifications be considered in the final discussion.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.