acceptodds
Under review as a conference paper at ICLR 2027

Search to Steer: Learning Desirability-Guided Fields for Flow Policy Refinement

Abstract

Flow matching provides an expressive policy representation for robot learning, but policies trained solely on offline demonstrations remain limited by data quality and coverage. Reinforcement learning can address these limitations, yet fine-tuning flow policy faces costly likelihood computation and the risk of mode collapse under reward maximization. In this paper, we propose a flow policy refinement framework that learns to steer action generation through desirability-guided search. Specifically, desirability-guided search explores local improvements in noisy action space, with the goal of increasing return without collapsing distinct behavioral modes. The resulting search directions then provide supervision for learing a guidance field that steers action generation toward higher-return outcomes. This formulation avoids calculation of policy likelihood ratios and backpropagation through the critic or the full sampling trajectory. The resulting guidance field can be integrated directly into flow sampling stage, enabling a lightweight inference pipeline without additional online planning.

Primary area:
reinforcement learning
Keywords:
Flow Matching, Reinforcement Learning, Policy Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qLTDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a desirability-guided search method for fine-tuning flow-matching policies. The goal is to identify local improvement directions and use them as supervision for learning a guidance field that steers action generation toward higher-return outcomes, while avoiding expensive policy likelihood calculations and backpropagation through the sampling process. To achieve this, the method adds noisy perturbations around a flow anchor and decodes them into executable action chunks. The resulting local improvement directions are distilled into a timestep-conditioned residual network, which is added to the base vector field during action generation.

Critical strengths

The paper proposes an alternative approach to guiding generative models toward higher-return actions without modifying the base policy. It trains an additional residual network that can be applied directly to the base vector field.

The paper provides a clear problem formulation and a comprehensive literature review of related methods.

The introduction of desirability values based on return distributions is interesting. The method aims to encourage the search procedure to favor actions with desirable long-term outcomes. The learned residual field is applied to the base vector field without requiring expensive policy fine-tuning or backpropagation through the sampling process.

Critical weaknesses

Limited performance improvements: From Table 1, FQL achieves better performance on Walker Walk and both Jaco Reach tasks than the proposed method. For several other tasks, the return improvements over strong baselines such as FQL are relatively small. The authors should provide further analysis to demonstrate whether these improvements are statistically significant and explain why the proposed method underperforms on some tasks.

Questions for authors

Missing baselines: The experimental evaluation could be strengthened by including additional baselines, such as DSRL, RFS, and residual steering methods that directly apply corrections to the actions generated by the base policy. These comparisons would help clarify the advantages of the proposed guidance-field approach.

Real-world evaluation: Although real-robot experiments are not strictly required, including real-world manipulation experiments would further strengthen the paper and demonstrate the practical applicability of the proposed method.

1 comment

Thank you for your careful review and constructive suggestions. We have revised the manuscript and conducted additional experiments to further examine the effectiveness and applicability of S2S.

Q1: Comparisons with additional baselines.

Thank you for suggesting these relevant baselines. S2S is designed as a lightweight framework for refining pretrained flow policies through explicit local exploration in noisy action space. Rather than directly optimizing the pretrained policy or relying on value gradients, S2S searches for promising action variations around intermediate flow states and distills the resulting search directions into a lightweight steering field.

An important distinction is that S2S does not require a differentiable Q-function, as candidate actions can be evaluated using different pretrained scoring modules. Moreover, once the search directions are distilled, S2S requires neither candidate evaluation nor utility-gradient computation during inference.

To further evaluate these distinctions, we conducted additional comparisons with QGF, which performs value-gradient guidance during inference, and FlowDPG, which learns value-driven policy distillation. These experiments were conducted on the Scene Play task using the same pretrained base policy and utility parameters, with the same residual architecture adopted for the distilled methods.

MethodSuccess rate (%)
S2S27.35 ± 5.28
QGF26.86 ± 6.25
FlowDPG21.46 ± 6.82

S2S achieves a higher mean success rate than FlowDPG and performance comparable to QGF at the evaluated guidance strength, without requiring online value-gradient computation.

Furthermore, we have included an additional experiment on the D3IL task to investigate the role of desirability in preserving multimodal behaviors. This experiment provides a complementary evaluation of the proposed desirability formulation beyond aggregate task success rates. The corresponding results and analysis are presented in the revised manuscript.

We also include a comparison with DSRL when adapting the large-scale pretrained π0.5\pi_{0.5} model, as discussed in our response to Q2.

Q2: Real-world evaluation and practical applicability.

Thank you for raising this important concern. To further investigate whether S2S can be extended beyond relatively small flow policies, we conducted an additional experiment adapting the pretrained π0.5\pi_{0.5} vision-language-action model on the Dual-moka-pot task in LIBERO.

We compare S2S with the frozen pretrained policy and DSRL under the same task setting.

MethodSuccess rate (%)
Frozen π0.5\pi_{0.5}70
π0.5\pi_{0.5} + DSRL75
π0.5\pi_{0.5} + S2S85

S2S improves the success rate of the frozen π0.5\pi_{0.5} policy from 70% to 85%, outperforming DSRL by 10 percentage points. These results provide evidence that the proposed search-and-distillation framework can also be applied to large-scale pretrained VLA models, suggesting its potential for broader robotic applications. Additional experimental details are provided in Section 5.6 of the revised manuscript.

Reviewer_uxi3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes refining a frozen flow-matching policy using a learned residual field. During training, a desirability value function is first learned from offline data. Perturbations are then applied to intermediate flow states, and the resulting decoded actions are evaluated using the desirability utility to identify promising search directions. These directions are distilled into a residual field. At deployment, the residual field augments the frozen policy’s vector field during action generation. Experiments on ExORL and OGBench show that the proposed method outperforms baselines such as FQL on many evaluated tasks.

Critical strengths

The method provides a practical mechanism for refining pretrained flow-matching policies. It constructs local search directions from candidate action scores and distills them into a residual guidance network while keeping the base policy frozen. This avoids backpropagation through the critic, and confines the search process to training. During deployment, it requires neither candidate search nor value-function queries, making the approach potentially useful for refining existing policies.

Critical weaknesses
  1. Insufficient ablations to identify the sources of the gains. The evaluation lacks sufficient ablations on β and the number of sampled search directions. It remains unclear which components drive the gains and whether trained-policy results support preserving multiple behavioral modes.

  2. Insufficient analysis of improvements and failure cases. The paper does not adequately explain why the method outperforms FQL and CFGRL on some tasks but substantially underperforms them on others. For example, it achieves 70±10 on Jaco Reach Top Left, compared with 222±42 for FQL. Further theoretical and empirical analysis of the individual components would help establish when refinement reliably improves the base policy and when it may fail.

Questions for authors
  1. The overview figure could better illustrate the details of the proposed pipeline.

  2. What explains the substantial performance gaps relative to competing methods in Tables 1 and 2? For example, why is Jaco Reach Top Left 70±10 versus 222±42 for FQL?

  3. Could the authors provide ablations on β? What trained-policy evidence demonstrates the effect of nonzero β on task performance and behavioral diversity under matched training settings?

  4. The offline-trained Q-function ranks decoded candidates after perturbation, which may place them outside the data distribution. How reliable are the resulting utility estimates and candidate rankings?

  5. Is the Section 5.4 search-direction consistency filter used in Tables 1 and 2? How are its two independent continuation samples generated, and how does this relate to the main algorithm’s frozen-Q scoring? Which task and return metric does Figure 4 use?

  6. Could the authors ablate the flow-anchor construction, particularly the choice of flow time t and its effect on performance?

  7. Why does the proposed method outperform CFGRL or FQL on some tasks? Which components explain the advantage?

3 comments

Thank you for your review and suggestions. We have conducted new experiments applying S2S to the π0.5 vision-language-action model, with component comparisons and ablations.

Q1: Overview figure.

We will refine the overview figure to illustrate the S2S pipeline and include it in the revised manuscript.

Q2 and W2: Analysis of Jaco tasks.

Jaco rewards are concentrated near the target and decay rapidly with distance. Our original search radius was not calibrated to this characteristic. We therefore evaluate different search scales while keeping the remaining settings fixed. The search radius and candidate perturbations are given by

ρt=cL(1−t),xt+ρtδi,\rho_t=c\sqrt{L}(1-t), \qquad x_t+\rho_t\delta_i,

where LL is the action-chunk length, tt is flow time, and δi\delta_i is a unit direction. Increasing cc expands the neighborhood around each anchor. The results show that tuning the search radius to the Jaco tasks improves returns:

Search scale ccJaco Top Left returnJaco Top Right return
0.2570 ± 10120 ± 14
1.0146 ± 23194 ± 18

Q3 and W1: Desirability, behavioral diversity, and search candidates.

The reference hyperparameter table contained a typographical error: the intended value is β=0.2\beta=0.2, and the entry has been corrected. We vary β\beta on Scene Play with other settings fixed. Success is averaged equally across five subtasks and reported as mean ± variance over three base value seeds, with 20 episodes per subtask and seed.

βSuccess rate (%)
024.33 ± 4.58
0.125.67 ± 6.43
0.227.35 ± 5.28
0.326.67 ± 5.58
0.526.38 ± 4.51

We further examine behavioral multimodality on D3IL Avoiding under a controlled reward setting. The results suggest that desirability preserves both behavioral modes while adjusting their relative preferences. Experimental details and visualizations appear in Section 5.5.

Besides, we also vary the number of antithetic search candidates KK, which counts both directions per pair (K=64K=64 means 32 directions and their opposites).

Number of candidates KKSuccess rate (%)
3225.45 ± 4.73
6427.35 ± 5.28
12827.63 ± 4.92

Q4: Reliability of utility estimates.

S2S constructs improvement directions from scalar comparisons among decoded candidates. Unlike gradient-based guidance, it does not require a differentiable learned Q-function: scores may come from different pretrained reward or world models. As the candidate-count ablation suggests, increasing KK can reduce sensitivity to random perturbation directions. For Q-based evaluation, S2S could also adopt clipped double Q-learning, using min⁡(Q1,Q2)\min(Q_1,Q_2) for conservative candidate scoring to mitigate overestimation.

Q5: Consistency filtering.

Thanks for your question and suggestion. We fix the state, goal, anchor xtx_t, flow time tt, and model parameters. Two searches use independent random keys to sample antithetic perturbations, decode candidates with the same frozen flow model, and score them with the same frozen utility. Their directions gAg_A and gBg_B are compared by cosine similarity. Randomness thus mainly comes from perturbations. We have clarified in the revised manuscript that Figure 4 reports results on the Cube Single task in OGBench. This experiment examines whether retaining anchors with consistent search directions improves supervision quality and how this trades off against the number of retained training samples. All data-filtering experiments use τ=0.7\tau=0.7.

Q6: Flow-anchor construction and flow time.

We sample flow time tt uniformly, by default from [0.2,0.85][0.2,0.85]. Smaller tt gives noisier anchors; larger tt places them closer to data actions. We further evaluate different training-time intervals on Scene Play Task while keeping other hyperparameters fixed.

Training-time distributionScene Play success rate (%)
t∼Uniform[0.2,0.85]t\sim\mathrm{Uniform}[0.2,0.85]27.35 ± 5.28
t∼Uniform[0.1,0.9]t\sim\mathrm{Uniform}[0.1,0.9]26.35 ± 4.73
t∼Uniform[0.05,0.95]t\sim\mathrm{Uniform}[0.05,0.95]24.35 ± 6.24

Q7 and W2: Which components contribute to the gains?

Beyond the ablations above, we investigate whether S2S can complement existing fine-tuning frameworks. When online Q-gradients are affordable, we combine S2S search with QGF to introduce explicit local exploration alongside gradient-based guidance. The result is:

MethodSuccess rate (%)
QGF26.86 ± 6.25
QGF + S2S search28.24 ± 5.67

This improvement further suggests the effectiveness of explicit exploration in noisy action space.

Thank you for the clarification and additional experiments. The rebuttal clarifies several issues in the original manuscript, such as the typo regarding β=0\beta=0, and provides the necessary ablation studies. However, some concerns remain unanswered. For example, the reliability of the Q-function is not empirically supported, although this could be evaluated by artificially injecting perturbations into the Q estimates. Moreover, the effectiveness of the method is still not very evident. Multiple experiments show that S2S performs only comparably to FQL, and its overall performance is not sufficiently strong. Overall, I find the method interesting, and the experimental evaluation is now relatively comprehensive, but the limited performance remain concerns.

Thanks for your response and for raising score.

Our goal is to provide a lightweight framework for refining pretrained flow policies through noisy-space exploration, without requiring critic gradients or relying exclusively on Q-functions. Our comparisons and integration with QGF demonstrate its potential to complement existing guidance methods and improve performance. We will further explore comparisons with FQL and incorporate additional more reliable measure to strengthen reliability of Q estimate.

Thank you again for your careful review and constructive feedback.

Reviewer_63ohOpenReview ↗
Rating
2
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

Search to Steer refines a frozen flow policy by distilling local, value-guided search directions into a residual steering field. Candidate perturbations are evaluated through a learned utility function during training; deployment adds the learned field to the base velocity without candidate search or critic queries. Experiments report competitive results on ExORL and state-based OGBench.

Critical strengths
  • Simple refinement mechanism. The framework separates policy refinement from base-policy training and requires no policy likelihood computation or differentiation through the critic or sampler to construct steering targets.
  • Practical deployment structure. Training-time search is amortized into a small residual network, eliminating candidate evaluation at inference.
  • Encouraging benchmark results. The method achieves the highest reported mean on several continuous-control and goal-conditioned tasks, suggesting that local-search supervision can be useful.
Critical weaknesses
  • The novelty boundary is insufficiently established. Frozen-policy adaptation is already explored by RL Token and DSRL, while QGF uses the same first-order endpoint approximation and FlowDPG distills critic-driven corrections into flow velocities. The identifiable contribution therefore appears to be the specific combination of antithetic, derivative-free search and residual-field regression. Comparisons with QGF and an otherwise identical residual network trained on analytic value gradients are needed to establish the benefit of this choice.

  • The desirability motivation is disconnected from the reported configuration. Appendix B specifies β=0\beta=0, where the proposed utility reduces to expected return. If this configuration underlies the main results, those results do not establish benefits from risk-sensitive desirability. Moreover, the analytical toy example shows that desirability can either preserve or concentrate mode occupancy; it does not demonstrate that the trained method prevents mode collapse.

  • Robotic applicability remains unverified. Experiments are limited to simulation, with no real-robot or pretrained-VLA evaluation. The only explicitly disclosed reference backbone is a four-layer, 512-wide MLP. Consequently, applicability to visual manipulation, large pretrained action models, and contact-rich physical execution remains unsupported.

  • Experimental validation is incomplete. The paper lacks systematic component ablations, measured training and inference costs, and sufficiently detailed statistical protocols. Figure 4 does not specify the evaluated task, statistical protocol, or definition of “return gain (arbitrary units),” making the downstream benefit difficult to interpret and reproduce.

Questions for authors
  • Were all main results obtained with β=0\beta=0? Can nonzero-β\beta experiments demonstrate gains in both task performance and measured behavioral diversity?
  • Under matched architectures and computational budgets, how does derivative-free search compare with analytic-gradient supervision for the same residual field?
1 comment

Thank you for your careful review and constructive suggestions. We have revised the manuscript and added experiments. We address the specific points below.

W1 and Q2: Comparison with related methods.

Thank you for suggesting these baselines. S2S is motivated by the need for a lightweight framework for refining flow policies through exploration in noisy action space. By performing local search around the frozen policy's intermediate states, S2S explores promising actions while preserving the policy's multimodal structure and subsequently distills these improvements into a lightweight steering field. In addition, local search can complement existing fine-tuning methods, as demonstrated in our combination experiment below.

S2S is not tied to a particular Q-function, as candidate scores can be obtained from different pretrained evaluation modules. Its focus is on explicit exploration in noisy action space and learning a steering field from the resulting local search directions. This exploration mechanism complements existing gradient-based guidance and distillation approaches. We examine two related approaches: online value-gradient guidance, represented by QGF, and distillation of value-driven corrections, represented by FlowDPG. The comparison is conducted on the Scene Play task, using the same pretrained base policy and utility parameters, as well as the same residual architecture for the distilled methods. Results are shown below.

MethodSuccess rate (%)
S2S27.35 ± 5.28
QGF26.86 ± 6.25
FlowDPG21.46 ± 6.82

At this fixed guidance strength, S2S achieves a higher mean success rate than FlowDPG and a success rate comparable to QGF, without requiring costly online value-gradient computation.

Furthermore, online Q-guidance requires repeated gradient evaluations during inference. This computational cost can become more substantial when evaluation relies on a large pretrained reward model, while gradient-based guidance may be infeasible when feedback comes from non-differentiable world models or human-in-the-loop preference rankings. In contrast, S2S requires neither candidate scoring nor utility-gradient computation at deployment. Inference costs are compared below.

MethodInference latency (ms/chunk)
S2S0.562
QGF1.409

We further investigate whether S2S can be effectively integrated with other fine-tuning frameworks. Specifically, when online Q-gradients are affordable, the search mechanism of S2S can also be combined with Q-guidance, introducing explicit local exploration alongside gradient-based guidance. We compare QGF with its QGF-enhanced variant below.

MethodSuccess rate (%)
QGF26.86 ± 6.25
QGF + S2S search28.24 ± 5.67

W2 and Q1: Desirability and behavioral diversity.

Thank you for your careful examination and suggestions. The reference hyperparameter table contained a typographical error: the intended value is β = 0.2. We have corrected the corresponding entry and further examined the following two questions.

  1. How does β affect task performance? We evaluate different values of β on Scene Play while keeping the remaining settings fixed. Success is averaged equally over five Scene Play subtasks and reported as mean ± variance across three base value seeds, with 20 test episodes per task and seed.
βSuccess rate (%)
024.33 ± 4.58
0.125.67 ± 6.43
0.227.35 ± 5.28
0.326.67 ± 5.58
0.526.38 ± 4.51
  1. How does desirability affect robot behavior? We have conducted an additional experiment on the D3IL Avoiding task and included the corresponding visualization results in the revised manuscript. The results demonstrate that desirability enables S2S to improve the performance of the pretrained policy while preserving its original behavioral multimodality. Please refer to Section 5.5 of the revised manuscript for details.

W3: Applicability to pretrained VLA models.

Thank you for this suggestion. To examine the applicability of S2S to large pretrained VLA models, we have conducted a new experiment by adapting π0.5 with S2S on the Dual-moka-pot task in LIBERO. The comparison with DSRL is presented below, and further experimental details are provided in Section 5.6 of the revised manuscript.

MethodSuccess rate (%)
Frozen π0.570
π0.5 + DSRL75
π0.5 + S2S85

W4: Experimental validation and Figure 4.

Thank you for pointing this out. We clarified the description in the revised manuscript. The consistency experiment is conducted on the Cube Single task in OGBench. Its purpose is to examine whether retaining anchors with agreement between independently estimated search directions improves supervision quality and how this trades off against the amount of retained training data. For all data-filtering experiments, we set the threshold (τ\tau = 0.7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.