Search to Steer: Learning Desirability-Guided Fields for Flow Policy Refinement

Flow matching provides an expressive policy representation for robot learning, but policies trained solely on offline demonstrations remain limited by data quality and coverage. Reinforcement learning can address these limitations, yet fine-tuning flow policy faces costly likelihood computation and the risk of mode collapse under reward maximization. In this paper, we propose a flow policy refinement framework that learns to steer action generation through desirability-guided search. Specifically, desirability-guided search explores local improvements in noisy action space, with the goal of increasing return without collapsing distinct behavioral modes. The resulting search directions then provide supervision for learing a guidance field that steers action generation toward higher-return outcomes. This formulation avoids calculation of policy likelihood ratios and backpropagation through the critic or the full sampling trajectory. The resulting guidance field can be integrated directly into flow sampling stage, enabling a lightweight inference pipeline without additional online planning.