- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a desirability-guided search method for fine-tuning flow-matching policies. The goal is to identify local improvement directions and use them as supervision for learning a guidance field that steers action generation toward higher-return outcomes, while avoiding expensive policy likelihood calculations and backpropagation through the sampling process. To achieve this, the method adds noisy perturbations around a flow anchor and decodes them into executable action chunks. The resulting local improvement directions are distilled into a timestep-conditioned residual network, which is added to the base vector field during action generation.
Critical strengths
The paper proposes an alternative approach to guiding generative models toward higher-return actions without modifying the base policy. It trains an additional residual network that can be applied directly to the base vector field.
The paper provides a clear problem formulation and a comprehensive literature review of related methods.
The introduction of desirability values based on return distributions is interesting. The method aims to encourage the search procedure to favor actions with desirable long-term outcomes. The learned residual field is applied to the base vector field without requiring expensive policy fine-tuning or backpropagation through the sampling process.
Critical weaknesses
Limited performance improvements: From Table 1, FQL achieves better performance on Walker Walk and both Jaco Reach tasks than the proposed method. For several other tasks, the return improvements over strong baselines such as FQL are relatively small. The authors should provide further analysis to demonstrate whether these improvements are statistically significant and explain why the proposed method underperforms on some tasks.
Questions for authors
Missing baselines: The experimental evaluation could be strengthened by including additional baselines, such as DSRL, RFS, and residual steering methods that directly apply corrections to the actions generated by the base policy. These comparisons would help clarify the advantages of the proposed guidance-field approach.
Real-world evaluation: Although real-robot experiments are not strictly required, including real-world manipulation experiments would further strengthen the paper and demonstrate the practical applicability of the proposed method.
Thank you for your careful review and constructive suggestions. We have revised the manuscript and conducted additional experiments to further examine the effectiveness and applicability of S2S.
Q1: Comparisons with additional baselines.
Thank you for suggesting these relevant baselines. S2S is designed as a lightweight framework for refining pretrained flow policies through explicit local exploration in noisy action space. Rather than directly optimizing the pretrained policy or relying on value gradients, S2S searches for promising action variations around intermediate flow states and distills the resulting search directions into a lightweight steering field.
An important distinction is that S2S does not require a differentiable Q-function, as candidate actions can be evaluated using different pretrained scoring modules. Moreover, once the search directions are distilled, S2S requires neither candidate evaluation nor utility-gradient computation during inference.
To further evaluate these distinctions, we conducted additional comparisons with QGF, which performs value-gradient guidance during inference, and FlowDPG, which learns value-driven policy distillation. These experiments were conducted on the Scene Play task using the same pretrained base policy and utility parameters, with the same residual architecture adopted for the distilled methods.
S2S achieves a higher mean success rate than FlowDPG and performance comparable to QGF at the evaluated guidance strength, without requiring online value-gradient computation.
Furthermore, we have included an additional experiment on the D3IL task to investigate the role of desirability in preserving multimodal behaviors. This experiment provides a complementary evaluation of the proposed desirability formulation beyond aggregate task success rates. The corresponding results and analysis are presented in the revised manuscript.
We also include a comparison with DSRL when adapting the large-scale pretrained π0.5 model, as discussed in our response to Q2.
Q2: Real-world evaluation and practical applicability.
Thank you for raising this important concern. To further investigate whether S2S can be extended beyond relatively small flow policies, we conducted an additional experiment adapting the pretrained π0.5 vision-language-action model on the Dual-moka-pot task in LIBERO.
We compare S2S with the frozen pretrained policy and DSRL under the same task setting.
S2S improves the success rate of the frozen π0.5 policy from 70% to 85%, outperforming DSRL by 10 percentage points. These results provide evidence that the proposed search-and-distillation framework can also be applied to large-scale pretrained VLA models, suggesting its potential for broader robotic applications. Additional experimental details are provided in Section 5.6 of the revised manuscript.