- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem of adapting a pretrained robot policy to a user’s preferred behavior when that behavior is rarely or never produced by the initial policy. It proposes PrefPI, an iterative preference-guided method that repeatedly selects relatively preferred trajectories from the current policy and uses classifier-free guidance to progressively steer the policy toward the desired behavior.
Critical strengths
-
Interesting Setup: The problem of incorporating human preferences when the desired behavior lies outside the initial policy’s effective support is very interesting.
-
Simplicity of the Idea: The idea of using classifier-free guidance (CFG) to incorporate human preferences is creative. I appreciate the authors for proposing a simple solution to an important problem.
-
Hardware Experiments: I appreciate the authors for conducting an extensive real-world hardware evaluation across multiple tasks.
-
Analysis of Failure Modes: I appreciate that the authors explicitly discuss the trade-off between stronger OOD steering and task success, and further analyze how the diversity of the initial policy affects the effectiveness of PrefPI.
Critical weaknesses
- Practicality and Scalability: My biggest concern with the paper is the practicality and scalability of the proposed approach. First, PrefPI appears to require the sampled trajectory set to contain sufficient preference-relevant variation for the user to provide an informative ranking. It is unclear what happens when none of the sampled trajectories meaningfully reflects the desired behavior. Second, each iteration requires collecting a new batch of trajectories, obtaining fresh human preference labels, and retraining the policy. This repeated human-in-the-loop interaction and optimization may become burdensome when several iterations are required to reach the desired OOD behavior. The paper also does not report the wall-clock training time or the end-to-end adaptation time per iteration, making the practical deployment cost difficult to assess. An interesting direction for future work would be to investigate whether lightweight human-provided priors, such as natural-language instructions, rough trajectory sketches, or waypoint specifications, could directly indicate the desired behavioral region and reduce the need for repeated trajectory selection.
Questions for authors
-
Coverage of Human Preferences: Could the authors analyze how often a sampled trajectory batch contains insufficient preference-relevant variation for the user to provide a meaningful preference signal? This seems particularly important in the OOD setting, since PrefPI relies on the sampled trajectories containing informative variations that can be ranked by the user.
-
Sensitivity to Key Hyperparameters: Could the authors comment on the sensitivity of PrefPI to its key hyperparameters and explain the basis for the chosen values? In particular, it would be useful to evaluate:
- different values of top-, or the preferred trajectory fraction;
- different numbers of sampled trajectories per iteration; and
- different CFG guidance weights controlling the strength of the conditional distribution.
3 comments
[Q2] Sensitivity Analysis to Key Hyperparameters
Following the reviewer's suggestion, we conducted extensive additional ablation studies on the sampling budget (), preferred trajectory selection size (), and CFG guidance strength (), each with three random seeds.
We have incorporated these results into the newly added Section 5.5 (Ablation Studies). We encourage the reviewer to refer to this section for visualizations of the results, while we provide the corresponding numerical results in tables below for completeness.
Ablation on Guidance Sterngth
We conducted an additional ablation study with guidance strengths on two detour tasks and added the results to Section 5.5 (Ablation Studies).
As shown in the tables below, smaller guidance weights () lead to slower preference steering while better preserving task success. In contrast, achieves faster steering but at the cost of substantial success-rate degradation, particularly in PushPlate task. The default provides a favorable trade-off between steering strength and task reliability across both tasks.
- Detour-BowlToStove
| w | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|---|
| w=2 | 15.9 (100.0) | 17.0 (97.5) | 17.5 (100.0) | 17.6 (100.0) | 18.7 (100.0) | 18.5 (100.0) | 19.1 (100.0) | 19.9 (96.2) | 20.1 (98.8) |
| w=3 | 15.7 (100.0) | 16.8 (99.2) | 17.9 (100.0) | 18.2 (100.0) | 18.8 (100.0) | 19.9 (100.0) | 20.1 (96.7) | 20.8 (98.3) | 21.5 (93.3) |
| Base (w=5) | 15.7 (99.5) | 16.8 (100.0) | 17.3 (100.0) | 18.4 (100.0) | 19.4 (100.0) | 20.8 (98.9) | 21.8 (89.5) | 23.6 (87.1) | 24.7 (80.1) |
| w=10 | 15.7 (100.0) | 17.9 (99.2) | 18.9 (100.0) | 22.5 (92.5) | 24.7 (80.0) | 26.7 (75.8) | 29.4 (60.0) | 29.5 (69.2) | 31.6 (57.5) |
- Detour-PushPlate
| w | 0 | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|---|
| w=2 | 3.8 (98.8) | 3.9 (100.0) | 4.3 (98.8) | 5.6 (87.5) | 5.7 (95.0) | 5.7 (76.2) |
| w=3 | 3.7 (98.3) | 3.8 (97.5) | 4.7 (100.0) | 5.5 (95.0) | 5.9 (80.0) | 7.1 (81.7) |
| Base (w=5) | 3.8 (98.0) | 4.7 (97.0) | 5.3 (98.9) | 6.2 (82.1) | 7.5 (77.5) | 7.9 (63.8) |
| w=10 | 3.9 (97.5) | 5.1 (89.2) | 6.8 (75.0) | 7.8 (52.5) | 8.2 (39.2) | 8.2 (41.7) |
Ablation on Sampling Budget and Selection Size
We have conducted an additional ablation study on the sampling budget and selection size, with results added in Section 5.5 (Ablation Studies). As shown in the table below, all four configurations achieve preference steering, although their steering rates and task success rates differ.
Notably, achieves the fastest preference steering on both tasks, reaching 30.6 cm on Detour-BowlToStove and 11.1 cm on Detour-PushPlate. However, this stronger steering comes with reduced task success rates, particularly on PushPlate. We hypothesize that a smaller selection ratio(m/N) increases the contrast between the conditional and unconditional distributions, resulting in more aggressive CFG-based steering.
The other three configurations, which use relatively similar selection ratios (40–50%), generally exhibit more comparable steering behavior. However, a sufficient sampling budget remains important, as yields weaker preference improvement and lower task success on PushPlate.
Overall, these results suggest that increasing the sampling budget and reducing the selection ratio can improve preference steering, but may introduce a trade-off with task success. In particular, extends the observed preference–success Pareto frontier, as illustrated in Appendix D.
The quantitative results are provided below (preference metric, with task success rate (%) in parentheses).
- Detour-BowlToStove
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | |
|---|---|---|---|---|---|---|---|---|---|
| Base (N40m16) | 15.7 (99.5) | 16.8 (100.0) | 17.3 (100.0) | 18.4 (100.0) | 19.4 (100.0) | 20.8 (98.9) | 21.8 (89.5) | 23.6 (87.1) | 24.7 (80.1) |
| N20m10 | 15.8 (100.0) | 16.8 (96.7) | 18.0 (100.0) | 19.3 (98.3) | 20.8 (93.3) | 21.6 (95.0) | 23.4 (88.3) | 24.9 (86.7) | 26.3 (70.0) |
| N100m20 | 15.6 (100.0) | 18.1 (99.7) | 19.0 (99.7) | 21.1 (100.0) | 23.9 (89.0) | 26.4 (71.7) | 27.6 (84.3) | 30.0 (64.0) | 30.6 (63.7) |
| N100m40 | 15.7 (100.0) | 17.4 (98.7) | 18.6 (100.0) | 19.8 (99.0) | 21.6 (96.3) | 22.9 (95.7) | 24.2 (89.7) | 26.1 (75.0) | 26.2 (72.3) |
- Detour-PushPlate
| 0 | 1 | 2 | 3 | 4 | 5 | |
|---|---|---|---|---|---|---|
| Base (N40m16) | 3.8 (98.0) | 4.7 (97.0) | 5.3 (98.9) | 6.2 (82.1) | 7.5 (77.5) | 7.9 (63.8) |
| N20m10 | 3.9 (96.7) | 5.2 (88.3) | 5.5 (71.7) | 6.4 (55.0) | 6.9 (31.7) | 6.8 (33.3) |
| N100m20 | 3.9 (97.3) | 5.8 (90.0) | 6.5 (85.3) | 8.5 (71.7) | 10.0 (51.0) | 11.1 (32.3) |
| N100m40 | 3.9 (97.3) | 4.6 (99.0) | 5.4 (94.0) | 6.9 (81.3) | 7.7 (80.7) | 9.0 (57.3) |
Thank you for the extensive ablation studies. All my concerns have been addressed
We appreciate the reviewer's valuable feedback and suggestions. Below, we address each weakness (W) and question (Q) individually.
[W1, Q1] What If None of the Sampled Trajectories Meaningfully Reflects the Desired Behavior?
We agree that sufficient preference-relevant diversity in the pretrained policy is an important prerequisite for PrefPI. In fact, we explicitly investigate this limitation through controlled experiments in Section 6, where we analyze how differences in the diversity of the pretrained prior affect preference steering.
However, we would like to clarify an important distinction: PrefPI does not require the initial policy to generate the desired behavior directly. Rather, it requires meaningful variation among sampled trajectories with respect to the user's preferences. Even when none of the sampled trajectories achieves the target behavior, as long as there is meaningful variation in their relative preference scores, PrefPI can progressively steer the policy toward the desired behavior.
Importantly, this limitation is not unique to PrefPI but is shared by conventional preference-learning approaches. For example, the seminal work Deep Reinforcement Learning from Human Preferences (Christiano et al., 2017) relies on human comparisons between trajectories generated by the current policy. If the policy produces nearly identical behaviors with no meaningful distinctions from the user's perspective, these comparisons cannot provide informative preference signals. Thus, sufficient behavioral variation is a fundamental requirement for learning from relative preferences.
Our approach leverages the strong pretrained prior of π₀.₅ to address this challenge in practice. Recent advances in diffusion and flow-matching policies have enabled expressive, multimodal action distributions that can capture diverse behaviors. By leveraging this inherent behavioral diversity in π₀.₅, PrefPI can progressively steer the policy toward preferred behaviors beyond its initially observed range. We believe this rich behavioral diversity is a key reason why PrefPI achieves effective preference steering across our π₀.₅ experiments.
As pretrained robot policies continue to scale and become more capable and diverse, we believe PrefPI provides an increasingly relevant framework for adapting such policies to user-specific preferences.
[W2] Practicality Concerns: Human Feedback Cost and Wall-Clock Adaptation Time
We believe feedback efficiency is one of the key strengths of PrefPI. We would like to highlight several findings from our real-world experiments:
While 150 preference annotations may still represent a nontrivial human effort, reducing the preference-feedback budget remains an important challenge in preference learning. Our results demonstrate that substantial behavioral changes can already be achieved with a relatively limited number of preference annotations. Further improving feedback efficiency, potentially through lightweight human-provided priors as the reviewer suggests, is a promising direction for future work.
Regarding computational cost, each PrefPI iteration requires only fine-tuning of a pretrained π₀.₅ policy rather than training a large VLA model from scratch. Specifically, our default configuration collects 40 trajectories per iteration, which takes approximately 20–30 minutes on a single real-world robot. Policy fine-tuning then involves 3,000 gradient updates with a batch size of 12, requiring approximately 1–1.5 hours on a single NVIDIA L40 GPU.
Thus, each iteration takes approximately 1.5–2 hours, including real-world rollout collection and policy fine-tuning. For example, the three iterations illustrated in Figure 1 would require approximately 5–6 hours in total. We have added these wall-clock statistics to Appendix C (page 15) of the revised manuscript to clarify the practical adaptation cost of PrefPI.