acceptodds
Under review as a conference paper at ICLR 2027

PrefPI: Preference-Guided Steering into Out-of-Distribution Behaviors

Abstract

We present PrefPI (Preference-Guided Policy Iteration), an iterative framework for steering pretrained generative robot policies using only relative preferences over self-generated trajectories. Unlike prior preference-learning methods that primarily sharpen modes already represented by the policy, we study steering beyond the initial effective support, where desired behaviors are rarely or never observed under the initial policy. Our key idea is to formulate preference learn- ing as preference-conditioned generative modeling: preferred trajectories define a conditional distribution, whose density ratio with the broader behavior prior pro- vides an implicit preference signal amplified by classifier-free guidance (CFG). Repeating this preference-conditioned modeling and guidance step yields a form of preference-guided policy iteration, turning incremental improvements toward previously inaccessible behaviors. Across diffusion policies and the PI0.5 flow- matching VLA in simulation and the real world, PrefPI produces substantial be- havioral shifts with limited feedback. In particular, PrefPI increases object trans- port height from 10.7 cm to 19.8 cm on real hardware with only 150 preference- labeled trajectories.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Preference Learning, Robot Learning, Vision-Language-Action Models, Generative Policies, Policy Iteration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dFgeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of adapting a pretrained robot policy to a user’s preferred behavior when that behavior is rarely or never produced by the initial policy. It proposes PrefPI, an iterative preference-guided method that repeatedly selects relatively preferred trajectories from the current policy and uses classifier-free guidance to progressively steer the policy toward the desired behavior.

Critical strengths
  • Interesting Setup: The problem of incorporating human preferences when the desired behavior lies outside the initial policy’s effective support is very interesting.

  • Simplicity of the Idea: The idea of using classifier-free guidance (CFG) to incorporate human preferences is creative. I appreciate the authors for proposing a simple solution to an important problem.

  • Hardware Experiments: I appreciate the authors for conducting an extensive real-world hardware evaluation across multiple tasks.

  • Analysis of Failure Modes: I appreciate that the authors explicitly discuss the trade-off between stronger OOD steering and task success, and further analyze how the diversity of the initial policy affects the effectiveness of PrefPI.

Critical weaknesses
  • Practicality and Scalability: My biggest concern with the paper is the practicality and scalability of the proposed approach. First, PrefPI appears to require the sampled trajectory set to contain sufficient preference-relevant variation for the user to provide an informative ranking. It is unclear what happens when none of the sampled trajectories meaningfully reflects the desired behavior. Second, each iteration requires collecting a new batch of trajectories, obtaining fresh human preference labels, and retraining the policy. This repeated human-in-the-loop interaction and optimization may become burdensome when several iterations are required to reach the desired OOD behavior. The paper also does not report the wall-clock training time or the end-to-end adaptation time per iteration, making the practical deployment cost difficult to assess. An interesting direction for future work would be to investigate whether lightweight human-provided priors, such as natural-language instructions, rough trajectory sketches, or waypoint specifications, could directly indicate the desired behavioral region and reduce the need for repeated trajectory selection.
Questions for authors
  • Coverage of Human Preferences: Could the authors analyze how often a sampled trajectory batch contains insufficient preference-relevant variation for the user to provide a meaningful preference signal? This seems particularly important in the OOD setting, since PrefPI relies on the sampled trajectories containing informative variations that can be ranked by the user.

  • Sensitivity to Key Hyperparameters: Could the authors comment on the sensitivity of PrefPI to its key hyperparameters and explain the basis for the chosen values? In particular, it would be useful to evaluate:

    • different values of top-mm, or the preferred trajectory fraction;
    • different numbers of sampled trajectories NN per iteration; and
    • different CFG guidance weights ww controlling the strength of the conditional distribution.
3 comments

We appreciate the reviewer's valuable feedback and suggestions. Below, we address each weakness (W) and question (Q) individually.

[W1, Q1] What If None of the Sampled Trajectories Meaningfully Reflects the Desired Behavior?

We agree that sufficient preference-relevant diversity in the pretrained policy is an important prerequisite for PrefPI. In fact, we explicitly investigate this limitation through controlled experiments in Section 6, where we analyze how differences in the diversity of the pretrained prior affect preference steering.

However, we would like to clarify an important distinction: PrefPI does not require the initial policy to generate the desired behavior directly. Rather, it requires meaningful variation among sampled trajectories with respect to the user's preferences. Even when none of the sampled trajectories achieves the target behavior, as long as there is meaningful variation in their relative preference scores, PrefPI can progressively steer the policy toward the desired behavior.

Importantly, this limitation is not unique to PrefPI but is shared by conventional preference-learning approaches. For example, the seminal work Deep Reinforcement Learning from Human Preferences (Christiano et al., 2017) relies on human comparisons between trajectories generated by the current policy. If the policy produces nearly identical behaviors with no meaningful distinctions from the user's perspective, these comparisons cannot provide informative preference signals. Thus, sufficient behavioral variation is a fundamental requirement for learning from relative preferences.

Our approach leverages the strong pretrained prior of π₀.₅ to address this challenge in practice. Recent advances in diffusion and flow-matching policies have enabled expressive, multimodal action distributions that can capture diverse behaviors. By leveraging this inherent behavioral diversity in π₀.₅, PrefPI can progressively steer the policy toward preferred behaviors beyond its initially observed range. We believe this rich behavioral diversity is a key reason why PrefPI achieves effective preference steering across our π₀.₅ experiments.

As pretrained robot policies continue to scale and become more capable and diverse, we believe PrefPI provides an increasingly relevant framework for adapting such policies to user-specific preferences.

[W2] Practicality Concerns: Human Feedback Cost and Wall-Clock Adaptation Time

We believe feedback efficiency is one of the key strengths of PrefPI. We would like to highlight several findings from our real-world experiments:

  • Detour: With only 150 human preference annotations, PrefPI increases the maximum detour height from 10.7 cm to 19.8 cm.
  • Speed: With the same feedback budget of 150 annotations, PrefPI reduces execution time from 23.8 s to 12.7 s.
  • Comparison with baselines: As demonstrated in Sections 5.3 and 5.4, PrefPI achieves faster preference steering than existing preference-learning baselines under the same feedback budget.

While 150 preference annotations may still represent a nontrivial human effort, reducing the preference-feedback budget remains an important challenge in preference learning. Our results demonstrate that substantial behavioral changes can already be achieved with a relatively limited number of preference annotations. Further improving feedback efficiency, potentially through lightweight human-provided priors as the reviewer suggests, is a promising direction for future work.

Regarding computational cost, each PrefPI iteration requires only fine-tuning of a pretrained π₀.₅ policy rather than training a large VLA model from scratch. Specifically, our default configuration collects 40 trajectories per iteration, which takes approximately 20–30 minutes on a single real-world robot. Policy fine-tuning then involves 3,000 gradient updates with a batch size of 12, requiring approximately 1–1.5 hours on a single NVIDIA L40 GPU.

Thus, each iteration takes approximately 1.5–2 hours, including real-world rollout collection and policy fine-tuning. For example, the three iterations illustrated in Figure 1 would require approximately 5–6 hours in total. We have added these wall-clock statistics to Appendix C (page 15) of the revised manuscript to clarify the practical adaptation cost of PrefPI.

[Q2] Sensitivity Analysis to Key Hyperparameters

Following the reviewer's suggestion, we conducted extensive additional ablation studies on the sampling budget (NN), preferred trajectory selection size (mm), and CFG guidance strength (ww), each with three random seeds.

We have incorporated these results into the newly added Section 5.5 (Ablation Studies). We encourage the reviewer to refer to this section for visualizations of the results, while we provide the corresponding numerical results in tables below for completeness.

Ablation on Guidance Sterngth ww

We conducted an additional ablation study with guidance strengths on two detour tasks and added the results to Section 5.5 (Ablation Studies).

As shown in the tables below, smaller guidance weights (w=2,3w=2,3) lead to slower preference steering while better preserving task success. In contrast, achieves faster steering but at the cost of substantial success-rate degradation, particularly in PushPlate task. The default provides a favorable trade-off between steering strength and task reliability across both tasks.

  1. Detour-BowlToStove
w012345678
w=215.9 (100.0)17.0 (97.5)17.5 (100.0)17.6 (100.0)18.7 (100.0)18.5 (100.0)19.1 (100.0)19.9 (96.2)20.1 (98.8)
w=315.7 (100.0)16.8 (99.2)17.9 (100.0)18.2 (100.0)18.8 (100.0)19.9 (100.0)20.1 (96.7)20.8 (98.3)21.5 (93.3)
Base (w=5)15.7 (99.5)16.8 (100.0)17.3 (100.0)18.4 (100.0)19.4 (100.0)20.8 (98.9)21.8 (89.5)23.6 (87.1)24.7 (80.1)
w=1015.7 (100.0)17.9 (99.2)18.9 (100.0)22.5 (92.5)24.7 (80.0)26.7 (75.8)29.4 (60.0)29.5 (69.2)31.6 (57.5)
  1. Detour-PushPlate
w012345
w=23.8 (98.8)3.9 (100.0)4.3 (98.8)5.6 (87.5)5.7 (95.0)5.7 (76.2)
w=33.7 (98.3)3.8 (97.5)4.7 (100.0)5.5 (95.0)5.9 (80.0)7.1 (81.7)
Base (w=5)3.8 (98.0)4.7 (97.0)5.3 (98.9)6.2 (82.1)7.5 (77.5)7.9 (63.8)
w=103.9 (97.5)5.1 (89.2)6.8 (75.0)7.8 (52.5)8.2 (39.2)8.2 (41.7)

Ablation on Sampling Budget NN and Selection Size mm

We have conducted an additional ablation study on the sampling budget and selection size, with results added in Section 5.5 (Ablation Studies). As shown in the table below, all four configurations achieve preference steering, although their steering rates and task success rates differ.

Notably, achieves the fastest preference steering on both tasks, reaching 30.6 cm on Detour-BowlToStove and 11.1 cm on Detour-PushPlate. However, this stronger steering comes with reduced task success rates, particularly on PushPlate. We hypothesize that a smaller selection ratio(m/N) increases the contrast between the conditional and unconditional distributions, resulting in more aggressive CFG-based steering.

The other three configurations, which use relatively similar selection ratios (40–50%), generally exhibit more comparable steering behavior. However, a sufficient sampling budget remains important, as yields weaker preference improvement and lower task success on PushPlate.

Overall, these results suggest that increasing the sampling budget and reducing the selection ratio can improve preference steering, but may introduce a trade-off with task success. In particular, extends the observed preference–success Pareto frontier, as illustrated in Appendix D.

The quantitative results are provided below (preference metric, with task success rate (%) in parentheses).

  1. Detour-BowlToStove
012345678
Base (N40m16)15.7 (99.5)16.8 (100.0)17.3 (100.0)18.4 (100.0)19.4 (100.0)20.8 (98.9)21.8 (89.5)23.6 (87.1)24.7 (80.1)
N20m1015.8 (100.0)16.8 (96.7)18.0 (100.0)19.3 (98.3)20.8 (93.3)21.6 (95.0)23.4 (88.3)24.9 (86.7)26.3 (70.0)
N100m2015.6 (100.0)18.1 (99.7)19.0 (99.7)21.1 (100.0)23.9 (89.0)26.4 (71.7)27.6 (84.3)30.0 (64.0)30.6 (63.7)
N100m4015.7 (100.0)17.4 (98.7)18.6 (100.0)19.8 (99.0)21.6 (96.3)22.9 (95.7)24.2 (89.7)26.1 (75.0)26.2 (72.3)
  1. Detour-PushPlate
012345
Base (N40m16)3.8 (98.0)4.7 (97.0)5.3 (98.9)6.2 (82.1)7.5 (77.5)7.9 (63.8)
N20m103.9 (96.7)5.2 (88.3)5.5 (71.7)6.4 (55.0)6.9 (31.7)6.8 (33.3)
N100m203.9 (97.3)5.8 (90.0)6.5 (85.3)8.5 (71.7)10.0 (51.0)11.1 (32.3)
N100m403.9 (97.3)4.6 (99.0)5.4 (94.0)6.9 (81.3)7.7 (80.7)9.0 (57.3)

Thank you for the extensive ablation studies. All my concerns have been addressed

Reviewer_kqqaOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PrefPI, an iterative framework that steers pretrained generative robot policies toward preferred behaviors beyond their initial typical behavior range. It repeatedly selects preferred policy-generated trajectories and uses preference-conditioned training with classifier-free guidance to progressively amplify the desired behavior, without learning an explicit reward model. The authors provide an idealized theoretical analysis of preference improvement and evaluate the method on diffusion policies, the PI0.5 vision-language-action model, and a real robot. Experiments demonstrate improvements in execution speed, transport height, and throwing distance, while showing that steering effectiveness depends on initial policy diversity and can come at the cost of task success.

Critical strengths
  1. Practical effectiveness: PrefPI produces substantial behavioral changes across diffusion policies, a VLA, and real hardware with limited feedback, supporting its value for robot adaptation.

  2. Insight into policy diversity: A controlled experiment shows that greater initial diversity enables stronger steering while preserving task success, clarifying when the method works well.

Critical weaknesses
  1. The theoretical guarantee does not directly cover the implemented algorithm. The analysis assumes idealized trajectory-level CFG with the current policy as the reference, whereas the implementation applies action-level guidance and uses an accumulated dataset of successful trajectories. Although the paper acknowledges this distinction, the guarantee does not establish monotonic improvement for the practical method or explain how it reaches behaviors beyond the initial effective support. This limits the theoretical support for the paper’s central claim.

  2. Large behavioral shifts can come at a substantial cost to task reliability. On LiftDetour, PrefPI’s success rate drops to roughly 30%, while FDPP achieves both a larger detour and higher success. This suggests that the reliability loss is avoidable in that setting and weakens the case for PrefPI’s practical advantage. Comparing preference improvements at matched success rates—for example, while maintaining at least 90% success—would provide a more meaningful assessment of useful steering.

Questions for authors
  1. At matched task success rates, does PrefPI achieve larger preference improvements than the baselines? Please provide preference–success trade-off curves for PrefPI and the main baselines, and report the best preference metric achieved while maintaining at least 90% success. Including LiftDetour would clarify whether PrefPI retains a practical advantage before its success rate substantially declines.

  2. What evidence supports treating action-level CFG with the accumulated behavioral prior as a policy-improvement operator, and which conclusions about the implemented method does the theoretical result support? Please clarify the scope of the guarantee and provide an ablation comparing current-iteration versus accumulated successful trajectories for the unconditional branch across several CFG strengths, with other settings and feedback budgets held fixed. Reporting both preference metrics and success rates would help assess the empirical consequences of the gap between theory and implementation.

3 comments

We thank the reviewer for these questions.

[W2, Q1] Preference Improvement at Matched Success Rates

RoboMimic vs. baselines. All methods were evaluated at every iteration, so we computed the best preference metric each method achieves while keeping ≥90% task success (mean ± std over 5 seeds), and added the full preference–success trade-off curves to Appendix D.

TaskPrefPI (ours)IFBCDPOFDPPDSRL
CanDropPos (cm ↑)2.19 ± 0.700.26 ± 0.081.00 ± 0.260.76 ± 1.990.34 ± 0.05
CanSpeed (time ↓)91.79 ± 1.02101.69 ± 1.28100.00 ± 2.72105.11 ± 0.48103.46 ± 0.35
LiftDetour (cm ↑)2.81 ± 0.241.93 ± 0.012.19 ± 0.024.01 ± 0.431.90 ± 0.00

PrefPI retains a clear advantage at matched success on CanDropPos and CanSpeed. On CanDropPos, PrefPI achieves more than twice the displacement of the strongest baseline (2.19 vs. 1.00 cm, DPO). On CanSpeed, it is the only method that substantially reduces completion time while maintaining ≥90% success (91.79 vs. 100.00 for the next best).

On LiftDetour, FDPP is stronger at matched success. We agree with the reviewer that PrefPI's reliability loss in this setting is avoidable rather than inherent to OOD steering. PrefPI still improves the detour by 48% (1.90 → 2.81 cm) at ≥90% success and ranks second, while IFBC, DPO, and DSRL show little or no shift. We attribute the gap to the exceptionally narrow initial prior on this task (relative std. of 1.1%, compared to 6.8% and 27.2% in our other detour settings; Section 6). Our controlled experiment (Fig. 9) shows that a narrow prior reduces both steering magnitude and reliability for PrefPI. We also note that FDPP uses approximately 2× more environment interactions than PrefPI, and that its performance is less consistent across tasks: it has high variance on CanDropPos (0.76 ± 1.99) and the slowest completion time on CanSpeed.

PI0.5 detour tasks. Using the ablations in Section 5.5, we report the best preference metric achieved while maintaining ≥90% success (success rate in parentheses):

SettingDetour-BowlToStove (init. 15.7 cm)Detour-PushPlate (init. 3.8 cm)
w=2w=220.1 (98.8)5.7 (95.0)
w=3w=321.5 (93.3)5.5 (95.0)
w=5w=5 (default)20.8 (98.9)5.3 (98.9)
w=10w=1022.5 (92.5)3.9 (97.5)
N=100,m=20N=100, m=2021.1 (100.0)5.8 (90.0)
N=100,m=40N=100, m=4022.9 (95.7)5.4 (94.0)
No accumulation (w=5w=5)19.7 (95.0)5.3 (93.3)

While maintaining ≥90% success, PrefPI improves the maximum bowl height by +46% (15.7 → 22.9 cm) and the push distance by +53% (3.8 → 5.8 cm), so roughly half of the total behavioral shift is achieved while task reliability is preserved. The guidance strength ww and the selection ratio m/Nm/N act as explicit knobs along this frontier (Fig. 8, Fig. 11).

[W1, Q2] Scope of the Theoretical Result and the Role of Accumulation

What the theory supports. The selection result (Eqs. 7–10) holds without idealization: for any rollout distribution, including the one induced by our implemented policy, top-mm selection reweights it by a non-decreasing function of the latent utility. The monotonic improvement result (Eq. 14), however, characterizes an idealized step: trajectory-level CFG whose two branches exactly represent pkp_k and pk+p_k^+. It motivates the improvement direction of PrefPI but is not a guarantee for the implemented algorithm. In particular, we apply guidance at the action level, which does not in general compose into the trajectory-level ratio. We will soften the corresponding contribution statement to "an idealized policy-improvement operator."

Why the idealized assumption is hard to satisfy in practice. The idealized step requires both branches to represent their target distributions accurately. That would require far more data than our setting provides: we use only 30–50 trajectories per iteration. With so few samples, if the unconditional branch is trained only on the current iteration's rollouts, both branches are fit to small and largely overlapping datasets. The true density ratio between them is then close to 1, and the learned ratio is dominated by estimation noise, which CFG with w>1w>1 amplifies rather than the preference signal.

Accumulation as a practical remedy. We therefore train the unconditional branch on the accumulated set of successful trajectories. This introduces a deviation from the idealized analysis: the reference reflects a mixture of recent policies rather than exactly pkp_k. In exchange, it gives a much larger and more stable estimate of the behavioral prior, and a stronger contrast with the preference-conditioned branch. In other words, we trade a small bias in the reference for a large reduction in variance, and we evaluate this choice empirically below.

Beyond the initial effective support. Similarly, the theory explains the direction of improvement but not how far the policy can move. Moving beyond the support of the initial policy relies on the generalization of the learned generative policy, which we support empirically: by the final iteration, 100% / 100% / 96.6% of real-world rollouts exceed the best rollout observed under the initial policy (Speed / Detour / Throw), as reported in our response to Reviewer z7PJ.


[Q2] Ablation: Current-Iteration vs. Accumulated Unconditional Branch

With all other settings and feedback budgets held fixed (Section 5.5, Fig. 8(a)):

Detour-BowlToStoveDetour-PushPlate
Accumulated, final iter.24.7 cm (80.1%)7.9 cm (63.8%)
No accumulation, final iter.17.7 cm (99.2%)5.7 cm (80.8%)
Accumulated, best at ≥90% success20.8 cm (98.9%)5.3 cm (98.9%)
No accumulation, best at ≥90% success19.7 cm (95.0%)5.3 cm (93.3%)

These results are consistent with the variance argument above. Without accumulation, which is the setting closest to the idealized analysis, steering oscillates across iterations (e.g., 17.5 → 16.9 → 18.1 → 16.9 cm on BowlToStove) and ends with limited improvement. With accumulation, the preference metric increases steadily, yielding substantially larger shifts, and its advantage on BowlToStove persists at matched success.


We hope these clarifications and analyses address the reviewer's concerns.

Thank you for the additional experiments and clarifications. All my concerns have been addressed.

Reviewer_z7PJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an iterative framework for steering a pre-trained generative robot policy using iterative preference labeling on its rollouts. The user selects top-m in each iteration, after which the dataset is used to train a preference-conditioned branch and an unconditional branch. These two branches can construct classifier-free guidance to produce the next policy. The authors show, in theory, that top-m selection induces an order-preserving factor as an optimality signal for CFG. Simulation and real-world results show that PrefPI can effectively improve speed, detour height, and throwing distance with limited feedback.

I recommend weak reject. The method is simple and the real-world robot experiments are compelling, but the main "beyond initial support" claim is not supported by the theory or meansured quantitatively. I would consider raising my score if the authors can (1) quantify how far, in each iteration, the learned behaviors lie outside the initial policy's support (e.g., distance to the nearest initial rollout), (2) provide a w ablation, (3) add true-reward baselines, and (4) demonstrate PrefPI on at least one task with a more complex target behavior or a stronger randomization.

Critical strengths
  1. The problem of steering a pre-trained policy towards behaviors it does not produce initially is well motivated and also a practical robotics problem in the real world.
  2. Real-world robot results are convincing, showing large behavior shifts while keeping high success rates. There are also extensive simulation experiments.
Critical weaknesses
  1. The algorithm does not support the main claim "steer a policy toward desirable behaviors that cannot be produced by the initial policy". The algorithm still requires the pretrained policy to sample trajectories that fit or move toward true preferences. The extrapolation still comes from the generalization ability of the pretrained policy and the large guidance weight (w=5), neither is analyzed. Although the authors define the "effective support" that makes the claim defensible, the paper never measures density.
  2. Practicality on real world robot tasks is unclear. All objectives are simple geometric or speed metrics, and the real-robot evaluations contain only single pick-and-place tasks, and the paper does not describe any layout randomization. It is unclear whether CFG-driven extrapolation works for preferences over more complex motions (e.g., rotations, contact-rich, or multi-stage behaviors) or under larger spatial randomization, and how many iterations those would require.
  3. It is unclear if the baseline comparisons are fair. DSRL uses a learned reward function from the preferences, where the reward learning method is not described. A few loops of preference labeling are far outside DSRL's intended regime. A natural comparison is to give DSRL a true reward function and then plot against the number of rollouts.
Questions for authors
  1. How is the guidance weight w chosen? How sensitive are the results to it?
  2. The paper only tests with almost fixed layouts. How does the number of iterations scale with task difficulty or spatial randomization?
  3. What is the effect of training the unconditional branch on D^acc versus only the current iteration's rollouts?
  4. How is m/N chosen, and how sensitive is PrefPI to it?
  5. For the real-robot experiments, please report the distribution of transport heights, speed, throw distances in the 100 fine-tuning demonstrations. Given Section 6's finding that steering strength depends strongly on demo diversity, it is important to verify that the final behaviors actually lie outside the range covered by these demonstrations.
6 comments

We thank the reviewer for their thoughtful and constructive feedback. We conducted all experiments requested by the reviewer, each with three random seeds, resulting in 51 additional training runs. We have revised the manuscript accordingly, incorporating these results primarily into the newly added Section 5.5 (Ablation Studies). We encourage the reviewer to refer to this section for visualizations of the results, while we provide the corresponding numerical results in tables below for completeness. We address each weakness (W) and question (Q) individually below.

[W1] Quantitative Measurement of Behavior Beyond the Initial Policy's Effective Support

We conducted an empirical analysis across all three real-world experiments (Speed, Detour, and Throw) to assess how far the generated behaviors extend beyond the initial policy's observed range. This is because the flow-matching policy used in π₀.₅ does not provide a tractable closed-form density function, directly measuring trajectory likelihoods is challenging, so we took empirical analysis.

Specifically, for each experiment, we identified the most preferred behavior among the 30–50 rollouts collected from the initial policy (Loop 0) and used its task-specific metric as a reference threshold. For example, in the Detour experiment, we used the maximum transport height achieved among the 50 initial rollouts.

We then measured two quantities at each subsequent iteration:

  1. Fraction beyond the initial threshold: The percentage of rollouts that outperform the best rollout from Loop 0.
  2. Magnitude of improvement: The average improvement among rollouts exceeding this threshold, reported both in absolute terms and as a percentage relative to the threshold.

The results are summarized below.

  1. Speed: Rollouts faster than the Loop-0 minimum (13.0 s, 390 steps)
Metric \ Loop012345678
Fraction (%)003.680.082.1100100100100
Mean improvement (s)––1.170.910.611.441.581.592.16
Relative improvement (%)––9.07.04.711.112.112.216.6
  1. Detour: Rollouts higher than the Loop-0 maximum (18.8 cm)
Metric \ Loop0123456
Fraction (%)0014.393.981.6100100
Mean improvement (cm)––0.561.090.961.441.55
Relative improvement (%)––2.95.85.17.78.2
  1. Throw: Rollouts farther than the Loop-0 maximum (16.7 cm)
Metric \ Loop01234567891011121314
Fraction (%)06.76.76.719.234.655.257.164.377.875.985.290.086.296.6
Mean improvement (cm)–1.839.784.483.236.705.525.555.007.118.018.368.3912.1712.04
Relative improvement (%)–10.958.426.819.340.033.033.229.942.547.950.050.172.771.9

Across all three experiments, the fraction of rollouts exceeding the best behavior observed under the initial policy increases substantially over successive iterations, eventually reaching 100% in both Speed and Detour and 96.6% in Throw.

The most pronounced behavioral shift occurs in the Throw experiment. By Loop 14, 96.6% of rollouts exceed the maximum throwing distance observed under the initial policy, with these rollouts achieving, on average, a 71.9% greater throwing distance than the initial maximum. Similarly, by the final iteration, all rollouts in the Speed and Detour experiments outperform their respective initial thresholds, with average improvements of 16.6% in execution time and 8.2% in transport height, respectively.

We also refer the reviewer to Figure 4, which visualizes trajectories sampled from the initial policy (Loop 0, red) and the steered policy (Loop 14, blue). The two sets of trajectories exhibit remarkably little overlap, providing further qualitative evidence that our method steers the policy toward behaviors substantially different from those observed under the initial policy.

These results provide quantitative evidence that PrefPI progressively shifts the rollout distribution toward behaviors beyond the range observed under the initial policy.

[W2, Q2] Robustness to Spatial Randomization and Complex Behaviors

We agree with the reviewer that evaluating our method under more diverse and challenging conditions would further strengthen the empirical results. To address the concern regarding spatial randomization, we conducted additional experiments on the Detour-BowlToStove task to investigate the following question:

Can our method consistently steer a pretrained π₀.₅ policy toward preferred behaviors across randomized initial object positions?

Specifically, we randomized the bowl's initial (x, y) position within circular regions of varying radius: 0 cm (fixed position), 5 cm, 7.5 cm, and 10 cm. In each rollout, the robot must pick up the bowl and place it on the stove. We retain the same preference objective as in the original experiment, favoring trajectories with higher maximum end-effector heights, regardless of the bowl's initial position.

SettingLift init (cm)Lift loop 8 (cm)Success init (%)Success loop 8 (%)
base (fixed bowl)15.7 ± 0.1724.7 ± 2.4399.5 ± 0.980.1 ± 22.1
r ≤ 5 cm15.3 ± 0.1421.1 ± 0.93100.0 ± 0.078.0 ± 20.1
r ≤ 7.5 cm15.5 ± 0.0422.1 ± 2.7299.3 ± 0.976.0 ± 3.3
r ≤ 10 cm15.7 ± 0.2521.7 ± 3.8298.0 ± 1.671.3 ± 13.2

As shown above, our method consistently increases the maximum trajectory height across all randomization settings. While the improvement is slower under larger spatial randomization compared to the fixed-position setting, the preferred behavior continues to emerge through iterative optimization. These results have been added to Section 5.5 (Ablation Studies).

[W3] DSRL with Ground-Truth Rewards

We agree that our setting differs from the regime for which DSRL was originally designed. We included DSRL to evaluate how conventional RL methods perform under a limited interaction budget, which is particularly relevant to preference learning on real-world robots, where rollout collection and human annotation are costly. Since PrefPI can optimize arbitrary trajectory-level preferences, our goal was to investigate whether it can achieve more effective preference steering than conventional RL approaches in this data-limited setting.

We also acknowledge that our description of DSRL in Appendix E was incorrect and misleading. In all DSRL experiments, we already used ground-truth preference labels, assigning a terminal reward of 1 to preferred trajectories and 0 to non-preferred trajectories, with zero rewards at intermediate steps. The statements suggesting that we trained a reward model for DSRL were incorrect. A learned reward model was used only for FDPP, and we have corrected the description accordingly.

[Q1] Ablation on Guidance Sterngth ww

We conducted an additional ablation study with guidance strengths w∈{2,3,5,10}w \in \{2, 3, 5, 10\} on two detour tasks and added the results to Section 5.5 (Ablation Studies).

As shown in the tables below, smaller guidance weights (w=2,3w=2,3) lead to slower preference steering while better preserving task success. In contrast, w=10w=10 achieves faster steering but at the cost of substantial success-rate degradation, particularly in PushPlate task. The default w=5w=5 provides a favorable trade-off between steering strength and task reliability across both tasks.

  1. Detour-BowlToStove
w012345678
w=215.9 (100.0)17.0 (97.5)17.5 (100.0)17.6 (100.0)18.7 (100.0)18.5 (100.0)19.1 (100.0)19.9 (96.2)20.1 (98.8)
w=315.7 (100.0)16.8 (99.2)17.9 (100.0)18.2 (100.0)18.8 (100.0)19.9 (100.0)20.1 (96.7)20.8 (98.3)21.5 (93.3)
Base (w=5)15.7 (99.5)16.8 (100.0)17.3 (100.0)18.4 (100.0)19.4 (100.0)20.8 (98.9)21.8 (89.5)23.6 (87.1)24.7 (80.1)
w=1015.7 (100.0)17.9 (99.2)18.9 (100.0)22.5 (92.5)24.7 (80.0)26.7 (75.8)29.4 (60.0)29.5 (69.2)31.6 (57.5)
  1. Detour-PushPlate
w012345
w=23.8 (98.8)3.9 (100.0)4.3 (98.8)5.6 (87.5)5.7 (95.0)5.7 (76.2)
w=33.7 (98.3)3.8 (97.5)4.7 (100.0)5.5 (95.0)5.9 (80.0)7.1 (81.7)
Base (w=5)3.8 (98.0)4.7 (97.0)5.3 (98.9)6.2 (82.1)7.5 (77.5)7.9 (63.8)
w=103.9 (97.5)5.1 (89.2)6.8 (75.0)7.8 (52.5)8.2 (39.2)8.2 (41.7)

[Q3] Ablation on Accumulating Trajectories (Dkacc\mathcal{D}^{\mathrm{acc}}_k)

Following the reviewer’s suggestion, we have conducted an additional ablation study to evaluate the effect of accumulating successful trajectories and included the results in Section 5.5 (Ablation Studies).

Specifically, we compare training the unconditional branch on accumulated successful trajectories (Dkacc\mathcal{D}^{\mathrm{acc}}_k) against using only the current iteration's trajectories (Dk\mathcal{D}_k). As shown in Figure 8(a) and the table below, removing accumulation leads to less stable preference improvement, with pronounced oscillations across iterations. For example, on Detour-BowlToStove, the accumulated variant reaches a maximum height of 24.7 cm after eight iterations, compared to 17.7 cm without accumulation. Similarly, on Detour-PushPlate, accumulation achieves a detour distance of 7.9 cm, compared to 5.7 cm without accumulation.

We believe that removing accumulation reduces the contrast between the conditional and unconditional branches and weakens the CFG steering signal. In contrast, accumulation allows the unconditional branch to evolve more gradually and helps maintain a stronger contrast between the two branches. Furthermore, training the unconditional branch on a larger and more diverse set of successful trajectories may help preserve its generalization ability and prevent overfitting to recent trajectories.

The quantitative results are provided below (preference metric, with task success rate (%) in parentheses).

  1. Detour-BowlToStove
012345678
Base (Accum.)15.7 (99.5)16.8 (100.0)17.3 (100.0)18.4 (100.0)19.4 (100.0)20.8 (98.9)21.8 (89.5)23.6 (87.1)24.7 (80.1)
No Accum.15.6 (100.0)17.5 (100.0)16.9 (99.2)18.1 (100.0)16.9 (99.2)19.7 (95.0)17.5 (100.0)21.0 (83.3)17.7 (99.2)
  1. Detour-PushPlate
012345
Base (Accum.)3.8 (98.0)4.7 (97.0)5.3 (98.9)6.2 (82.1)7.5 (77.5)7.9 (63.8)
No Accum.3.9 (95.8)5.3 (93.3)4.4 (99.2)5.4 (83.3)4.9 (96.7)5.7 (80.8)

[Q4] Ablation on Sampling Budget (NN) and Selection Size (mm)

We have conducted an additional ablation study on the sampling budget and selection size, with results added in Section 5.5 (Ablation Studies). As shown in the table below, all four configurations achieve preference steering, although their steering rates and task success rates differ.

Notably, N=100,m=20N=100, m=20 achieves the fastest preference steering on both tasks, reaching 30.6 cm on Detour-BowlToStove and 11.1 cm on Detour-PushPlate. However, this stronger steering comes with reduced task success rates, particularly on PushPlate. We hypothesize that a smaller selection ratio(m/N) increases the contrast between the conditional and unconditional distributions, resulting in more aggressive CFG-based steering.

The other three configurations, which use relatively similar selection ratios (40–50%), generally exhibit more comparable steering behavior. However, a sufficient sampling budget remains important, as N=20,m=10N=20, m=10 yields weaker preference improvement and lower task success on PushPlate.

Overall, these results suggest that increasing the sampling budget and reducing the selection ratio can improve preference steering, but may introduce a trade-off with task success. In particular, N=100,m=20N=100, m=20 extends the observed preference–success Pareto frontier, as illustrated in Appendix D.

The quantitative results are provided below (preference metric, with task success rate (%) in parentheses).

  1. Detour-BowlToStove
012345678
Base (N40m16)15.7 (99.5)16.8 (100.0)17.3 (100.0)18.4 (100.0)19.4 (100.0)20.8 (98.9)21.8 (89.5)23.6 (87.1)24.7 (80.1)
N20m1015.8 (100.0)16.8 (96.7)18.0 (100.0)19.3 (98.3)20.8 (93.3)21.6 (95.0)23.4 (88.3)24.9 (86.7)26.3 (70.0)
N100m2015.6 (100.0)18.1 (99.7)19.0 (99.7)21.1 (100.0)23.9 (89.0)26.4 (71.7)27.6 (84.3)30.0 (64.0)30.6 (63.7)
N100m4015.7 (100.0)17.4 (98.7)18.6 (100.0)19.8 (99.0)21.6 (96.3)22.9 (95.7)24.2 (89.7)26.1 (75.0)26.2 (72.3)
  1. Detour-PushPlate
012345
Base (N40m16)3.8 (98.0)4.7 (97.0)5.3 (98.9)6.2 (82.1)7.5 (77.5)7.9 (63.8)
N20m103.9 (96.7)5.2 (88.3)5.5 (71.7)6.4 (55.0)6.9 (31.7)6.8 (33.3)
N100m203.9 (97.3)5.8 (90.0)6.5 (85.3)8.5 (71.7)10.0 (51.0)11.1 (32.3)
N100m403.9 (97.3)4.6 (99.0)5.4 (94.0)6.9 (81.3)7.7 (80.7)9.0 (57.3)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.