acceptodds
Under review as a conference paper at ICLR 2027

UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning

Abstract

Preference-based RL provides an approach to learning reward models from pairwise comparisons between behaviors, bypassing the need for explicit reward design. However, existing methods typically rely on passive data collection and suffer from poor sample efficiency, especially during the early stages of learning. We introduce a model-based approach that actively directs exploration by jointly reasoning over uncertainties in the reward, dynamics and value functions. Our method, Uncertainty-Balanced Preference Planning (UBP2), uses ensembles of reward, dynamics and value function models to evaluate candidate trajectories according to a unified score combining expected reward, terminal value and epistemic uncertainty. Planning under this objective yields an explicit tradeoff between exploitation and information acquisition without requiring ad hoc exploration heuristics. Empirically, experiments on the Meta-World benchmarks show that UBP2 achieves substantially higher sample efficiency than model-free preference-based methods and non-optimistic model-based baselines.

Primary area:
reinforcement learning
Keywords:
Preference based Reinforcement Learning, Model Based Learning, Planning, Uncertainty, Exploration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WhAFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces UBP2, a model-based preference-based RL method built on TD-MPC2. It uses ensembles to estimate reward, dynamics, and value uncertainty, incorporating uncertainty bonuses into MPC planning. Preference queries are selected from the replay buffer using a UCB criterion, and a learned policy takes over once the query budget is exhausted. The paper establishes a sublinear regret bound under GP/RKHS assumptions. Across 10 MetaWorld tasks, UBP2 achieves an average IQM success rate of 76.1%, outperforming RUNE, MRN, and MBP. A small-scale study with real human preferences further compares UBP2 with MBP.

Critical strengths
  1. The non-optimistic variant MBP (62.6) performs on par with the model-free baselines, so the improvement comes mainly from optimistic exploration rather than from the model-based backbone.
  2. Exploration during planning and query selection share one uncertainty view, and the uncertainty coefficients are tuned automatically instead of by hand-designed schedules (Table 10).
  3. Turning the uncertainty bonus into a penalty at test time sharply degrades performance (Table 7), which indicates that the learned uncertainty acts as an exploration signal rather than a risk proxy.
Critical weaknesses
  • The components are established: TD-MPC2, ensemble-based optimism (H-UCRL, SOMBRL, MaxInfoRL) and reward-uncertainty bonuses (RUNE).
    • The ablation (Table 2) does not support the claim that all three uncertainties are needed. Removing reward uncertainty from planning (0111) costs only 0.6 points on average and improves 4/10 tasks; most optimistic variants fall within 67.7–76.1, with per-task SEs of 10–20.
    • The theory treats the reward as directly observed noisy values (GP regression, Eqs. 12–13), omits query selection, and largely restates the bound of Sukhija et al. (2025b).
    • Query–policy alignment work (QPA, Hu et al., ICLR 2024), whose motivation overlaps closely with the query-selection argument, is neither discussed nor compared.
    • The human study (2 annotators, 2 tasks, compared only against MBP) contains no measurement supporting "easier to label".
    • With 10% flipped labels, Window Close drops from 92.8 to 42.4, yet this is described as robust.
    • Fig. 4(b) reports consistent gains from horizon 7 to 11, but Table 8 shows horizon 11 is worse on five other tasks.
    • Wall-clock time is 12–20× that of the baselines (Table 6).
Questions for authors
  1. How does selection perform if the score is made order-invariant, or if it uses the predicted return of the segments plus uncertainty? How does it compare with QPA-style policy-aligned selection?
  2. What does the Wilcoxon test pair, seeds or time points? Please report error bars for the time-averaged IQM actually shown (not only the final-step SE), and the number of seeds used in the ablation, noise and human experiments.
  3. How many runs per method were performed? What fraction of queries were marked "no preference" and discarded for each method, and what was the inter-annotator agreement?
  4. How do the model-free baselines perform with a compute budget comparable to UBP2's, and how does UBP2 perform with cheaper planning (fewer CEM samples or iterations)? Please also show success versus number of labels, not only versus environment steps.
  5. How was the per-task horizon chosen, and on which data? What are the results with a single horizon for all tasks?
7 comments
AuthorsOpenReview ↗

Q1, W4 , Q2: Comparison to QPA policy aligned selection. Report error bars for the time-averaged IQM

We thank the reviewer for the careful reading and for suggestions that have strengthened the paper. In response, we add QPA and PPE on all 10 tasks. We extend the statistics: error bars are now bootstrap standard errors of the time-averaged IQM (10,000 resamples over runs), significance uses a two-sided Mann–Whitney U test on per-run scores, and we report seed counts. We also test an order-invariant query score and report human-study label statistics.

Q1, W4, Q2: "Comparison to QPA. Request to report error bars for the time-averaged IQM"

We have added QPA and PPE as baselines on all 10 tasks, using the same feedback budgets as the other methods. UBP2 still ranks first on every task except Coffee Button, where RUNE already outperformed UBP2 in Table 1 and remains the best method on that task.

We report below IQM success over seeds at each evaluation, averaged over training, with the bootstrap standard error of this time-averaged IQM in parentheses (10,000 resamples of runs). The best result per task is shown in bold. The Avg row is bootstrapped by resampling runs within each task. Asterisks here mark a significant difference from UBP2 under a two-sided unpaired Wilcoxon rank-sum (Mann–Whitney U) test on per-run success averaged over training (p<0.05p<0.05). We report this test because it uses all seeds of each method, whereas the paired test in the original Table 1 can only use matched seeds. The significant differences for MBP, Rune and MRN are unchanged except Hammer vs. MRN (p=0.074p=0.074); the unpaired test additionally detects significant differences against QPA and PPE, which have only 5 seeds.

TaskMax FBUBP2MBPRuneMRNQPAPPETD-MPC2
Door Close50097.8 (0.4)95.6 (0.6)*91.5 (4.8)*94.2 (5.1)*92.2 (19.3)94.2 (19.1)*98.8 (0.2)
Window Close50093.1 (0.9)74.7 (13.5)*85.2 (8.4)*84.9 (8.5)*77.9 (13.8)*64.2 (16.8)*98.7 (0.3)
Handle Press100093.4 (1.1)90.8 (1.5)*84.5 (3.3)*81.5 (8.1)*82.0 (14.0)*80.5 (2.0)*98.6 (0.2)
Coffee Button100086.2 (4.5)75.3 (8.3)90.4 (1.4)87.7 (5.0)89.3 (1.3)80.6 (5.8)98.4 (0.4)
Faucet Open200089.5 (2.1)83.6 (2.7)*83.1 (2.2)*80.1 (2.8)*83.1 (3.9)70.9 (16.6)*95.9 (1.1)
Door Open200070.7 (9.8)61.5 (11.3)41.7 (13.5)*48.2 (11.5)31.4 (19.6)64.9 (14.0)96.1 (1.1)
Door Unlock250085.8 (2.6)85.2 (4.7)56.1 (9.1)*65.6 (6.7)*69.2 (5.4)*49.5 (5.7)*97.7 (0.3)
Sweep Into500036.4 (10.0)1.1 (2.8)*35.5 (6.7)28.3 (9.1)6.1 (4.8)11.9 (9.6)94.6 (0.8)
Drawer Open500077.7 (2.7)43.7 (10.6)*35.9 (14.1)*38.1 (12.7)*33.0 (17.2)*32.0 (18.4)*96.1 (1.2)
Hammer1000029.9 (9.2)14.8 (5.9)4.6 (1.7)*7.5 (3.3)29.7 (13.1)16.1 (10.9)94.5 (1.6)
Avg ↑–76.1 (1.8)62.6 (2.4)60.8 (2.5)61.6 (2.5)59.4 (4.1)56.5 (4.2)96.9 (0.3)
Rank ↓–1.303.403.753.853.954.75–

Q1, W4, Q2: "Comparison of order invariance to QPA policy aligned selection"

Both QPA and UBP2 select KK segment pairs (τ1,τ2)(\tau_1,\tau_2) per query round, but from different data and by different rules. QPA samples pairs uniformly from DNon\mathcal{D}^{\text{on}}_N, the NN most recently collected trajectories, so that queries stay close to the current policy's visitation distribution dπd_\pi: SQPA=(τ1(k),τ2(k))k=1K,(τ1(k),τ2(k))∼Unif(DNon).\mathcal{S}_{\text{QPA}} = \\{(\tau_1^{(k)},\tau_2^{(k)})\\}_{k=1}^{K}, \qquad (\tau_1^{(k)},\tau_2^{(k)}) \sim \mathrm{Unif}\big(\mathcal{D}^{\text{on}}_N\big). UBP2 instead draws a candidate set C\mathcal{C} of pairs uniformly from the full replay buffer D⊃DNon\mathcal{D} \supset \mathcal{D}^{\text{on}}_N and keeps the KK pairs with the highest optimistic score: SUBP2=top-K(τ1,τ2)∈C[μE(τ1,τ2)+σE(τ1,τ2)],\mathcal{S}_{\text{UBP2}} = \mathop{\text{top-}K}\limits_{(\tau_1,\tau_2)\in\mathcal{C}} \big[\mu_E(\tau_1,\tau_2) + \sigma_E(\tau_1,\tau_2)\big], where μE\mu_E and σE\sigma_E are the mean and standard deviation, across the EE reward-ensemble members, of the predicted probability that τ1\tau_1 is preferred over τ2\tau_2. QPA therefore restricts where queries come from, while UBP2 ranks queries from the whole buffer by the reward model's prediction and uncertainty.

The following results show the effect of order invariance over 5 seeds on 2 tasks whose optimistic preference selection had the largest gains in success rate (Table 2 in the paper). With the above table, these results compare QPA, UBP2 and UBP2 using the order invariance below.

Query scoreDoor UnlockDrawer OpenOverall
μ+σ\mu+\sigma, order-sensitive (UBP2)84.875.580.4
max⁡(μ,,1−μ)+σ\max(\mu,\\,1-\mu)+\sigma, order-invariant82.765.973.5
AuthorsOpenReview ↗

Q1, W4, Q2: Input to Wilcoxon test, and Number of seeds

Q1, W4, Q2: "What is the input for the Wilcoxon test"

In both the original (paper) and the revised analysis in our previous comment, the test compares seeds, not time points. Each run is summarized by a single number computed from its evaluation success curve over training, so evaluation steps within a run are never treated as separate observations.

Original submission: We used a two-sided paired Wilcoxon signed-rank test per task, comparing UBP2 with each baseline. The per-run value was the area under the evaluation success curve. The ii-th seed of UBP2 was paired with the ii-th seed of the baseline (all methods use seeds 1,…,n1,\dots,n). When the two methods had different numbers of seeds, only the first n=min⁡(nUBP2,,nbaseline)n=\min(n_{\mathrm{UBP2}},\\, n_{\mathrm{baseline}}) seeds were used.

Revised version: Runs of different methods with the same seed index are independent, so pairing them has no statistical basis, and truncating to the smaller seed count discards data.

Q2: "What are the number of seeds in ablations and additional experiments?"

The number of seeds used in our experiments:

ExperimentTasksMethods / variantsSeeds
Main comparison10 Meta-World tasksUBP2, MBP, Rune, MRN10 or 15 per task
QPA, PPE5
Uncertainty ablation10 Meta-World tasks7 ablated variants of UBP25
UBP2, MBP10 or 15 per task
Autotune ablationDrawer OpenVU / DU / RU / all off5
Label noiseHandle Press, Window CloseUBP2, MBP, Rune, MRN5
Human labelsHandle Press, Window CloseUBP2, MBP1
AuthorsOpenReview ↗

Q3, W5: Human Feedback Experiment Label Statistics

Q3, W5: "What is the number of queries marked as no preference? What is the evidence for UBP2 being easier to label?"

Not all "no preference" labels are equal. Most of UBP2's additional no-preference labels come from pairs of two successful segments after the task is solved where both segments are equally good, so "no preference" is the correct label, and it reflects that UBP2 reached stable success early. MBP's no-preference labels come almost entirely from pairs of two failures, which are genuinely hard to tell apart.

TaskMethodNo-pref / pairsBoth successSuccess vs. failBoth fail
Handle PressUBP2769 / 1000 (76.9%)518 (67.4%)3 (0.4%)248 (32.2%)
MBP444 / 1000 (44.4%)34 (7.7%)7 (1.6%)403 (90.8%)
Window CloseUBP2246 / 500 (49.2%)42 (17.1%)30 (12.2%)174 (70.7%)
MBP191 / 500 (38.2%)4 (2.1%)10 (5.2%)177 (92.7%)

The observation that UBP2's segments were easier to label came from the annotators. It is supported by the table below, which shows the outcome of the pairs shown to annotators: success-vs.-failure pairs and pairs where both segments succeed are easier to label than pairs where both fail. A vertical line marks the query index after which UBP2 reached a stable 100% success rate.

TaskMethodPair type / Query index1–100101–200201–300301–400401–500501–600601–700701–800801–900901–1000
Window CloseUBP2Success vs. failure53437 ┃1720–––––
Both succeed1618 ┃3642–––––
Both fail946045 ┃4738–––––
MBPSuccess vs. failure088 ┃1732–––––
Both succeed010 ┃37–––––
Both fail1009192 ┃8061–––––
Handle PressUBP2Success vs. failure1410161413822 ┃34
Both succeed110234456658785 ┃9188
Both fail8580614231271113 ┃68
MBPSuccess vs. failure56132135373841 ┃3954
Both succeed001155814 ┃157
Both fail9594867860585445 ┃4639

These results show that UBP2's advantage holds under real human feedback, and that its queries are easier for annotators to tell apart.

Q3, W5: "How many runs per human feedback experiment method and what is the inter-annotator agreement?"

Our human feedback study has one human-labeled run per method and task (four runs in total), each labeled by a single annotator. Each run required 500 or 1,000 online labels, so labeling every query twice was not feasible, and we therefore cannot report inter-annotator agreement. For scale, these four runs collected 3,000 human labels, more than the prior online PbRL human studies we surveyed, as shown in the table below.

MethodTasks (simulated)Total human labels
PEBBLE [1]DMC Cart, Quadruped, Hopper, Walker~500
Few-Shot PbRL [2]Meta-World Window Close, Door Close; DMC Point Mass, Reacher~1,280
QPA [3]DMC Cheetah Run~200
Uni-RLHF [4]Walker front flip200
UBP2 (ours)Meta-World Window Close, Handle Press3,000
  1. K. Lee, L. Smith, and P. Abbeel. PEBBLE: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning (ICML), 2021.
  2. J. Hejna and D. Sadigh. Few-shot preference learning for human-in-the-loop RL. In Conference on Robot Learning (CoRL), 2022.
  3. X. Hu, J. Li, X. Zhan, Q.-S. Jia, and Y.-Q. Zhang. Query-policy misalignment in preference-based reinforcement learning. In International Conference on Learning Representations (ICLR), 2024.
  4. Y. Yuan et al. Uni-RLHF: Universal platform and benchmark suite for reinforcement learning with diverse human feedback. In International Conference on Learning Representations (ICLR), 2024.
Reviewer_XfHwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work takes a model-based approach for preference based, using TD-MPC2 objective by learning ensemble models of rewards and dynamics. These are used to get an uncertainty guided planning, which combines uncertainty over reward dynamics and value estimates to obtain an optimistic exploration. This work also introduces an optimistic preference label query strategy that focuses on pairs of comparisons that have both high predicted reward and high epistemic uncertainty.

Critical strengths

None

Critical weaknesses

Results are overclaimed:

  • Sample efficiency: this is only shown by curves "reaching high success rates earlier", making this characterization a bit vague in a sense with no quantitative measure. It's easy to argue that looking at the same figures, this notion of "earliness", if ever relevant, is marginal on most of these envs, except maybe for 3 of them (f,g,i). So for a method where one of the main contributions is sample efficiency, this remains a marginal improvement. On a side note, even though the sample efficiency was slightly motivated in the intro, query efficiency remains more important in preference based RL context.
  • Baselines: Lack of important baselines an comparisons to recent state of the art methods: QPA [1] and PPE [2] for instance are recent methods that have a better performance than the proposed older ones in this work. These works might turn out better in overall performance and/or might have a better sample efficiency than what was proposed.
  • Regarding component ablations: The table shows all configs, yet there are no conclusions or evidence of importance of each piece, enabling everything is good because it's just the best balance overall. "Slightly improve individual tasks" and "inconsistent and incur higher regret elsewhere" are almost contradictory, the notion of regret defined here is also based on individual task not an aggregation or average ("maximum per-task drop relative to the best variant")
  • Scope: The experiments are only limited to Metaworld, which is insufficient.
  • Reduced feedback: "retains its effectiveness even when the available feedback is significantly reduced" is an overstatement : (c) is just one method tried on two environments of the same suite. Having only 2 envs is already not significant by iteself, but the fact that only a single method was tested on the envs, does not remove the possibility of the environments not needing less feedback in the end. The result thus can be also interpreted as Drawer Open and Door close can be learned with fewer feedback, which has nothing to do with UPB2.

[1] Xiao Hu, Jianxiong Li, Xianyuan Zhan, Qing-Shan Jia, and Ya-Qin Zhang. Query-policy mis- alignment in preference-based reinforcement learning. In Proceedings of the 12th International Conference on Learning Representations, 2024.

[2] Yiwen Zhu, Jinyi Liu, Pengjie Gu, Yifu Yuan, Zhenxing Ge, Wenya Wei, Zhou Fang, Yujing Hu, and Bo An. Improving reward models with proximal policy exploration for preference-based reinforcement learning. In Advances in Neural Information Processing Systems, volume 38, 2025.

Questions for authors
  • Previous work [1][2] showed and discussed that uniform/random sampling for queries selection remains better in general than disagreement, why is this trivial baseline removed from the comparison ?

[1] Ibarz, Borja, et al. "Reward learning from human preferences and demonstrations in atari." Advances in neural information processing systems 31 (2018)

[2] Xiao Hu, Jianxiong Li, Xianyuan Zhan, Qing-Shan Jia, and Ya-Qin Zhang. Query-policy mis- alignment in preference-based reinforcement learning. In Proceedings of the 12th International Conference on Learning Representations, 2024.

6 comments
AuthorsOpenReview ↗

W5: Reduced Feedback Results, W1: Sample Efficiency

We thank the reviewer for the detailed review. We address every point with new results and clarifications. The uniform sampling baseline was already in the submitted paper. We added QPA and PPE on all 10 tasks; UBP2 ranks first on 9 out of 10 and maintains the best average success. Reduced budget results for all four methods on five tasks show that UBP2 has the highest average success at most budget settings.

W5: "Reduced Feedback is only reported on only 2 envs which can be interpreted as Drawer Open and Door close can be learned with fewer feedback, which has nothing to do with UBP2"

W5: We agree that Fig. 4(c) alone does not separate UBP2's robustness from task difficulty. We had run all four methods on 5 tasks at reduced budgets, but omitted these results for space; we include them below (each run on at least 5 seeds) and will add them to the appendix. UBP2 has the highest average success at every budget. The results of MRN and Rune show that the two tasks in Fig. 4(c) are not simply easy with little feedback.

TaskBudgetFBUBP2MBPRUNEMRN
Drawer OpenFull500078.4 (1.9)57.0 (12.6)43.2 (16.9)27.6 (14.1)
Half250078.7 (2.8)51.9 (14.0)37.7 (13.7)25.9 (16.8)
Least100040.1 (19.5)0.8 (1.5)16.0 (13.0)33.9 (15.7)
Door CloseFull50097.8 (0.5)96.3 (1.1)93.7 (2.3)93.5 (17.7)
Half25097.2 (0.5)94.8 (4.4)95.1 (1.0)95.6 (19.3)
Least10092.5 (18.0)63.9 (23.6)30.0 (20.2)56.4 (22.0)
Window CloseFull50091.8 (1.2)81.1 (11.2)77.8 (16.2)81.6 (10.9)
Half25087.4 (17.3)80.3 (17.1)30.2 (20.6)15.3 (15.7)
Faucet OpenFull200085.9 (15.9)84.1 (3.6)82.4 (3.5)81.1 (3.5)
Half100090.1 (8.0)82.7 (5.6)54.9 (19.7)63.8 (16.1)
Least10016.8 (15.6)0.0 (0.2)0.0 (0.0)0.0 (0.0)
Handle PressFull100093.1 (0.8)91.2 (4.6)84.6 (8.4)79.4 (9.1)
Least10018.9 (11.3)41.9 (16.2)7.1 (5.8)10.9 (4.9)
AverageFull–87.479.173.167.4
Half–88.776.562.661.8
Least–49.821.615.330.1

W1: "Sample efficiency...a bit vague in a sense with no quantitative measure...this remains a marginal improvement."

We respectfully disagree. Our method is quantified following standard methods adopted by prior work and measures both sample efficiency and exhibits substantial gains.

  • Under a fixed query schedule, success over environment steps also measures query efficiency. As in PEBBLE [1] and the methods built on it, including RUNE, MRN, QPA and PPE, preference queries are issued at a fixed rate: in UBP2, a batch of 12–50 queries every 2,500 environment steps until the budget is used (Table 3). Reaching a given success rate in fewer steps therefore also means reaching it with fewer queries, as Section 6 (line 452) states: "We report Interquartile Mean (IQM) success rate over environment steps where faster gains indicate higher sample efficiency in preference-based learning."
  • The gains are quantified. Table 1 reports success averaged over all evaluations during training, i.e. the normalized area under the learning curve. This is a stricter test of query efficiency than averaging only the final evaluations (e.g. the last 10 in PPE), which ignores how quickly success is reached. On this metric UBP2 is best on 9 of 10 tasks, and a Wilcoxon signed-rank test shows a significant improvement over RUNE on 8 of 10 tasks, over MRN on 7 and over MBP on 6, not only on the three tasks the reviewer notes.

[1]. K. Lee, L. Smith, and P. Abbeel. PEBBLE: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning (ICML), 2021.

AuthorsOpenReview ↗

W3: Component Ablations, W4: Only Metaworld Experiments, Q1: Random Sampling Baseline

W3: "No conclusions or evidence of importance of each piece in component ablations..,the notion of regret defined here is also based on individual task not an aggregation or average"

W3: Regret, as we use it, is a standard measure in RL for comparing decisions across settings, and follows prior work: "Regret is defined as the difference between the payoff obtained for a decision, and the optimal payoff that could have been obtained in the same setting with a different decision" [1]. The maximum over tasks is itself an aggregate, the worst-case one, and it complements the average: UBP2 has both the best average success (76.1) and the lowest maximum regret (13.8), so it is the minimax-regret choice among the configurations tested. For example, removing value uncertainty (1101) beats UBP2 on five tasks, such as Door Unlock (88.6 vs. 85.8) and Coffee Button (90.1 vs. 86.2), but collapses on Sweep Into (4.0 vs. 36.4): its largest gain (8.3 points, on Hammer) is a quarter of that loss (32.4). Prior work measures the regret of a policy on an environment against the best policy for that environment, and selects the policy that minimizes its maximum regret over environments [1, App. B; 2, Sec. 2.2; 3, Sec. 3]. We apologize if the sentence the reviewer quoted is confusing, the sentence describes a specific pattern rather than a contradiction. We will rephrase it in the paper.

W4: "The experiments are only limited to Metaworld"

W4: We focus on Meta-World because it is the standard benchmark for robotic manipulation in PbRL; RUNE, for example, evaluates only on Meta-World. Within it, our 10 tasks span a wide range of difficulty: feedback budgets from 500 to 10,000 queries, and success of the model-free baselines from above 90% to below 10%. This is more Meta-World tasks than the methods we compare with use: RUNE 8, MRN 6, PEBBLE 6, QPA 3 and PPE 7.

Q1: "Uniform/random sampling for queries selection ... removed from the comparison"

Q1: We apologize for the confusion: the uniform baseline is in the paper, but we did not state this clearly. Both MBP and the ablation variant 1110 select preference queries uniformly at random; 1110 is UBP2 with optimistic selection replaced by uniform sampling. Table 2 therefore compares the two directly: 1111 outperforms 1110 on all 10 tasks. This does not contradict prior work [4,5], which found uniform sampling better than disagreement-based selection. UBP2 does not use disagreement alone: its score adds the ensemble's mean predicted preference to its disagreement.

[1] M. Dennis, N. Jaques, E. Vinitsky, A. Bayen, S. Russell, A. Critch, and S. Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Advances in Neural Information Processing Systems (NeurIPS), 2020.

[2] M. Beukman, S. Coward, M. Matthews, M. Fellows, M. Jiang, M. Dennis, and J. Foerster. Refining minimax regret for unsupervised environment design. In International Conference on Machine Learning (ICML), 2024.

[3] L. Xu, A. Perrault, F. Fang, H. Chen, and M. Tambe. Robust reinforcement learning under minimax regret for green security. In Conference on Uncertainty in Artificial Intelligence (UAI), 2021.

[4] B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei. Reward learning from human preferences and demonstrations in Atari. In Advances in Neural Information Processing Systems (NeurIPS), 2018.

[5] X. Hu, J. Li, X. Zhan, Q.-S. Jia, and Y.-Q. Zhang. Query-policy misalignment in preference-based reinforcement learning. In International Conference on Learning Representations (ICLR), 2024.

AuthorsOpenReview ↗

W2: Extended baselines comparison

W2: "Lack of important baselines comparisons to recent state of the art methods: QPA [1] and PPE [2]"

W2: We have added QPA and PPE as baselines on all 10 tasks, using the same feedback budgets as the other methods. UBP2 still ranks first on every task except Coffee Button, where RUNE already outperformed UBP2 in Table 1 and remains the best method on that task.

TaskMax FBUBP2MBPRuneMRNQPAPPETD-MPC2
Door Close50097.8 (1.0)95.6 (0.0)*91.5 (10.0)*94.2 (0.0)*92.2 (20.0)94.2 (20.0)98.8 (0.0)
Window Close50093.1 (1.0)74.7 (13.5)*85.2 (9.2)*84.9 (10.2)*77.9 (14.0)64.2 (20.0)98.7 (0.0)
Handle Press100093.4 (0.0)90.8 (4.0)*84.5 (8.1)*81.5 (10.3)*82.0 (13.6)80.5 (2.0)98.6 (0.0)
Coffee Button100086.2 (3.7)75.3 (10.7)90.4 (1.0)87.7 (11.0)89.3 (5.8)80.6 (13.0)98.4 (0.0)
Faucet Open200089.5 (7.9)83.6 (5.0)*83.1 (1.4)*80.1 (0.0)*83.1 (0.0)70.9 (19.4)95.9 (0.0)
Door Open200070.7 (10.4)61.5 (11.7)41.7 (13.1)*48.2 (11.4)31.4 (18.3)64.9 (19.6)96.1 (0.0)
Door Unlock250085.8 (2.1)85.2 (10.0)56.1 (13.1)*65.6 (9.8)*69.2 (0.0)49.5 (4.0)97.7 (0.0)
Sweep Into500036.4 (12.3)1.1 (2.0)*35.5 (9.3)28.3 (14.5)6.1 (14.6)11.9 (18.3)94.6 (2.0)
Drawer Open500077.7 (4.9)43.7 (12.8)*35.9 (12.8)*38.1 (14.9)*33.0 (24.5)32.0 (24.5)96.1 (0.0)
Hammer1000029.9 (11.5)14.8 (9.1)4.6 (0.9)*7.5 (7.2)*29.7 (21.6)16.1 (20.9)94.5 (0.0)
Avg ↑–76.162.660.861.659.456.596.9
Rank ↓–1.303.403.753.853.954.75–
Reviewer_hs8yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UBP2 addresses preference-based reinforcement learning, where an agent learns a reward model from pairwise comparisons of trajectory segments.

The method maintains ensembles for reward, dynamics, and value prediction. It combines their uncertainty estimates with predicted returns during planning and uses an optimistic criterion to select preference queries.

On 10 MetaWorld tasks, the reported average training-performance metric increases from 62.6 for the non-optimistic model-based baseline to 76.1 for UBP2. These values summarize performance over training and should not be interpreted simply as final-policy success rates.

Critical strengths
  1. A reasonably broad benchmark evaluation. The paper evaluates 10 manipulation tasks with 10–15 seeds per task.
  2. Useful ablations and robustness experiments. The experiments examine uncertainty components, feedback budgets, and noisy preferences.
  3. Some validation with human feedback. The human-preference experiment complements results based on simulated preferences, although its scale is small.
  4. Compute costs are reported. This makes the trade-off between interaction efficiency and computation easier to assess.
Critical weaknesses
  1. Interaction efficiency comes with substantial computation. Reported training times are approximately 30–47 hours, compared with roughly 2.3–2.4 hours for RUNE and MRN in the listed comparisons. The evidence supports improved environment-interaction efficiency, while overall computational efficiency remains a limitation.
  2. Human-feedback evidence is preliminary. The study involves only two tasks and two annotators, which limits conclusions about robustness to diverse human preferences.
Questions for authors

It seems that the preference-query score appears sensitive to arbitrary pair ordering. Algorithm 2 scores a pair using μ(p)+σ(p),p=P(τ1≻τ2).\mu(p)+\sigma(p), \qquad p=P(\tau_1\succ\tau_2). Swapping the segments changes the score to 1−μ(p)+σ(p).1-\mu(p)+\sigma(p). Thus, the same unordered comparison can receive a different priority depending on which segment is listed first. Could authors explain more in detail motivations behind this design choice, and how this will affect experimental results?

3 comments
AuthorsOpenReview ↗

W2: Human-feedback evidence

We thank the reviewer for the careful review and for recognizing the breadth of our benchmark. We give our hypothesis for why the order-sensitive optimistic preference selection works, together with an experiment showing that it outperforms an order-invariant version. We clarify the scope and insights from the human-feedback experiments. Finally we outline concrete steps for reducing the compute time, including a parallelized implementation that trains in about 5 hours.

W2: The study involves only two tasks and two annotators.

We agree that more tasks would strengthen our evidence, however, our experiments use more than twice the human labels than the relevant prior work shown in the table below. Human studies in online PbRL are costly [1], because the annotator must label throughout training: each of our human-labelled runs required 500 or 1,000 labels and several hours of labelling, for 3,000 labels in total.

MethodTasks (simulated)Total human labels
PEBBLE [1]DMC Cart, Quadruped, Hopper, Walker~500
Few-Shot PbRL [2]Meta-World Window Close, Door Close; DMC Point Mass, Reacher~1,280
QPA [3]DMC Cheetah Run~200
Uni-RLHF [4]Walker front flip200
UBP2 (ours)Meta-World Window Close, Handle Press3,000

Regarding robustness to diverse human preferences: although humans are more prone to mislabeling, we attribute the matched (window close) or improved (Handle press) performance to UBP2 optimistic preference selection's easier queries for annotations. When the choice is clear, annotators have less room to disagree, so we expect UBP2's clearer queries to also reduce the effect of annotator diversity on performance. This is supported by the following table showing the state of the pairs that were shown to annotators; it is easier to annotate successful vs failure pairs or both succeed than it is for both failures. The vertical lines indicate that UBP2 has reached a stable 100% success rate after that query index.

TaskMethodPair type / Query index1–100101–200201–300301–400401–500501–600601–700701–800801–900901–1000
Window CloseUBP2Success vs. failure53437 ┃1720–––––
Both succeed1618 ┃3642–––––
Both fail946045 ┃4738–––––
MBPSuccess vs. failure088 ┃1732–––––
Both succeed010 ┃37–––––
Both fail1009192 ┃8061–––––
Handle PressUBP2Success vs. failure1410161413822 ┃34
Both succeed110234456658785 ┃9188
Both fail8580614231271113 ┃68
MBPSuccess vs. failure56132135373841 ┃3954
Both succeed001155814 ┃157
Both fail9594867860585445 ┃4639

These results show that UBP2's advantage holds under real human feedback, and that its queries are easier for annotators to tell apart, which we hope make the case that the current set of human feedback experiments are sufficient for our intended claims.

The main aim of our human feedback experiment is to prove that UBP2's gains, measured with an oracle that labels preferences from the hand-crafted reward, also translate to gains under real human feedback. Our experiments show UBP2's gains do translate: UBP2 performed as well with human labels as with oracle labels on both tasks, while MBP's average success on Handle Press fell from 91.6 to 45.4. On Handle Press, UBP2 with human labels was even more query efficient than with oracle labels, reaching stable 100% success after 384 queries vs. 624.

TaskMethodAvg. success over 1M steps (%), HumanAvg. success over 1M steps (%), OracleQueries to stable 100% success, HumanQueries to stable 100% success, Oracle
Window CloseUBP295.095.0276236
MBP91.189.1467500
Handle PressUBP295.593.1707808
MBP45.491.6never1000
  1. K. Lee, L. Smith, and P. Abbeel. PEBBLE: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. (ICML), 2021.
  2. J. Hejna and D. Sadigh. Few-shot preference learning for human-in-the-loop RL. (CoRL), 2022.
  3. X. Hu, J. Li, X. Zhan, Q.-S. Jia, and Y.-Q. Zhang. Query-policy misalignment in preference-based reinforcement learning.(ICLR), 2024.
  4. Y. Yuan et al. Uni-RLHF: Universal platform and benchmark suite for reinforcement learning with diverse human feedback.(ICLR), 2024.
AuthorsOpenReview ↗

W1: Computational Efficiency and Q1: Order Invariance

W1 : Overall computational efficiency remains a limitation

W1: We agree that UBP2's training time is higher than that of the model-free baselines (30-47 hours vs. 2.3-2.4 hours). Much of this gap (quantified below) is inherent to model-based planning. A model-free agent chooses each action with one pass through its policy network, whereas UBP2 plans each action by rolling out its learned dynamics for many candidate trajectories and evaluating its reward, dynamics and value ensembles on each of them for 6 iterations. The table below reports the training time of TD-MPC2[1], which reflects the cost of model-based planning itself, alongside that of UBP2. We also report training times in Appendix B.2 and will state this cost as a limitation in the main text.

TaskMethodHorizonHorizon incr.Planner iterationsSamplesGPUTime (h)Time incr.
Window CloseTD-MPC2 (baseline)7—6512H10020.4—
Window CloseUBP27—6512H10029.5+45%
Window CloseUBP211+57%6512H10041.7+104%

Reducing the compute time is future work, but it is feasible; we have implemented a version of UBP2 on a vectorized TD-MPC2[1] that collects data from 10 environments in parallel and performs fewer gradient updates per environment step, and trains in under 5 hours [on drawer open for 1M steps]. None of the reported results in this work use this version, and fewer updates per step may change sample efficiency, so we do not claim that it matches the reported performance; it shows only that the training time can be reduced substantially.

Q1 : It seems that the preference-query score appears sensitive to arbitrary pair ordering...Could authors explain...this design choice, and how this will affect experimental results?

The reviewer is right, this is a design choice and we hypothesize below why it matters, and will explain this in the paper. The score μ+σ\mu+\sigma depends on which segment is listed first, where μ\mu is the ensemble's mean probability that the first segment is preferred and σ\sigma is its standard deviation across ensemble members. The two segments are sampled independently from the buffer, so the order is random, and a confident pair scores high only when its preferred segment happens to be listed first. We have found this to work well in practice. We hypothesize that it balances the two terms: by randomly discarding about half of the confident pairs, it limits how much μ\mu dominates σ\sigma without a tuned weight between them, unlike the planning objective (Paper Eq.1), whose uncertainty weights are auto-tuned (Eq.2). The results below show such sensitivity for 2 tasks whose optimistic preference selection contributed to the highest gains in success rate Door Unlock and Drawer Open over 5 seeds (Paper Table 2)

Query scoreDoor UnlockDrawer OpenOverall
μ+σ\mu+\sigma, order-sensitive (UBP2)84.875.580.4
max⁡(μ,,1−μ)+σ\max(\mu,\\,1-\mu)+\sigma, order-invariant82.765.973.5

[1] N. Hansen, H. Su, and X. Wang. TD-MPC2: Scalable, robust world models for continuous control. In International Conference on Learning Representations (ICLR), 2024.

AuthorsOpenReview ↗

Kind reminder for feedback or clarification before the conclusion of the discussion period.

With less than 2 days remaining in the discussion period, we wanted to follow up on our previous message. We have done our best to address the concerns raised within the constraints of the rebuttal process, and we would greatly appreciate any further feedback or clarification before the discussion period concludes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.