- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces UBP2, a model-based preference-based RL method built on TD-MPC2. It uses ensembles to estimate reward, dynamics, and value uncertainty, incorporating uncertainty bonuses into MPC planning. Preference queries are selected from the replay buffer using a UCB criterion, and a learned policy takes over once the query budget is exhausted. The paper establishes a sublinear regret bound under GP/RKHS assumptions. Across 10 MetaWorld tasks, UBP2 achieves an average IQM success rate of 76.1%, outperforming RUNE, MRN, and MBP. A small-scale study with real human preferences further compares UBP2 with MBP.
Critical strengths
- The non-optimistic variant MBP (62.6) performs on par with the model-free baselines, so the improvement comes mainly from optimistic exploration rather than from the model-based backbone.
- Exploration during planning and query selection share one uncertainty view, and the uncertainty coefficients are tuned automatically instead of by hand-designed schedules (Table 10).
- Turning the uncertainty bonus into a penalty at test time sharply degrades performance (Table 7), which indicates that the learned uncertainty acts as an exploration signal rather than a risk proxy.
Critical weaknesses
- The components are established: TD-MPC2, ensemble-based optimism (H-UCRL, SOMBRL, MaxInfoRL) and reward-uncertainty bonuses (RUNE).
- The ablation (Table 2) does not support the claim that all three uncertainties are needed. Removing reward uncertainty from planning (0111) costs only 0.6 points on average and improves 4/10 tasks; most optimistic variants fall within 67.7–76.1, with per-task SEs of 10–20.
- The theory treats the reward as directly observed noisy values (GP regression, Eqs. 12–13), omits query selection, and largely restates the bound of Sukhija et al. (2025b).
- Query–policy alignment work (QPA, Hu et al., ICLR 2024), whose motivation overlaps closely with the query-selection argument, is neither discussed nor compared.
- The human study (2 annotators, 2 tasks, compared only against MBP) contains no measurement supporting "easier to label".
- With 10% flipped labels, Window Close drops from 92.8 to 42.4, yet this is described as robust.
- Fig. 4(b) reports consistent gains from horizon 7 to 11, but Table 8 shows horizon 11 is worse on five other tasks.
- Wall-clock time is 12–20× that of the baselines (Table 6).
Questions for authors
- How does selection perform if the score is made order-invariant, or if it uses the predicted return of the segments plus uncertainty? How does it compare with QPA-style policy-aligned selection?
- What does the Wilcoxon test pair, seeds or time points? Please report error bars for the time-averaged IQM actually shown (not only the final-step SE), and the number of seeds used in the ablation, noise and human experiments.
- How many runs per method were performed? What fraction of queries were marked "no preference" and discarded for each method, and what was the inter-annotator agreement?
- How do the model-free baselines perform with a compute budget comparable to UBP2's, and how does UBP2 perform with cheaper planning (fewer CEM samples or iterations)? Please also show success versus number of labels, not only versus environment steps.
- How was the per-task horizon chosen, and on which data? What are the results with a single horizon for all tasks?
7 comments
Q1, W4, Q2: Input to Wilcoxon test, and Number of seeds
Q1, W4, Q2: "What is the input for the Wilcoxon test"
In both the original (paper) and the revised analysis in our previous comment, the test compares seeds, not time points. Each run is summarized by a single number computed from its evaluation success curve over training, so evaluation steps within a run are never treated as separate observations.
Original submission: We used a two-sided paired Wilcoxon signed-rank test per task, comparing UBP2 with each baseline. The per-run value was the area under the evaluation success curve. The -th seed of UBP2 was paired with the -th seed of the baseline (all methods use seeds ). When the two methods had different numbers of seeds, only the first seeds were used.
Revised version: Runs of different methods with the same seed index are independent, so pairing them has no statistical basis, and truncating to the smaller seed count discards data.
Q2: "What are the number of seeds in ablations and additional experiments?"
The number of seeds used in our experiments:
| Experiment | Tasks | Methods / variants | Seeds |
|---|---|---|---|
| Main comparison | 10 Meta-World tasks | UBP2, MBP, Rune, MRN | 10 or 15 per task |
| QPA, PPE | 5 | ||
| Uncertainty ablation | 10 Meta-World tasks | 7 ablated variants of UBP2 | 5 |
| UBP2, MBP | 10 or 15 per task | ||
| Autotune ablation | Drawer Open | VU / DU / RU / all off | 5 |
| Label noise | Handle Press, Window Close | UBP2, MBP, Rune, MRN | 5 |
| Human labels | Handle Press, Window Close | UBP2, MBP | 1 |
Q3, W5: Human Feedback Experiment Label Statistics
Q3, W5: "What is the number of queries marked as no preference? What is the evidence for UBP2 being easier to label?"
Not all "no preference" labels are equal. Most of UBP2's additional no-preference labels come from pairs of two successful segments after the task is solved where both segments are equally good, so "no preference" is the correct label, and it reflects that UBP2 reached stable success early. MBP's no-preference labels come almost entirely from pairs of two failures, which are genuinely hard to tell apart.
| Task | Method | No-pref / pairs | Both success | Success vs. fail | Both fail |
|---|---|---|---|---|---|
| Handle Press | UBP2 | 769 / 1000 (76.9%) | 518 (67.4%) | 3 (0.4%) | 248 (32.2%) |
| MBP | 444 / 1000 (44.4%) | 34 (7.7%) | 7 (1.6%) | 403 (90.8%) | |
| Window Close | UBP2 | 246 / 500 (49.2%) | 42 (17.1%) | 30 (12.2%) | 174 (70.7%) |
| MBP | 191 / 500 (38.2%) | 4 (2.1%) | 10 (5.2%) | 177 (92.7%) |
The observation that UBP2's segments were easier to label came from the annotators. It is supported by the table below, which shows the outcome of the pairs shown to annotators: success-vs.-failure pairs and pairs where both segments succeed are easier to label than pairs where both fail. A vertical line marks the query index after which UBP2 reached a stable 100% success rate.
| Task | Method | Pair type / Query index | 1–100 | 101–200 | 201–300 | 301–400 | 401–500 | 501–600 | 601–700 | 701–800 | 801–900 | 901–1000 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Window Close | UBP2 | Success vs. failure | 5 | 34 | 37 ┃ | 17 | 20 | – | – | – | – | – |
| Both succeed | 1 | 6 | 18 ┃ | 36 | 42 | – | – | – | – | – | ||
| Both fail | 94 | 60 | 45 ┃ | 47 | 38 | – | – | – | – | – | ||
| MBP | Success vs. failure | 0 | 8 | 8 ┃ | 17 | 32 | – | – | – | – | – | |
| Both succeed | 0 | 1 | 0 ┃ | 3 | 7 | – | – | – | – | – | ||
| Both fail | 100 | 91 | 92 ┃ | 80 | 61 | – | – | – | – | – | ||
| Handle Press | UBP2 | Success vs. failure | 14 | 10 | 16 | 14 | 13 | 8 | 2 | 2 ┃ | 3 | 4 |
| Both succeed | 1 | 10 | 23 | 44 | 56 | 65 | 87 | 85 ┃ | 91 | 88 | ||
| Both fail | 85 | 80 | 61 | 42 | 31 | 27 | 11 | 13 ┃ | 6 | 8 | ||
| MBP | Success vs. failure | 5 | 6 | 13 | 21 | 35 | 37 | 38 | 41 ┃ | 39 | 54 | |
| Both succeed | 0 | 0 | 1 | 1 | 5 | 5 | 8 | 14 ┃ | 15 | 7 | ||
| Both fail | 95 | 94 | 86 | 78 | 60 | 58 | 54 | 45 ┃ | 46 | 39 |
These results show that UBP2's advantage holds under real human feedback, and that its queries are easier for annotators to tell apart.
Q3, W5: "How many runs per human feedback experiment method and what is the inter-annotator agreement?"
Our human feedback study has one human-labeled run per method and task (four runs in total), each labeled by a single annotator. Each run required 500 or 1,000 online labels, so labeling every query twice was not feasible, and we therefore cannot report inter-annotator agreement. For scale, these four runs collected 3,000 human labels, more than the prior online PbRL human studies we surveyed, as shown in the table below.
| Method | Tasks (simulated) | Total human labels |
|---|---|---|
| PEBBLE [1] | DMC Cart, Quadruped, Hopper, Walker | ~500 |
| Few-Shot PbRL [2] | Meta-World Window Close, Door Close; DMC Point Mass, Reacher | ~1,280 |
| QPA [3] | DMC Cheetah Run | ~200 |
| Uni-RLHF [4] | Walker front flip | 200 |
| UBP2 (ours) | Meta-World Window Close, Handle Press | 3,000 |
- K. Lee, L. Smith, and P. Abbeel. PEBBLE: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning (ICML), 2021.
- J. Hejna and D. Sadigh. Few-shot preference learning for human-in-the-loop RL. In Conference on Robot Learning (CoRL), 2022.
- X. Hu, J. Li, X. Zhan, Q.-S. Jia, and Y.-Q. Zhang. Query-policy misalignment in preference-based reinforcement learning. In International Conference on Learning Representations (ICLR), 2024.
- Y. Yuan et al. Uni-RLHF: Universal platform and benchmark suite for reinforcement learning with diverse human feedback. In International Conference on Learning Representations (ICLR), 2024.
Q1, W4 , Q2: Comparison to QPA policy aligned selection. Report error bars for the time-averaged IQM
We thank the reviewer for the careful reading and for suggestions that have strengthened the paper. In response, we add QPA and PPE on all 10 tasks. We extend the statistics: error bars are now bootstrap standard errors of the time-averaged IQM (10,000 resamples over runs), significance uses a two-sided Mann–Whitney U test on per-run scores, and we report seed counts. We also test an order-invariant query score and report human-study label statistics.
We have added QPA and PPE as baselines on all 10 tasks, using the same feedback budgets as the other methods. UBP2 still ranks first on every task except Coffee Button, where RUNE already outperformed UBP2 in Table 1 and remains the best method on that task.
We report below IQM success over seeds at each evaluation, averaged over training, with the bootstrap standard error of this time-averaged IQM in parentheses (10,000 resamples of runs). The best result per task is shown in bold. The Avg row is bootstrapped by resampling runs within each task. Asterisks here mark a significant difference from UBP2 under a two-sided unpaired Wilcoxon rank-sum (Mann–Whitney U) test on per-run success averaged over training (p<0.05). We report this test because it uses all seeds of each method, whereas the paired test in the original Table 1 can only use matched seeds. The significant differences for MBP, Rune and MRN are unchanged except Hammer vs. MRN (p=0.074); the unpaired test additionally detects significant differences against QPA and PPE, which have only 5 seeds.
Both QPA and UBP2 select K segment pairs (τ1,τ2) per query round, but from different data and by different rules. QPA samples pairs uniformly from DNon, the N most recently collected trajectories, so that queries stay close to the current policy's visitation distribution dπ: SQPA=(τ1(k),τ2(k))k=1K,(τ1(k),τ2(k))∼Unif(DNon). UBP2 instead draws a candidate set C of pairs uniformly from the full replay buffer D⊃DNon and keeps the K pairs with the highest optimistic score: SUBP2=(τ1,τ2)∈Ctop-K[μE(τ1,τ2)+σE(τ1,τ2)], where μE and σE are the mean and standard deviation, across the E reward-ensemble members, of the predicted probability that τ1 is preferred over τ2. QPA therefore restricts where queries come from, while UBP2 ranks queries from the whole buffer by the reward model's prediction and uncertainty.
The following results show the effect of order invariance over 5 seeds on 2 tasks whose optimistic preference selection had the largest gains in success rate (Table 2 in the paper). With the above table, these results compare QPA, UBP2 and UBP2 using the order invariance below.