acceptodds
Under review as a conference paper at ICLR 2027

Learning Robotic Skills from Offline Preference-based RL

Abstract

Preference-based reinforcement learning (PbRL) has become a promising framework for training agents in the absence of explicit reward functions, offering a principled approach to steer robot agents toward fulfilling human intentions. Existing methods typically follow a two-stage pipeline: learning a reward model from preferences, then applying off-the-shelf RL algorithms with the learned reward. However, since these learned proxy rewards are not the ground truth, their direct use in standard RL algorithms inevitably introduces compounding errors. To address this issue, we propose Preference-Score Actor-Critic (PSAC), an actor-critic framework that augments the critic update with a preference-score function derived directly from preference data, instead of relying solely on the learned reward model. Extensive experiments demonstrate that PSAC achieves state-of-the-art performance on 14 out of 17 D4RL benchmark tasks. Furthermore, we validate the efficacy of the proposed preference score augmentation through ablation studies, demonstrating significant performance improvements. Finally, using fewer than 700 manually labeled preference pairs, PSAC effectively controls a Unitree G1 robot to navigate rough terrain. Code and video demonstrations are available at https://anonymous.4open.science/r/PSAC-6686.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Preference-based RL, Actor-critic

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zqPPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a method for learning from offline demonstrations for which a labelled preference dataset is also available. The method combines a diffusion policy approach to behavior cloning with a q-value regularization, where the q-value objective if modified to include an estimate of the value of a state action pair according to the preference data. The preference score is distinct from the reward function learned from the labelled preference data. The D4RL dataset is used as the source of training data, and several baselines are compared against. The results suggest the proposed method PSAC improves upon prior work.

Critical strengths

None

Critical weaknesses
  • The paper does not provide sufficient evidence to justify the proposed modification to policy learning is the source of the gains. The policy architecture and behavior cloning objective is based on a diffusion policy, whereas the baselines that are compared against would have relied on a MLP. The use of a diffusion policy alone could explain a large portion of the observed performance gains. Additionally, another contributing factor could be the increase in number of parameters that are used in the PSAC method. As the overall size of the models used in the different approaches (PSAC vs. baselines) is not mentioned, this is difficult to assess. Contributing to understanding the benefits of the proposed preference score, there are no experiments examining how the preference score mitigates negative effects on policy learning from a poor reward function. The need for the preference score is motivated to address errors in the reward function. However, the preference score appears to be learned from the same data as the reward function, so it is not clear how the preference score addresses errors in the learned reward function. There should be experiments explicitly looking at noise in the reward model to show that the introduced preference score is able to mitigate different amounts of noise in the reward function. Additionally, there should be experiments looking at the extent to which the preference score and the reward are different and provide distinct information to the learning process.
  • The paper does not appropriately motivate the design decisions made and how these are distinct from what has been done in prior work (Note about prior work: there are only two sentences to cover all prior work on offline RL). For example, why exactly is the preference score expected to address errors in the reward function when it is trained on the same data? What is the motivation and reason for select the given form of the preference score? How is the preference score meaningfully distinct from the reward? Can the learned reward be used in place of the preference score? What is the motivation for incorporating the Q-value as an auxiliary to the BC loss? Is the addition of the Q-value as an auxiliary loss as contribution of this work, because there isn't a citation for it? What about the prior works motivates that modification to critic learning should have performance gains?
  • Much of the space of the paper is taken up with unnecessary results. For example, the complete Table 1 with results per demonstration quality despite no analysis at that level, an image of the UniTree task that provides no additional information, and the weirdly massive Table 3 for an experiment that bring limited additional analysis and learnings as an experiment distinct from the D4RL experiments. This has led to a lack of space for related work, a conclusion, and appropriate ablations.
Questions for authors
  • Why exactly is the preference score expected to address errors in the reward function when it is trained on the same data?
  • What is the motivation and reason for select the given form of the preference score?
  • How is the preference score meaningfully distinct from the reward?
  • Can the learned reward be used in place of the preference score?
  • What is the motivation for incorporating the Q-value as an auxiliary to the BC loss?
  • Is the addition of the Q-value as an auxiliary loss to the BC loss a contribution of this work, because there isn't a citation for it?
  • What about the prior works motivates that modification to critic learning should have performance gains?
Reviewer_ShBgOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

this paper proposes Preference-Score Actor-Critic (PSAC) for offline PbRL. Next to a Preference Transformer reward, it fits a state-desirability function V_ψ directly on the preference labels and scores an action by its predicted next-state desirability minus the κ-quantile over actions from the behavior distribution; this preference score is added inside the bootstrapped target of a Diffusion-QL style critic. PSAC has the highest mean among the compared preference-based methods on 14 of 17 D4RL tasks, with the biggest gain on AntMaze, and in a Unitree G1 simulation it learns rough-terrain navigation from 676 preference pairs.

Critical strengths

(1) The “PSAC w/o ps” column is a clean ablation. With the same reward model, backbone and data, adding the score lifts the AntMaze average from 59.7 to 79.4, and all four AntMaze datasets improve.

(2) Using the same labels twice, once for a proxy reward and once for a state ranking fitted without Bellman backups, is simple and could be bolted onto other offline PbRL pipelines.

Critical weaknesses

(1) Table 1 can’t be checked as submitted. Section 5.1 says baseline numbers are taken “either from their original paper or from our own reproduction”, but not which is which, so I checked the likely sources. The DQL column matches [1], which reports mean ± standard error over 5 models with offline checkpoint selection; the AntMaze rows of PT and LSTM match [2] (IQL, 8 runs, ± standard deviation); the Gym, pen and kitchen rows of IQL, PT and DPPO match [3] (5 seeds); PSAC is the mean of the final 10 evaluations over 4 seeds. The PT column mixes sources for the same method and labels: on hopper-medium-replay and walker2d-medium-replay it uses the rerun from [3] (59.7±25.8 and 43.3±39.8), while [2] reports 84.54±4.07 and 71.27±10.30. With the number from [2], PSAC’s lead on hopper-medium-replay over the best preference-based entry drops from 15.1 points (DPPO, 73.2) to 3.8. Every traced PT entry is PT+IQL, so that column also changes the backbone. Two copied entries look mistranscribed (IQL’s kitchen-partial and kitchen-mixed are swapped relative to [3], and [2] gives LSTM 62.88±5.99 on medium-play, not ±6.9).

(2) AntMaze has the biggest gain and the weakest comparison. The AntMaze results in [1] are on the antmaze-*-v0 datasets with a modified reward, while the human labels from [2] come with v2. The DPPO paper [3] left AntMaze out (Appendix F) because of a goal-setting bug in the environment, so the DPPO AntMaze entries (5.5, 3.6, 0.0, 0.0) are presumably your own runs of a method its authors never set up for AntMaze, and the IQL AntMaze entries match no source (large-diverse 30.3, against 44.38 in [2]). A preference-only method at 86.8 on large-play, about twice the task-reward baselines in the same table, needs an explanation.

(3) With λP̂(s′, a′) inside the bootstrapped target, the fixed point of Eq. 11 is the Q-function of the reward r_φ(s, a) + λγE[P̂(s′, a′)], so PSAC adds a shaping term to the objective. Section 4 instead says the score helps “rectify the Q-value learning, effectively compensating for potential biases or errors inherent in the reward model”, and no experiment checks that mechanism.

References
[1] Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning, ICLR 2023.
[2] Preference Transformer: Modeling Human Preferences using Transformers for RL, ICLR 2023.
[3] Direct Preference-based Policy Optimization without Reward Modeling, NeurIPS 2023.

Questions for authors

Q1. Please give the source and protocol of every Table 1 entry, and rerun IQL, DQL and PT under PSAC’s protocol. Which AntMaze version does PSAC use, and how were the DPPO and IQL AntMaze runs configured?

Q2. Does P̂ still help when added on top of the ground-truth reward, and does the per-task gain track the reward model’s error? Without these checks I’d describe the contribution as preference-based reward shaping.

Q3. Several details are missing or inconsistent. κ and N aren’t given, nor how a_1, …, a_N ∼ µ(·|s) are drawn when the behavior policy is unknown (from the diffusion BC model?), nor the transition model’s architecture and accuracy. Eq. 4–5 sum V_ψ over the states of a segment, but Appendix B.1 (Eq. 17) trains it on single-step state pairs; the critic has L = 16 residual blocks in Appendix B.1 and 4 in Table 4; and since the Preference Transformer outputs a segment score s(τ), how is the per-step r_φ(s, a) in Eq. 11 obtained?

Q4. The α values in Table 5 equal Diffusion-QL’s η for every task covered in [1], and those were tuned with task rewards. That’s fine, but please say so. How were max-Q backup (on for halfcheetah-mr/me, hopper-mr and all four Adroit tasks, where [1] uses it only on AntMaze) and the policy learning rate chosen without task rewards?

References
[1] Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning, ICLR 2023.

Reviewer_TVs3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to train a "value" function on preference data, on top of the ordinary reward function, and shows that such a value function is helpful in PBRL settings when used to predict the "desirability" of the next state. The authors show good results on D4RL benchmarks and also use it to train policies for humanoids.

Critical strengths
  • The paper is overall easy to follow.
  • Good ablations (PSAC vs PSAC w/o ps) to show the benefit of the core component of the proposed method.
Critical weaknesses
  • The proposed "value" function is not the canonical value function in RL, i.e. it does not measure the expected future return given a state and policy, but it shares the same notation. The authors have acknowledged this, but it adds unnecessary confusion when reading the paper. For example, the proposed V and the traditional V (in Eq 6) measure completely different quantities, and calling this comparison direct vs indirect is quite confusing to me.
  • The setup for comparison against DPPO is not documented. Specifically, I wonder if everything else is held the same except for the learning algorithm, i.e. does your DPPO implementation use the same diffusion-based policy architecture? In fact, the PSAC w/o ps baseline performs on par with or better than LSTM, PT and DPPO on average, indicating that there seem to be improvements brought by the policy architecture choice or other implementation details that are not central to the core thesis (the proposed value function) of this paper.

However, both issues are not fundamental, and overall the experimental results are positive, so I lean towards acceptance. The reasons for not rating higher are partially due to the previous two issues (fixable), and also the fact that the overall method is on the incremental side.

Questions for authors
  • Does your DPPO implementation use the same diffusion-based policy architecture?
  • If I understand correctly, PSAC w/o PS is essentially PT with the diffusion actor, right? Then the gap between the two is pretty big. What are the contributing factors?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.