- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a new reward factorization framework named PrefGeo for preference-based reinforcement learning. PrefGeo addresses the factorization ambiguity and balances different reward directions by decoupling the learning of reward bases and coefficients. PrefGeo first learns multiple reward heads on each source context, then extracts compact bases from learned heads by minimizing the volume spanned by the bases while ensuring reconstruction and non-degeneracy. The bases enable few-shot adaptation by only learning the coefficients. Experiments are conducted in both simulation and real-robot settings, showing superior performance of the proposed method and the necessity of each module.
Critical strengths
- The paper is well-written. The methodology is clearly described. Limitations are explicitly discussed.
- The paper presents a real-robot experiment.
Critical weaknesses
- This paper assumes that a target context shares the preference criteria underlying the source contexts. This assumption can make the algorithm sensitive to the selection of source tasks. However, this assumption is not validated experimentally, and the sensitivity of performance to the source tasks is not sufficiently evaluated.
- This paper considers an idealized preference setting. Noise in the labeling process is not considered (page 22, line 1147). Additionally, no human preference labels are collected in the real robot experiment (page 28, line 1482). The constructed preference could largely deviate from that of real human labelers.
Questions for authors
- The Fisher metric G is derived at a neutral head with probability 0.5. Will it induce a large bias when the actual probability deviates from 0.5? Can you report the accuracies for transitions with different probability of being preferred, respectively?
- How does the source domain selection affect the performance of PrefGeo? I suggest the authors report the performance under several different source domain selection schemes, including the number and the selection of the source domains. In the current experiments, the source tasks are randomized for some domains, which could entangle the effect of source domain selection and the randomness in the algorithm. Additionally, It would be better if the reward simplex (Fig. 2(b)) could be visualized in these experiments.
- Can you quantify to which extent the target domain preferences can be expressed by the learned reward bases?
- Is the proposed method sensitive to labeling noise, such as those mentioned in B-Pref?
- Please provide a detailed pseudocode for PrefGeo so that the readers can directly know the overall process and all key parameters.
8 comments
W1 and Q2 Influence of source task selection (2/2)
(iii) A criterion absent from the source preferences. To stress-test the shared-criteria assumption, we add an open-hand criterion on Meta-World Handle Press that never appears in the source preference labels. Success requires completing the press while keeping the gripper opening ≥ 0.8 for ≥ 90% of evaluation steps. Both methods freeze their source-learned basis and adapt only the mixture coefficients, using 20 or 200 target labels (Table m).
Table m. Stress test on Meta-World Handle Press with an open-hand criterion absent from all source preference labels: original and open-hand success rate (SR, %) with frozen source-learned bases and 20 or 200 target labels. Mean ± SD over 3 seeds.
| Method | Target labels | Original SR (%) ↑ | Open-hand SR (%) ↑ |
|---|---|---|---|
| PrefGeo | 20 | 97.0 ± 1.0 | 75.0 ± 5.3 |
| LoRe | 20 | 73.0 ± 2.6 | 0.0 ± 0.0 |
| PrefGeo | 200 | 99.3 ± 0.6 | 96.7 ± 0.6 |
| LoRe | 200 | 88.3 ± 2.5 | 0.0 ± 0.0 |
PrefGeo reaches 75.0% with only 20 labels and 96.7% with 200, close to its original-task success rate. LoRe improves on the original task with more labels but stays at 0% on open-hand. Since both bases are frozen, the gap comes from what each basis encodes, not from the amount of target feedback.
To verify this, we compute the Spearman correlation between reweighted combinations of each frozen basis and the target scoring functions (Table n).
Table n. Spearman correlation between reweighted combinations of each frozen basis (200 labels) and the target scoring functions in the open-hand stress test.
| Target scoring function | PrefGeo | LoRe |
|---|---|---|
| Open-hand attribute | +0.51 | −0.27 |
| Task + open-hand reward | +0.55 | −0.32 |
We observe that PrefGeo's basis can express the open-hand criterion to a meaningful extent, while LoRe's is negatively correlated with both targets. This suggests that PrefGeo's basis transfers better to criteria beyond those in the source preferences.
Scope and limitations. What PrefGeo needs is therefore coverage by its learned basis, not that the criterion appeared in the source preferences. A criterion outside that coverage cannot be recovered by adjusting mixture coefficients alone. Appendix G discusses this limitation and extensions, such as a residual reward head or a new basis component fitted as target feedback accumulates.
Reward-simplex visualization. We will add the reward-simplex visualization (as in Fig. 2(b)) in the revision.
References
- Sadigh, D., Dragan, A. D., Sastry, S., & Seshia, S. A. Active Preference-Based Learning of Reward Functions. RSS, 2017.
- Bıyık, E., Palan, M., Landolfi, N. C., Losey, D. P., & Sadigh, D. Asking Easy Questions: A User-Friendly Approach to Active Reward Learning. CoRL, 2019.
- Barreto, A., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., van Hasselt, H., & Silver, D. Successor Features for Transfer in Reinforcement Learning. NeurIPS, 2017.
W2 and Q4 Preference realism and labeling noise
Thank you for these comments. We have added experiments with human preference labels and with noisy labels. The tables below also appear in our response to Reviewer vnQ6 and are reproduced here for convenience.
Human preference labels. With labels from three human annotators, PrefGeo remains effective: it keeps its advantage over the baselines on LIBERO-Goal (Table e) and maintains its success rate on the real-robot tasks (Table f).
Table e. Success rate (%) on held-out LIBERO-Goal tasks under the leave-one-task-out protocol, with the original scripted labels and with human preference labels. Both settings use 20 labels for adaptation to the target task.
| Method | Original success rate (%) ↑ | Success rate with human labels (%) ↑ |
|---|---|---|
| Base policy | 51.63 ± 0.95 | 51.63 ± 0.95 |
| LoRe | 55.73 ± 0.85 | 56.70 ± 1.12 |
| PrefGeo (Ours) | 74.71 ± 1.00 | 80.65 ± 1.15 |
Table f. Real-world success (successes / 10 trials) of PrefGeo on the Empty and Marker tasks, with the original demonstration-preferred labels and with human preference labels from three annotators.
| Task | PrefGeo with demonstration-preferred labels | PrefGeo with human labels |
|---|---|---|
| Empty | 9/10 | 9/10 |
| Marker | 5/10 | 6/10 |
Labeling noise. We flip each training label with probability ε up to 0.2, as in the "mistake" teacher of B-Pref (Table i). PrefGeo is robust to this noise, staying ahead of LoRe at every noise level with consistently smaller performance drops.
Table i. PrefGeo / LoRe under label noise, where each source and target training label is flipped with probability ε; evaluation uses clean labels. Each cell reports final success rate (%) on Meta-World and LIBERO-Goal, and win rate (%) on the remaining benchmarks, as mean ± std over 5 runs. LIBERO-Goal uses the human-label setting of Table e. Δ denotes the change from ε = 0 to ε = 0.2 in percentage points (PrefGeo / LoRe).
| Benchmark | Target | ε = 0 | ε = 0.1 | ε = 0.2 | Δ(0 → 0.2) |
|---|---|---|---|---|---|
| Meta-World | drawer-open | 92.5 ± 3.5 / 61.5 ± 4.2 | 91.5 ± 5.4 / 58.2 ± 4.3 | 90.6 ± 4.1 / 56.1 ± 5.2 | −1.9 / −5.4 |
| Meta-World | sweep-into | 90.2 ± 3.8 / 43.7 ± 5.8 | 89.7 ± 4.9 / 41.5 ± 5.2 | 85.1 ± 5.4 / 36.0 ± 7.9 | −5.1 / −7.8 |
| LIBERO-Goal | 10-task average | 80.7 ± 1.2 / 56.7 ± 1.1 | 77.4 ± 1.6 / 53.5 ± 1.2 | 77.6 ± 1.0 / 51.3 ± 2.1 | −3.1 / −5.4 |
| PersonalLLM | Unseen users | 79.2 ± 0.5 / 76.4 ± 1.1 | 78.4 ± 0.5 / 73.4 ± 1.3 | 76.7 ± 0.4 / 72.9 ± 1.2 | −2.5 / −3.5 |
| PRISM | Unseen users | 64.1 ± 3.8 / 59.9 ± 3.9 | 63.8 ± 3.0 / 57.0 ± 3.2 | 63.6 ± 4.3 / 55.6 ± 2.7 | −0.5 / −4.3 |
| Assistive Feeding | Unseen users | 83.1 ± 2.9 / 69.3 ± 3.0 | 81.6 ± 3.0 / 65.4 ± 3.3 | 81.9 ± 2.5 / 58.9 ± 5.1 | −1.2 / −10.4 |
W1 and Q2 Influence of source task selection (1/2)
Thank you for this important question. We add three ablations on source selection: (i) source-to-target similarity, (ii) source diversity, and (iii) a target criterion that never appears in the source preferences. PrefGeo is robust in all three.
Shared-criteria assumption. This is a standard structural assumption: preference-based reward learning in robotics models human rewards as shared features with user-specific weights (Sadigh et al., 2017; Bıyık et al., 2019), and successor features do the same across tasks (Barreto et al., 2017). PrefGeo applies it across contexts (tasks or users). We test it below, including a stress test (iii) with a criterion absent from all source preferences.
Number of sources. Fig. 4 (rightmost) varies the number of source users from 8 to 1,000 on PersonalLLM. PrefGeo outperforms all baselines across the range and already reaches low held-out NLL with 8 users.
Selection of sources. In Tables j and m, the source set is fixed per condition and only the seed varies, so the effect of source selection is separated from algorithmic randomness. Table k additionally varies the source pool within each diversity level (pools listed in Table l) to show that the result does not depend on one particular pool.
(i) Source-to-target similarity. Sources share only the target's action family (Action ID), only its object family (Object ID), or neither (OOD) (Table j).
Table j. Effect of source-to-target similarity on Meta-World: success rate (SR, %) on two target tasks when the sources share only the target's action family (Action ID), only its object family (Object ID), or neither (OOD). Mean ± SD over 3 seeds; the last column averages the two target means.
Similarity helps both methods, most of all a shared action family. But from Action ID to OOD, PrefGeo drops 8.1 points on average while LoRe drops 27.6, and PrefGeo still reaches 86.1% with neither family shared.
(ii) Source diversity. We hold similarity fixed at the OOD level of (i) (no shared object or action family) and vary the diversity of three-task source pools, measured as D = number of distinct object families + number of distinct action families (Low: D ≤ 3; High: D = 6). Results are shown in Table k, and the source pools are listed in Table l.
Table k. Effect of source diversity at the OOD similarity level: success rate (SR, %) with low-diversity (D ≤ 3) and high-diversity (D = 6) three-task source pools. Mean ± SD over 3 pools × 2 seeds.
PrefGeo is essentially insensitive to diversity: +1.9 and +2.1 points, well within one SD. LoRe gains +18.3 and +19.5 points yet stays more than 40 points below PrefGeo.
Table l. Source pools used in Table k. D = number of distinct object families + number of distinct action families.