acceptodds
Under review as a conference paper at ICLR 2027

PrefGeo: Generalizing Preference Learning Across Contexts via Geometric Reward Factorization

Abstract

Preference-based reinforcement learning is a popular paradigm for aligning AI models with human intent, but generalizing it to a new context, whether an unseen task or a new user, typically requires substantial additional feedback. In this work, we introduce PrefGeo, a geometric reward factorization framework for efficient preference learning across contexts. Unlike prior methods that fit a shared reward basis directly from preference data, PrefGeo recovers it from independently estimated context-specific rewards via regularized minimum-volume unmixing in a preference-induced Fisher geometry, capturing reusable preference structure. A new context is then adapted by updating only low-dimensional mixture weights. We evaluate PrefGeo on cross-task generalization in robotic manipulation, with both conventional policies and vision-language-action models, and on cross-user personalization in large language models and assistive human-robot interaction. PrefGeo consistently outperforms baselines in preference alignment and downstream task performance, especially when target feedback is scarce. Real-robot experiments and an assistive-feeding user study confirm its effectiveness in the real world.

Primary area:
reinforcement learning
Keywords:
Preference-based Reinforcement Learning, Robot Learning from Human Feedback, Reward Factorization, Preference Transfer, User Personalization, Cross-Context Generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xcVJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new reward factorization framework named PrefGeo for preference-based reinforcement learning. PrefGeo addresses the factorization ambiguity and balances different reward directions by decoupling the learning of reward bases and coefficients. PrefGeo first learns multiple reward heads on each source context, then extracts compact bases from learned heads by minimizing the volume spanned by the bases while ensuring reconstruction and non-degeneracy. The bases enable few-shot adaptation by only learning the coefficients. Experiments are conducted in both simulation and real-robot settings, showing superior performance of the proposed method and the necessity of each module.

Critical strengths
  1. The paper is well-written. The methodology is clearly described. Limitations are explicitly discussed.
  2. The paper presents a real-robot experiment.
Critical weaknesses
  1. This paper assumes that a target context shares the preference criteria underlying the source contexts. This assumption can make the algorithm sensitive to the selection of source tasks. However, this assumption is not validated experimentally, and the sensitivity of performance to the source tasks is not sufficiently evaluated.
  2. This paper considers an idealized preference setting. Noise in the labeling process is not considered (page 22, line 1147). Additionally, no human preference labels are collected in the real robot experiment (page 28, line 1482). The constructed preference could largely deviate from that of real human labelers.
Questions for authors
  1. The Fisher metric G is derived at a neutral head with probability 0.5. Will it induce a large bias when the actual probability deviates from 0.5? Can you report the accuracies for transitions with different probability of being preferred, respectively?
  2. How does the source domain selection affect the performance of PrefGeo? I suggest the authors report the performance under several different source domain selection schemes, including the number and the selection of the source domains. In the current experiments, the source tasks are randomized for some domains, which could entangle the effect of source domain selection and the randomness in the algorithm. Additionally, It would be better if the reward simplex (Fig. 2(b)) could be visualized in these experiments.
  3. Can you quantify to which extent the target domain preferences can be expressed by the learned reward bases?
  4. Is the proposed method sensitive to labeling noise, such as those mentioned in B-Pref?
  5. Please provide a detailed pseudocode for PrefGeo so that the readers can directly know the overall process and all key parameters.
8 comments

W1 and Q2 Influence of source task selection (1/2)

Thank you for this important question. We add three ablations on source selection: (i) source-to-target similarity, (ii) source diversity, and (iii) a target criterion that never appears in the source preferences. PrefGeo is robust in all three.

Shared-criteria assumption. This is a standard structural assumption: preference-based reward learning in robotics models human rewards as shared features with user-specific weights (Sadigh et al., 2017; Bıyık et al., 2019), and successor features do the same across tasks (Barreto et al., 2017). PrefGeo applies it across contexts (tasks or users). We test it below, including a stress test (iii) with a criterion absent from all source preferences.

Number of sources. Fig. 4 (rightmost) varies the number of source users from 8 to 1,000 on PersonalLLM. PrefGeo outperforms all baselines across the range and already reaches low held-out NLL with 8 users.

Selection of sources. In Tables j and m, the source set is fixed per condition and only the seed varies, so the effect of source selection is separated from algorithmic randomness. Table k additionally varies the source pool within each diversity level (pools listed in Table l) to show that the result does not depend on one particular pool.

(i) Source-to-target similarity. Sources share only the target's action family (Action ID), only its object family (Object ID), or neither (OOD) (Table j).


Table j. Effect of source-to-target similarity on Meta-World: success rate (SR, %) on two target tasks when the sources share only the target's action family (Action ID), only its object family (Object ID), or neither (OOD). Mean ± SD over 3 seeds; the last column averages the two target means.

Source selectionMethodFaucet Open SR (%) ↑Window Open SR (%) ↑Average SR (%) ↑
Action IDPrefGeo93.8 ± 3.094.6 ± 3.194.2
Action IDLoRe61.5 ± 7.464.7 ± 8.063.1
Object IDPrefGeo87.1 ± 3.688.0 ± 3.987.6
Object IDLoRe43.6 ± 8.648.2 ± 7.145.9
OODPrefGeo85.4 ± 6.386.7 ± 5.986.1
OODLoRe33.8 ± 7.937.1 ± 8.335.5

Similarity helps both methods, most of all a shared action family. But from Action ID to OOD, PrefGeo drops 8.1 points on average while LoRe drops 27.6, and PrefGeo still reaches 86.1% with neither family shared.

(ii) Source diversity. We hold similarity fixed at the OOD level of (i) (no shared object or action family) and vary the diversity of three-task source pools, measured as D = number of distinct object families + number of distinct action families (Low: D ≤ 3; High: D = 6). Results are shown in Table k, and the source pools are listed in Table l.


Table k. Effect of source diversity at the OOD similarity level: success rate (SR, %) with low-diversity (D ≤ 3) and high-diversity (D = 6) three-task source pools. Mean ± SD over 3 pools × 2 seeds.

TargetDiversityPrefGeo SR (%) ↑LoRe SR (%) ↑
Faucet OpenLow (D ≤ 3)84.3 ± 5.824.5 ± 7.2
Faucet OpenHigh (D = 6)86.2 ± 5.142.8 ± 8.4
Faucet OpenHigh − Low+1.9+18.3
Window OpenLow (D ≤ 3)85.5 ± 5.527.0 ± 7.6
Window OpenHigh (D = 6)87.6 ± 4.946.5 ± 8.0
Window OpenHigh − Low+2.1+19.5

PrefGeo is essentially insensitive to diversity: +1.9 and +2.1 points, well within one SD. LoRe gains +18.3 and +19.5 points yet stays more than 40 points below PrefGeo.


Table l. Source pools used in Table k. D = number of distinct object families + number of distinct action families.

LevelTargetSource tasksD
LowBothbutton-press, coffee-button, handle-press3
LowBothbutton-press, button-press-wall, coffee-button2
LowBothbutton-press-topdown, coffee-button, handle-press-side3
HighWindow Openbutton-press, handle-pull, faucet-close6
HighWindow Opencoffee-button, handle-pull, door-close6
HighWindow Openbutton-press, handle-pull, drawer-close6
HighFaucet Openbutton-press, handle-pull, window-close6
HighFaucet Opencoffee-button, handle-pull, door-close6
HighFaucet Openbutton-press, handle-pull, drawer-close6

W1 and Q2 Influence of source task selection (2/2)

(iii) A criterion absent from the source preferences. To stress-test the shared-criteria assumption, we add an open-hand criterion on Meta-World Handle Press that never appears in the source preference labels. Success requires completing the press while keeping the gripper opening ≥ 0.8 for ≥ 90% of evaluation steps. Both methods freeze their source-learned basis and adapt only the mixture coefficients, using 20 or 200 target labels (Table m).


Table m. Stress test on Meta-World Handle Press with an open-hand criterion absent from all source preference labels: original and open-hand success rate (SR, %) with frozen source-learned bases and 20 or 200 target labels. Mean ± SD over 3 seeds.

MethodTarget labelsOriginal SR (%) ↑Open-hand SR (%) ↑
PrefGeo2097.0 ± 1.075.0 ± 5.3
LoRe2073.0 ± 2.60.0 ± 0.0
PrefGeo20099.3 ± 0.696.7 ± 0.6
LoRe20088.3 ± 2.50.0 ± 0.0

PrefGeo reaches 75.0% with only 20 labels and 96.7% with 200, close to its original-task success rate. LoRe improves on the original task with more labels but stays at 0% on open-hand. Since both bases are frozen, the gap comes from what each basis encodes, not from the amount of target feedback.

To verify this, we compute the Spearman correlation between reweighted combinations of each frozen basis and the target scoring functions (Table n).


Table n. Spearman correlation between reweighted combinations of each frozen basis (200 labels) and the target scoring functions in the open-hand stress test.

Target scoring functionPrefGeoLoRe
Open-hand attribute+0.51−0.27
Task + open-hand reward+0.55−0.32

We observe that PrefGeo's basis can express the open-hand criterion to a meaningful extent, while LoRe's is negatively correlated with both targets. This suggests that PrefGeo's basis transfers better to criteria beyond those in the source preferences.

Scope and limitations. What PrefGeo needs is therefore coverage by its learned basis, not that the criterion appeared in the source preferences. A criterion outside that coverage cannot be recovered by adjusting mixture coefficients alone. Appendix G discusses this limitation and extensions, such as a residual reward head or a new basis component fitted as target feedback accumulates.

Reward-simplex visualization. We will add the reward-simplex visualization (as in Fig. 2(b)) in the revision.

References

  • Sadigh, D., Dragan, A. D., Sastry, S., & Seshia, S. A. Active Preference-Based Learning of Reward Functions. RSS, 2017.
  • Bıyık, E., Palan, M., Landolfi, N. C., Losey, D. P., & Sadigh, D. Asking Easy Questions: A User-Friendly Approach to Active Reward Learning. CoRL, 2019.
  • Barreto, A., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., van Hasselt, H., & Silver, D. Successor Features for Transfer in Reinforcement Learning. NeurIPS, 2017.

W2 and Q4 Preference realism and labeling noise

Thank you for these comments. We have added experiments with human preference labels and with noisy labels. The tables below also appear in our response to Reviewer vnQ6 and are reproduced here for convenience.

Human preference labels. With labels from three human annotators, PrefGeo remains effective: it keeps its advantage over the baselines on LIBERO-Goal (Table e) and maintains its success rate on the real-robot tasks (Table f).


Table e. Success rate (%) on held-out LIBERO-Goal tasks under the leave-one-task-out protocol, with the original scripted labels and with human preference labels. Both settings use 20 labels for adaptation to the target task.

MethodOriginal success rate (%) ↑Success rate with human labels (%) ↑
Base policy51.63 ± 0.9551.63 ± 0.95
LoRe55.73 ± 0.8556.70 ± 1.12
PrefGeo (Ours)74.71 ± 1.0080.65 ± 1.15

Table f. Real-world success (successes / 10 trials) of PrefGeo on the Empty and Marker tasks, with the original demonstration-preferred labels and with human preference labels from three annotators.

TaskPrefGeo with demonstration-preferred labelsPrefGeo with human labels
Empty9/109/10
Marker5/106/10

Labeling noise. We flip each training label with probability ε up to 0.2, as in the "mistake" teacher of B-Pref (Table i). PrefGeo is robust to this noise, staying ahead of LoRe at every noise level with consistently smaller performance drops.


Table i. PrefGeo / LoRe under label noise, where each source and target training label is flipped with probability ε; evaluation uses clean labels. Each cell reports final success rate (%) on Meta-World and LIBERO-Goal, and win rate (%) on the remaining benchmarks, as mean ± std over 5 runs. LIBERO-Goal uses the human-label setting of Table e. Δ denotes the change from ε = 0 to ε = 0.2 in percentage points (PrefGeo / LoRe).

BenchmarkTargetε = 0ε = 0.1ε = 0.2Δ(0 → 0.2)
Meta-Worlddrawer-open92.5 ± 3.5 / 61.5 ± 4.291.5 ± 5.4 / 58.2 ± 4.390.6 ± 4.1 / 56.1 ± 5.2−1.9 / −5.4
Meta-Worldsweep-into90.2 ± 3.8 / 43.7 ± 5.889.7 ± 4.9 / 41.5 ± 5.285.1 ± 5.4 / 36.0 ± 7.9−5.1 / −7.8
LIBERO-Goal10-task average80.7 ± 1.2 / 56.7 ± 1.177.4 ± 1.6 / 53.5 ± 1.277.6 ± 1.0 / 51.3 ± 2.1−3.1 / −5.4
PersonalLLMUnseen users79.2 ± 0.5 / 76.4 ± 1.178.4 ± 0.5 / 73.4 ± 1.376.7 ± 0.4 / 72.9 ± 1.2−2.5 / −3.5
PRISMUnseen users64.1 ± 3.8 / 59.9 ± 3.963.8 ± 3.0 / 57.0 ± 3.263.6 ± 4.3 / 55.6 ± 2.7−0.5 / −4.3
Assistive FeedingUnseen users83.1 ± 2.9 / 69.3 ± 3.081.6 ± 3.0 / 65.4 ± 3.381.9 ± 2.5 / 58.9 ± 5.1−1.2 / −10.4
Reviewer_vnQ6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper works on preference reward factorization (PRF), where context-specific reward heads are mixtures of a shared reward basis and a new task or user only needs new mixture weights. It argues that existing PRF methods suffer from factorization ambiguity and geometry bias, and proposes PrefGeo: fit each source reward head on its own, recover the basis by regularized minimum-volume unmixing under the preference-induced Fisher metric G = I(0), then adapt a new context by updating only the mixture weights, starting from the source mean. Experiments cover cross-task transfer on Meta-World and LIBERO-Goal, personalization on Assistive Feeding, PersonalLLM, PRISM and Reddit TL;DR, real-robot π0.5-DROID tasks, and an 8-participant feeding study.

Critical strengths

(1) The analysis is the nice part. Appendix A works through both failure modes with small examples: a stretched simplex that fits the source heads just as well but makes moving toward smoothness 4× cheaper than toward comfort, and a Euclidean penalty that keeps 20% of the smoothness weight where the Fisher penalty keeps 96%. Table 5 then answers the obvious objection that G is only feature preprocessing: LoRe on G-whitened features gets worse on every metric. That is the most convincing experiment in the paper.

(2) Where the setup really tests unseen users, the gains are clear. On Assistive Feeding, WR goes from 69.30 (LoRe) to 83.14 with 20 labels per OOD target user, and the Figure 4 ablation lines up with the two stated problems: dropping the Fisher geometry or decoupling costs about 10 points, the volume and non-degeneracy terms 3–4 each.

Critical weaknesses

(1) Most experiments don’t test adaptation to a new preference. LIBERO-Goal uses one scripted utility U with the same weights for all ten tasks (Appendix D.2), and on the real robot every pair has the demonstration preferred over a base-policy rollout (Appendix F.1), so target and source preferences are the same. The only 0-label result points the same way: on TL;DR (Figure 15), PrefGeo’s NLL is about the same at B = 0 as at B = 10 (about 0.86–0.87) and already below every baseline at B = 10. Assistive Feeding, with simulated users, is the one benchmark that tests adaptation to new preferences.

(2) The abstract claims more than Table 1 shows. It says PrefGeo “consistently outperforms baselines in preference alignment and downstream task performance”, but accuracy isn’t the best on PRISM (MAML 58.11 vs 57.92) or TL;DR (LoRe 65.38 vs 65.14), and the PRISM WR gap over LoRe (64.12±3.84 vs 59.89±3.85) is about one standard deviation over five runs. Part of the NLL advantage may be calibration: on PRISM, LoRe has 57.99% accuracy with an NLL of 1.67, while VPL has similar accuracy and an NLL of 0.66 (PrefGeo 0.62).

Questions for authors

Q1. Can you report θ0 alone, with zero target labels, on every benchmark, so that basis quality and adaptation can be told apart? And for Assistive Feeding, how far out are the target users? Section 4.1 says their weights lie outside the range spanned by the source users, Appendix D.2 says the shift preserves the support; the fraction outside the convex hull of the 40 source users, or the reconstruction residual from Appendix G, would settle it.

Q2. Does the NLL gap survive temperature-scaling each baseline’s adapted reward on the target few-shot set? If it mostly goes away, the personalization claim should rest on WR. Also, VPL’s curve in Figure 15 stays at about 1.37 for every label budget, which suggests its encoder isn’t using the target comparisons; please show that setup.

Q3. On Meta-World, PrefGT uses 10,000 ground-truth target labels, 500× PrefGeo’s budget, yet falls below PrefGeo on four of five tasks. Does it use the same frozen z(x) and the same number of environment steps? Since the labels come from a noise-free scripted teacher on a shaped reward built from shared reach, grasp and place terms, the low-rank assumption holds almost by construction, so one run with noisy labels would help. A table of final success with standard deviations would too: Figure 3 has only curves, and Figure 13 has no error bars or episode counts.

7 comments

W1 Adaptation to new preferences

Thank you for raising this concern. We believe it comes from what counts as a "new preference" in our setting, so we first clarify this and then address each point .

Adaptation to new preferences is tested beyond Assistive Feeding. As shown in Figure 1(a), we study two kinds of adaptation: cross-task adaptation, where shared expert criteria are applied to a new task with its own goal and behavioral requirements, and cross-user personalization, where the target is an unseen user with individual preferences.

  • Meta-World: each task has its own ground-truth reward function provided by the benchmark, and comparisons are labeled using that task's return. Adapting to a held-out task therefore means fitting feedback governed by the target task's own reward.
  • LLM benchmarks: PersonalLLM, PRISM, and TL;DR use disjoint source and target user sets, with separate adaptation and evaluation comparisons for each target user. PRISM and TL;DR contain feedback from real participants or annotators; PersonalLLM uses simulated users with different reward-model mixtures. On TL;DR, the similar NLL at B = 0 and B = 10 likely reflects that annotators largely share criteria for summary quality, so the source-mean initialization already fits most unseen annotators well; it does not mean that source and target preferences are identical. Please also refer to our response to Q1 (Table h), which separates the contributions of zero-shot initialization and few-shot adaptation.

LIBERO-Goal. We use the same utility coefficients across tasks to model an expert applying consistent criteria such as success, safety, and motion quality. Although these criteria are fixed, the goals and the behaviors required to satisfy them change across tasks, so this setting evaluates whether the shared reward basis supports adaptation to a held-out task. Target feedback matters here: on two LIBERO-Goal targets, few-shot labels consistently improve success rate over the zero-shot initialization (Table h).

To address the concern about a fixed scripted utility, we repeated LIBERO-Goal with human preference labels. For each task, three annotators (non-authors) each provided 100 preference labels, yielding 300 labels per task. We followed the same leave-one-task-out protocol as in the paper, using 300 labels for each source task and 20 labels for adaptation to the held-out target task (Table e).


Table e. Success rate (%) on held-out LIBERO-Goal tasks under the leave-one-task-out protocol, with the original scripted labels and with human preference labels. Both settings use 20 labels for adaptation to the target task.

MethodOriginal success rate (%) ↑Success rate with human labels (%) ↑
Base policy51.63 ± 0.9551.63 ± 0.95
LoRe55.73 ± 0.8556.70 ± 1.12
PrefGeo (Ours)74.71 ± 1.0080.65 ± 1.15

With the same feedback budget, human labels increase PrefGeo's success rate from 74.71% to 80.65%. One possible explanation is that human judgments capture task-specific details of execution quality that the fixed scripted utility may miss. PrefGeo thus remains effective when adaptation is driven by direct human feedback.

Real-world tasks with human labels. We also repeated the Empty and Marker tasks using preference labels from three annotators watching rollout videos (Table f). This replaces the original automatic labeling rule, which paired demonstrations with sampled base-policy rollouts and always marked the demonstration as preferred.


Table f. Real-world success (successes / 10 trials) of PrefGeo on the Empty and Marker tasks, with the original demonstration-preferred labels and with human preference labels from three annotators.

TaskPrefGeo with demonstration-preferred labelsPrefGeo with human labels
Empty9/109/10
Marker5/106/10

With direct human judgments and the same feedback budget, PrefGeo maintains its success rate on both tasks.

We will clarify the two adaptation settings and the interpretation of the TL;DR curve in the paper, and add both human-feedback experiments.

W2 and Q2 NLL calibration and VPL inference

Thank you for these comments. We agree that the abstract's claim should be more precise, and we have added a calibration-controlled comparison.

Temperature scaling. Following your suggestion, we recalibrated every method using the target few-shot labels. For each target user, each support comparison is scored by a model adapted on (or, for VPL, conditioned on) the remaining support comparisons. We pool these leave-one-out logits over target users and fit one positive temperature per method, benchmark, and seed (Table g).


Table g. NLL (↓) on the test comparisons of unseen target users after temperature scaling. Each method's temperature is fit on leave-one-out logits from the target few-shot labels.

MethodPersonalLLM ↓PRISM ↓TL;DR ↓Assistive Feeding ↓
LoRe0.27 ± 0.0190.68 ± 0.0060.64 ± 0.0210.60 ± 0.005
PReF0.28 ± 0.0080.68 ± 0.0090.68 ± 0.0160.68 ± 0.002
MAML0.32 ± 0.0120.67 ± 0.0050.65 ± 0.0100.61 ± 0.011
VPL0.31 ± 0.0210.68 ± 0.0210.65 ± 0.0260.57 ± 0.021
PrefGeo0.17 ± 0.0130.62 ± 0.0050.61 ± 0.0140.53 ± 0.010

Calibration narrows the raw gaps, but PrefGeo still has the lowest NLL on all four benchmarks.

For the abstract, we will revise the claim to "consistently outperforms baselines in preference prediction in terms of NLL and in downstream performance in terms of success rate and win rate". The PRISM WR gain also holds at the user level: PrefGeo improves WR for 85.8% of unseen PRISM users (Table d in our response to Reviewer mRGG).

VPL inference in Figure 15. We follow VPL's original inference protocol: the preference encoder and conditional reward model are trained on source users and frozen at test time. For each nonzero feedback budget, the corresponding target comparisons and their labels are passed to the encoder to recompute the user latent, which then conditions reward prediction.

This amortized inference does not guarantee that held-out NLL decreases as the feedback budget increases: additional comparisons may change the inferred latent and individual predictions without substantially improving aggregate NLL. The flat curve therefore indicates limited average benefit from additional feedback in this setting; it does not establish that the encoder ignores that feedback. Consistent with this, VPL's TL;DR NLL drops to 0.65 after temperature scaling (Table g), comparable to the other baselines, so its high raw NLL mainly reflects miscalibration rather than uninformative predictions. We will make the target-feedback inputs and budget-specific latent inference more explicit in Appendix D.5.

Q1 Zero-shot performance and Assistive Feeding target users

Thank you for these suggestions.

Zero-shot performance of θ₀. As an additional ablation, we report PrefGeo's zero-shot performance in six settings spanning Meta-World, LIBERO-Goal, and LLM benchmarks (Table h).


Table h. Zero-shot (θ₀, no target labels) vs. few-shot performance of PrefGeo, in % (mean ± std over 5 runs). Gain is reported in percentage points. Note that LIBERO-Goal rows use the human preference labels and are separate runs from the per-task results in our Q3 response.

Benchmark / TaskMetric ↑Zero-shotFew-shotGain (pp)
Meta-World: drawer-openSuccess rate83.6 ± 12.192.5 ± 3.5+8.9
Meta-World: sweep-intoSuccess rate79.4 ± 6.890.2 ± 3.8+10.8
LIBERO-Goal: wine bottleSuccess rate71.0 ± 4.285.0 ± 3.6+14.0
LIBERO-Goal: drawerSuccess rate77.0 ± 4.390.0 ± 3.0+13.0
PRISMWin rate62.7 ± 4.164.1 ± 3.8+1.4
PersonalLLMWin rate76.3 ± 0.779.2 ± 0.5+2.9

Few-shot labels improve performance in all six settings, with larger gains on robot manipulation (8.9–14.0 points), where held-out tasks differ more from the source tasks, and smaller gains on the LLM benchmarks (1.4–2.9 points). For TL;DR, the zero-label result is the B = 0 point in Figure 15.

For Assistive Feeding, we conducted a follow-up user study on real-world robot-assisted feeding (Table c in our response to Reviewer mRGG). With the same basis, adapting θ₀ with ten preference labels per participant raised satisfaction from 3.00 ± 0.71 to 3.80 ± 0.45 on a 1–5 scale (five participants), indicating a further benefit of few-shot adaptation with real users.

Assistive Feeding target users. As suggested, we computed the fraction of target users whose ground-truth preference weights lie outside the convex hull of the 40 source users: all 60 do (100%), so no convex combination of source users reproduces any target user. This is consistent with Section 4.1, and we will correct the "preserves the support" statement in Appendix D.2 accordingly.

Reviewer_mRGGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper improves preference reward factorization for generalization to new tasks and users. It separates context-specific reward estimation from shared basis recovery, using Fisher geometry and a minimum-volume objective to address geometry bias and factorization ambiguity. Adaptation to a new context updates only the weights over the learned basis. The experiments evaluate preference prediction, best-of-N selection, and robot policy performance, including user satisfaction in a real-world feeding study.

Critical strengths
  1. A clear separation between reward learning and basis recovery

    The paper separates context-specific reward learning from basis recovery, unlike the joint learning used in PRF baselines. The Fisher metric and minimum-volume objective provide explicit criteria for choosing a basis that can be reused in new contexts.

  2. Evaluation of policy performance and user satisfaction using the learned rewards

    The paper trains policies from scratch with the learned rewards on Meta-World and fine-tunes existing policies using high-reward trajectories on LIBERO. The real-world feeding study also evaluates user satisfaction after fine-tuning policies with personalized rewards.

  3. Thorough ablation studies

    The paper separately removes the Fisher geometry, decoupling, volume term, and non-degeneracy term. It also tests different feedback budgets and basis dimensions. Using the same representations and comparison data, with hyperparameters selected only from source data, helps make the comparisons fair.

Critical weaknesses
  1. The interpretation of the learned basis needs more support.

    Figure 6 examines each basis component separately, but their combination may still represent effort economy. LoRe’s failure may therefore come from difficulty learning suitable weights with limited feedback, rather than a limitation of its basis as suggested in Section 4.4.

  2. It is unclear how much of the improvement in the feeding study comes from personalization.

    The real-world robot-assisted feeding study compares Base, LoRe, and PrefGeo, but does not include PrefGeo with the source-mean weights. The improvement could partly come from a better shared reward before personalization.

  3. The average results do not show how many users benefit.

    Table 1 reports improvements in average NLL and win rate, but it is unclear whether these gains occur for most unseen users.

Questions for authors
  1. In the feeding study, how much does personalizing PrefGeo help compared with simply using the average weights from source users? How much improvement comes from the participant’s ten preference labels?
  2. On PRISM and TLDR, what proportion of unseen users benefit from PrefGeo compared with LoRe? Are the average gains spread across users, or mainly driven by a smaller group?
5 comments

W1. Interpretation of the learned basis.

Thank you for this point. We agree that examining components individually does not establish what their combinations can express, so the statement in Section 4.4 needs further support. We therefore conducted two additional diagnostics of the learned reward bases.

Diagnostic 1: more target labels. We freeze the bases learned by PrefGeo and LoRe on the source tasks and fit the mixture coefficients for each target task using either 20 (the original setting in the paper) or 2,000 labels. This larger budget allows us to assess whether the performance gap persists with substantially more target feedback. As a reference, we also fit a task-specific preference reward head for each target task from scratch with the same 2,000 labels. All models are evaluated on the same held-out target comparisons, and we report NLL averaged over the target tasks of each benchmark (Table a).


Table a. NLL (↓) on held-out target comparisons with frozen source bases, averaged over the target tasks of each benchmark. Mixture coefficients are fit per target task with 20 or 2,000 labels; the reference is a task-specific reward head trained from scratch with the same 2,000 labels.

BenchmarkPrefGeo (20)PrefGeo (2,000)LoRe (20)LoRe (2,000)Task-specific reference (2,000)
LIBERO-Goal0.3500.2890.3910.3770.295
Meta-World0.6760.5840.7580.7360.572

More feedback barely helps LoRe, and LoRe with 2,000 labels still does not outperform PrefGeo with 20. In contrast, PrefGeo becomes comparable to the task-specific reference once given the same 2,000 labels, so its gap to the reference at 20 labels reflects the 100× smaller label budget rather than a limitation of the basis. The PrefGeo–LoRe gap is therefore not explained by estimating coefficients from limited feedback.

Diagnostic 2: span of the basis. We also test directly whether the target reward lies in the span of each basis. For each target task, we take its reference head's reward differences on the held-out comparisons and fit unconstrained coefficients so that each frozen basis reproduces them as closely as possible (Table b). Because this uses all components jointly, it covers information that only appears through their combination.


Table b. Relative error (↓) in reproducing each target task's reference-head reward differences on held-out comparisons, using each frozen basis with unconstrained coefficients, averaged over the target tasks of each benchmark. Relative error is ∥Δrbasis−Δrref∥2/∥Δrref∥2\lVert \Delta r^{\text{basis}} - \Delta r^{\text{ref}} \rVert_2 / \lVert \Delta r^{\text{ref}} \rVert_2, where Δrref\Delta r^{\text{ref}} is the reference head's reward difference and Δrbasis\Delta r^{\text{basis}} is its reconstruction from the frozen basis with all coefficients fit jointly.

BenchmarkPrefGeoLoRe
LIBERO-Goal0.3190.420
Meta-World0.4510.836

We observe that LoRe still cannot reproduce the target reward as well as PrefGeo, even when all of its components are combined with unconstrained coefficients.

The same pattern appears at the policy level. In a stress test with a target criterion absent from all source preference labels (Tables m and n in our response to Reviewer xcVJ), LoRe's frozen basis stays at 0% success even with 200 target labels, and its reweighted combinations correlate negatively with the target criterion, whereas PrefGeo reaches 96.7%.

Together, these results show that the gap lies in the basis itself rather than in estimating the coefficients from limited feedback: LoRe's components may still encode some useful criteria such as effort economy in combination, but less of the target preference lies in their span. This matches Appendix A.2: fitting the source preferences well does not by itself guarantee a reusable basis. We will add both diagnostics and revise the statement in Section 4.4.

W2 / Q1. Initialization vs. personalization.

Thank you, this is a fair question. To quantify the separate contribution of downstream adaptation, we conducted a follow-up user study with five participants: three from the original study and two new participants. We add a new ablation, PrefGeo-zero, which uses the source-mean weights without any personal labels (Table c).


Table c. Follow-up user study (n = 5): user satisfaction (1–5, mean ± SD). PrefGeo-zero uses the source-mean weights without personal labels; PrefGeo adapts with ten personal preference labels.

MethodSatisfaction (1–5), mean ± SD
Base1.80 ± 0.84
LoRe2.20 ± 0.84
PrefGeo-zero3.00 ± 0.71
PrefGeo3.80 ± 0.45

Ten personal preference labels raise mean satisfaction from 3.00 to 3.80, indicating a further benefit from few-shot adaptation in PrefGeo. Even without personal labels, PrefGeo-zero already exceeds both Base and few-shot LoRe, reflecting the transfer value of the shared reward basis.

The same pattern holds beyond the user study: across six settings spanning Meta-World, LIBERO-Goal, and LLM benchmarks (Table h in our response to Reviewer vnQ6), few-shot labels improve on the zero-shot source-mean initialization in every case.

Together, these results support our goal of learning a reusable basis that provides an effective starting point and enables further personalization with limited feedback.

We have also invited the remaining five participants from the original study and will include any additional completed evaluations in the revised manuscript.

W3 / Q2. How many unseen users benefit.

Thank you for suggesting this. We compared PrefGeo and LoRe for each unseen user, using the per-user metrics behind Table 1 (Table d).


Table d. Per-user comparison of PrefGeo against LoRe on unseen users: percentage of users for whom PrefGeo is better, tied, or worse. Improved means lower NLL or higher WR for PrefGeo.

DatasetMetricImproved (%)Tied (%)Worse (%)
PRISMNLL ↓90.20.09.8
PRISMWR ↑85.811.82.4
TL;DRNLL ↓97.40.02.6
Assistive FeedingNLL ↓90.00.010.0
Assistive FeedingWR ↑86.78.35.0

We observe that the improvements extend to the vast majority of evaluated users, rather than being confined to a small subset.

Thank you for your time and effort in reviewing our paper and for helping us improve the manuscript. If you have any follow-up questions, please let us know!

Dear Reviewer, thank you again for your time in reviewing our work! We wanted to follow up on our responses and the additional experiments above. If you have had a chance to review them, we would appreciate hearing whether they address your questions. Please let us know if anything needs further clarification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Discussion Summary

As the discussion period draws to a close, we wanted to sincerely thank all reviewers, ACs, and SACs for their time and constructive feedback.

We are glad that all three reviewers recognized PrefGeo's methodological and empirical strengths, including its analysis of factorization ambiguity and geometry bias (vnQ6), decoupled reward estimation and geometric basis recovery (mRGG, xcVJ), and broad evaluation across simulated and real-world settings (mRGG, vnQ6, xcVJ).

The reviewers' comments motivated additional analyses and ablations that further strengthen the evidence for PrefGeo's key contributions:

  • More expressive and transferable reward basis (Tables a–b, m–n). PrefGeo outperforms LoRe with 100× fewer target labels and successfully adapts to a criterion absent from source preference labels (96.7% vs. 0% success).
  • Robustness to source-task selection (Tables j–l). PrefGeo maintains strong performance across source similarity and diversity levels.
  • Robustness to human feedback and label noise (Tables e–f, i, p–q). Human-labeled experiments confirm PrefGeo's effectiveness, while noise tests and component ablations further support the benefits of decoupling and Fisher geometry.
  • Effective few-shot personalization (Tables c–d, h). Zero-shot and few-shot comparisons verify the additional benefit of adaptation beyond the transferred basis. Per-user analyses show improvements for most unseen users.

We also clarified the scope of cross-task generalization and cross-user personalization, calibration, and Fisher geometry.

We are grateful that all three reviewers confirmed that our responses addressed their concerns. Their constructive feedback has helped further strengthen the paper.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.