acceptodds
Under review as a conference paper at ICLR 2027

Behavior-Aware Reward Shaping for Personalized Preference Optimization

Abstract

Large Language Models (LLMs) increasingly power web applications such as chatbots and recommender systems, making personalization critical for enhancing user experience. Existing methods for personalized preference alignment often extend Reinforcement Learning from Human Feedback (RLHF) by incorporating user characteristics into reward shaping. While effective, these approaches mainly rely on semantic information and overlook behavioral similarity, which encodes collaborative relationships among users and responses—important cues for personalization. Yet, capturing such signals is challenging: at the response level, users interact with dynamically generated responses, causing cross-user interactions to appear orthogonal with no overlaps, preventing the capture of collaborative signals. To address this challenge, we propose BehaviorRM, a reward shaping framework that explicitly models behavioral similarity. First, Response Semantic Decomposition represents each query-response pair as a composition of shared semantic identifiers (SIDs) obtained via residual quantization, which induces interaction overlaps at the SID level and exposes latent collaborative structures. Next, Behavioral Interaction Modeling fits the SID-level interaction matrix by jointly learning embeddings for users and SIDs, capturing behavioral similarity beyond pure semantics. Finally, Behavior-aware Reward Shaping integrates the behavioral rewards with semantic rewards to form a unified signal, enhancing personalized preference modeling. Extensive experiments on three real-world datasets show that BehaviorRM significantly outperforms baseline methods. All codes are included in the Supplemental Material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
preference alignment, LLM personalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_7E5JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes BehaviorRM, a personalized reward modeling framework that jointly captures individual semantic consistency and behavioral similarity across users. The framework consists of three components. First, it represents each response as a composition of shared Semantic Identifiers (SIDs) through quantization, alleviating the sparsity of user interaction data and establishing a basis for information sharing across users. Second, it learns user and SID embeddings and models a SID-level interaction matrix to capture behavioral similarity across users. Finally, it combines a semantic reward reflecting individual semantic preferences with a behavioral reward reflecting behavioral similarity across users to obtain a unified reward signal. Through this approach, BehaviorRM aims to improve personalized preference alignment by leveraging interaction information from similar users while preserving individual preferences.

Critical strengths
  • Well-motivated solution to interaction sparsity. The paper addresses interaction sparsity by representing query-response pairs as shared SIDs and constructing a user-SID interaction matrix. This allows semantically similar interactions to share identifiers, increasing interaction overlap across users compared with raw query-response representations and potentially alleviating sparsity.

  • Complementary modeling of individual preferences and cross-user behavior. BehaviorRM models individual semantic preferences and behavioral similarity across users as separate reward signals and combines them. This is a sensible design, as individual preferences capture user-specific characteristics, while behavioral similarity provides additional information from similar users. Combining the two allows the model to leverage cross-user information without relying solely on other users' preferences.

Critical weaknesses
  • Limited statistical validation. In 12 of the 16 settings in Table 1, the improvement over the strongest baseline is below 1%p, and BehaviorRM underperforms LoRe on PRISM with both Qwen backbones (54.56 vs. 55.30 and 54.24 vs. 54.87). Nevertheless, significance tests are reported only for Llama-3.2-1B, which shows the largest average margin among the four backbones (+0.9%p vs. +0.13%p for Qwen3-1.7B). Significance tests or mean ± std over multiple seed for all backbones are needed to establish that the improvements are reliable.

  • Unclear hyperparameter tuning data. Appendix F describes the search ranges for α, weight decay, and the embedding dimension, but does not specify on which data the search was performed. Notably, the accuracy at the adopted configuration in Appendix G.3 (89.44%) is identical to the main result for PersonalLLM with Qwen3-4B in Table 1, suggesting that the sensitivity analysis was conducted on the test set. If the hyperparameter search in Appendix F was also performed on the test set, the reported results may be optimistic.

Questions for authors

The paper presents a novel approach to personalized reward modeling by representing query-response pairs as shared SIDs to capture behavior similarity across users. However, statistical significance is reported for only one backbone, and the improvements over the strongest baseline are marginal in many settings. Reporting statistical significance across backbones would help establish the robustness of the results and strengthen the paper.

Reviewer_CNhEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets personalized reward modeling. It argues that existing methods such as PAL, VPL and LoRe rely on semantic information and don't model behavioral similarity between users, and that standard collaborative filtering doesn't apply because LLM responses are generated on the fly, so different users almost never rate the same (query, response) pair. BehaviorRM quantizes each (query, response) embedding into hierarchical semantic IDs with an RQ-VAE, learns a matrix-factorization-style "behavioral" reward between user embeddings and semantic-ID embeddings, and linearly combines it with a LoRe-style semantic reward, training everything with a Bradley-Terry loss. Experiments on ChatbotArena, PRISM, RedditTLDR and PersonalLLM with four Skywork-Reward-V2 backbones show small improvements over LoRe in most settings.

Critical strengths
  • The motivation is clear and intuitive: generated responses don't overlap across users, so the user–response matrix is nearly empty, and semantic IDs are a reasonable way to create shared structure.
  • The method is simple, lightweight and easy to add on top of an existing reward model.
  • Results go in the same direction across most dataset and backbone combinations.
  • There are several ablations (fusion variants, RQ-VAE vs. RQ-KMeans, hyperparameter sensitivity) and a visualization of how semantic IDs create overlap.
Critical weaknesses
  1. The gains are small and the evidence that they are real is thin. Improvements over LoRe are mostly around one point or less, and BehaviorRM loses to LoRe on two of four backbones on PRISM. The main table appears to be single-run, and significance is tested only on one backbone with three seeds, although the text says significance was "verified". LoRe's number for the same setting also differs between Table 1 and Table 4. Tuning is uneven: baselines use their default hyperparameters, while BehaviorRM's mixing weight is tuned in steps of 0.01. Most tellingly, the "Semantic Only" ablation is structurally LoRe, yet on PRISM it scores clearly below the LoRe numbers in Table 1. That gap is larger than BehaviorRM's own improvement, which suggests implementation and tuning differences are as large as the effect being claimed.
  2. It isn't shown that the behavioral branch adds information beyond semantics. The behavioral branch is built from the same embedding the semantic branch already uses, just quantized, and the paper's own ablation shows plain residual k-means works about as well as RQ-VAE. So the semantic-ID step is essentially clustering of the response embeddings. LoRe also already shares a low-rank basis across users, so the claim that existing methods lack collaborative modeling is overstated. Key controls are missing: a behavioral branch on the continuous embedding without quantization, a parameter-matched LoRe (e.g. higher rank), and a shuffled-ID sanity check. Without them the gain could simply come from extra parameters.
  3. The "nearly orthogonal" premise only holds for some datasets, and the simplest collaborative baselines are missing. In PersonalLLM every prompt comes with the same fixed set of responses for all users, and in RedditTLDR the summaries come from a few fixed policies and are compared repeatedly, so plain collaborative filtering on response IDs is possible there without semantic IDs. In PRISM and ChatbotArena, where the premise does hold, responses come from a small fixed set of models, so a user × model factorization is a natural and cheap baseline. The results pattern is also hard to square with the motivation: the gains are largest on the semi-synthetic PersonalLLM, mixed on PRISM, the sparsest real-user dataset, and on RedditTLDR the Semantic Only variant beats the full model.
  4. Novelty is overstated and related work is incomplete. CoPL (EMNLP 2025) already applies collaborative filtering to personalized reward models and evaluates on TL;DR and PersonalLLM, but it is neither cited nor compared. Using RQ-VAE semantic IDs in place of item IDs in ranking models has also been studied in recommender systems (e.g. RecSys 2024), and the training objective is essentially BPR-style matrix factorization with content features. These connections should be discussed. PReF, SynthesizeMe and MiCRo are cited but not compared.
  5. The title promises more than the experiments show. Only reward-model accuracy on seen users is evaluated. No preference optimization (DPO, PPO, or best-of-n with the learned reward) is done, and new users are not considered.
  6. Minor:
    • The sensitivity analysis says that at low α the behavioral component is underweighted, but under Eq. 11 a low α gives the behavioral term more weight.
    • The zero-shot Ref baseline is well below 50% in several settings, which suggests a possible evaluation issue.
    • The abstract mentions three real-world datasets, but PersonalLLM is semi-synthetic.
Questions for authors
  1. Why does Semantic Only, which is structurally LoRe, score so far below LoRe in Table 1? What happens if the semantic-ID branch is added on top of the official LoRe implementation?
  2. What is the raw cross-user overlap of (query, response) pairs in each dataset? How often does a test pair or its semantic-ID sequence appear in training, for example for another user?
  3. How do a continuous-embedding behavioral branch, a parameter-matched LoRe, raw-ID matrix factorization and a user × model factorization compare?
  4. Is Table 1 a single run, why do LoRe's numbers differ between Tables 1 and 4, and were hyperparameters selected on the validation split?
Reviewer_Cj38OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

BehaviorRM combines semantic and behavioral rewards for personalized preference modeling. It quantizes query-response representations into shared Semantic IDs using RQ-VAE, then learns user–SID embeddings to capture collaborative signals. Experiments on three datasets report modest improvements over the evaluated baselines.

Critical strengths
  • Shared SIDs provide a practical way to connect otherwise sparse user-response interactions.
  • The reported gains support combining semantic and behavioral signals.
Critical weaknesses
  • The learned SID structure needs more direct analysis. Please show examples of query-response pairs sharing SIDs and quantify their semantic or preference consistency. A clustering ablation comparing RQ-VAE with k-means and random assignments under the same code budget would help test whether the learned grouping explains the gains.
  • The representation construction needs clearer specification: encoder input, last-token hidden-state extraction, and frozen versus trained components. A last-token versus mean-pooling ablation would help assess sensitivity to this design choice.
Questions for authors
  1. Can you provide within-SID examples and consistency statistics, and compare RQ-VAE grouping against k-means and random assignments with a matched code budget?
  2. Please clarify the encoder and pooling implementation. Which hidden states are used exactly and how does last-token pooling compared to other type of pooling methods?

These analyses would help resolve my concerns and could move my recommendation toward acceptance.

Reviewer_QbCEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BehaviorRM, a personalized reward model intended to combine semantic relevance with collaborative preference information. Because different users rarely evaluate exactly the same generated responses, the method quantizes query–response representations into shared semantic identifiers and learns user–identifier embeddings from preference data. A similarity-based score from these embeddings is combined with a low-rank semantic personalized reward and trained using a pairwise preference objective. Experiments cover four datasets and four reward-model backbones, with analyses of component choices, quantization, and hyperparameter sensitivity. The central question is whether this shared discrete representation captures useful preference structure beyond what existing semantic personalized reward models already learn.

Critical strengths
  1. A practical response to the lack of shared items in personalized reward modeling. Standard collaborative filtering relies on users interacting with common items, whereas generated responses are typically distinct. Mapping query–response pairs to shared semantic identifiers creates a concrete mechanism for pooling preference supervision across nonidentical responses. This is a sensible adaptation of ideas from recommendation to a genuine difficulty in personalized reward learning.

  2. Evaluation across several data sources and model configurations. Table 1 covers four datasets and four backbones from two model families, rather than relying on a single favorable setting. The paper also compares semantic-only, behavior-only, and fusion variants, and includes an alternative quantization method in Appendix G.2. Appendix G.1 further reports three-seed results and significance tests for Llama3.2-1B, providing some evidence that the improvements are repeatable on that backbone.

Critical weaknesses
  1. The evidence does not clearly separate collaborative preference structure from added representational capacity. Both reward branches ultimately use the same query–response representation: Equation 10 applies a learned projection, while Equation 9 uses embeddings indexed by its quantization. The semantic projection is itself shared across users, so cross-user learning is not unique to the behavioral branch. The fusion ablations and embedding-dimension sweep are useful, but do not provide a parameter-matched semantic alternative. As a result, the gain could partly reflect a more flexible feature map rather than the specific behavioral complementarity claimed by the paper.

  2. The semantic-only baseline is not reconciled with the strongest existing baseline. Table 2's Semantic Only variant appears closely related to the low-rank semantic scoring used by LoRe, yet the same-backbone PRISM scores differ substantially: 52.44 versus 55.30 in Table 1. Conversely, Semantic Only slightly exceeds the full model on RedditTLDR, 69.97 versus 69.80. These results do not rule out benefits from fusion, but they make the component comparison difficult to interpret without a precise account of implementation, normalization, regularization, and tuning differences. The strongest semantic baseline is the relevant reference for attributing gains to the additional branch.

  3. Increased interaction overlap is not sufficient evidence of meaningful behavioral similarity. Section 5.4 visualizes densification using positive interactions from 20 users. Coarsening distinct responses into shared identifiers naturally increases overlap, but does not establish that this overlap identifies users with similar preferences beyond ordinary semantic similarity. The alternative quantizer experiment helps compare implementations, not for resolving this interpretation. A held-out analysis relating identifier overlap to preference agreement would more directly test the proposed explanation of why the method works.

  4. The statistical evidence has limited coverage, and an important baseline discrepancy remains unexplained. Appendix G.1 reports three-seed results for one backbone, with significant improvements on three datasets, but this does not establish robustness across all Table 1 settings. In addition, LoRe's PersonalLLM result for Llama3.2-1B is 89.20 in Table 1 and 87.54 ± 0.16 in Table 4. That difference materially changes the apparent size of the improvement. Clarifying whether the tables use different runs, settings, or aggregation rules is necessary before drawing a firm conclusion from the reported margins.

Questions for authors
  1. What exactly distinguishes Semantic Only in Table 2 from LoRe with the same backbone? Is it the full model with the behavioral weight set to zero, and are parameter counts, normalization, regularization, and tuning budgets matched?

  2. Can you compare against a parameter-matched nonlinear semantic model and a control that randomizes assignments of examples to semantic identifiers while preserving identifier frequencies? Randomizing assignments, rather than merely renaming identifiers, would help test whether the learned grouping contributes beyond capacity.

  3. Does identifier overlap predict held-out preference agreement after accounting for query–response embedding similarity? Such an analysis would provide more direct evidence of the proposed behavioral signal than interaction densification alone.

  4. Can you reconcile the LoRe PersonalLLM results in Tables 1 and 4 and identify the runs or aggregation procedures behind Table 1? Extending matched repeated-seed comparisons to another backbone would help establish the scope of the observed improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.