Behavior-Aware Reward Shaping for Personalized Preference Optimization

Large Language Models (LLMs) increasingly power web applications such as chatbots and recommender systems, making personalization critical for enhancing user experience. Existing methods for personalized preference alignment often extend Reinforcement Learning from Human Feedback (RLHF) by incorporating user characteristics into reward shaping. While effective, these approaches mainly rely on semantic information and overlook behavioral similarity, which encodes collaborative relationships among users and responses—important cues for personalization. Yet, capturing such signals is challenging: at the response level, users interact with dynamically generated responses, causing cross-user interactions to appear orthogonal with no overlaps, preventing the capture of collaborative signals. To address this challenge, we propose BehaviorRM, a reward shaping framework that explicitly models behavioral similarity. First, Response Semantic Decomposition represents each query-response pair as a composition of shared semantic identifiers (SIDs) obtained via residual quantization, which induces interaction overlaps at the SID level and exposes latent collaborative structures. Next, Behavioral Interaction Modeling fits the SID-level interaction matrix by jointly learning embeddings for users and SIDs, capturing behavioral similarity beyond pure semantics. Finally, Behavior-aware Reward Shaping integrates the behavioral rewards with semantic rewards to form a unified signal, enhancing personalized preference modeling. Extensive experiments on three real-world datasets show that BehaviorRM significantly outperforms baseline methods. All codes are included in the Supplemental Material.