- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes BehaviorRM, a personalized reward modeling framework that jointly captures individual semantic consistency and behavioral similarity across users. The framework consists of three components. First, it represents each response as a composition of shared Semantic Identifiers (SIDs) through quantization, alleviating the sparsity of user interaction data and establishing a basis for information sharing across users. Second, it learns user and SID embeddings and models a SID-level interaction matrix to capture behavioral similarity across users. Finally, it combines a semantic reward reflecting individual semantic preferences with a behavioral reward reflecting behavioral similarity across users to obtain a unified reward signal. Through this approach, BehaviorRM aims to improve personalized preference alignment by leveraging interaction information from similar users while preserving individual preferences.
Critical strengths
-
Well-motivated solution to interaction sparsity. The paper addresses interaction sparsity by representing query-response pairs as shared SIDs and constructing a user-SID interaction matrix. This allows semantically similar interactions to share identifiers, increasing interaction overlap across users compared with raw query-response representations and potentially alleviating sparsity.
-
Complementary modeling of individual preferences and cross-user behavior. BehaviorRM models individual semantic preferences and behavioral similarity across users as separate reward signals and combines them. This is a sensible design, as individual preferences capture user-specific characteristics, while behavioral similarity provides additional information from similar users. Combining the two allows the model to leverage cross-user information without relying solely on other users' preferences.
Critical weaknesses
-
Limited statistical validation. In 12 of the 16 settings in Table 1, the improvement over the strongest baseline is below 1%p, and BehaviorRM underperforms LoRe on PRISM with both Qwen backbones (54.56 vs. 55.30 and 54.24 vs. 54.87). Nevertheless, significance tests are reported only for Llama-3.2-1B, which shows the largest average margin among the four backbones (+0.9%p vs. +0.13%p for Qwen3-1.7B). Significance tests or mean ± std over multiple seed for all backbones are needed to establish that the improvements are reliable.
-
Unclear hyperparameter tuning data. Appendix F describes the search ranges for α, weight decay, and the embedding dimension, but does not specify on which data the search was performed. Notably, the accuracy at the adopted configuration in Appendix G.3 (89.44%) is identical to the main result for PersonalLLM with Qwen3-4B in Table 1, suggesting that the sensitivity analysis was conducted on the test set. If the hyperparameter search in Appendix F was also performed on the test set, the reported results may be optimistic.
Questions for authors
The paper presents a novel approach to personalized reward modeling by representing query-response pairs as shared SIDs to capture behavior similarity across users. However, statistical significance is reported for only one backbone, and the improvements over the strongest baseline are marginal in many settings. Reporting statistical significance across backbones would help establish the robustness of the results and strengthen the paper.