- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies the problem of personalized reward modeling, where the goal is to learn a new user’s preference model quickly from a few examples. Given the data sparsity in this setting, it is important to leverage as much existing data as possible to inform this learning. Towards this end, this paper proposes PerSPL (Personalized Semi-supervised Preference Learning), a framework for leveraging unlabeled data (along with labeled data) for improving quick personalization. Its main technical contribution is to center rewards in a way that enables meaningful comparisons between responses to different prompts; training is iterative rounds of model updates and pseudo-label updates. Experiments across a range of benchmarks show consistent improvements over baselines. Method design choices are justified with ablations.
Critical strengths
I find this paper to be clear and well organized. The research topic is interesting, and I believe that the problem of learning new user preferences quickly from limited feedback remains quite understudied relative to its prevalence in the real world. The setting is described clearly, which allows the paper to effectively motivate the main algorithm. The PerSPL algorithm is presented clearly, and the method design choices are intuitive.
Empirical validation is relatively strong (with a small set of caveats, see weaknesses), evaluating across a range of benchmark settings and offering a comprehensive comparison to baselines. The experiments in Sections 5.4 and 5.5 help justify algorithmic choices.
Critical weaknesses
The algorithmic novelty presented in PerSPL is limited, recombining known ingredients (i.e., centering rewards, recalibration) for a new purpose. Though the empirical wins are consistent, they are generally small in magnitude. These factors can limit the potential contribution of this paper.
Also, though this critique extends beyond this paper to much of the published related work, its practical impact is limited by a focus on personalized reward modeling, which is ultimately just a step on the journey to personalized generation; extending the method and/or experiments to actually generating better responses would bolster the contribution of the work.
Finally, an experiment comparing PerSPL to other approaches for using the unlabeled data would strengthen the argument for the algorithm. For example, what if we just generate a second response for each unpaired response and label them with the current model, doesn’t that then avoid the need for solving the cross-prompt problem (although it incurs some cost)?
Minor thing, many citations should be in parantheses (e.g., using \citep{} in latex) instead of inline (e.g., using \citet).
Questions for authors
Are there other reasonable baseline ways to use this unlabeled data in some semi-supervised fashion? E.g. by generating extra responses and giving pairwise labels, and then maybe weighting data based on its provenance? It would strengthen the experiments to compare to the closest simple/reasonable semi-supervised baseline.
5 comments
Response to Reviewer QgyL (2/3)
W2
W2. Showing that the method helps generate better responses would strengthen the contribution.
Thank you for your advice. We also agree that a personalized reward model is not an end in itself. Its value ultimately lies in whether it helps generate better responses for each user.
We therefore added a generation experiment to test whether the personalized reward learned by PerSPL can select a better response for each user among newly generated responses. The results show that personalized rewards select each user's best response more often than a general reward model, and PerSPL does so most often among all methods.
We connect the reward model to generation via best-of-N. For each prompt, a language model generates N candidates, each user's personalized reward selects one, and the selected response is evaluated by that user's true utility. Since all methods select from the same candidates, any difference comes from the reward model alone. We use PersonalLLM because it is the only dataset on which the true utility of new responses can be computed. Its users are defined on the scores of 10 public reward models, which can be rerun on new responses.
Setup. For each of the 1,000 PersonalLLM test prompts, Qwen2.5-7B-Instruct generates N = 16 responses, and each user's true utility is obtained by rerunning the 10 reward models that define the PersonalLLM users. PerSPL, MRM and LoRe use the paper's checkpoints and users (100 seen and 100 unseen per seed), and Skywork serves as a non-personalized reference. We report top-1, whether the selected response is the user's best (6.25% for random selection). Please see Appendix D of the revision for details..
Table W2-1. Best-of-16 selection with Qwen2.5-7B-Instruct responses
| Method | Seen: top-1 | Unseen: top-1 | Seen: PerSPL − method (pp) | Unseen: PerSPL − method (pp) |
|---|---|---|---|---|
| Random selection | 6.3% | 6.3% | – | – |
| Skywork | 21.8% | 22.1% | +5.3 (5/5, p < .001) | +5.3 (5/5, p < .001) |
| MRM | 24.8% | 25.1% | +2.2 (5/5, p = .023) | +2.2 (5/5, p = .020) |
| LoRe | 25.7% | 26.2% | +1.4 (5/5, p < .001) | +1.1 (5/5, p = .002) |
| PerSPL | 27.1% | 27.4% | – | – |
- Personalized rewards select the user's best response clearly more often than a general reward model. All three personalized methods are above Skywork, and PerSPL's top-1 is 5.3pp higher, in all 5 seeds. Skywork is itself a strong general reward model, so selecting by general response quality alone is not enough, and selecting by each user's own preferences brings additional gains.
- Among the personalized methods, PerSPL most often selects the user's best response. Its top-1 is 2.2pp higher than MRM's and 1.4 / 1.1pp higher than LoRe's (seen / unseen), positive in all 5 seeds for every comparison and user group.
We will add this experiment to the revision (Section 5.2, Table 1, Appendix A.2 and Table 11).
Response to Reviewer QgyL (3/3)
W3 and Q1
W3/Q1. A comparison with other ways of using the unlabeled data is needed, e.g., generating a second response for each unpaired response and pseudo-labeling the pair.
Following your suggestion, we implement and include your suggested approach as an additional baseline, which corresponds to SSRM-style pairwise pseudo-labeling [2]. Moreover, we also include a strong semi-supervised method corresponding to SSPO [1] as a baseline that uses the unlabeled data in an un-paired response setting. We compare with these additional baselines in all five settings.
The results in Table W3-1 below show that:
- Even when the second response is free and drawn from the same distribution as the pool, pairing does not outperform PerSPL, which does not require pairing or generation.
- PerSPL outperforms per-user SSPO in all five settings, with or without our centering, which demonstrates the necessity of centering and PerSPL's effectiveness in utilizing the unlabeled pool in an un-paired response setting.
Setup.
- SSRM-style (the pairwise approach, suggested by the reviewer). We pair each response in the unlabeled pool with another response to the same prompt, compute with the current model, keep only pairs whose confidence exceeds the same threshold as in PerSPL, and train with the BT loss. Like PerSPL, it weights labeled and pseudo-labeled pairs differently by their provenance, using the same labeled-loss weight , which corresponds to the weighting you suggest in Q1. And unlike in your description, the second response is not generated but taken from an existing training response to the same prompt in the dataset, which is a more favorable case for this approach.
- Per-user SSPO-style. This threshold pseudo-labeling approach estimates a reward threshold for each user and assigns hard pseudo-labels to un-paired responses in the unlabeled pool. We test two versions, with and without our pair-mean centering.
Results.
Table W3-1. Comparison with SSRM-style and per-user SSPO-style baselines
| Overall accuracy (%) | HS | HS | Reddit 100 | Reddit 150 | PLLM |
|---|---|---|---|---|---|
| SSRM-style | 71.20 | 65.45 | 69.25 | 69.83 | 95.59 |
| Per-user SSPO (no centering) | 60.96 | 57.29 | 62.16 | 60.88 | 88.75 |
| Per-user SSPO + our centering | 70.54 | 64.77 | 69.36 | 69.60 | 95.34 |
| PerSPL | 71.40 | 65.51 | 70.44 | 70.61 | 95.54 |
The results show that for SSRM-style, PerSPL is 0.2–1.2pp higher on HelpSteer2 and both Reddit settings, and within ±0.06pp on HelpSteer2 and PersonalLLM. For Per-user SSPO, without centering, it falls 6.8–10.4pp below PerSPL, because it thresholds rewards that are not comparable across prompts. Our centering closes most of this gap, and PerSPL remains 0.2–1.1pp ahead in every setting.
We will add both baselines and this discussion to the revision.
Minor. Citations should be in parentheses (e.g., using \citep{} in LaTeX) instead of inline (e.g., using \citet).
We sincerely apologize for this oversight and really appreciate your carefulness for pointing this out. We will fix this and use the correct format in the revision.
Thank you again for your invaluable suggestions!
References
[1] S. Lee, S. Lim, S. Park, S. Cheon, and K. Song. Semi-Supervised Preference Optimization with Limited Feedback. ICLR, 2026.
[2] Y. He, H. Wang, Z. Jiang, A. Papangelis, and H. Zhao. Semi-Supervised Reward Modeling via Iterative Self-Training. Findings of EMNLP, 2024.
Response to Reviewer QgyL (1/3)
Thank you for the thoughtful review and the constructive suggestions. Motivated by them, we added a we added a label-equivalence analysis of the unlabeled pool (W1), a best-of-N generation experiment (W2) and two semi-supervised baselines that use the unlabeled pool (W3). We respond to each point below.
W1
Thank you for this comment. We think our algorithmic contribution is to show how these components make personalized pseudo-labeling on un-paired responses feasible.
We will revise the contribution statements (Section 1) to make this unambiguous.
While the gains are small in absolute terms, we believe they are indeed valuable. On these benchmarks, we show this in two ways:
C1. Collecting more labels does improve MRM, but at a high cost. In Figure 3b of the paper (HelpSteer2, α=0.01), MRM needs four times the paper's label budget to reach 65.68%, close to the 65.51% that PerSPL reaches with the paper's budget, while with twice the budget it reaches only 64.82%. PerSPL thus obtains, without additional labels, an accuracy that MRM needs about four times as many labels to match.
C2. To further quantify the value of the pool on this scale, following the data-advantage analysis of Le Scao & Rush [3], we additionally measured how many real labeled pairs the unlabeled pool is worth. We trained labeled-only with larger label budgets and found the budget at which it matches PerSPL. On HelpSteer2, the unlabeled pool is worth about two thirds to all of the label budget, at no additional labeling cost.
Table W1-1. Label equivalence of the unlabeled pool
The results show that the unlabelled pool for PerSPL is worth 70-100% of the label budget for seen users and about two-thirds for unseen users. The gain for unseen users is smaller because their adaptation updates only the r-dimensional weights wu, while the shared basis V stays frozen.
The revision will add this analysis (Section 5.4, Table 4) and state the size of the gains and the conditions under which they appear (Appendix F.2).
References
[1] S. Lee, S. Lim, S. Park, S. Cheon, and K. Song. Semi-Supervised Preference Optimization with Limited Feedback. ICLR, 2026.
[2] Y. He, H. Wang, Z. Jiang, A. Papangelis, and H. Zhao. Semi-Supervised Reward Modeling via Iterative Self-Training. Findings of EMNLP, 2024.
[3] T. Le Scao and A. M. Rush. How many data points is a prompt worth? NAACL, 2021.