acceptodds
Under review as a conference paper at ICLR 2027

PerSPL: Personalized Semi-supervised Preference Learning

Abstract

Personalized reward models require user-specific preference comparisons, but collecting such feedback is costly and provides only limited supervision for each user. At the same time, large pools of responses can be generated without user identities or preference labels, yet existing personalized reward models leave this data unused. Directly learning from these responses is difficult because pairwise objectives constrain relative scores within a prompt, whereas pseudo-labeling individual responses requires scores that are comparable across prompts for the same user. We propose PerSPL (Personalized Semi-supervised Preference Learning), a backbone-agnostic framework that uses pair-mean score centering to stabilize this cross-prompt score coordinate and iteratively turns unlabeled responses into user-specific training signals. Across three primary benchmarks with a full suite of baselines, including Reddit TLDR with real-user preference comparisons, PerSPL achieves the highest mean accuracy for both seen and unseen users. A focused evaluation on Chatbot Arena further shows gains over the two strongest personalized baselines from the primary suite on real-user open-domain preferences. These results show that unlabeled responses can provide additional supervision for personalized reward modeling when per-user preference labels are scarce.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Personalized reward modeling, semi-supervised learning, large language model alignment, reinforcement learning from human feedback

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QgyLOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies the problem of personalized reward modeling, where the goal is to learn a new user’s preference model quickly from a few examples. Given the data sparsity in this setting, it is important to leverage as much existing data as possible to inform this learning. Towards this end, this paper proposes PerSPL (Personalized Semi-supervised Preference Learning), a framework for leveraging unlabeled data (along with labeled data) for improving quick personalization. Its main technical contribution is to center rewards in a way that enables meaningful comparisons between responses to different prompts; training is iterative rounds of model updates and pseudo-label updates. Experiments across a range of benchmarks show consistent improvements over baselines. Method design choices are justified with ablations.

Critical strengths

I find this paper to be clear and well organized. The research topic is interesting, and I believe that the problem of learning new user preferences quickly from limited feedback remains quite understudied relative to its prevalence in the real world. The setting is described clearly, which allows the paper to effectively motivate the main algorithm. The PerSPL algorithm is presented clearly, and the method design choices are intuitive.

Empirical validation is relatively strong (with a small set of caveats, see weaknesses), evaluating across a range of benchmark settings and offering a comprehensive comparison to baselines. The experiments in Sections 5.4 and 5.5 help justify algorithmic choices.

Critical weaknesses

The algorithmic novelty presented in PerSPL is limited, recombining known ingredients (i.e., centering rewards, recalibration) for a new purpose. Though the empirical wins are consistent, they are generally small in magnitude. These factors can limit the potential contribution of this paper.

Also, though this critique extends beyond this paper to much of the published related work, its practical impact is limited by a focus on personalized reward modeling, which is ultimately just a step on the journey to personalized generation; extending the method and/or experiments to actually generating better responses would bolster the contribution of the work.

Finally, an experiment comparing PerSPL to other approaches for using the unlabeled data would strengthen the argument for the algorithm. For example, what if we just generate a second response for each unpaired response and label them with the current model, doesn’t that then avoid the need for solving the cross-prompt problem (although it incurs some cost)?

Minor thing, many citations should be in parantheses (e.g., using \citep{} in latex) instead of inline (e.g., using \citet).

Questions for authors

Are there other reasonable baseline ways to use this unlabeled data in some semi-supervised fashion? E.g. by generating extra responses and giving pairwise labels, and then maybe weighting data based on its provenance? It would strengthen the experiments to compare to the closest simple/reasonable semi-supervised baseline.

5 comments
AuthorsOpenReview ↗

Response to Reviewer QgyL (1/3)

Thank you for the thoughtful review and the constructive suggestions. Motivated by them, we added a we added a label-equivalence analysis of the unlabeled pool (W1), a best-of-N generation experiment (W2) and two semi-supervised baselines that use the unlabeled pool (W3). We respond to each point below.

W1

W1 (a). The algorithmic novelty presented in PerSPL is limited, recombining known ingredients (i.e., centering rewards, recalibration) for a new purpose.

Thank you for this comment. We think our algorithmic contribution is to show how these components make personalized pseudo-labeling on un-paired responses feasible.

  • Identifying the obstacle. As your summary notes, the key step is making responses to different prompts comparable. BT training leaves a prompt-specific offset in the reward, so single responses cannot be compared across prompts or pseudo-labeled directly.
  • Applying existing methods directly does not work. In the per-user SSPO baseline we introduced in W3 below [1], thresholding such rewards falls 6.9-10.0pp below labeled-only, consistent with the ablation in Table 3. Pair-mean centering turns unpaired training from harmful into helpful.
  • How the components are combined also matters. Even with our centering, per-user SSPO stays below PerSPL in all 5 settings. SSRM-style pairing [2], which needs a second response to each prompt, is not more effective either. This shows that the use of unlabelled data in PerSPL is not trivial.

We will revise the contribution statements (Section 1) to make this unambiguous.

W1 (b). The empirical gains are consistent but small in magnitude.

While the gains are small in absolute terms, we believe they are indeed valuable. On these benchmarks, we show this in two ways:

  • C1. By comparing PerSPL with MRM trained with more labels
  • C2. By measuring how many labeled pairs the unlabeled pool is worth.

C1. Collecting more labels does improve MRM, but at a high cost. In Figure 3b of the paper (HelpSteer2, α=0.01\alpha=0.01), MRM needs four times the paper's label budget to reach 65.68%, close to the 65.51% that PerSPL reaches with the paper's budget, while with twice the budget it reaches only 64.82%. PerSPL thus obtains, without additional labels, an accuracy that MRM needs about four times as many labels to match.

C2. To further quantify the value of the pool on this scale, following the data-advantage analysis of Le Scao & Rush [3], we additionally measured how many real labeled pairs the unlabeled pool is worth. We trained labeled-only with larger label budgets and found the budget at which it matches PerSPL. On HelpSteer2, the unlabeled pool is worth about two thirds to all of the label budget, at no additional labeling cost.

Table W1-1. Label equivalence of the unlabeled pool

Labeled-only accuracy (%)Paper's budget+50%+100%+200%+400%PerSPL (with unlabeled pool)Pool is worth
HelpSteer2 α=0.1\alpha=0.1, unseen71.0071.1771.2871.3671.6171.21+66%
HelpSteer2 α=0.01\alpha=0.01, unseen64.6064.8565.1865.4665.9764.97+68%
HelpSteer2 α=0.1\alpha=0.1, seen70.9071.4271.5872.01–71.59+100%
HelpSteer2 α=0.01\alpha=0.01, seen65.5065.9566.1766.51–66.04+70%

The results show that the unlabelled pool for PerSPL is worth 70-100% of the label budget for seen users and about two-thirds for unseen users. The gain for unseen users is smaller because their adaptation updates only the rr-dimensional weights wuw_u, while the shared basis VV stays frozen.

The revision will add this analysis (Section 5.4, Table 4) and state the size of the gains and the conditions under which they appear (Appendix F.2).

References
[1] S. Lee, S. Lim, S. Park, S. Cheon, and K. Song. Semi-Supervised Preference Optimization with Limited Feedback. ICLR, 2026.
[2] Y. He, H. Wang, Z. Jiang, A. Papangelis, and H. Zhao. Semi-Supervised Reward Modeling via Iterative Self-Training. Findings of EMNLP, 2024.
[3] T. Le Scao and A. M. Rush. How many data points is a prompt worth? NAACL, 2021.

AuthorsOpenReview ↗

Response to Reviewer QgyL (2/3)

W2

W2. Showing that the method helps generate better responses would strengthen the contribution.

Thank you for your advice. We also agree that a personalized reward model is not an end in itself. Its value ultimately lies in whether it helps generate better responses for each user.

We therefore added a generation experiment to test whether the personalized reward learned by PerSPL can select a better response for each user among newly generated responses. The results show that personalized rewards select each user's best response more often than a general reward model, and PerSPL does so most often among all methods.

We connect the reward model to generation via best-of-N. For each prompt, a language model generates N candidates, each user's personalized reward selects one, and the selected response is evaluated by that user's true utility. Since all methods select from the same candidates, any difference comes from the reward model alone. We use PersonalLLM because it is the only dataset on which the true utility of new responses can be computed. Its users are defined on the scores of 10 public reward models, which can be rerun on new responses.

Setup. For each of the 1,000 PersonalLLM test prompts, Qwen2.5-7B-Instruct generates N = 16 responses, and each user's true utility is obtained by rerunning the 10 reward models that define the PersonalLLM users. PerSPL, MRM and LoRe use the paper's checkpoints and users (100 seen and 100 unseen per seed), and Skywork serves as a non-personalized reference. We report top-1, whether the selected response is the user's best (6.25% for random selection). Please see Appendix D of the revision for details..

Table W2-1. Best-of-16 selection with Qwen2.5-7B-Instruct responses

MethodSeen: top-1Unseen: top-1Seen: PerSPL − method (pp)Unseen: PerSPL − method (pp)
Random selection6.3%6.3%––
Skywork21.8%22.1%+5.3 (5/5, p < .001)+5.3 (5/5, p < .001)
MRM24.8%25.1%+2.2 (5/5, p = .023)+2.2 (5/5, p = .020)
LoRe25.7%26.2%+1.4 (5/5, p < .001)+1.1 (5/5, p = .002)
PerSPL27.1%27.4%––
  • Personalized rewards select the user's best response clearly more often than a general reward model. All three personalized methods are above Skywork, and PerSPL's top-1 is 5.3pp higher, in all 5 seeds. Skywork is itself a strong general reward model, so selecting by general response quality alone is not enough, and selecting by each user's own preferences brings additional gains.
  • Among the personalized methods, PerSPL most often selects the user's best response. Its top-1 is 2.2pp higher than MRM's and 1.4 / 1.1pp higher than LoRe's (seen / unseen), positive in all 5 seeds for every comparison and user group.

We will add this experiment to the revision (Section 5.2, Table 1, Appendix A.2 and Table 11).

AuthorsOpenReview ↗

Response to Reviewer QgyL (3/3)

W3 and Q1

W3/Q1. A comparison with other ways of using the unlabeled data is needed, e.g., generating a second response for each unpaired response and pseudo-labeling the pair.

Following your suggestion, we implement and include your suggested approach as an additional baseline, which corresponds to SSRM-style pairwise pseudo-labeling [2]. Moreover, we also include a strong semi-supervised method corresponding to SSPO [1] as a baseline that uses the unlabeled data in an un-paired response setting. We compare with these additional baselines in all five settings.

The results in Table W3-1 below show that:

  • Even when the second response is free and drawn from the same distribution as the pool, pairing does not outperform PerSPL, which does not require pairing or generation.
  • PerSPL outperforms per-user SSPO in all five settings, with or without our centering, which demonstrates the necessity of centering and PerSPL's effectiveness in utilizing the unlabeled pool in an un-paired response setting.

Setup.

  1. SSRM-style (the pairwise approach, suggested by the reviewer). We pair each response in the unlabeled pool with another response to the same prompt, compute p=σ(ra−rb)p = \sigma(r_a - r_b) with the current model, keep only pairs whose confidence exceeds the same threshold as in PerSPL, and train with the BT loss. Like PerSPL, it weights labeled and pseudo-labeled pairs differently by their provenance, using the same labeled-loss weight α\alpha, which corresponds to the weighting you suggest in Q1. And unlike in your description, the second response is not generated but taken from an existing training response to the same prompt in the dataset, which is a more favorable case for this approach.
  2. Per-user SSPO-style. This threshold pseudo-labeling approach estimates a reward threshold for each user and assigns hard pseudo-labels to un-paired responses in the unlabeled pool. We test two versions, with and without our pair-mean centering.

Results.
Table W3-1. Comparison with SSRM-style and per-user SSPO-style baselines

Overall accuracy (%)HS α=0.1\alpha=0.1HS α=0.01\alpha=0.01Reddit 100Reddit 150PLLM
SSRM-style71.2065.4569.2569.8395.59
Per-user SSPO (no centering)60.9657.2962.1660.8888.75
Per-user SSPO + our centering70.5464.7769.3669.6095.34
PerSPL71.4065.5170.4470.6195.54

The results show that for SSRM-style, PerSPL is 0.2–1.2pp higher on HelpSteer2 α=0.1\alpha=0.1 and both Reddit settings, and within ±0.06pp on HelpSteer2 α=0.01\alpha=0.01 and PersonalLLM. For Per-user SSPO, without centering, it falls 6.8–10.4pp below PerSPL, because it thresholds rewards that are not comparable across prompts. Our centering closes most of this gap, and PerSPL remains 0.2–1.1pp ahead in every setting.

We will add both baselines and this discussion to the revision.

Minor. Citations should be in parentheses (e.g., using \citep{} in LaTeX) instead of inline (e.g., using \citet).

We sincerely apologize for this oversight and really appreciate your carefulness for pointing this out. We will fix this and use the correct format in the revision.


Thank you again for your invaluable suggestions!

References
[1] S. Lee, S. Lim, S. Park, S. Cheon, and K. Song. Semi-Supervised Preference Optimization with Limited Feedback. ICLR, 2026.
[2] Y. He, H. Wang, Z. Jiang, A. Papangelis, and H. Zhao. Semi-Supervised Reward Modeling via Iterative Self-Training. Findings of EMNLP, 2024.

Reviewer_PM86OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PerSPL uses unlabeled prompt–response examples to supplement scarce user-specific preference comparisons. It combines pair-mean score centering with per-user isotonic calibration, confidence-based pseudo-labeling, and iterative self-training. Centering addresses the cross-prompt score ambiguity left by pairwise reward training. Experiments on PersonalLLM, HelpSteer2, and Reddit TLDR, with an additional Chatbot Arena evaluation, report higher mean pairwise accuracy than the evaluated baselines for both seen and unseen users. Component ablations support the role of centering in making effective use of unpaired data.

Critical strengths

Section 3 clearly distinguishes within-prompt ranking from cross-prompt score comparability. Pair-mean centering targets a score component that Bradley–Terry training leaves unconstrained, and Appendix B connects pair-center variation to cross-prompt risk. This explains why scores that work for pairwise ranking may be unsuitable for labeling individual responses and gives a concrete rationale for the proposed regularizer.

Table 3 shows a clear interaction between centering and unpaired self-training. Across its four settings, adding unpaired training reduces seen-user accuracy by 2.9–6.5 percentage points without centering, but improves it by 0.5–1.8 percentage points with centering. Under the reported training schedule, these results support centering as an important component of the self-training procedure. The Reddit TLDR and additional Chatbot Arena evaluations also extend the evidence beyond simulated user preferences.

Critical weaknesses

The reliability of the per-user confidence estimates remains insufficiently established. Appendix B.4 reports a precision improvement for the highest-confidence 10% of candidate pseudo-labels on HelpSteer2, using labeled-only models on seen users. It does not report per-user precision and coverage under the filtering rule actually used during training. The isotonic calibrator is fitted on the same comparisons used to train the reward model. With only nine support pairs for an unseen PersonalLLM user (Table 5), high fitted confidence may not translate into reliable pseudo-labels on held-out responses.

The incremental benefit of unpaired self-training for unseen users is less clearly established in the four settings of Table 3. Relative to the centered supervised model, the gains are 0.2, 0.4, 0.6, and 0.0 percentage points; Appendix C.3 reports paired-bootstrap 95% confidence intervals that include zero in three cases. Table 10 provides additional evidence of larger mean gains under some controlled PersonalLLM conditions, so the concern is whether the benefit is consistent across settings and users. The paper does not show whether the small average gains in Table 3 are broadly shared or concentrated in a subset of unseen users.

Questions for authors

At the confidence threshold actually used during training, what are the per-user pseudo-label precision and coverage, particularly for users with very small support sets? Please evaluate these on held-out labeled examples and specify the correctness criterion or proxy. Since the calibrator is fitted on reward scores from the same comparisons used for model training, how well do its confidence estimates transfer to held-out responses?

Across the existing runs, how are the unseen-user gains from unpaired self-training distributed across users in the four settings of Table 3? A paired per-user comparison against the centered supervised model would help establish whether the small average improvements are broadly shared or driven by a few users.

7 comments
AuthorsOpenReview ↗

Response to Reviewer PM86 (1/3)

Thank you for the careful review and the precise analysis suggestions. Following your comments, we

  • measured per-user pseudo-label precision and coverage at the thresholds actually used (W1/Q1),
  • checked how the calibrator's confidence transfers to held-out responses (W1/Q1),
  • checked whether the gains are shared across users or driven by a few, by comparing PerSPL with the supervised model for each user (W2/Q2),
  • clarified the value of unlabelled pool by measuring how many real labeled pairs the unlabeled pool is worth for unseen users (W2),
  • additionally analyzed which users gain more from the unlabeled pool.

We respond to each point below.

W1 and Q1

W1 (a). Per-user pseudo-label precision and coverage under the threshold actually used in training are not reported, especially for users with very few labeled pairs.
W1 (b). Since the calibrator is fitted on the training comparisons, its confidence may be overconfident on held-out responses, so the pseudo-labels may not be reliable.

We agree that the reliability of the per-user confidence estimates should be evaluated under the filtering rule actually used in training on the held out set. To address your two concerns, we measured
1. per-user pseudo-label precision and coverage at the thresholds actually used
2. how the calibrator's confidence transfers to held-out responses.

Together, they show that the pseudo-labels are indeed reliable. In particular, we observe that

  • Accepted pseudo-labels accuracy are above chance in every setting and in all but 7 of the 4,315 per-user measurements.
  • The calibrator's ranking and direction transfer to held-out responses, and these determine the reliability of pseudo-labels.

(a) Precision and Coverage in Real Settings

To obtain these measurements, we reran training with the paper's configurations and recorded, at every pseudo-labeling round, the pseudo-labels that PerSPL actually assigns to each user. For each user and round, we then measure how many candidates are accepted (coverage) and how many accepted pseudo-labels are correct (precision).

In particular, we use two correctness criteria for precision.

  • Held-out proxy. We apply the same calibrator and threshold τ\tau to the user's test pairs, where the chosen response should be labeled 1 and the rejected one 0.
  • Ground truth on the unlabeled pool. For the candidates actually accepted, we use the simulated user's true utility to check whether the response beats the median of all responses to the same prompt. If it does, the pseudo-label should be 1, and otherwise 0.

On Reddit, users are real, so we cannot know a user's preference for responses in the unlabeled pool, and we report only the held-out proxy of the test set. We will report these results in the revision.

Each cell in Table W1-1 is the median over users, seeds, and pseudo-labeling rounds. The last column counts the cases whose held-out precision is below 50%, to check whether any user is systematically misled by pseudo-labels.

Table W1-1. Per-user coverage and precision of pseudo-labels at the thresholds used in training

DatasetUsersτ\tauCoveragePrecision: held-out proxyPrecision: unlabeled-pool ground truthUser-seed-rounds with held-out precision < 50%
HelpSteer2 α=0.1\alpha=0.1unseen0.666.7 [38.8, 95.8]61.8 [59.3, 65.5]63.2 [60.4, 67.0]0 / 498
HelpSteer2 α=0.01\alpha=0.01unseen0.661.1 [37.5, 91.5]58.9 [56.3, 61.9]63.2 [60.0, 67.7]1 / 489 (0.2%)
Reddit 100unseen0.816.2 [7.8, 27.9]72.8 [68.4, 77.6]—1 / 93 (1.1%)
Reddit 150unseen0.816.1 [10.1, 25.2]73.1 [68.9, 76.8]—1 / 97 (1.0%)
PersonalLLMunseen0.887.7 [77.0, 94.0]94.2 [88.7, 96.1]90.6 [84.7, 92.8]0 / 1000
HelpSteer2 α=0.1\alpha=0.1seen0.657.7 [42.1, 74.6]64.3 [62.1, 66.9]66.0 [63.3, 68.9]0 / 500
HelpSteer2 α=0.01\alpha=0.01seen0.652.2 [40.3, 68.7]62.1 [58.4, 65.2]67.7 [63.8, 72.3]2 / 250 (0.8%)
Reddit 100seen0.821.1 [13.7, 28.4]72.1 [67.9, 75.7]—0 / 194
Reddit 150seen0.820.0 [11.9, 28.0]72.8 [68.6, 77.1]—2 / 194 (1.0%)
PersonalLLMseen0.885.9 [76.6, 92.2]94.8 [90.6, 96.4]91.1 [86.6, 93.3]0 / 1000

The results show that pseudo-labels are above chance for nearly every user in held out sets, including unseen users with very few labeled pairs, and no user is systematically misled by them. In every setting, the median precision of accepted pseudo-labels is above 50%. At most about 1% in any setting have held-out precision below 50%.

AuthorsOpenReview ↗

Response to Reviewer PM86 (2/3)

(b) Pseudo-labels are reliable on held-out responses

Results above show that the pseudo-labels are above chance for nearly every user, including on held-out test pairs. Here we explain why this holds even though the calibrator is fitted on the training comparisons. The ranking and direction of the fitted confidence transfer to held-out responses, and these are what determine the reliability of pseudo-labels. The table bins candidates by their confidence qq and shows the fraction actually preferred in each bin (HelpSteer2 α=0.1\alpha=0.1, unseen users).

Table W1-2. Calibration of the confidence q

qq bin0–0.10.1–0.20.2–0.30.3–0.40.4–0.50.5–0.60.6–0.70.7–0.80.8–0.90.9–1
Mean qq in bin0.000.150.250.350.430.530.640.740.831.00
Training pairs0.000.190.240.360.460.520.640.740.821.00
Test pairs0.300.380.410.460.480.530.570.590.610.65
Unlabeled-pool candidates0.290.370.390.440.480.530.570.600.620.66

On the training pairs used to fit the calibrator, the fraction preferred matches qq almost exactly. On new responses, it still rises monotonically with qq and is close to qq near 0.5, only more extreme than the actual fraction at the two ends. Since a pseudo-label depends only on whether qq is above or below 0.5, and responses on either side are still labeled correctly more often than not, this does not reverse the pseudo-labels. Together with (a), this shows that the pseudo-labels are reliable.

W2 and Q2

W2 (a). For unseen users, the gains from unpaired self-training in Table 3 are small and mostly not significant.

Thank you for raising this important question. We believe the gain of unseen users is restricted by the adaptation mechanism (which parameters it can update). In particular, when an unseen user is adapted, the shared basis VV is frozen, and only the rr-dimensional weights wuw_u are updated, which leaves limited room for gain.

While the absolute scale of the gain appears small, we evaluate it from the perspective of data value, i.e., how much additional labelled data can lead to an equivalent scale of performance gain, following the data-advantage analysis of Le Scao & Rush [1]. On HelpSteer2, the unlabelled pool is worth about 2/3 of the unseen users' label budget.

Table W2-1. Label equivalence of the unlabeled pool on HelpSteer2

Labeled-only accuracy (%)Paper's budget+50%+100%+200%+400%PerSPL (with unlabeled pool)Pool is worth
HelpSteer2 α=0.1\alpha=0.1, unseen71.0071.1771.2871.3671.6171.21+66%
HelpSteer2 α=0.01\alpha=0.01, unseen64.6064.8565.1865.4665.9764.97+68%
HelpSteer2 α=0.1\alpha=0.1, seen70.9071.4271.5872.01–71.59+100%
HelpSteer2 α=0.01\alpha=0.01, seen65.5065.9566.1766.51–66.04+70%

For unseen users, the unlabeled pool is equivalent to about two thirds of their label budget. In the revision, we will add this analysis (Section 5.4, Table 4) and explain how freezing VV limits the gain for unseen users (Appendix A.3).

References
[1] T. Le Scao and A. M. Rush. How many data points is a prompt worth? NAACL, 2021.

AuthorsOpenReview ↗

Response to Reviewer PM86 (3/3)

W2 and Q2 (continued)

W2 (b) / Q2. It is unclear whether these small average gains are broadly shared across settings and users or driven by a few users.

Following your suggestion, we performed per-user paired comparisons using the datasets in Table 3 and Table 4, comparing PerSPL with the corresponding supervised model for each user. In every setting, PerSPL improves a majority of user-seed pairs.

Table W2-2. Per-user gain of PerSPL over the centered supervised model

SettingUsersnMean Δ\DeltaMedian Δ\DeltaWin Rate
HelpSteer2 α=0.1\alpha=0.1unseen250+0.20+0.1157.6%
HelpSteer2 α=0.01\alpha=0.01unseen250+0.38+0.3963.2%
Reddit 100unseen100+0.59+0.1157%
Reddit 150unseen100+0.00+0.1258%
HelpSteer2 α=0.1\alpha=0.1seen250+0.69+0.5867.6%
HelpSteer2 α=0.01\alpha=0.01seen250+0.54+0.5865.2%
Reddit 100seen100+1.38+0.3766%
Reddit 150seen100+1.84+0.3359%

Each row above counts user-seed pairs, and the win rate counts ties as half. The table shows that on HelpSteer2, PerSPL improves 58-68% of the user-seed pairs, and the mean and median are close (e.g., +0.38 vs. +0.39 for unseen users at α=0.01\alpha=0.01). On Reddit, PerSPL improves 57-66% of the user-seed pairs, with the gain varying more across workers.

We also performed the same per-user analysis on the controlled experiments of Table 4 and Table 10, comparing PerSPL with labeled-only in the same setting. Here ρ\rho is the fraction of OOD generators in the unlabeled pool, and each row covers 1,000 user-seed pairs.

Table W2-3. Per-user gain in the controlled experiments

ρ\rhoMean Δ\DeltaMedian Δ\DeltaWin Rate
0+4.25+2.067%
0.5+3.24+1.161%
1+1.54+1.264.5%

In this setting, the gain is also broad. PerSPL improves 61-67% of user-seed pairs, and the mean gain is positive in all seeds. The mean gain becomes smaller as more of the unlabeled pool comes from OOD generators.

Which users gain more and when unlabelled data helps more

Prompted by your question, we further analyzed when the unlabeled pool yields larger gains. Besides substituting for part of the labels when labels are scarce (see our response to Reviewer 4hNs W2), it also makes the model more robust to errors in the support labels.

In the controlled experiment, the less reliable a user's own support labels, the larger the gain from the unlabeled pool. The table below groups users by the number of their training labels that were flipped (ρ=0\rho=0).

Table W2-4. Gain from the unlabeled pool by number of flipped support labels

Number of flipped labels12345
Δ\Delta+2.9+4.9+7.5+13.7+15.4

The gain increases monotonically with the number of flipped labels. This highlights a potential advantage of PerSPL: compared with training on labels alone, the unlabeled pool makes adaptation more robust to noise.


Thank you again for your constructive and insightful questions!

Reviewer_4hNsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies personalized reward models when each user has only a few labeled preference pairs, and asks whether a large shared pool of unlabeled single responses (no user IDs, no labels) can help. The key observation is that the Bradley-Terry loss only fixes score differences within a prompt, so single-response scores aren't comparable across prompts and can't be thresholded directly. PerSPL adds a pair-mean centering penalty to fix the score location, fits a per-user isotonic calibrator on each user's labeled pairs, uses it to give confident pseudo-labels to pool responses for each user, and then retrains iteratively (self-training). The reward model is a low-rank personalized head on top of frozen Skywork-Reward-V2 embeddings, the same architecture as LoRe. Experiments cover PersonalLLM, HelpSteer2, Reddit TLDR and a Chatbot Arena subset.

Critical strengths
  • The setting makes sense. Preference labels per user are expensive while responses are cheap, and using an unlabeled response pool for personalized reward models is a reasonable and practical idea.
  • The paper clearly explains why pseudo-labeling single responses doesn't work out of the box with BT-trained rewards (the score offset per prompt is not identified), and the fix is simple.
  • The 2×2 ablation in Table 3 is well designed and honestly reported. It shows that centering on its own does almost nothing for seen users, unpaired training without centering actually hurts, and the two only help together.
  • Results are consistent: PerSPL has the best mean accuracy across benchmarks, and the comparison with LoRe uses the same architecture.
Critical weaknesses
  1. The centering penalty is not new. Eisenstein et al. (2023, "Helping or Herding?") already use essentially the same regularizer, with the same motivation (an arbitrary prompt-dependent constant doesn't change the BT likelihood). The paper doesn't cite it, and the contributions claim both identifying this ambiguity and introducing pair-mean centering. To be fair, the way it is used here is different, since the ablation suggests centering matters mainly because it makes cross-prompt pseudo-labeling possible. But both the identifiability observation and the regularizer itself should be credited to prior work, and the contribution should focus on showing that this known regularizer enables personalized pointwise pseudo-labeling.
  2. The benefit from the unlabeled data itself is small in most settings. With centering in place, adding the pool improves seen users by around one point, and for unseen users the improvement is mostly not significant, even though new users are part of the claimed contribution. PerSPL's own labeled-only variant already beats LoRe in several columns, so part of the advantage over baselines seems to come from the training setup rather than the pool. The clearest gain appears in the controlled study in Table 4, but that setting adds label noise on purpose because clean PersonalLLM is close to ceiling, so it doesn't say much about the clean case.
  3. I'm worried about how realistic the benchmarks are. In PersonalLLM and HelpSteer2 the users are simulated as random linear mixtures of reward-model scores or attribute ratings. The model is a low-rank head (as in LoRe) on top of a frozen reward-model embedding, so in these benchmarks the task closely matches the model's own structure, which may make pseudo-labels unusually easy to get right. The real-user evaluations (Reddit TLDR and Chatbot Arena) only have around 40 users each, with very few labeled pairs for the Arena users. It's hard to tell how well the conclusions carry over to real users with more complex preferences.
  4. Some natural baselines are missing. At least for PersonalLLM and Reddit TLDR, the "unpaired" pool comes from datasets that have several responses per prompt, so one could just form pairs within a prompt and do pairwise pseudo-labeling (SSRM-style) on top of LoRe, which avoids the cross-prompt problem entirely. A per-user version of SSPO-style thresholding would also be a fair comparison. Without these, it's not clear that the single-response formulation is needed, and the unpaired premise feels partly artificial on these benchmarks.
  5. The statistical evidence is thin given how small the differences are. Intervals are computed by bootstrapping five seed differences, which tends to give overly narrow intervals and ignores variation across users and test pairs. Several of the margins over the strongest baseline are about the same size as the per-column standard deviations.
  6. Minor:
    • It's not clear whether the unlabeled pool overlaps with test prompts or test responses (e.g. Reddit TLDR summaries are compared by many workers). If it does, PerSPL gets a transductive advantage over the baselines.
    • How hyperparameters were selected, and how much tuning the baselines got, isn't clearly described.
Questions for authors
  1. Was the frozen Skywork-Reward-V2 model trained on preference data that overlaps with HelpSteer2, Reddit TLDR or Chatbot Arena? If so, some of the accuracy may come from the backbone rather than from personalization.
  2. In Figure 3a, with a 50% label flip rate the labels carry no information, yet PerSPL is still clearly above chance and well ahead of MRM. Where does this signal come from?
  3. How exactly are unseen users adapted? Does PerSPL also pseudo-label the pool for unseen users, which parameters are frozen, and do baselines follow the same procedure?
  4. How is overlap between the unlabeled pool and the test data handled in each dataset?
9 comments
AuthorsOpenReview ↗

Response to Reviewer 4hNs (1/7)

Thank you for the detailed and invaluable review. Following your comments, we added

  • two additional semi-supervised baselines, namely SSRM-style and SSPO-style (W4),
  • a label-equivalence analysis for the value of the unlabeled data, i.e., how many real labeled pairs its gain is worth (W2),
  • an analysis of whether pseudo-labels are unusually easy to get right, by measuring their precision (W3),
  • an additional real-user benchmark, PRISM-Personalized, with 723 users from 75 countries (W3),
  • a stricter statistical test based on a random-effects model, beyond the common seed bootstrap (W5), and a per-user comparison with MRM (W5),
  • overlap checks of the unlabeled pool and of the backbone's training data against the test sets (W6, Q1).

We respond to each point below.

W1

W1. The centering regularizer is not new and should be credited to prior work, and the contribution should focus on showing that it enables personalized pseudo-labeling.

Thank you for pointing us to Eisenstein et al. [1], who indeed already proposed the same centering regularizer. We will properly cite this work in the updated version (Sections 1, 3 and 4), and credit both the identifiability observation and the regularizer to it.

We will also revise our contribution statement (Section 1). Specifically, our contribution is to show that the centering penalty makes the un-paired unlabeled pool usable for personalized preference learning. In particular, we show that

  • Without centering, adding the unlabeled pool lowers the accuracy by 2.9-6.5pp (Table 3 in the revision).
  • The per-user SSPO baseline we added based on your suggestion also underperforms without centering and falls 6.9-10.0pp below labeled-only (see our response to W4).

References
[1] J. Eisenstein et al. Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking. arXiv:2312.09244, 2023.

AuthorsOpenReview ↗

Response to Reviewer 4hNs (2/7)

W2

W2 (a). New users are part of the claimed contribution, yet the gain in unseen users is small.

This is a great question. We would like to clarify by the following two points:

  • P1. The unseen users’ gain is smaller than that of seen users due to the adaptation mechanism, and importantly, it is unlikely to keep them on par. Please see our detailed explanation below.
  • P2. While the absolute value of the gain appears small, from the perspective of value of data, the observed gain is significant and worth about 70% of the labelled budget. Please see the detailed explanation in our response for W2(b)

Explanation of P1. The paper covers two kinds of users: seen users and new users who are unseen during training. Both have only limited labeled comparisons, so the paper's core question is the same for both: whether an unlabeled pool can improve accuracy in this situation. Unseen and seen differ only in which stage of our pipeline they enter.

  • Stage 1 (seen users). The shared basis VV and the users' weights wuw_u are learned jointly, and pseudo-labels update both.
  • Stage 2 (unseen users). VV is frozen, and only each user's weights wuw_u are fitted and updated by pseudo-labels.

This difference explains why the additional gain from the pool is small for unseen users. The value of unlabeled data depends on the structure of the problem, not only on how scarce labels are [2, 3]. Here, it depends on which parameters the pseudo-labels can update. For seen users, pseudo-labels update both the shared basis VV and their weights wuw_u. Instead, unseen user's pseudo-labels can only adjust the rr-dimensional weights wuw_u, which leaves much less room for gain. Under the same adaptation procedure, PerSPL's point estimates on unseen users are still higher than those of every baseline.

W2 (b). Part of PerSPL's advantage comes from the training setup, and the gain from the unlabeled pool itself is small.

We agree that the training setup contributes to PerSPL's advantage. To separate the contribution of the unlabeled pool from that of the training setup, we measured how many additional labeled pairs labeled-only training needs, to match PerSPL trained with the paper's label budget plus the unlabeled pool. This follows the data-advantage analysis of Le Scao & Rush [4]. On HelpSteer2, the pool is worth roughly 70% to all of the label budget, for both seen and unseen users.

Table W2-1. Label equivalence of the unlabeled pool

Labeled-only+50%+100%+200%PerSPLPool is worth
HelpSteer2 α=0.1\alpha=0.1, seen70.9071.4271.5872.0171.59+100%
HelpSteer2 α=0.1\alpha=0.1, unseen71.0071.1771.2871.3671.21+66%
HelpSteer2 α=0.01\alpha=0.01, seen65.5065.9566.1766.5166.04+70%
HelpSteer2 α=0.01\alpha=0.01, unseen64.6064.8565.1865.4664.97+68%

Measured against the gain from adding labels, the pool's gain is substantial. PerSPL matches labeled-only trained with about 1.7-2 times the paper's label budget. The training setup thus explains part of the advantage, but the unlabeled pool has practical value of its own: when each user has few effective labels, it substitutes for a substantial number of real labels.

References
[2] A. Singh, R. D. Nowak, and X. Zhu. Unlabeled data: Now it helps, now it doesn't. NeurIPS, 2008.
[3] V. Castelli and T. M. Cover. The relative value of labeled and unlabeled samples in pattern recognition with an unknown mixing parameter. IEEE Transactions on Information Theory, 42(6), 1996.
[4] T. Le Scao and A. M. Rush. How many data points is a prompt worth? NAACL, 2021.

AuthorsOpenReview ↗

Response to Reviewer 4hNs (3/7)

W3

W3 (a). Pseudo-labels may be unusually easy to get right. This could inflate the gain from the unlabeled pool.

We appreciate your concern that pseudo-labels may be unusually easy to get right on the simulated benchmarks and inflate the gain from the unlabeled pool.

To address your concern, we measured how accurate PerSPL's pseudo-labels are before the confidence filter, using the simulated users' true utilities. We find that the pseudo-labels are far from easy to get right, so PerSPL’s gain does not rely on unusually accurate pseudo-labels.

We measured the accuracy of the pseudo-labels of all PerSPL’s candidate pseudo-labels. For the simulated users of HelpSteer2 and PersonalLLM, correctness is judged by the users' true utilities. For comparison, we also report Reddit TL;DR, where users are real and we use a held-out proxy (see Appendix E.1 of the revision). The results show that, on every benchmark, the unfiltered pseudo-labels are 4-11 points less accurate than PerSPL's own pairwise test accuracy.

Table W3-1. Accuracy of pseudo-labels vs. PerSPL's test accuracy

Seen / unseen users (%, median)HelpSteer2 α=0.1\alpha=0.1HelpSteer2 α=0.01\alpha=0.01PersonalLLMReddit 100 (real users)Reddit 150 (real users)
Unfiltered pseudo accuracy61.0 / 60.161.7 / 59.988.2 / 87.860.2 / 59.860.3 / 59.8
PerSPL test accuracy71.6 / 71.266.0 / 65.095.7 / 95.570.7 / 70.170.9 / 70.3
  • HelpSteer2. Only about 60% of pseudo-labels are correct, the same as for the real users of Reddit.
  • PersonalLLM. Pseudo-labels are more accurate, but they remain below PerSPL's own test accuracy. They are accurate partly because the benchmark is easy for all methods (LoRe and MRM also reach 92-95% in Table 1).

W3 (b). The real-user evaluations have only about 40 users each, with very few labeled pairs for Arena users. It's hard to tell how well the conclusions carry over to real users with more complex preferences.

Thank you for raising this point. The real-user evidence in the paper comes from about 40 users per dataset. To test whether the conclusions carry over to a larger and more diverse population of real users, we added an evaluation on PRISM-Personalized [5, 6]. The personalized dataset includes 723 real users, about 18 times as many as Reddit or Arena, with 16,705 preference pairs in total (see Appendices A.1 and C.2 of the revision).

Table W3-2. Performance on PRISM-Personalized

SeenUnseenOverall
LoRe56.2857.1656.72
MRM60.2759.7460.01
PerSPL61.6161.4161.51

As can be observed in Table W3-2, PerSPL indeed achieves the best accuracy for both seen and unseen users in this more complex setting. Under the more strict crossed random-effects model [7] that considers the variations across users and test pairs, PerSPL improves over MRM by +1.50 pp overall (95% CI

+0.47,+2.54+0.47, +2.54

, p = 0.006), +1.33 pp for seen users (p = 0.037) and +1.67 pp for unseen users (p = 0.005).

References
[5] H. R. Kirk et al. The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models. NeurIPS, 2024.
[6] M. J. Ryan et al. SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs. ACL, 2025.
[7] R. H. Baayen, D. J. Davidson, and D. M. Bates. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 2008.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/2)

We thank the AC and the reviewers for their careful and invaluable feedback. We are encouraged that the reviewers recognized the following strengths:

  • This work studies an important and understudied problem (Reviewers QgyL, 4hNs).
  • The method is intuitive and lightweight (Reviewers 4hNs, QgyL).
  • The method offers a well-supported approach to explore the value of unpaired unlabeled data, whose amount can be considerable in practice (Reviewers 4hNs, PM86).
  • The method shows consistent empirical gains (Reviewers 4hNs, PM86, QgyL).

Following the suggestions of reviewers, we have conducted an array of additional experiments and analysis to further verify the efficacy and value of our algorithm. We appreciate the reviewers’ constructive feedback and we believe these additional results helped us improve the quality of our work.

A. Stronger Evidence and Analysis for Our Method

  • Ablations to Show the Value of Unlabelled Data, i.e., How Much Labeled Data Is Unlabeled Data Worth. We quantify the value of unlabeled data. On HelpSteer2, the unlabeled pool is worth about 65-100% of the label budget, as PerSPL matches labeled-only trained with 1.7-2 times the paper's label budget, for both seen and unseen users. This matters when effective labels are limited.
  • Applying Our Framework for Personalized Generation. We demonstrate the value of our framework by directly applying our reward model to the end goal of personalized generation. We ran best-of-16 selection on PersonalLLM with newly generated Qwen2.5-7B responses. PerSPL selects clearly better responses than all baselines and most often selects each user's best response*.*
  • Additional Semi-Supervised Baselines. SSRM [1] and SSPO [2] are two strong semi-supervised methods for reward modeling and preference optimization: SSRM pseudo-labels response pairs, and SSPO pseudo-labels un-paired responses by thresholding their rewards. Neither is personalized, so we adapted both to the personalized setting and compared them with PerSPL under exactly the same setup. Comparison with SSRM-style proves pairing is not more effective than PerSPL, and PerSPL’s superior performance over SSPO-style confirms centering as the prerequisite for un-paired response pseudo-labeling.
    Combined, they show that PerSPL is a simple and practical way to use unlabeled data for personalization.
  • Consistency of the Gain Across Users. We test whether the average gains reflect most users rather than a few. We compared PerSPL with the supervised model for each user, for Table 3 and for the controlled experiments of Table 4 / Table 10. PerSPL improves about 57-68% users in all settings, so on these benchmarks the gain is not driven by a few users.
  • Reliability of Pseudo-labels. We measured the per-user precision and coverage of pseudo-labels at the confidence thresholds actually used in training. Accepted pseudo-labels are above chance on held-out responses in nearly all per-user measurements, so they provide a reliable signal even for unseen users with very few labels.
  • Additional Benchmark with More Diverse User Preference. We added PRISM-Personalized [3, 4], with 723 real users from 75 countries who write their own prompts, about 18 times as many users as Reddit or Arena. PerSPL still achieves the best accuracy for both seen and unseen users.

References
[1] Y. He, H. Wang, Z. Jiang, A. Papangelis, and H. Zhao. Semi-Supervised Reward Modeling via Iterative Self-Training. Findings of EMNLP, 2024.
[2] S. Lee, S. Lim, S. Park, S. Cheon, and K. Song. Semi-Supervised Preference Optimization with Limited Feedback. ICLR, 2026.
[3] H. R. Kirk et al. The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models. NeurIPS, 2024.
[4] M. J. Ryan et al. SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs. ACL, 2025.

AuthorsOpenReview ↗

General Response to All Reviewers (2/2)

B. Rigorous Verification of Performance Gains

  • Experiments Showing that the Performance Gain Doesn’t Originate From Degenerate (Trivially Correct) Pseudo-labels. On every benchmark, pseudo-labels are 4-11 points less accurate than PerSPL's own test accuracy, showing the benchmark task is not unusually easy.
  • Statistical Robustness. Beyond the commonly used seed bootstrap, we added a more conservative random-effects model that also accounts for variation across users and test pairs, as the reviewer suggested. The results show that even under a stricter test than is standard, PerSPL's overall advantage over MRM remains significant in 3 settings. In addition, our method significantly outperforms MRM on the newly included PRISM benchmark.
  • Leakage Checks. We checked the unlabeled pool against the test sets, and the public training data of Skywork-Reward-V2 against the HelpSteer2 test set. No detectable overlap was found.

In addition, we will clarify the following in the revision. First, we will credit the identifiability observation and the centering regularizer to Eisenstein et al. [5] (Sections 1, 3 and 4), and restate our contribution as showing how these components make personalized pseudo-labeling on un-paired responses feasible. Second, we will describe the construction of the unlabeled pool (Appendices A.1 and G.1), hyperparameter selection (Appendix C.4), unseen-user adaptation (Section 5.2 and Appendix A.3), and the backbone's training data (Appendix G.2).

We hope these additions address the reviewers' concerns, and we are happy to answer any further questions.

References
[5] J. Eisenstein et al. Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking. arXiv:2312.09244, 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.