acceptodds
Under review as a conference paper at ICLR 2027

Multi-Dimensional Comparative Scale Construction for Efficient Personalized Subjective Judgment in High-Traffic Applications

Abstract

Subjective judgments are central to many high-traffic applications, but subjective intensity is difficult to quantify and perceptions vary substantially across individuals. To address these challenges, we propose a pairwise comparative framework for multi-dimensional scale construction. By comparing case–person pairs along case and profile dimensions, the framework constructs relative scales that capture both fine-grained intensity and individual variation. To support practical high-traffic deployment, we optimize both offline scale construction and online inference. For scale construction, we combine sparse Elo comparisons with multi-judge voting, cutting the comparison cost from to for objects and a budget of opponents per object, while limiting reliance on any single judge. For inference, we propose SubJudge, a System One model for personalized scoring with Batchwise Preference Optimization (BPO). Using Bradley–Terry comparisons, BPO trains the model to learn relative orderings, and SubJudge reads a continuous score from digit-token probabilities at the first response position, requiring only one forward pass per criterion and reducing the inference complexity to . Experiments on PluriHarms and iNews show that our 9B models match or surpass the evaluated frontier LLMs on multiple metrics. On the H100 GPU, SubJudge achieves an approximately to speedup in mean inference latency over Qwen3.5-9B with different thinking budgets.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
High-Traffic Applications, Subjective Judgment, Safety, affective

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XvSbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper looks at the task of predicting user judgments on items, where these judgments can be highly subjective and personalized. To do so, they propose a 2-stage pipeline. First, data on users relevant to the subjective judgements is collected, forming a profile; as well as their judgment on input cases. These (user profile, case) tuples are then compared in a pairwise manner against each other using a jury of 5 LLM models to determine which should be ranked higher on a subjective response scale. These pairwise comparisons consider the same user with different cases, and different users on the same case. To avoid quadratic blowup in the number of pairwise comparisons, the authors have each tuple compared pairwise only against K (K << N) other tuples, allowing only KN many comparisons. To order the tuples using only a limited number of comparisons, the authors use an Elo rating system and updates to obtain a total ordering over the tuples. This ordering forms a subjective scoring scale across all the tuples. Following this, the authors use pairwise preferences from this scale to train an autoregressive model (using Qwen 3.5 as backbone) to predict user judgments which they name SubJudge. Following this, predictions for (user profile, case) can be done in a single-shot with SubJudge, without any need for pairwise comparisons.

Critical strengths

The paper proposes a method that can significantly improve computational and time requirements to predict subjective judgements from users. This task is certainly important due to downstream applications in critical areas like AI safety and LLM alignment. Direct previous work on the datasets used in this paper use frontier LLM models to do so; this work proposes training a much smaller parameter LLM for this task. The reduced computational and time requirements allow scalability for large data applications, as well as accessibility on this task; therefore this work can have positive impact in the area for applied AI.

Critical weaknesses

Results and novelty seem somewhat incremental.

  • It combines ideas from the literature (e.g. learning subjective preferences, using multiple LLM judges for pairwise comparisons, training a smaller-parameter judge to obtain comparable/stronger accuracy). The combination of using pairwise comparisons with elo-based scoring to design a subjective ruler is the most significant and interesting novelty. SubJudge tends to perform slightly better on the two datasets than the baselines of previous work that use frontier LLM models; but it is not clear that these are the best baselines to compare to - for example, other simple learned models might prove to also perform well, and would more clearly show the value of this pairwise comparison and subjective ruler pipeline.

  • Some ablation tests studying how the elo rankings perform with respect to ordering pairwise playoffs and the set of LLM judges used are done, but this leave many questions. For example, is using the K pairwise comparisons to train, without the elo rating step able to produce a sufficiently good judge? If not, then it suggests that methods such as the proposed elo rating help in ranking the items much better. How does something simple like ridge regression that is simple and small (compared even to say the 9B parameter backbone) work?

Questions for authors
  1. Given that the user scores are directly available, it seems that it would be quite possible to train a model using these alongside user profiles, without going through the pairwise comparisons. For example, works such as - When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks (Fleisig, Adebe, Klein EMNLP 2024) look at a similar issue of modelling individual annotators, but directly train on ratings, instead of using such a pairwise comparison pipeline. Table 3, GT-label seems to do this direct training from my understanding of the paper, and performs quite well. Is there a reason not to discuss such approaches further?

  2. Related to 1 as well. Why is the pairwise comparisons pipeline necessary? What advantage does it give since the result is a model that predicts scores, and not results of pairwise comparisons? For example, if the goal is a model that can make pairwise comparisons to use in downstream tasks such as LLM alignment training that uses pairwise comparisons, this pipeline makes sense. One expectation is that ranking users on the subjective ruler via comparisons on the same case allows better generalization to unseen users which may have different profiles from those in training. Table 18 has some results on this against baselines, but the results are mixed. So I would like to ask the authors if they have further insights onto the importance/value of this pipeline.

Reviewer_SnXNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SubJudge, which is a framework for multi-dimensional personalized subjective judgment. The framework aims to address two challenges: (1) subjective intensity is hard to quantify, and (2) subjective judgments exhibit individual variation. The framework applies a pairwise comparison for multi-dimensional scale construction by combining multi-judge voting and an ELO-based sparse scale. The framework is extensively evaluated on 2 datasets, iNews and PluriHarms.

Critical strengths
  • The framework allows sparse comparison along two axes, which gives more flexibility in score rankings.
  • The framework is optimized for efficiency in high-traffic applications.
Critical weaknesses
  • The improvement brought directly by the ELO-based comparison is not clear. The student may be trained directly on the original teacher comparisons without first converting them into ELO rankings.
  • The novelty of including the BPO training objective is limited due to the lack of baselines. Showing the improvement of training with both SFT and BPO does not justify the use of BPO.
Questions for authors
  • How does the proposed framework compare to directly training the student model?
Reviewer_H5BfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for efficient personalized subjective judgment based on multi-dimensional pairwise comparisons. Each judgment is represented as a case–person pair, and comparisons are constructed along two axes: different cases evaluated for the same person, and different people evaluating the same case. The authors employ multiple LLM judges with majority voting and sparse Elo updates to construct relative subjective scales, reducing the offline comparison complexity from (O(N^2)) to (O(NK)).

To support efficient inference, the paper introduces SubJudge, a scoring model trained through two-stage post-training: digit-based supervised fine-tuning followed by Batchwise Preference Optimization (BPO). BPO uses a Bradley–Terry objective to distill the Elo-induced rankings into a differentiable continuous score obtained from digit-token probabilities at the first response position.

Experiments on PluriHarms and iNews demonstrate competitive personalized prediction performance against several frontier LLMs. The paper also reports training ablations, judge-panel analyses, comparison-budget studies, and substantial inference latency improvements over reasoning-enabled baselines.

Critical strengths
  1. An effective framework for learning personalized subjective judgments through comparative supervision.

The paper addresses an important problem: subjective evaluations are difficult to quantify through absolute ratings and may vary substantially across individuals. The proposed framework combines pairwise comparisons, multi-judge aggregation, and Elo-based scale construction to produce supervision for personalized scoring. The experimental results are promising. In particular, SubJudge achieves an Individual MAE of 0.1659 on PluriHarms, compared with 0.203 for the same-backbone Qwen3.5-9B K-shot baseline. The results on iNews also demonstrate improvements in predicting individual affective responses, particularly for Arousal and Dominance. These findings support the usefulness of comparative supervision for subjective evaluation.

  1. Efficient inference with a simple and practical scoring mechanism.

The proposed SubJudge model uses the expected value of digit-token probabilities to produce continuous scores without autoregressive generation. Combined with BPO, this enables the model to internalize comparative supervision and predict scores using a single forward pass per criterion. The method is particularly appealing for high-traffic applications, where repeatedly querying LLM judges or comparing against reference examples would be costly. The paper also provides useful ablations demonstrating the complementary benefits of digit SFT and BPO, along with evaluations of judge-panel robustness and inference efficiency.

Critical weaknesses
  1. Insufficient evidence for the benefits of multi-dimensional comparisons.

The paper's main contribution is constructing subjective scales through comparisons along both case and person dimensions. However, no ablation isolates the contribution of each comparison axis. Moreover, Table 21 shows lower agreement for cross-person comparisons (72.96%) than same-person comparisons (75.66%). It remains unclear whether cross-person comparisons meaningfully improve personalized prediction.

  1. Limited comparisons with simpler distillation baselines.

The reported improvements may largely result from supervision provided by powerful LLM judges rather than the proposed comparative scale construction. Although Appendix C.4 compares pairwise judgments with direct scoring, no downstream baseline trains the same student on direct teacher scores under a matched budget. Additionally, the main iNews frontier baselines do not receive the same label-based calibration as SubJudge. These limitations make it difficult to isolate the effectiveness of the proposed method.

Questions for authors

No.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions in the Updated Manuscript

We sincerely thank the reviewers H5Bf\color{blue}{\text{H5Bf}}, SnXN\color{red}{\text{SnXN}}, XvSb\color{purple}{\text{XvSb}}, and the AC for their valuable time and constructive feedback. Their comments greatly improved the clarity and completeness of the paper. All revisions to the manuscript are marked in blue\color{blue}{\text{blue}}.

Positive Feedback from Reviewers

  • Effective framework for an important problem
    • H5Bf\color{blue}{\text{H5Bf}}. An effective framework for learning personalized subjective judgments through comparative supervision. The paper addresses an important problem.
    • XvSb\color{purple}{\text{XvSb}}. This task is certainly important due to downstream applications in critical areas like AI safety and LLM alignment.
  • Comparative scale construction along two axes
    • SnXN\color{red}{\text{SnXN}}. The framework allows sparse comparison along two axes, which gives more flexibility in score rankings.
    • XvSb\color{purple}{\text{XvSb}}. The combination of using pairwise comparisons with Elo-based scoring to design a subjective ruler is the most significant and interesting novelty.
  • Efficient inference for high-traffic applications
    • H5Bf\color{blue}{\text{H5Bf}}. Efficient inference with a simple and practical scoring mechanism.
    • SnXN\color{red}{\text{SnXN}}. The framework is optimized for efficiency in high-traffic applications.
    • XvSb\color{purple}{\text{XvSb}}. The reduced computational and time requirements allow scalability for large data applications.
  • Promising results and useful ablations
    • H5Bf\color{blue}{\text{H5Bf}}. The experimental results are promising. The paper also provides useful ablations demonstrating the complementary benefits of digit SFT and BPO.
    • SnXN\color{red}{\text{SnXN}}. The framework is extensively evaluated on 2 datasets, iNews and PluriHarms.

Reviewer Concerns and Our Responses

  1. Contribution of each comparison axis (H5Bf\color{blue}{\text{H5Bf}}, XvSb\color{purple}{\text{XvSb}}). We added an axis ablation to Sec. 3.2 (Table 6). Each axis contributes, and with roughly the same number of comparisons, using both axes together is still better than either axis alone.
  2. Distillation from direct teacher scores (H5Bf\color{blue}{\text{H5Bf}}, XvSb\color{purple}{\text{XvSb}}). We added this comparison to Sec. 3.2 (Table 7). The same student is trained with the same three judges, on their direct scores and on the Elo scale built from their pairwise votes. The comparative scale gives lower Individual and Aggregated MAE.
  3. Calibration of the iNews baselines (H5Bf\color{blue}{\text{H5Bf}}). Under matched calibration (App. B.5, Table 17), SubJudge is still better than the same-backbone Qwen3.5-9B on every dimension. With all baselines calibrated in the same way, SubJudge remains the best on Arousal and essentially on par with the best baseline on Dominance.
  4. Training on the judged pairs without Elo (SnXN\color{red}{\text{SnXN}}, XvSb\color{purple}{\text{XvSb}}). We added a comparison with Elo-based BPO to Sec. 3.2 (Table 8). Training on the judged pairs also gives a usable scorer, but the Elo scale gives lower MAE on both metrics.
  5. Justification of BPO (SnXN\color{red}{\text{SnXN}}). We clarified that Table 4 already compares BPO with SFT, where BPO further improves performance.
  6. Simple baselines such as ridge regression (SnXN\color{red}{\text{SnXN}}, XvSb\color{purple}{\text{XvSb}}). We added ridge regression models trained directly on the PluriHarms human ratings to App. C.5 (Table 26). Every variant has much higher error than SubJudge.
  7. Training directly on human ratings (XvSb\color{purple}{\text{XvSb}}). We clarified that large amounts of human-annotated GT data are costly in high-traffic applications. Our method avoids them and needs only a small amount of GT data for alignment.
  8. Generalization to unseen users (XvSb\color{purple}{\text{XvSb}}). On the three public test splits (App. C.1, Tables 19–21), SubJudge has better Arousal and Dominance MAE than every frontier LLM. Valence is mixed, but SubJudge is still better than the same backbone on all three splits.

We believe these updates address the reviewers' concerns and further clarify the value of SubJudge.

Thank you very much for your time and consideration!

Best regards,

Authors of Submission #21725

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.