- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper looks at the task of predicting user judgments on items, where these judgments can be highly subjective and personalized. To do so, they propose a 2-stage pipeline. First, data on users relevant to the subjective judgements is collected, forming a profile; as well as their judgment on input cases. These (user profile, case) tuples are then compared in a pairwise manner against each other using a jury of 5 LLM models to determine which should be ranked higher on a subjective response scale. These pairwise comparisons consider the same user with different cases, and different users on the same case. To avoid quadratic blowup in the number of pairwise comparisons, the authors have each tuple compared pairwise only against K (K << N) other tuples, allowing only KN many comparisons. To order the tuples using only a limited number of comparisons, the authors use an Elo rating system and updates to obtain a total ordering over the tuples. This ordering forms a subjective scoring scale across all the tuples. Following this, the authors use pairwise preferences from this scale to train an autoregressive model (using Qwen 3.5 as backbone) to predict user judgments which they name SubJudge. Following this, predictions for (user profile, case) can be done in a single-shot with SubJudge, without any need for pairwise comparisons.
Critical strengths
The paper proposes a method that can significantly improve computational and time requirements to predict subjective judgements from users. This task is certainly important due to downstream applications in critical areas like AI safety and LLM alignment. Direct previous work on the datasets used in this paper use frontier LLM models to do so; this work proposes training a much smaller parameter LLM for this task. The reduced computational and time requirements allow scalability for large data applications, as well as accessibility on this task; therefore this work can have positive impact in the area for applied AI.
Critical weaknesses
Results and novelty seem somewhat incremental.
-
It combines ideas from the literature (e.g. learning subjective preferences, using multiple LLM judges for pairwise comparisons, training a smaller-parameter judge to obtain comparable/stronger accuracy). The combination of using pairwise comparisons with elo-based scoring to design a subjective ruler is the most significant and interesting novelty. SubJudge tends to perform slightly better on the two datasets than the baselines of previous work that use frontier LLM models; but it is not clear that these are the best baselines to compare to - for example, other simple learned models might prove to also perform well, and would more clearly show the value of this pairwise comparison and subjective ruler pipeline.
-
Some ablation tests studying how the elo rankings perform with respect to ordering pairwise playoffs and the set of LLM judges used are done, but this leave many questions. For example, is using the K pairwise comparisons to train, without the elo rating step able to produce a sufficiently good judge? If not, then it suggests that methods such as the proposed elo rating help in ranking the items much better. How does something simple like ridge regression that is simple and small (compared even to say the 9B parameter backbone) work?
Questions for authors
-
Given that the user scores are directly available, it seems that it would be quite possible to train a model using these alongside user profiles, without going through the pairwise comparisons. For example, works such as - When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks (Fleisig, Adebe, Klein EMNLP 2024) look at a similar issue of modelling individual annotators, but directly train on ratings, instead of using such a pairwise comparison pipeline. Table 3, GT-label seems to do this direct training from my understanding of the paper, and performs quite well. Is there a reason not to discuss such approaches further?
-
Related to 1 as well. Why is the pairwise comparisons pipeline necessary? What advantage does it give since the result is a model that predicts scores, and not results of pairwise comparisons? For example, if the goal is a model that can make pairwise comparisons to use in downstream tasks such as LLM alignment training that uses pairwise comparisons, this pipeline makes sense. One expectation is that ranking users on the subjective ruler via comparisons on the same case allows better generalization to unseen users which may have different profiles from those in training. Table 18 has some results on this against baselines, but the results are mixed. So I would like to ask the authors if they have further insights onto the importance/value of this pipeline.
General Response and Summary of Revisions in the Updated Manuscript
We sincerely thank the reviewers H5Bf, SnXN, XvSb, and the AC for their valuable time and constructive feedback. Their comments greatly improved the clarity and completeness of the paper. All revisions to the manuscript are marked in blue.
Positive Feedback from Reviewers
Reviewer Concerns and Our Responses
We believe these updates address the reviewers' concerns and further clarify the value of SubJudge.
Thank you very much for your time and consideration!
Best regards,
Authors of Submission #21725