- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors present a methodology to address the underlying mismatch between continuous and shifting human opinions/annotations and the single-label training and evaluation setup. Their method primarily involves a principled way to create a neighborhood of similar annotator-content samples, and extend single annotations by averaging over the neighborhood, and then use this target then for training. They show how this better captures the uncertainty and the plurality of human opinions in a test set of their construction.
Critical strengths
- The paper identifies, poses and solves a genuinely interesting and important problem.
- The authors provide a principled approach to control the amount of neighbor information used to construct the new training distribution, controlled by the "subjectivity" of the problem (at least at the moment for the current model being trained), shown to be superior to simpler alternatives.
- The paper's motivation, and hence the provided solution, are easy to follow, keep readers interested and engaged, and provide genuine insight.
Critical weaknesses
- Lack of detail on the dataset construction and some basic statistics in the main paper, important since the takeaways rely on the soundness of the dataset. The authors should not delegate all of the discussion to the appendix.
- Lack of other datasets and models to show generalization of findings: The findings are supported only by the test set of the authors and for one model. This brings up another point about the generalizability. The method was motivated as using a neighborhood to generate a distribution that is otherwise not present, yet the examined dataset does already have such a distribution (I understand it is not used during training, correct?). It would be interesting to see generalization in a dataset that does not have such a property, where the neighbors are used to construct a distribution that is not there. Moreover, the authors have annotated each and every sample with all annotators, which is usually prohibitively expensive in practice. Generalization in a setting where each item has a few annotations and each annotator only sees a fraction of the dataset (i.e., the usual case) has not been established.
Questions for authors
- Why are the RL objectives used in the paper? My assessment is that, since this is only done for classification purposes with no reasoning, it appears as if this was done for the sake of their inclusion, without any actual expected benefit. Is this accurate?
- It would be useful to describe the complexity of the each gradient descent step, as efficiency is a point of emphasis for any training algorithm.
- A clarification: the grouping, since it happens on x, take into account the persona embeddings and the content embeddings? Therefore, the grouping can happen from similar personas and/or similar content. Empirically, what neighbors are fetched? In your dataset with its high density of annotations and a less dense dataset?
- For the formulation in Section 3.1 and the preference of one model over the other for accuracy: What would the case be on average for a perfect underlying model with sampling (iii.k2) compared to a simpler model (iii.k1)? You presented a specific example with a failure case, and theoretically it establishes your point, but in a whole population, empirically and on average, wouldn’t phi_theta_2 (its observed k2s) be better?
- I had a hard time wrapping my head around definition 2 and theorem 1. Could you provide more intuition? Why is the definition what it is? Why are the terms called bias and statistical error? etc.
7 comments
Response (2/3)
Question1: Why are the RL objectives used in the paper? My assessment is that, since this is only done for classification purposes with no reasoning, it appears as if this was done for the sake of their inclusion, without any actual expected benefit. Is this accurate?
Response: This is not entirely accurate. Our task is to learn from human actions, and RL and preference optimization are the standard training paradigm for exactly this; they are not specific to reasoning tasks. In social simulation and opinion alignment, they are widely used: Sotopia-RL trains agents with GRPO in simulated social environments [1], and SDPO optimizes social agents with a DPO-style objective [2]. Closest to our setting, preference optimization has also been applied to multi-choice opinion tasks: GPO learns group preference alignment on OpinionQA [3], and GDPO aligns the group opinion distribution of GlobalOpinionQA with a distributional variant of DPO [4]. Including DPO and PPO as baselines therefore follows established practice in this line of work.
[1] Yu et al., 2025. Sotopia-RL: Reward Design for Social Intelligence. arXiv:2508.03905.
[2] Kong et al., 2025. SDPO: Segment-Level Direct Preference Optimization for Social Agents. ACL 2025.
[3] Zhao et al., 2023. Group Preference Optimization: Few-Shot Alignment of Large Language Models. arXiv:2310.11523.
[4] Yao et al., 2024. GDPO: Group Distributional Preference Optimization. arXiv:2412.20299.
Question2: It would be useful to describe the complexity of the each gradient descent step, as efficiency is a point of emphasis for any training algorithm.
Response: The per-step cost of SALT is essentially the same as SFT. Each training step reads the soft label vector and then runs a standard forward and backward pass. The only difference from SFT is the loss. SFT computes cross-entropy against a single observed label, while SALT computes KL divergence over the K options. The extra cost is O(K), negligible relative to the network itself.
Neighbor retrieval and soft-label aggregation do not occur inside training steps. The soft labels are refreshed every 30 steps by one gradient-free inference pass over the training set, and embeddings are precomputed once before training.
Measured costs are reported in Appendix D (Computational Cost). Embedding precomputation is a one-time cost of 40 seconds. Each training epoch takes about 35 minutes for SALT and 31 minutes for SFT on 8 H20 GPUs, an increase of about 13%. We will add this per-step analysis to Appendix D in the revision.
Question 3: A clarification: the grouping, since it happens on x, take into account the persona embeddings and the content embeddings? Therefore, the grouping can happen from similar personas and/or similar content. Empirically, what neighbors are fetched? In your dataset with its high density of annotations and a less dense dataset?. Response: We thank the reviewer for this question.
-
Embedding composition. Yes, the embedding of x contains both the persona and the question content, so both can in principle shape the neighbor ranking. In SubjSim, however, neighborhood retrieval is restricted to the same action-space group (Eqs. 11–12). A group corresponds to exactly one question. All contexts in a group share the same question text, so the content component is constant within a group. The neighbor ranking is therefore driven by the persona embedding (see Appendix D).
-
What the fetched neighbors look like. The fetched neighbors are therefore contexts with the same question and the most similar personas. We verified that they are behaviorally meaningful: their response distributions are measurably closer than those of random personas (7.1% lower JSD for the nearest neighbor; details in our response to Reviewer MrP8, Weakness 3).
-
Annotation density: In a less dense dataset, the radius rule automatically widens the neighborhood, so neighbors are fetched from a broader range of personas. Please see our response to Weakness 2 for the full analysis and the subsampling experiment.
Response (3/3)
Question 4: For the formulation in Section 3.1 and the preference of one model over the other for accuracy: What would the case be on average for a perfect underlying model with sampling (iii.k2) compared to a simpler model (iii.k1)? You presented a specific example with a failure case, and theoretically it establishes your point, but in a whole population, empirically and on average, wouldn’t phi_theta_2 (its observed k2s) be better?
Response: We thank the reviewer for this question. The cases in Section 3.1 are meant to show that accuracy can be wrong, not that it is always wrong. For a fixed model θ, its expected accuracy equals the true probability of its top action: . So on average, the ideal model does score highest under accuracy. We agree with the reviewer on this. The problem is that, for a metric meant to measure distributional fidelity, "can be wrong" is already a defect.
First, the failure probability grows with subjectivity. The reversal in case (iii) happens whenever the observed answer misses the mode, with probability . And since each context has only one observation, this risk cannot be averaged away by repeated sampling.
Second, even in expectation, accuracy only sees the top action. A perfect model and a point-mass model on the mode have identical expected accuracy (both p_max), while their true errors differ by .
So our claim is not that accuracy is always wrong, but that it is neither sensitive nor reliable, and fails more often on more subjective contexts which are exactly where a reliable metric matters most.
Question 5: I had a hard time wrapping my head around definition 2 and theorem 1. Could you provide more intuition? Why is the definition what it is? Why are the terms called bias and statistical error? etc.
Response: We thank the reviewer for this question.
Intuition behind Definition 2. measures how many actions the probability mass is effectively spread over. It has a calibration property: if the probability is uniform over actions, equals exactly . So it serves as the "equivalent number of equally likely actions": for a point mass, for the uniform distribution.
Why this form. It was not chosen by us; it is exactly the combination that falls out of the derivation of the statistical error. The aggregated label of action averages independent Bernoulli observations with mean (the neighborhood average), whose noise is . Summing the per-action errors, as the TV distance does, gives , and the combination appearing here is exactly .
Why "bias" and "statistical error". The names follow the bias–variance decomposition in kernel regression. The statistical error is the random fluctuation from finite sampling, and it vanishes as the neighborhood grows. The bias is the systematic shift from using neighbors whose distribution is rather than . By the Lipschitz assumption it is at most , and collecting more neighbors does not remove it. The two terms move in opposite directions in , and balancing them gives . Since the statistical error is controlled by , more subjective contexts call for a larger radius, which is the basis for SALT's subjectivity-adaptive radius.
Response (1/3)
We sincerely thank the reviewer for the positive assessment and for recognizing the importance of the problem. We are also grateful for the reviewer's questions, which are inspiring and help us improve the paper. We address each point below.
Weakness 1: Lack of detail on the dataset construction and some basic statistics in the main paper, important since the takeaways rely on the soundness of the dataset. The authors should not delegate all of the discussion to the appendix.
Response: We thank the reviewer for this suggestion. We agree that our conclusions rely on the soundness of the dataset, so its construction should be explained in the main paper itself. In the revision, we will expand Section 5.1 to cover the question source, the annotation protocol and quality control, the annotator pool, and key statistics of the dataset. Full details will remain in Appendix G.
Weakness2: Lack of other datasets and models to show generalization of findings: The findings are supported only by the test set of the authors and for one model. This brings up another point about the generalizability. The method was motivated as using a neighborhood to generate a distribution that is otherwise not present, yet the examined dataset does already have such a distribution (I understand it is not used during training, correct?). It would be interesting to see generalization in a dataset that does not have such a property, where the neighbors are used to construct a distribution that is not there. Moreover, the authors have annotated each and every sample with all annotators, which is usually prohibitively expensive in practice. Generalization in a setting where each item has a few annotations and each annotator only sees a fraction of the dataset (i.e., the usual case) has not been established.
Response:
On the dataset already having a distribution. Correct: no distribution enters training. Our modeling assumption is that every subjective context has an underlying response distribution, whether it is annotated or not. This is not a property specific to SubjSim; it is a property of this class of tasks. No existing dataset provides such distributions, which is why we constructed SubjSim ourselves. On the training side, each context carries only a single label, and the soft label is built entirely by neighborhood aggregation. From the algorithm's perspective, SubjSim is therefore indistinguishable from a dataset without distributions: they exist only behind the scenes, as the evaluation reference.
Annotation density. SALT adapts to annotation density automatically. Sparse annotation shrinks the group size ng, and the radius rule compensates on its own: r∗≍(ngL2Keff∗)1/(dX+2) As ng shrinks, the radius grows as ng−1/(dX+2), pooling more neighbors to keep the statistical term controlled. The cost is small. The optimized error bound also scales as ng−1/(dX+2), so halving the annotations multiplies it by 21/(dX+2), and at our estimated dX=28, halving the annotations would increase the bound by only about 2%.
We further verified this empirically. We subsampled the SubjSim training set from 100% down to 10% (about 16 personas per question instead of 165) and retrained SALT:
Two observations:
a. Performance is insensitive to density: removing 90% of the training data changes each distribution metric by only 3% to 6% in relative terms (KL 0.2925 → 0.3069).
b. Even with only 10% of the data, SALT still clearly outperforms SFT trained on the full data (KL 0.3069 vs. 1.2871). This shows that the aggregation mechanism itself borrows statistical strength across contexts, consistent with the radius-rule analysis above.
We thank the reviewer for raising this question. The subsampling results will be added to the revised paper.