acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning with Comparative Evidence for Social Intelligence

Abstract

Developing socially intelligent AI remains heavily dependent on human-annotated data, limiting the scale and breadth of social understanding that models can acquire. Methods that derive training signals from unlabeled data offer a path beyond this dependence, but social predictions lack the verification oracles available in domains such as mathematics and coding. Moreover, core social targets such as affect, intent, preference, and pragmatic meaning are often ambiguous. The same observed behavior can support multiple plausible interpretations, making it difficult to verify which is best supported. To address this challenge, we introduce Reinforcement Learning with Comparative Evidence (RLCE), a reinforcement learning method that learns social understanding from unlabeled training data without constructing rewards from ground-truth annotations. Given distinct answers in a rollout group, RLCE constructs evidence tests that identify observable evidence favoring an answer over another, validates these tests against the input sample, and aggregates test outcomes to determine the best-supported interpretation. Evidence tests are then regenerated as the policy produces new answers during training, enabling tests to evolve with the policy’s answers. Across four benchmarks spanning affect, pragmatics, communicative intent, and preference, RLCE attains the strongest performance among seven methods that use no ground-truth training labels for rewards, including consensus, policy LLM-judge verification, multimodal co-evolution, and rubric-based rewards. Gains over the strongest baseline are +3.58 on MOSEI, +4.50 points on UR-FUNNY, +1.15 on IntentQA, and +18.93 on PrefEval. Analyses further show that RLCE exhibits a larger share of reward variation between correct and incorrect predictions than compared rubric methods, can overturn erroneous policy-derived preferences, and benefits from pairwise test construction, compositional test aggregation, and on-policy test evolution. Code and models will be released after the review process; additional implementation details are in the appendix, included in a separate supplementary file.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
social intelligence, autonomous learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SxcxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Reinforcement Learning with Comparative Evidence (RLCE), a critic-free reinforcement learning framework for learning social understanding from unlabeled data. RLCE compares candidate predictions using pairwise evidence tests, validates whether the corresponding evidence is present in the input, and aggregates the results to determine the reward. Experiments on four benchmarks spanning affect, pragmatics, communicative intent, and preference show consistent improvements over several unlabeled-reward baselines. The paper also provides analyses and ablations supporting the roles of pairwise evidence construction, compositional aggregation, and on-policy test evolution.

Critical strengths
  • The core verification mechanism is well motivated. ****RLCE addresses the ambiguity inherent in social understanding by directly comparing competing interpretations through observable evidence, providing a more targeted verification signal than independently evaluating each answer against generic criteria.
  • The empirical evaluation provides meaningful support for the proposed design. RLCE shows consistent improvements across four diverse social-understanding benchmarks and relevant unlabeled-reward baselines. The paper further provides controlled ablations and analyses, as well as an extension to test-time training that demonstrates the applicability of its label-free verification signal.
Critical weaknesses
  • The computational cost and scalability of comparative verification are insufficiently characterized. RLCE constructs evidence tests for pairs of distinct rollout answers and then validates multiple tests for each comparison. This appears potentially more expensive than pointwise rubric or direct-judge rewards as the number of distinct candidates increases. This is particularly important in reinforcement learning, where repeated reward construction can make verification cost a significant factor in the overall efficiency and scalability of training.
  • The aggregation does not explicitly account for redundancy among evidence tests. The paper shows that composing more evidence tests improves aggregate verdict accuracy, suggesting that additional tests can strengthen verification. However, semantically overlapping tests may repeatedly capture the same underlying cue, causing that evidence to be effectively counted multiple times in the aggregation. The current system design and analysis do not appear to explicitly account for semantic redundancy or correlation among evidence tests.
  • The comparison with closely related pairwise verification methods is incomplete. For example, prior work has explored pairwise self-evaluation for selecting among candidate answers (e.g., REPS [1]) and contrastive rubric generation that derives discriminative evaluation criteria from response pairs (e.g., OpenRubrics [2]). Including these approaches would help better establish the novelty boundary of the proposed comparative verification framework.

[1] Akira Kawabata and Saku Sugawara. 2024. Rationale-Aware Answer Verification by Pairwise Self-Evaluation. EMNLP2024

[2] Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, and Haoyu Wang. 2026. OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment. ACL 2026

Questions for authors
  1. Could the authors provide the end-to-end computational cost of RLCE relative to the baselines, including the pairwise comparisons, generated evidence tests, constructor and validator calls per example, as well as wall-clock time or throughput? Understanding this cost-performance trade-off is important for assessing the practical significance and scalability of the method.
  2. Does RLCE explicitly control for semantic redundancy among evidence tests? Could the authors quantify test redundancy or evaluate a diversity-aware/deduplicated aggregation variant?
  3. What are the key methodological differences between RLCE and prior pairwise verification or contrastive rubric-generation approaches? In particular, what does RLCE introduce beyond these existing formulations, and why are these differences particularly effective for social understanding?
Reviewer_L4eUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Reinforcement Learning with Comparative Evidence (RLCE) for social understanding without using ground-truth training annotations to construct rewards. It canonicalizes answers within a rollout group and uses a frozen copy of the initial policy to generate directional evidence tests for distinct answer pairs. The same frozen model checks these tests against the input, with an abstention option. Aggregated supporting and contradicting evidence selects a canonical answer, whose matching rollouts receive verification rewards for GRPO alongside a structural format reward. Tests are regenerated as policy answers evolve. Experiments on sentiment, humor, intent, and preference report improvements over seven reward-construction baselines, supplemented by test-time adaptation and component analyses.

Critical strengths
  1. S1: A clear, inspectable verification design. Sections 3.2–3.3 (pp. 4–5) separate construction, input-based validation, aggregation, and optimization. Canonicalizing duplicate answers prevents multiplicity from directly determining verification scores. Appendix C.1 (p. 25) additionally specifies that candidate answers are removed before validation. Using the frozen initial model throughout verification makes the improvement over direct policy judgment interesting without introducing a stronger external reward model.

  2. S2: Encouraging results across distinct tasks. Tables 1 and 6 (pp. 6, 23) report three-run means and standard deviations from a common initialization. Gains over the strongest competing means are 3.58 points on MOSEI, 4.50 on UR-FUNNY, and 18.93 on PrefEval. The authors appropriately qualify the smaller IntentQA improvement. Table 7 further reveals substantial Choice/Persona gains, while showing that the advantage does not extend uniformly to Explicit preferences.

  3. S3: Substantial implementation documentation. Appendix A specifies preprocessing, pair-level PrefEval splitting, answer extraction, evaluators, and checkpoint selection; Appendices C–D provide prompts, optimization settings, and baseline adaptations. Validation-label use is explicitly distinguished from reward construction (p. 14). The evolving/static comparison in Table 3 (p. 8) also provides useful evidence that updating tests matters, particularly for free-form answers.

Critical weaknesses
  1. W1: The construction ablation needs clarification to establish an isolated pairwise benefit. Section 4.3 and Table 4 (p. 9) describe pointwise tests that reward an answer when a majority are satisfied, while also stating that remaining components are unchanged. RLCE instead selects one highest-scoring canonical answer and rewards matching rollouts (p. 5, Eqs. 3–5; Appendix C.2, p. 25). If the pointwise variant independently thresholds each answer, several answers or none can receive verification rewards, changing within-group reward patterns independently of test construction. The reported 3.00-point IntentQA and 9.26-point PrefEval differences would then compare two design packages rather than isolate pairwise construction. This is a substantive but potentially clarification-resolvable limitation of mechanism attribution, not a refutation of the full pipeline's performance. State the exact ablation reward rule. If selection also changes, provide a matched comparison or narrow the isolation claim.

  2. W2: The RLAC adaptation has an unresolved format-reward specification inconsistency. Appendix D.6 (p. 45, “Rewards”) gives R = 0.8 I[u] + 0.2 I[format], r_C = R, and r_G = 1 - R, where u denotes a validated unsupported claim. The prose says well-formed rollouts receive higher rewards. On the format-compliance interpretation indicated by that prose, however, the Generator reward is 1 - 0.8 I[u] - 0.2 I[format], so compliance lowers it by 0.2 at fixed u. The no-verdict fallback to the group mean (pp. 45–46) concerns a different branch and does not reconcile this expression; role-specific indicators could reconcile it, but are not defined. This is a minor-to-moderate specification issue with possible baseline implications, not an established implementation failure. If the format indicator is constant within a group, its contribution cancels under group normalization, so training damage cannot be inferred from the sign alone. Clarify the implemented role-specific rewards and whether mixed-format groups occur; reevaluation is needed only if an implementation discrepancy materially affects updates.

  3. W3: The relationship to close pairwise rubric approaches should be discussed more explicitly. Section 2 (p. 3) and the bibliography (pp. 10–12) omit Jia et al.'s Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric (first posted February 2026) and Rezaei et al.'s Online Rubrics Elicitation from Pairwise Comparisons (October 2025). OpenRS builds pair-conditioned criteria and aggregates criterion-level preferences for RL; OnlineRubrics elicits additional criteria from current/control-policy response pairs. These overlap with important ingredients of RLCE. However, OpenRS refines meta-rubrics using preference-label feedback, and OnlineRubrics augments an initial rubric set, whereas RLCE emphasizes sample-grounded evidence validation with a frozen initial policy. This is a moderate, repairable positioning limitation, not evidence that RLCE is redundant or that these methods are directly comparable under identical supervision. Subject to the applicable cutoff, explain these distinctions and identify the specific new contribution. A discussion is necessary; new experiments are not automatically required merely because these references are absent.

  4. W4: The variance diagnostic needs a precise definition and a directional interpretation. Section 4.2 and Figure 2 (pp. 7–8) relate the between-correct/incorrect share of reward variation to learning-signal quality, but the exact computation is unspecified. A conventional between-class variance fraction is unchanged when rewards are negated: binary rewards r = y and r = 1 - y can both put all variation between correctness classes while favoring opposite updates. This is a minor, repairable limitation of this diagnostic alone. Define the statistic and report a signed correctness–reward relationship, or explain whether direction is already incorporated. The separate conditional error-reversal analyses do support their stated claim that some erroneous preferences can be overturned; this concern does not apply to those results.

Questions for authors
  1. W1: What exact reward rule is used for the pointwise ablation? Does it retain RLCE's single-winner selection, or independently threshold each answer's tests?
  2. W2: Please provide the implemented Generator and Critic reward expressions, including the meanings of their format indicators and the no-verdict branch. Do existing logs indicate mixed-format groups during training?
  3. W3: How do input-evidence validation and supervision requirements distinguish RLCE from OpenRS and OnlineRubrics?
  4. W4: What is the exact Figure 2 variance statistic, and does its definition incorporate the direction of the reward–correctness relationship?
Reviewer_YypVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

RLCE learns social understanding from unlabeled data by constructing pairwise evidence tests over rollout answers and aggregating them into GRPO rewards, with tests regenerated on-policy. The paper reports the best mean performance among seven label-free methods on four benchmarks, with gains over the strongest baseline of +1.15/+3.58/+4.50/+18.93.

Critical strengths

S1. Starting from the same Qwen3.5-9B base, RLCE has the best mean on all four benchmarks. On MOSEI, UR-FUNNY and PrefEval its margins over the strongest baseline (+3.58/+4.50/+18.93) are reported as "each exceeding the seed standard deviation of either method". S2. Under test-time training RLCE (G=4) improves over the base model on all four benchmarks (+3.09/+5.79/+7.22/+28.98), while TTRL regresses on two of them with G=4 and on three with G=16.

Critical weaknesses

W1. All results in the main text use one base model, Qwen3.5-9B, whose frozen copy serves as both test constructor and validator, even though the paper states that "RLCE is validator-agnostic". One run with another base model, or with a validator other than the policy's own frozen copy, would show whether the method transfers beyond this single configuration. W2. The paper motivates RLCE as a way to reduce reliance on labels, but neither Table 1 nor the main text includes GRPO trained with ground-truth rewards on the same data. Without this reference, readers cannot tell how much of the supervised gain RLCE recovers, which is needed to put the +1.15 to +18.93 improvements in context.

Questions for authors

See Critical Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.