- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Reinforcement Learning with Comparative Evidence (RLCE), a critic-free reinforcement learning framework for learning social understanding from unlabeled data. RLCE compares candidate predictions using pairwise evidence tests, validates whether the corresponding evidence is present in the input, and aggregates the results to determine the reward. Experiments on four benchmarks spanning affect, pragmatics, communicative intent, and preference show consistent improvements over several unlabeled-reward baselines. The paper also provides analyses and ablations supporting the roles of pairwise evidence construction, compositional aggregation, and on-policy test evolution.
Critical strengths
- The core verification mechanism is well motivated. ****RLCE addresses the ambiguity inherent in social understanding by directly comparing competing interpretations through observable evidence, providing a more targeted verification signal than independently evaluating each answer against generic criteria.
- The empirical evaluation provides meaningful support for the proposed design. RLCE shows consistent improvements across four diverse social-understanding benchmarks and relevant unlabeled-reward baselines. The paper further provides controlled ablations and analyses, as well as an extension to test-time training that demonstrates the applicability of its label-free verification signal.
Critical weaknesses
- The computational cost and scalability of comparative verification are insufficiently characterized. RLCE constructs evidence tests for pairs of distinct rollout answers and then validates multiple tests for each comparison. This appears potentially more expensive than pointwise rubric or direct-judge rewards as the number of distinct candidates increases. This is particularly important in reinforcement learning, where repeated reward construction can make verification cost a significant factor in the overall efficiency and scalability of training.
- The aggregation does not explicitly account for redundancy among evidence tests. The paper shows that composing more evidence tests improves aggregate verdict accuracy, suggesting that additional tests can strengthen verification. However, semantically overlapping tests may repeatedly capture the same underlying cue, causing that evidence to be effectively counted multiple times in the aggregation. The current system design and analysis do not appear to explicitly account for semantic redundancy or correlation among evidence tests.
- The comparison with closely related pairwise verification methods is incomplete. For example, prior work has explored pairwise self-evaluation for selecting among candidate answers (e.g., REPS [1]) and contrastive rubric generation that derives discriminative evaluation criteria from response pairs (e.g., OpenRubrics [2]). Including these approaches would help better establish the novelty boundary of the proposed comparative verification framework.
[1] Akira Kawabata and Saku Sugawara. 2024. Rationale-Aware Answer Verification by Pairwise Self-Evaluation. EMNLP2024
[2] Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, and Haoyu Wang. 2026. OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment. ACL 2026
Questions for authors
- Could the authors provide the end-to-end computational cost of RLCE relative to the baselines, including the pairwise comparisons, generated evidence tests, constructor and validator calls per example, as well as wall-clock time or throughput? Understanding this cost-performance trade-off is important for assessing the practical significance and scalability of the method.
- Does RLCE explicitly control for semantic redundancy among evidence tests? Could the authors quantify test redundancy or evaluate a diversity-aware/deduplicated aggregation variant?
- What are the key methodological differences between RLCE and prior pairwise verification or contrastive rubric-generation approaches? In particular, what does RLCE introduce beyond these existing formulations, and why are these differences particularly effective for social understanding?