- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents SocialRL, a reinforcement learning framework for improving LLMs’ ability to pursue social goals while maintaining relationships during multi-turn interactions. The framework combines multi-turn PPO with a six-dimensional process reward. An LLM generates context-specific binary evaluation criteria for each dimension, judges whether the current response satisfies them, and assigns context-dependent weights. Process rewards are combined with a terminal outcome reward. The authors evaluate four policy backbones against multiple opponents on SOTOPIA-π, SOTOPIA-All, SOTOPIA-Hard, and AgentSense, reporting improvements over base models and selected training baselines.
Critical strengths
-
Substantial experimental effort and broad evaluation coverage. The study evaluates multiple policy backbones and opponent models across several benchmark settings, including AgentSense. The reported goal-achievement improvements suggest potential practical value and provide evidence that the overall training recipe can benefit more than a single model or interaction setting.
-
Attention to both task success and relationship quality. Evaluating both objectives makes the study more informative than evaluating goal completion alone. The human-alignment audit also provides useful evidence about the automatic evaluators, although it does not resolve the reward-stability concerns discussed below.
Critical weaknesses
- The experiments do not sufficiently establish the contribution of the proposed domain-specific designs beyond standard multi-turn PPO with dense feedback. The optimization procedure largely follows standard PPO and GAE. I acknowledge the comparisons with existing methods and the ablations on process rewards, reward dimensions, and dynamic weights. In particular, the NoProcess ablation supports the usefulness of process supervision in the tested setting. However, removing all process rewards does not establish that the proposed rubric construction is superior to an existing or simpler process reward.
A particularly informative missing comparison is standard multi-turn PPO using an existing social reward design, such as that of Sotopia-RL, under matched reward-model and training budgets. Comparisons that change both the optimization procedure and the reward design cannot isolate their respective contributions. There is also no direct ablation of dynamically generated criteria against fixed criteria under the same PPO setup. Although the dynamic-weight ablation provides some positive evidence, it is restricted to one policy–opponent setting and lacks uncertainty estimates. Consequently, the evidence does not convincingly establish the claimed methodological contribution beyond applying standard multi-turn PPO with dense social feedback.
- The stability and comparability of the dynamically generated rubrics are insufficiently validated. According to Figure 4, the evaluated response is provided to the same LLM call that generates the criteria, assigns pass/fail judgments, and determines dimension weights. Different responses to the same dialogue state may therefore be assessed using different criteria and weights. This raises a concern that the criteria could adapt to the response being evaluated, producing post-hoc justification or scores that are not comparable across candidate responses.
Binary judgments, weight caps, normalization, and fallback rules constrain the output format and numerical range, but do not establish consistency in what is being evaluated. The reported PRM–human correlation of 0.816 is useful evidence of alignment across sampled records; it does not directly measure repeated-scoring reliability or ranking consistency among alternative responses to the same context. Without these checks, the reliability of this central training signal remains unclear.
Questions for authors
-
Under the same multi-turn PPO implementation, reward-model capability, and rollout budget, how does the proposed reward design compare with an existing social reward design or a simpler fixed-rubric process reward? Please clarify which baseline settings and budgets were matched and provide uncertainty estimates for the key comparisons.
-
Why are the evaluation criteria and dimension weights conditioned on the response being evaluated? How do you ensure comparable evaluation of alternative responses to the same dialogue state? A comparison with criteria generated solely from the scenario, goal, and dialogue history—and then shared across candidate responses—would directly address this concern.
-
How reproducible are the generated rubrics and resulting scores for identical inputs? Do repeated evaluations preserve candidate-response rankings, and how well do those rankings agree with human judgments within the same dialogue context?
2 comments
rebuttal2
W2 & Q2 & Q3: Reliability, Comparability, and Reproducibility of Dynamic Rubrics
Reviewer concern: The original mechanism generates rubrics and weights based on the current response, so different candidate responses may be evaluated under different criteria, creating a risk of post-hoc rationalization; the PRM–Human correlation coefficient cannot directly prove repeated scoring stability, nor can it prove the reliability of ranking candidate responses within the same context.
The original design incorporates the current response into rubric generation to make the evaluation criteria provide fine-grained feedback for specific behaviors. We agree that if different candidate responses use different evaluation criteria, additional validation of ranking comparability and repeated scoring reliability is needed. But please note: the original design goal is not for the evaluation model to find favorable reasons for the current response, but to generate observable and verifiable binary judgment criteria based on the current dialogue context and specific behavior. Directly presuming "response-conditioned evaluation" to be "post-hoc rationalization" lacks evidence. To address this issue, we add a Response-independent Frozen Rubric control experiment: given the current dialogue state, first generate evaluation criteria and dimension weights only based on the scenario, role information, social goal, and dialogue history, fix the rubric and weights before candidate responses are generated, and then use exactly the same criteria to perform pass/fail evaluation on different candidate responses under the same dialogue state. We compare three mechanisms:
| Rubric Mechanism | Ranking Consistency | Human Alignment (200 samples) |
|---|---|---|
| Response-conditioned | 0.947 | 0.823 |
| Context-only Frozen | 0.96 | 0.819 |
| Fixed Generic | 0.942 | 0.864 |
The focus is on whether scores for different candidate responses have a stable relative ranking, whether the ranking is consistent with human judgment, and whether frozen criteria improve or weaken final training performance. This set of experiments can distinguish two possibilities: response-conditioned rubrics indeed provide finer-grained behavioral feedback; or the mechanism may introduce unnecessary scoring differences but the impact is not significant. The results show that ranking consistency and human alignment under all three mechanisms are at a high level, and there is no support for the strong accusation that "post-hoc rationalization makes evaluation incomparable."
rebuttal1
W1 & Q1: Attribution Between Multi-turn Optimization and Reward Design
SocialRL's methodological contribution is not proposing a completely new PPO or GAE algorithm, but constructing dynamic multi-dimensional process rewards usable for long-term policy optimization in multi-turn social interaction. PPO and GAE themselves are existing algorithms, and this paper does not present their mathematical update rules as a new general reinforcement learning algorithm contribution. To describe "not reinventing PPO" as "insufficient algorithmic contribution" is a misplaced evaluation of the paper's contribution. Multi-turn PPO is responsible for using complete trajectories and future returns for cross-turn credit assignment; Dynamic Process Rewards are responsible for providing intermediate supervision reflecting goal progress, relationship state, and interaction validity. The original manuscript has already provided component-level evidence: Appendix E.1 compares PPO and GRPO, Appendix E.2 shows that removing process rewards reduces Goal Achievement from 63.3% to 53.2%, and Appendix E.4 shows that Dynamic Weighting outperforms Uniform, Manual, and Grid. These results respectively verify the roles of the optimization algorithm, process supervision, and dynamic weighting. We agree that direct comparison with alternative process rewards is still needed, so we add:
We compare Single-turn PPO and Multi-turn PPO under a fixed SocialRL Reward, unifying policy initialization, training data, Rollout Budget, reward model capability, and evaluation protocol, and separately report the mean, standard deviation, and confidence intervals across training seeds. Please do not expand "need to add seed experiments" into "all existing conclusions are invalid."