- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OCPO trains social agents using the likelihood of recorded partner continuations as local supervision. Following behavioral cloning, it combines sensitivity-weighted likelihood reward (SWLR), mean-only advantage centering, and GSPO.
Experiments on SOTOPIA and SOTOPIA-Hard with two backbones report improved overall scores in most settings. It also lowers training costs than comparison methods for social-policy optimization.
Critical strengths
- Practical training efficiency. Reusing recorded continuations avoids additional dialogue rollouts, external evaluator calls, and separate reward-model training during policy optimization. This offers a useful way to reduce adaptation costs when suitable expert trajectories are available.
- Useful component analysis. The ablations show improvements from sensitivity weighting over uniform token weighting. The intervention study also provides some evidence that SWLR preferences align with better final social outcomes.
Critical weaknesses
- Limited technical contribution and insufficient validation of the central reward assumption. Follow-up likelihood rewards, mean-only centering, and GSPO build on existing methods. Sensitivity weighting is a reasonable refinement, but my main concern is whether compatibility with one recorded next response is a reliable proxy for downstream social success. A good alternative action may induce a different, equally valid continuation and therefore receive a low reward. The authors acknowledge this limitation, and the intervention study partially addresses it, but the highest-versus-lowest SWLR comparison does not sufficiently establish when this proxy is reliable or where it fails.
- The empirical evidence does not firmly establish the claimed advantage. Several improvements in Table 1 are small, with no uncertainty estimates or significance tests. Most baseline results are taken from prior papers, leaving comparability unclear. The absence of stronger contemporary closed-source/open-source models further limits the broader performance claims. Clearer evidence of robustness would materially strengthen the paper.
Questions for authors
- Beyond selecting the highest- versus lowest-SWLR candidates, any additional evidence shows that SWLR can reliably rank plausible alternative actions by their final social outcomes? Particularly, a good alternative action may naturally prompt a different reply from the partner, but it can be scored poorly against the recorded response.
- Can you provide uncertainty estimates or paired significance tests for the main improvements? Are the gains consistent across independent training seeds, rather than only evaluation repeats?
5 comments
Response to Q2: Training Stability and Consistent Performance Gains
Q2. Independent training seeds, uncertainty, and comparability of gains
We thank the reviewer for emphasizing independent training repetitions. In the original manuscript, Table 1 reports averages over four evaluation runs (evaluation seeds 0, 1, 2, 3; Appendix E.2). To additionally measure optimization stability, we have now run Stage-2 OCPO with two new training seeds for each backbone and combined these runs with the original training seed. The three Stage-2 runs for each backbone use its corresponding BC initialization and are assessed in Self-Play on SOTOPIA and SOTOPIA-Hard.
Table R1. Stability across three independent Stage-2 OCPO training seeds (Self-Play Overall).
| Backbone | Training seed | SOTOPIA | SOTOPIA-Hard |
|---|---|---|---|
| Qwen2.5-7B | 20260810 (original) | 3.87 | 3.57 |
| 20261108 | 3.88 | 3.60 | |
| 20261118 | 3.88 | 3.55 | |
| Mean ± SD | 3.877 ± 0.006 | 3.573 ± 0.025 | |
| Llama3.1-8B | 20260809 (original) | 3.88 | 3.70 |
| 20261108 | 3.90 | 3.66 | |
| 20261118 | 3.89 | 3.73 | |
| Mean ± SD | 3.890 ± 0.010 | 3.697 ± 0.035 |
SD is the sample standard deviation across independent Stage-2 training seeds (n = 3). The newly trained models closely reproduce the original results for both backbones on both benchmarks. The cross-seed SD is only 0.006/0.025 for Qwen and 0.010/0.035 for Llama on SOTOPIA/SOTOPIA-Hard. These experiments directly address variability across independently optimized policies, beyond the evaluation repetitions already specified in the manuscript.
The gains are also consistent against a locally evaluated RL baseline. Appendix F.1 explicitly distinguishes externally cited baselines from results obtained using our own pipeline. In particular, we run ALSO locally for both Qwen2.5-7B and Llama3.1-8B. Table 1 reports the following comparisons under Self-Play and GPT-4o-as-Partner:
Table R2. OCPO versus locally run ALSO (Overall).
| Backbone / method | Self-Play All | Self-Play Hard | GPT-4o Partner All | GPT-4o Partner Hard |
|---|---|---|---|---|
| Qwen + ALSO | 3.75 | 3.20 | 3.80 | 3.14 |
| Qwen + OCPO | 3.87 | 3.57 | 3.98 | 3.69 |
| Gain | +0.12 | +0.37 | +0.18 | +0.55 |
| Llama + ALSO | 3.68 | 3.16 | 3.73 | 3.13 |
| Llama + OCPO | 3.88 | 3.70 | 3.92 | 3.76 |
| Gain | +0.20 | +0.54 | +0.19 | +0.63 |
OCPO improves all eight backbone-by-benchmark-by-interaction comparisons. The gains are particularly pronounced on SOTOPIA-Hard: +0.37/+0.54 in Self-Play and +0.55/+0.63 with GPT-4o-as-Partner (Qwen/Llama). This consistent pattern across model families and interaction protocols, together with the new training-seed SDs, provides substantially stronger evidence for reproducible benefits than an isolated endpoint improvement.
Technical Contribution, Frontier-Model Context, and Summary
Technical contribution and training efficiency
We appreciate the recognition of OCPO's efficiency and wish to clarify the integrated technical contribution. OCPO converts recorded partner continuations into candidate-action supervision without requiring a separate reward model or fresh full-dialogue rollouts during optimization. The action-conditioned versus ablated continuation likelihood provides a local learning signal; token sensitivity identifies the most action-responsive parts of that continuation; mean-only centering preserves meaningful differences in preference strength across candidate groups; and GSPO optimizes complete structured utterances at sequence level while preserving the coupling between social intent and linguistic realization. The controlled ablations above substantiate the complementary contribution of these choices.
For Qwen2.5-7B, the complete BC + OCPO pipeline requires 56.33 A100 GPU-hours, compared with 213 for SOTOPIA-RL and 160 for SAVOIR, representing reported reductions of 73.6% and 64.8%, respectively. OCPO also avoids the additional LLM calls used to produce reward-model training data and the cost of training a separate reward model. This favorable combination of strong social performance, reuse of existing trajectories, and substantially lower adaptation cost is a central practical contribution.
Coverage of contemporary open- and closed-source frontier models
To address the request for newer model references, we additionally surveyed publicly reported social-interaction results. Kang et al. (2026), What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds, report seven SOTOPIA-style social dimensions for recent frontier models in HICO-Bench (Appendix D, Table 10). We calculate the arithmetic mean of the seven published scores (Goal, Knowledge, Believability, Relationship, Secret, Social Rules, and Material Benefits):
Table R3. Recent frontier models: derived seven-dimension means.
| Model | Seven-dimension mean (derived) |
|---|---|
| Qwen3.8-Max | 4.40 |
| DeepSeek-V4-Pro | 4.33 |
| GPT-6 Astra | 4.30 |
For clarity, this source uses the HICO-Bench protocol: 180 paired-world coordination scenarios overall, with a test set of 60 scenarios × 2 worlds × 2 roles = 240 tasks, a fixed Qwen3-8B partner, and a Qwen3.8-Max judge. The seven-dimension means above are our calculations from the published component scores; the source does not label them as an Overall column. HICO-Bench uses different scenarios, partners, and evaluation settings from our original SOTOPIA/SOTOPIA-Hard experiments with a GPT-4o evaluator, so these frontier-model results are presented separately as contemporary context, with no cross-protocol ranking asserted.
This additional evidence situates OCPO alongside recent frontier-model developments while preserving the scope of our main claim: efficient, effective policy adaptation for 7B/8B social agents using existing expert trajectories.
Summary
Validity of the reward signal: Our existing free-running early/middle/late interventions favor high-SWLR branches in terminal Overall, Goal, and Relationship. Matched semantic-supervision comparisons and component ablations independently establish the contribution of SWLR and its design choices.
Robustness: New independent Stage-2 training runs give small cross-seed SDs for both backbones on both benchmarks, extending the original four-run evaluation averages with direct optimization-stability evidence.
Strength of the gains: Against locally run ALSO, OCPO improves all eight Overall results, with especially substantial advantages on SOTOPIA-Hard.
Practical contribution: OCPO combines action-sensitive continuation supervision with stable sequence-level policy optimization, while reducing compute and eliminating separate reward-model training.
We believe these complementary results materially strengthen the evidence for OCPO's technical soundness, empirical reliability, and practical value. We respectfully ask the reviewer to reconsider the assessment and overall recommendation in light of the added evidence.
Reference: Kang et al. (2026). What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds. arXiv:2610.20195. https://arxiv.org/abs/2610.20195
Response to Q1: SWLR Validity and Downstream Outcome Evidence
Dear Reviewer yo4T,
Thank you for recognizing OCPO's practical training efficiency and the value of our ablation and intervention analyses. Your questions concern the validity of SWLR supervision, the distinct contribution of our design, the robustness of the reported gains, and coverage of contemporary frontier models. We address each point below, including new results from three independent Stage-2 training seeds per backbone and additional published frontier-model evidence.
Q1. Validity of SWLR and the reliability of action preferences
We appreciate the concern that a plausible alternative action may elicit a different, equally appropriate partner continuation. OCPO explicitly accommodates the diversity of open-ended interactions by combining conditional likelihood gain, token-sensitivity weighting, and within-group policy optimization. The value of this signal is further tested through actual dialogue continuations and controlled learning comparisons.
(1) SWLR isolates action-sensitive information in an observed continuation. For a fixed dialogue state and recorded partner continuation, SWLR compares the continuation's token likelihoods with a candidate action present versus with that action ablated. This isolates changes associated with the candidate beyond the context-only likelihood. Sensitivity weighting then emphasizes continuation tokens whose likelihoods respond most strongly to the action. For each training state, OCPO samples eight candidate actions and constructs relative advantages for sequence-level GSPO updates, with auxiliary SFT and KL regularization.
A single continuation may contain stochastic variation; the learned policy aggregates supervision across many dialogue states and alternatives. Our Stage-2 training uses 12,800 complete eight-action groups (102,400 accepted sampled actions). The relevant empirical test is whether this local preference signal consistently improves subsequent social interactions.
(2) Our existing intervention study directly connects SWLR preferences to realized terminal outcomes. Figure 4 and Appendix G.2 sample 200 starting states in each of the early, middle, and late dialogue stages. From each state, a high-SWLR branch and a low-SWLR branch undergo three consecutive action interventions; both branches then proceed naturally to termination. Crucially, the original partner responds naturally after each executed action. The ensuing conversation can therefore diverge freely from the recorded continuation used for scoring.
Win rates follow the paper's Figure 4 / Appendix G.2 definition, with ties contributing half to each branch. The high-SWLR branch is favored in Overall, Goal, and Relationship at all three stages. The strongest advantage occurs with early interventions, consistent with beneficial local decisions propagating through longer-term dialogue dynamics. Importantly, these outcomes are evaluated on naturally evolving interactions rather than on whether the partner reproduces the logged response.
(3) Controlled comparisons demonstrate the contribution of the SWLR signal and its components. Appendix G.1 compares SWLR against semantic similarity with matched backbones, BC initialization, sampling budget, and optimization settings. SWLR yields higher terminal Overall on both SOTOPIA benchmarks for both Qwen2.5-7B and Llama3.1-8B. Table 3 further isolates the contributions of the main components under Qwen2.5-7B and GPT-4o-as-Partner:
Removing token weighting decreases Overall by 0.31/0.21, and removing mean-only centering decreases it by 0.35/0.28, on SOTOPIA/SOTOPIA-Hard respectively. These sizable, controlled differences show that the sensitivity-aware reward and its optimization choices make material contributions to final performance.
Taken together, the free-running intervention, matched semantic baseline, and component ablations support our central claim: SWLR supplies actionable local preferences that improve downstream social outcomes when learned over diverse states. Its usefulness does not rely on perfectly predicting the counterfactual terminal return of every possible alternative action.