acceptodds
Under review as a conference paper at ICLR 2027

OCPO: Observed Continuation Policy Optimization for Social Agents

Abstract

Social intelligence simulation challenges language agents to navigate open-ended, multi-turn interactions while balancing social goals and interpersonal relationships. However, training signals are typically available only at the dialogue level, making credit assignment to individual utterances difficult and resulting in sparse, delayed supervision. Existing methods often obtain finer-grained feedback through additional dialogue rollouts, external evaluation, or separately trained reward models, making supervision costly to acquire. To address this, we propose Observed Continuation Policy Optimization (OCPO), a framework that turns immediate partner responses recorded in high-quality dialogue trajectories into local policy supervision. OCPO scores policy-sampled utterances by their compatibility with the same observed partner continuation. It uses sensitivity-based token weighting to emphasize tokens sensitive to candidate utterances and mean-only advantage centering to preserve reward-difference magnitudes in sequence-level updates. By reusing recorded continuations, OCPO acquires rewards without additional full-dialogue rollouts, a separately trained reward model, or external evaluator calls during policy optimization. Experiments on the benchmark social intelligence environment demonstrate that OCPO consistently outperforms large reasoning models and existing optimization methods. Notably, OCPO cuts GPU-hours by 65% compared with previous social-policy optimization pipelines, offering an effective and cost-efficient paradigm for building social agents.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
social intelligence simulation, language agents, social policy optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yo4TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OCPO trains social agents using the likelihood of recorded partner continuations as local supervision. Following behavioral cloning, it combines sensitivity-weighted likelihood reward (SWLR), mean-only advantage centering, and GSPO.

Experiments on SOTOPIA and SOTOPIA-Hard with two backbones report improved overall scores in most settings. It also lowers training costs than comparison methods for social-policy optimization.

Critical strengths
  • Practical training efficiency. Reusing recorded continuations avoids additional dialogue rollouts, external evaluator calls, and separate reward-model training during policy optimization. This offers a useful way to reduce adaptation costs when suitable expert trajectories are available.
  • Useful component analysis. The ablations show improvements from sensitivity weighting over uniform token weighting. The intervention study also provides some evidence that SWLR preferences align with better final social outcomes.
Critical weaknesses
  • Limited technical contribution and insufficient validation of the central reward assumption. Follow-up likelihood rewards, mean-only centering, and GSPO build on existing methods. Sensitivity weighting is a reasonable refinement, but my main concern is whether compatibility with one recorded next response is a reliable proxy for downstream social success. A good alternative action may induce a different, equally valid continuation and therefore receive a low reward. The authors acknowledge this limitation, and the intervention study partially addresses it, but the highest-versus-lowest SWLR comparison does not sufficiently establish when this proxy is reliable or where it fails.
  • The empirical evidence does not firmly establish the claimed advantage. Several improvements in Table 1 are small, with no uncertainty estimates or significance tests. Most baseline results are taken from prior papers, leaving comparability unclear. The absence of stronger contemporary closed-source/open-source models further limits the broader performance claims. Clearer evidence of robustness would materially strengthen the paper.
Questions for authors
  • Beyond selecting the highest- versus lowest-SWLR candidates, any additional evidence shows that SWLR can reliably rank plausible alternative actions by their final social outcomes? Particularly, a good alternative action may naturally prompt a different reply from the partner, but it can be scored poorly against the recorded response.
  • Can you provide uncertainty estimates or paired significance tests for the main improvements? Are the gains consistent across independent training seeds, rather than only evaluation repeats?
5 comments
AuthorsOpenReview ↗

Response to Q1: SWLR Validity and Downstream Outcome Evidence

Dear Reviewer yo4T,

Thank you for recognizing OCPO's practical training efficiency and the value of our ablation and intervention analyses. Your questions concern the validity of SWLR supervision, the distinct contribution of our design, the robustness of the reported gains, and coverage of contemporary frontier models. We address each point below, including new results from three independent Stage-2 training seeds per backbone and additional published frontier-model evidence.

Q1. Validity of SWLR and the reliability of action preferences

We appreciate the concern that a plausible alternative action may elicit a different, equally appropriate partner continuation. OCPO explicitly accommodates the diversity of open-ended interactions by combining conditional likelihood gain, token-sensitivity weighting, and within-group policy optimization. The value of this signal is further tested through actual dialogue continuations and controlled learning comparisons.

(1) SWLR isolates action-sensitive information in an observed continuation. For a fixed dialogue state and recorded partner continuation, SWLR compares the continuation's token likelihoods with a candidate action present versus with that action ablated. This isolates changes associated with the candidate beyond the context-only likelihood. Sensitivity weighting then emphasizes continuation tokens whose likelihoods respond most strongly to the action. For each training state, OCPO samples eight candidate actions and constructs relative advantages for sequence-level GSPO updates, with auxiliary SFT and KL regularization.

A single continuation may contain stochastic variation; the learned policy aggregates supervision across many dialogue states and alternatives. Our Stage-2 training uses 12,800 complete eight-action groups (102,400 accepted sampled actions). The relevant empirical test is whether this local preference signal consistently improves subsequent social interactions.

(2) Our existing intervention study directly connects SWLR preferences to realized terminal outcomes. Figure 4 and Appendix G.2 sample 200 starting states in each of the early, middle, and late dialogue stages. From each state, a high-SWLR branch and a low-SWLR branch undergo three consecutive action interventions; both branches then proceed naturally to termination. Crucially, the original partner responds naturally after each executed action. The ensuing conversation can therefore diverge freely from the recorded continuation used for scoring.

Intervention stageHigh-SWLR Overall win rateGoal win rateRelationship win rate
Early66%71%63%
Middle61%64%58%
Late55%57%53%

Win rates follow the paper's Figure 4 / Appendix G.2 definition, with ties contributing half to each branch. The high-SWLR branch is favored in Overall, Goal, and Relationship at all three stages. The strongest advantage occurs with early interventions, consistent with beneficial local decisions propagating through longer-term dialogue dynamics. Importantly, these outcomes are evaluated on naturally evolving interactions rather than on whether the partner reproduces the logged response.

(3) Controlled comparisons demonstrate the contribution of the SWLR signal and its components. Appendix G.1 compares SWLR against semantic similarity with matched backbones, BC initialization, sampling budget, and optimization settings. SWLR yields higher terminal Overall on both SOTOPIA benchmarks for both Qwen2.5-7B and Llama3.1-8B. Table 3 further isolates the contributions of the main components under Qwen2.5-7B and GPT-4o-as-Partner:

VariantSOTOPIA OverallSOTOPIA-Hard Overall
Full OCPO3.983.69
Without token weighting3.673.48
Without mean-only advantage3.633.41
GSPO replaced by GRPO3.803.51

Removing token weighting decreases Overall by 0.31/0.21, and removing mean-only centering decreases it by 0.35/0.28, on SOTOPIA/SOTOPIA-Hard respectively. These sizable, controlled differences show that the sensitivity-aware reward and its optimization choices make material contributions to final performance.

Taken together, the free-running intervention, matched semantic baseline, and component ablations support our central claim: SWLR supplies actionable local preferences that improve downstream social outcomes when learned over diverse states. Its usefulness does not rely on perfectly predicting the counterfactual terminal return of every possible alternative action.

AuthorsOpenReview ↗

Response to Q2: Training Stability and Consistent Performance Gains

Q2. Independent training seeds, uncertainty, and comparability of gains

We thank the reviewer for emphasizing independent training repetitions. In the original manuscript, Table 1 reports averages over four evaluation runs (evaluation seeds 0, 1, 2, 3; Appendix E.2). To additionally measure optimization stability, we have now run Stage-2 OCPO with two new training seeds for each backbone and combined these runs with the original training seed. The three Stage-2 runs for each backbone use its corresponding BC initialization and are assessed in Self-Play on SOTOPIA and SOTOPIA-Hard.

Table R1. Stability across three independent Stage-2 OCPO training seeds (Self-Play Overall).

BackboneTraining seedSOTOPIASOTOPIA-Hard
Qwen2.5-7B20260810 (original)3.873.57
202611083.883.60
202611183.883.55
Mean ± SD3.877 ± 0.0063.573 ± 0.025
Llama3.1-8B20260809 (original)3.883.70
202611083.903.66
202611183.893.73
Mean ± SD3.890 ± 0.0103.697 ± 0.035

SD is the sample standard deviation across independent Stage-2 training seeds (n = 3). The newly trained models closely reproduce the original results for both backbones on both benchmarks. The cross-seed SD is only 0.006/0.025 for Qwen and 0.010/0.035 for Llama on SOTOPIA/SOTOPIA-Hard. These experiments directly address variability across independently optimized policies, beyond the evaluation repetitions already specified in the manuscript.

The gains are also consistent against a locally evaluated RL baseline. Appendix F.1 explicitly distinguishes externally cited baselines from results obtained using our own pipeline. In particular, we run ALSO locally for both Qwen2.5-7B and Llama3.1-8B. Table 1 reports the following comparisons under Self-Play and GPT-4o-as-Partner:

Table R2. OCPO versus locally run ALSO (Overall).

Backbone / methodSelf-Play AllSelf-Play HardGPT-4o Partner AllGPT-4o Partner Hard
Qwen + ALSO3.753.203.803.14
Qwen + OCPO3.873.573.983.69
Gain+0.12+0.37+0.18+0.55
Llama + ALSO3.683.163.733.13
Llama + OCPO3.883.703.923.76
Gain+0.20+0.54+0.19+0.63

OCPO improves all eight backbone-by-benchmark-by-interaction comparisons. The gains are particularly pronounced on SOTOPIA-Hard: +0.37/+0.54 in Self-Play and +0.55/+0.63 with GPT-4o-as-Partner (Qwen/Llama). This consistent pattern across model families and interaction protocols, together with the new training-seed SDs, provides substantially stronger evidence for reproducible benefits than an isolated endpoint improvement.

AuthorsOpenReview ↗

Technical Contribution, Frontier-Model Context, and Summary

Technical contribution and training efficiency

We appreciate the recognition of OCPO's efficiency and wish to clarify the integrated technical contribution. OCPO converts recorded partner continuations into candidate-action supervision without requiring a separate reward model or fresh full-dialogue rollouts during optimization. The action-conditioned versus ablated continuation likelihood provides a local learning signal; token sensitivity identifies the most action-responsive parts of that continuation; mean-only centering preserves meaningful differences in preference strength across candidate groups; and GSPO optimizes complete structured utterances at sequence level while preserving the coupling between social intent and linguistic realization. The controlled ablations above substantiate the complementary contribution of these choices.

For Qwen2.5-7B, the complete BC + OCPO pipeline requires 56.33 A100 GPU-hours, compared with 213 for SOTOPIA-RL and 160 for SAVOIR, representing reported reductions of 73.6% and 64.8%, respectively. OCPO also avoids the additional LLM calls used to produce reward-model training data and the cost of training a separate reward model. This favorable combination of strong social performance, reuse of existing trajectories, and substantially lower adaptation cost is a central practical contribution.

Coverage of contemporary open- and closed-source frontier models

To address the request for newer model references, we additionally surveyed publicly reported social-interaction results. Kang et al. (2026), What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds, report seven SOTOPIA-style social dimensions for recent frontier models in HICO-Bench (Appendix D, Table 10). We calculate the arithmetic mean of the seven published scores (Goal, Knowledge, Believability, Relationship, Secret, Social Rules, and Material Benefits):

Table R3. Recent frontier models: derived seven-dimension means.

ModelSeven-dimension mean (derived)
Qwen3.8-Max4.40
DeepSeek-V4-Pro4.33
GPT-6 Astra4.30

For clarity, this source uses the HICO-Bench protocol: 180 paired-world coordination scenarios overall, with a test set of 60 scenarios × 2 worlds × 2 roles = 240 tasks, a fixed Qwen3-8B partner, and a Qwen3.8-Max judge. The seven-dimension means above are our calculations from the published component scores; the source does not label them as an Overall column. HICO-Bench uses different scenarios, partners, and evaluation settings from our original SOTOPIA/SOTOPIA-Hard experiments with a GPT-4o evaluator, so these frontier-model results are presented separately as contemporary context, with no cross-protocol ranking asserted.

This additional evidence situates OCPO alongside recent frontier-model developments while preserving the scope of our main claim: efficient, effective policy adaptation for 7B/8B social agents using existing expert trajectories.

Summary

Validity of the reward signal: Our existing free-running early/middle/late interventions favor high-SWLR branches in terminal Overall, Goal, and Relationship. Matched semantic-supervision comparisons and component ablations independently establish the contribution of SWLR and its design choices.

Robustness: New independent Stage-2 training runs give small cross-seed SDs for both backbones on both benchmarks, extending the original four-run evaluation averages with direct optimization-stability evidence.

Strength of the gains: Against locally run ALSO, OCPO improves all eight Overall results, with especially substantial advantages on SOTOPIA-Hard.

Practical contribution: OCPO combines action-sensitive continuation supervision with stable sequence-level policy optimization, while reducing compute and eliminating separate reward-model training.

We believe these complementary results materially strengthen the evidence for OCPO's technical soundness, empirical reliability, and practical value. We respectfully ask the reviewer to reconsider the assessment and overall recommendation in light of the added evidence.

Reference: Kang et al. (2026). What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds. arXiv:2610.20195. https://arxiv.org/abs/2610.20195

Reviewer_WmmzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Observed Continuation Policy Optimization (OCPO), a framework for improving multi-turn social agents using observed partner responses from expert dialogue trajectories as local reward signals. The method introduces a sensitivity-weighted likelihood reward (SWLR) to evaluate candidate utterances, combined with mean-only advantage centering and GSPO for policy optimization. Experiments on SOTOPIA and SOTOPIA-Hard demonstrate competitive performance while substantially reducing training costs compared with existing methods.

Critical strengths

The paper addresses the important challenge of obtaining fine-grained supervision for multi-turn social agents without costly dialogue rollouts or reward models. OCPO achieves competitive performance while substantially reducing reported training costs.

Critical weaknesses

W1. Missing SFT-only baseline. The paper does not report performance after SFT alone, making it difficult to distinguish the improvements from SFT and subsequent RL optimization.

W2. Limited experimental coverage. Experiments are restricted to SOTOPIA and SOTOPIA-Hard. Evaluation on additional benchmarks or interaction settings would strengthen the generalizability claims.

Questions for authors

Q1. Dependence on expert trajectories. OCPO relies on high-quality expert trajectories with observed partner continuations. How well would it work with limited or lower-quality data, and could it generalize to broader RL settings?

Q2. Reward validity. A candidate action may lead to a different yet equally valid partner response. How well does SWLR reflect actual action quality? Could the authors examine its correlation with long-term returns?

7 comments
AuthorsOpenReview ↗

Response to Wmmz (1/3): SFT Baseline and Generalization

Thank you for recognizing OCPO's training-efficiency contribution and highlighting concerns regarding SFT attribution, evaluation coverage, expert-data dependence, and SWLR validity. We provide new SFT-only, AgentSense, and data-dependency experiments, together with clarification of our existing intervention and reward-design evidence.

W1. Missing SFT-only Baseline

We agree that the gains attributable to supervised fine-tuning (SFT) and subsequent reinforcement learning (RL) should be distinguished. We therefore evaluated the SFT-only checkpoints of both Qwen2.5-7B-Instruct and Llama3.1-8B-Instruct under all four original settings. These checkpoints correspond exactly to the behavior-cloning (BC) initialization before OCPO optimization.

Table R1. SFT-only vs. OCPO (Overall ↑)

BackboneMethodSelf SOTOPIASelf HardGPT-4o SOTOPIAGPT-4o Hard
Qwen2.5-7BSFT-only3.503.133.523.37
OCPO3.873.573.983.69
Δ+0.37+0.44+0.46+0.32
Llama3.1-8BSFT-only3.553.173.613.44
OCPO3.883.703.923.76
Δ+0.33+0.53+0.31+0.32

OCPO improves over SFT-only in all eight backbone/evaluation configurations. Qwen2.5-7B gains up to +0.44 in Self-Play and +0.46 with GPT-4o as partner; Llama3.1-8B gains +0.53 on SOTOPIA-Hard Self-Play. These comparisons establish the incremental benefit of RL beyond BC. We will add the SFT-only baseline to the revised manuscript.

W2. Limited Experimental Coverage

Thank you for suggesting evaluation beyond SOTOPIA. We added AgentSense (NAACL 2025), an independently constructed benchmark of multi-turn social interaction involving distinct identities, private information, and social goals. Its scenarios and evaluation criteria differ from those of SOTOPIA.

We follow the original AgentSense protocol: 245 scenarios, Self-Play, three repeated evaluations, and the same three judges (GPT-4o, Qwen2.5-72B, and Llama-3-70B). We evaluate Qwen2.5-7B and additionally train Llama-3-8B to match the backbone used for the published AgentSense baseline. Published Base results are retained; BC and OCPO are evaluated by us under the corresponding protocol.

Table R2. AgentSense Results (↑)

ModelMethodSelfOtherGPT-4oQwen JudgeLlama JudgeMajorityInfo Acc.
Qwen2.5-7BBase84.6961.3779.7978.5970.7477.8270.59
BC85.7163.9281.5480.2873.4679.8371.32
OCPO87.9468.7384.9784.1477.1183.4272.06
Llama-3-8BBase86.7165.3680.8481.0273.3379.4764.90
BC87.4867.6982.8783.0475.6281.4365.72
OCPO89.3672.4186.1886.0778.8684.7666.91

Base scores are taken from the original AgentSense paper. Our AgentSense experiment uses Llama-3-8B for backbone matching; the original SOTOPIA experiments use Llama3.1-8B.

For Majority, OCPO improves over BC from 79.83 to 83.42 (+3.59 percentage points) on Qwen2.5-7B, and from 81.43 to 84.76 (+3.33 pp) on Llama-3-8B. Both backbones improve across Self, Other, all three independent judges, Majority, and information accuracy. The consistent pattern across judges further supports the robustness of the observed gains.

We also examine selected social-goal categories to identify where OCPO's gains are especially pronounced.

Table R3. Selected Social Goal Types (↑)

Social GoalQwen BCQwen OCPOΔ (pp)Llama BCLlama OCPOΔ (pp)
Conflict Resolution57.6866.21+8.5356.8764.11+7.24
Relationship Building88.7391.64+2.9189.1292.06+2.94

Conflict Resolution gains of +8.53 pp and +7.24 pp highlight the method's effectiveness in interactions requiring the management of conflicting interests and strategic coordination. Consistent positive gains in Relationship Building further show improvements in relationship-oriented social behavior. These are selected categories; the complete AgentSense results in Table R2 form the basis of our overall generalization claim.

Crucially, the trained policies transfer to this independent interactive benchmark without additional AgentSense-specific policy optimization. The improvements across both backbones and multiple evaluation dimensions demonstrate that OCPO learns transferable social behaviors extending beyond the SOTOPIA evaluation distribution.

AuthorsOpenReview ↗

Response to Wmmz (2/3): Expert Data Dependency

Q1. Dependence on Expert Trajectories

Thank you for asking how OCPO behaves with limited data, a weaker data generator, and broader RL settings. We conducted two controlled ablations using Qwen2.5-7B-Instruct as the policy backbone.

(1) Sensitivity to expert-data quantity. We subsampled the Stage-2 expert dataset at the episode level to 25%, 50%, and 100% of its original size. All settings share the same original BC initialization, 800 RL updates, eight sampled actions per state, SWLR, mean-only advantage centering, and GSPO. Thus, this ablation isolates the amount of expert data available to the RL stage.

As Table R5 shows, with only 25% of the expert data, OCPO achieves Overall scores of 3.66/3.39/3.70/3.52 across Self-Play SOTOPIA, Self-Play Hard, GPT-4o-partner SOTOPIA, and GPT-4o-partner Hard. These improve over BC by +0.16/+0.26/+0.18/+0.15. With 50% of the data, the corresponding results rise to 3.76/3.48/3.82/3.59, improving over BC by +0.26/+0.35/+0.30/+0.22. OCPO therefore retains consistent RL gains even when the available expert data is reduced by 75%, with further gains as data availability increases.

(2) Robustness to a weaker data generator. We replaced the Qwen2.5-72B-Instruct generator used in SOTOPIA-Ω with Qwen2.5-32B-Instruct, while preserving the original Self-Play, dynamic strategy injection (DSI), scenario distribution, generation settings, and quality-filtering procedure. We matched the number of eligible OCPO training instances and kept the BC initialization and RL hyperparameters fixed.

First, we evaluated the two unmodified generator models on the same SOTOPIA Self-Play benchmarks:

Table R4. Social Ability of Data Generators (Overall ↑)

Base GeneratorSOTOPIASOTOPIA-Hard
Qwen2.5-72B-Instruct4.033.55
Qwen2.5-32B-Instruct3.703.31

The 72B scores are reported by SOTOPIA-Ω; the 32B scores are from our evaluation using the same protocol. The 32B generator scores 0.33/0.24 lower than 72B, establishing a clear difference in their baseline social capability.

Table R5. Robustness to Data Quantity and Generator Strength (Overall ↑)

Training DataSelf SOTOPIASelf HardGPT-4o SOTOPIAGPT-4o Hard
BC-only3.503.133.523.37
Expert 25%3.663.393.703.52
Expert 50%3.763.483.823.59
Expert 100% (72B + DSI)3.873.573.983.69
32B + DSI (100%)3.703.423.803.56

Using the equally sized 32B + DSI dataset, OCPO still improves over BC by +0.20/+0.29/+0.28/+0.19 across the four settings. Relative to the original 72B + DSI dataset, the resulting policies are lower by only 0.17/0.15/0.18/0.13. Thus, the observed-continuation signal remains useful even when the original expert generator is replaced by a substantially smaller, weaker social model. The generator-level scores in R4 characterize generator strength; we make no unsupported assumption that quality filtering preserves the same per-trajectory quality gap.

(3) Broader RL settings. OCPO requires logged action–continuation pairs and a model that can score the conditional likelihood of the observed feedback under candidate actions. This construction can be applied to sequential interactions with observable, modelable feedback. Its applicability depends on the informativeness of feedback and adequate data coverage. Within multi-turn social interaction, our two backbones, SOTOPIA/SOTOPIA-Hard, Self-Play/GPT-4o-partner settings, and new AgentSense results collectively demonstrate transfer across scenarios, partners, and evaluation protocols. We will clarify these applicability conditions and identify other RL domains as future extensions.

AuthorsOpenReview ↗

Response to Wmmz (3/3): Reward Validity and Contributions

Q2. Reward Validity

We appreciate the concern that an action may elicit a different yet equally valid response, as acknowledged in Appendix A. The key question is whether SWLR preferences improve realized social outcomes. Our temporal interventions, reward-design ablations, and new cross-benchmark results directly inform this question.

(1) Why fixed-continuation scoring provides informative supervision. SWLR scores sampled actions against the same observed partner continuation, fixing the state and reference target for controlled within-state comparisons. Sensitivity-based token weighting emphasizes response tokens whose likelihood changes most with the candidate action, concentrating supervision on action-relevant feedback. This reuses logged expert interactions without additional full-dialogue rollouts or a separate reward model.

A single continuation cannot cover every valid response. We therefore test the resulting preferences through complete, naturally evolving dialogues.

(2) Temporal interventions directly test realized long-term outcomes. Figure 4 and Appendix G.2 examine 200 paired states per early/middle/late stage. Matched high- and low-SWLR branches execute the highest- or lowest-scoring action among eight candidates at three consecutive decision points. After execution, the partner responds naturally; the scoring continuation is never inserted into the actual dialogue. Later interventions use a branch-local continuation probe as their scoring reference. Both branches then proceed to termination under the official SOTOPIA evaluation.

Original Figure 4. High-SWLR Branch Win Rates in Final Scores

Final MetricEarlyMiddleLate
Overall66%61%55%
Goal71%64%57%
Relationship63%58%53%

High-SWLR branches achieve superior final Overall, Goal, and Relationship outcomes more often at all stages. Early-stage win rates reach 66%, 71%, and 63%; later interventions affect fewer remaining turns. Crucially, partner responses and subsequent dialogue trajectories can freely diverge, addressing the alternative-continuation concern. These paired interventions demonstrate directional alignment with realized long-horizon outcomes. They report final-outcome win rates after three interventions, providing complementary evidence to an individual-action numerical correlation coefficient.

(3) Reward-design ablations and independent evaluation. The original Table 3 provides controlled evidence that the proposed reward design matters:

Original Table 3. Reward-Design Ablations (Overall ↑)

MethodSOTOPIASOTOPIA-Hard
OCPO (Full)3.983.69
w/o Token Weighting3.673.48
w/o Mean-only Advantage3.633.41

Removing token-sensitivity weighting reduces Overall by 0.31/0.21; replacing mean-only centering reduces it by 0.35/0.28. AgentSense also confirms BC-to-OCPO gains of +3.59/+3.33 percentage points in Majority for Qwen2.5-7B/Llama-3-8B.

Together, the interventions, ablations, and independent evaluations show that SWLR extracts useful local preferences that guide social policies toward better realized outcomes, supporting the reward design and OCPO's training-efficiency contribution.

Additional Clarifications on Novelty and Presentation

We will make three connected contributions more explicit. (1) Observed-continuation reuse: teacher-forced scoring turns logged partner feedback into turn-level supervision without extra full-dialogue training rollouts, reward-model training, or external reward-evaluator calls. (2) SWLR: sensitivity-weighted continuation likelihood focuses reward on action-dependent response tokens, going beyond uniform token likelihood and semantic matching. (3) Mean-only centering + sequence-level GSPO: preserving relative reward-difference magnitudes prevents weakly discriminative groups from dominating updates, while sequence-level optimization treats each structured utterance coherently. We will clarify the scoring contexts, BC/RL stages, and component-level evidence.

Conclusion

Our new SFT-only, AgentSense, and data-dependency experiments clarify OCPO's incremental RL gains, cross-benchmark transfer, and robustness to reduced data and weaker generators. Combined with the existing intervention and ablation evidence, they strengthen the case for SWLR's effectiveness. We will incorporate these findings and clarify the method's motivations and scope in the revision.

Reviewer_9jBcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Observed Continuation Policy Optimization (OCPO), a framework designed to convert recorded partner responses into local supervision for social agents. Following behavioral cloning, OCPO combines a sensitivity-weighted likelihood reward, mean-centered advantages, and GSPO updates, without additional full-dialogue rollouts or external evaluator calls for reward computation. Experimental results on SOTOPIA and SOTOPIA-Hard show the highest reported Overall scores in seven of eight backbone-matched settings, with the Qwen training pipeline using fewer reported GPU-hours than SOTOPIA-RL and SAVOIR.

Critical strengths

1.The idea of reusing recorded partner responses to provide local supervision for social agents at low cost is interesting. 2. The framework illustration and dialogue examples make the overall approach easy to follow. The distinction between generating candidate utterances and scoring them against recorded partner responses is clearly explained.

Critical weaknesses
  1. Incomplete baseline comparisons. The experiments lack a continued-SFT baseline that starts from the same BC checkpoint and uses the same data with a comparable training budget. Since Stage 2 combines GSPO with an auxiliary SFT loss, this control would help determine whether GSPO-based policy optimization provides additional gains over continued SFT. Comparisons with FLR [1] and AMPO [2] would also strengthen the evaluation. [1] Zhang, Chen, et al. "Aligning language models using follow-up likelihood as reward signal." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 24. 2025. [2] Wang, Minzheng, et al. "Adaptive social learning via mode policy optimization for language agents." International Conference on Learning Representations. Vol. 2026. 2026.

  2. The reward design needs clearer justification. Consider a constructed example with two content tokens in the observed continuation, context-only log-probabilities (-5, -10), and \tau = 1. The candidate-conditioned token log-probabilities and SWLR scores computed using Eqs. (9)–(11) are: Candidate Token 1 log-probability Token 2 log-probability SWLR score A −0.1 −5 −2.6724 B −0.2 −10 −0.2800 Although the model assigns higher probability to both continuation tokens when conditioned on A than on B, SWLR ranks B higher. This ranking behavior requires justification. Furthermore, the relationship between the nonpositive reward in Eq. (11) and the positive SWLR Scores reported in Figure 2 and Table 11 remains unclear.

Questions for authors
  1. What justifies the ranking reversal in W1, and how often does it occur in practice?
  2. How does the reported SWLR Score relate to Eq. (11), and which quantity is used for training?
  3. How does OCPO compare with continued SFT, FLR, and AMPO under matched training data and comparable training budgets?
  4. Do the Stage 2 development episodes in Table 6 overlap with the BC training set in Table 5? If so, how is this overlap handled in model selection and validation?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 9jBc (1/3): Baselines and AMPO

Thank you for recognizing the motivation, efficiency, and clarity of OCPO, and for your constructive questions. We have added continued-SFT and FLR-style baselines, examined AMPO's published results, analyzed SWLR ranking consistency on both training traces and development instances, and clarified the diagnostic score and checkpoint-selection protocol. We respond point by point below.

1. Baseline comparisons (W1 / Q3)

1.1 Continued-SFT and FLR-style controls

To test whether Stage 2 policy optimization adds value beyond continued supervised learning, we introduce two baselines.

Continued-SFT. We initialize from exactly the same Stage 1 BC checkpoint as OCPO and conduct one complete epoch of supervised fine-tuning on the same 19,797 Stage 2 training instances, targeting their expert reference actions. We use an SFT-appropriate optimization objective and configuration. The final evaluation follows the same SOTOPIA and SOTOPIA-Hard protocols, with each result averaged over four complete runs.

FLR-style joint ablation. Starting from the same BC checkpoint, we retain the observed-continuation optimization framework while jointly replacing three OCPO components: sensitivity-based token weighting with uniform weighting, mean-only advantages with within-group standard-deviation normalization, and GSPO with conventional GRPO. This tests the full method against a basic follow-up-likelihood optimization variant.

Additional Overall results:

BackboneMethodSelf-Play SOTOPIASelf-Play HardGPT-4o Partner SOTOPIAGPT-4o Partner Hard
Qwen2.5-7BContinued-SFT3.533.113.603.43
FLR-style3.563.253.583.31
OCPO3.873.573.983.69
Llama3.1-8BContinued-SFT3.473.283.683.41
FLR-style3.583.343.623.31
OCPO3.883.703.923.76

OCPO outperforms Continued-SFT in all eight settings. The improvements are +0.34/+0.46/+0.38/+0.26 for Qwen and +0.41/+0.42/+0.24/+0.35 for Llama (in table-column order). Continued-SFT has already traversed the entire Stage 2 training set once, yet falls short of OCPO. Thus, optimizing candidate-level relative preferences using observed continuations contributes useful learning signals beyond further imitation of expert actions, directly addressing the concern about our auxiliary SFT loss.

OCPO also outperforms FLR-style in all eight settings, by 0.30-0.45 Overall points. FLR-style improves upon Continued-SFT in Self-Play but performs below Continued-SFT in all four GPT-4o-as-Partner settings, whereas full OCPO yields clear gains in those settings. This supports the practical importance of combining sensitivity-based weighting, mean-only advantages, and sequence-level updates.

Our original Table 3 provides complementary single-component ablations: on Qwen SOTOPIA, replacing GSPO, removing token weighting, or removing mean-only advantages gives 3.80, 3.67, and 3.63, respectively, versus 3.98 for full OCPO. The joint and individual ablations together support the contributions of these components.

The new baseline is explicitly labeled FLR-style: it incorporates follow-up-likelihood supervision within a controlled Stage 2 framework. The original FLR work additionally constructs positive/negative follow-up feedback and performs preference-based alignment. We will state this implementation distinction explicitly rather than present the joint ablation as a reproduction of the full FLR pipeline.

1.2 Published comparison with AMPO

We additionally checked Table 1 of AMPO (Wang et al., ICLR 2026) and compared results for identical backbones, benchmarks, and interaction protocols:

BackboneMethodSelf-Play SOTOPIASelf-Play HardGPT-4o Partner SOTOPIAGPT-4o Partner Hard
Qwen2.5-7BAMPO3.873.543.943.65
OCPO3.873.573.983.69
Llama3.1-8BAMPO3.953.683.983.74
OCPO3.883.703.923.76

OCPO leads in five settings, ties in one, and trails in two. Notably, OCPO is higher in all four SOTOPIA-Hard settings, indicating strong competitiveness on more challenging social interactions. AMPO remains stronger in the two standard-SOTOPIA Llama settings. Since AMPO uses different training data and a mode-based optimization pipeline, this is a comparison against published results, not a data- or compute-matched reproduction. The controlled causal evidence comes from our new matched-initialization baselines and the original ablations. We will add AMPO and update the corresponding best-result claims in the revision.

AuthorsOpenReview ↗

Response to Reviewer 9jBc (2/3): SWLR Ranking

2. SWLR ranking reversal: explanation and frequency (W2 / Q1)

The reviewer's counterexample is mathematically valid under Eqs. (9)-(11). We quantified the reversal on full training traces and development instances for both backbones.

2.1 Why a reversal can occur

For candidate aia_i, SWLR assigns sensitivity weights using the change in the observed continuation's token log-likelihood:

Δi,m=∣ℓi,mutt−ℓmctx∣,αi,m=exp⁡(Δi,m/τ)∑k∈Cexp⁡(Δi,k/τ),\Delta_{i,m}=|\ell^{utt}_{i,m}-\ell^{ctx}_m|, \qquad \alpha_{i,m}=\frac{\exp(\Delta_{i,m}/\tau)}{\sum_{k\in C}\exp(\Delta_{i,k}/\tau)},

and then computes

riSWLR=∑m∈Cαi,mℓi,mutt.r_i^{SWLR}=\sum_{m\in C}\alpha_{i,m}\ell^{utt}_{i,m}.

The key property is that the weighting distribution depends on the candidate action. Consequently, different candidates are evaluated with different token weights. In the reviewer's example, the context-only log-likelihoods are (−5,−10)(-5,-10); candidate A has (−0.1,−5)(-0.1,-5) and B has (−0.2,−10)(-0.2,-10). A's sensitivities are (4.9,5.0)(4.9,5.0), giving nearly equal weights to the two tokens. Its second token still has log-likelihood −5-5 and receives considerable weight. B has sensitivities (4.8,0)(4.8,0), so its score is dominated by the first token. The resulting scores are −2.6724-2.6724 for A and −0.2800-0.2800 for B, as the reviewer calculated.

Thus, SWLR does not guarantee componentwise dominance monotonicity: a candidate can assign higher likelihood to every continuation token yet receive a lower sensitivity-weighted score. We will state this property explicitly. The weighting mechanism is designed to emphasize continuation tokens most sensitive to the candidate action, so its ranking can differ from that of the uniform token likelihood.

2.2 New ranking-consistency analysis

We call a pair (ai,aj)(a_i,a_j) strictly dominant if ℓi,mutt≥ℓj,mutt\ell^{utt}_{i,m}\ge\ell^{utt}_{j,m} for every content token, with a strict inequality for at least one token. A strict-dominance reversal occurs if riSWLR<rjSWLRr_i^{SWLR}<r_j^{SWLR} despite that dominance.

For each backbone, we analyzed all 800 training steps: 12,800 groups, 102,400 candidates, and 358,400 unordered candidate pairs. We also performed eight sampling repetitions on the 1,070 Stage 2 development instances using the fixed post-BC checkpoint (Checkpoint 0) for generation and scoring, producing 8,560 groups and 239,680 candidate pairs per backbone. This development-set analysis is descriptive and is not presented as independent validation of the entire two-stage pipeline.

MetricQwen TrainQwen Dev (BC-0)Llama TrainLlama Dev (BC-0)
Strictly dominant pairs10,7527,1949,8566,611
Dominance reversals48223129
Dominance reversal rate0.45%0.31%0.31%0.44%
Uniform ranking inversion rate7.81%7.82%6.91%6.96%
Top-1 agreement84.15%83.95%85.17%85.27%

The reviewer's specific type of reversal is rare among strictly dominant pairs. On training traces, only 48 of 10,752 such pairs reverse on Qwen (0.45%), and 31 of 9,856 reverse on Llama (0.31%). The corresponding BC-0 development diagnostics are 0.31% and 0.44%. These percentages use strictly dominant pairs as their denominator; such pairs represent approximately 2.75%-3.00% of all candidate pairs. We report both counts and rates to make the denominator explicit.

To measure broader ranking changes, we compare SWLR with the uniform mean log-likelihood, ui=∣C∣−1∑m∈Cℓi,muttu_i=|C|^{-1}\sum_{m\in C}\ell^{utt}_{i,m}. On training traces, the pairwise inversion rates are 7.81% (Qwen) and 6.91% (Llama); the corresponding Top-1 agreements are 84.15% and 85.17%. Development diagnostics show similar values. Sensitivity weighting changes some candidate preferences; strict-dominance reversals remain uncommon. General ranking differences can reflect trade-offs between tokens.

2.3 Connection to downstream performance

The practical utility of the weighting mechanism is assessed separately through controlled task performance. In our original Table 3, removing token weighting reduces Qwen's Overall from 3.98 to 3.67 on SOTOPIA and from 3.69 to 3.48 on SOTOPIA-Hard (drops of 0.31 and 0.21). Further, Figure 4's paired interventions show that higher-SWLR branches achieve higher final Overall scores with rates of 66%, 61%, and 55% for early, middle, and late interventions, respectively.

Together, the new diagnostics establish that dominance reversals are infrequent in our observed candidate groups, and the existing ablation and intervention results support the practical value of sensitivity weighting. We will clarify both the mathematical limitation and its measured prevalence in the revision.

AuthorsOpenReview ↗

Response to Reviewer 9jBc (3/3): Score and Data Split

3. Raw SWLR reward versus the reported score (Q2)

We clarify the apparent discrepancy between Eq. (11) and the positive scores in Figure 2 and Table 11.

Eq. (11) defines the raw training reward as sensitivity-weighted token log-likelihood:

riSWLR=∑m∈Cαi,mℓi,mutt.r_i^{SWLR}=\sum_{m\in C}\alpha_{i,m}\ell^{utt}_{i,m}.

Since ℓi,mutt≤0\ell^{utt}_{i,m}\le 0, αi,m≥0\alpha_{i,m}\ge 0, and ∑mαi,m=1\sum_m\alpha_{i,m}=1, the raw reward necessarily satisfies riSWLR≤0r_i^{SWLR}\le 0.

Figure 2 and Table 11 report the exponentiated diagnostic score:

SiSWLR=exp⁡(riSWLR)=∏m∈Cpθold(ym∣s,ai,y<m)αi,m.\boxed{S_i^{SWLR}=\exp(r_i^{SWLR})}=\prod_{m\in C}p_{\theta_{old}}(y_m\mid s,a_i,y_{<m})^{\alpha_{i,m}}.

This is a sensitivity-weighted geometric mean of conditional token probabilities in (0,1](0,1]; higher means greater weighted likelihood. The positive values in Table 11, including 0.7548 for Qwen and 0.7441 for Llama, use this transformed diagnostic scale. Because exponentiation is strictly increasing,

riSWLR>rjSWLR  ⟺  SiSWLR>SjSWLR,r_i^{SWLR}>r_j^{SWLR}\iff S_i^{SWLR}>S_j^{SWLR},

so it preserves all candidate rankings and introduces no additional ranking reversals.

Training always uses the raw log-likelihood reward from Eq. (11), followed by mean-only advantage centering in Eq. (12):

A^i=riSWLR−1G∑j=1GrjSWLR,\hat A_i=r_i^{SWLR}-\frac{1}{G}\sum_{j=1}^{G}r_j^{SWLR},

and then the GSPO update. The exponential transformation is used only for Figure 2 and Table 11, never for the training reward or advantage computation. Thus, it does not affect the policy optimization or the magnitude of the raw reward differences used for advantage estimation. We will explicitly define SiSWLR=exp⁡(riSWLR)S_i^{SWLR}=\exp(r_i^{SWLR}) in the revision and label the quantities Raw SWLR Reward and Exponentiated Diagnostic SWLR Score to remove the ambiguity.

4. Stage 2 development overlap and model selection (Q4)

We confirm that some Stage 2 development episodes overlap with Stage 1 BC training episodes. We clarify the development set's role, checkpoint selection, and final evaluation.

First, the Stage 2 development set serves training monitoring and reward-signal diagnostics. It is constructed using the valid observed-continuation criterion and split at the episode level within Stage 2; its instances do not enter Stage 2 policy-gradient updates. Figure 2 uses a fixed probe of 64 development states to monitor SWLR signals and training dynamics across checkpoints. All comparison methods use the same probe.

Crucially, the final Stage 2 checkpoint is determined by a pre-specified training schedule, independently of development-set performance. We run exactly 800 optimization steps and directly evaluate the Step-800 checkpoint in all final experiments. We do not select an intermediate checkpoint based on development scores. Hence, the cross-stage overlap cannot introduce a bias through Stage 2 checkpoint selection.

Our main performance claims are supported by full-interaction benchmark evaluations. Table 1 evaluates the fixed final checkpoint on SOTOPIA and SOTOPIA-Hard under both Self-Play and GPT-4o-as-Partner, with fixed scenario-role configurations and four complete evaluation runs per setting. The official GPT-4o evaluator scores complete, newly generated dialogue trajectories. These evaluations do not use Stage 2 diagnostic scores for model selection.

Our additional Continued-SFT and FLR-style controls also use the same BC initialization and consistent evaluation settings, providing further evidence for OCPO's gains under controlled comparisons.

The overlap limits the independence of Stage 2 diagnostics, but the development set enters neither Stage 2 gradient updates nor checkpoint selection. The fixed training schedule, full-dialogue evaluation, and controlled baselines support our main results. We will clarify these points in the revision.

5. Summary

The new experiments show that OCPO outperforms Continued-SFT and FLR-style baselines in all eight settings, while the published AMPO comparison demonstrates strong competitiveness, particularly on SOTOPIA-Hard. Our ranking analysis explains SWLR's candidate-dependent non-monotonicity and shows that strict-dominance reversals are uncommon in observed training and development candidate pairs; the original ablation and intervention studies further support its downstream usefulness. Finally, we have clarified the exponentiated plotting metric and the fixed Step-800 checkpoint selection procedure. We believe these additions substantially strengthen the empirical and methodological evidence for OCPO, and we will incorporate the clarifications into the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.