acceptodds
Under review as a conference paper at ICLR 2027

What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds

Abstract

Effective social interaction requires more than fluent responses: agents must reach plans that fit the participants' needs and circumstances. Existing evaluations of social agents mainly rely on LLM judges that rate the conversation, without checking the resulting plan. We introduce HICO-Bench, a benchmark of 180 everyday coordination scenarios, such as resource allocation, scheduling, and authorization, in which two agents with private information converse and then independently record the plan they agreed on. A program checks Feasibility (the two records match and satisfy both participants' private conditions), and an LLM judge checks Integrity (the conversation supports the recorded plan); passing both is a Verified Success. Each scenario has two versions that share the same public context but differ in one private condition, so the correct plan differs between them. Across thirteen LLMs, 36% of conversations rated highly for goal completion fail Verified Success. To learn from outcomes that arrive only at the end of a conversation, we propose Paired-Credit Policy Optimization (PaCPO). After each turn, PaCPO asks the agent for a tentative plan, checks it in both versions of the scenario, and rewards the turns after which the tentative plans fit their respective versions. PaCPO raises Qwen3-8B's Verified Success from 20.4% to 28.3%.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Social reasoning and interaction, Social intelligence, Task-oriented dialogue and negotiation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_h35gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces HICO-Bench, a benchmark of 180 paired coordination scenarios that checks whether two agents reach a shared, feasible plan supported by their dialogue. It also proposes PaCPO, which compares tentative plans across the two versions of each scenario to assign rewards across dialogue turns.

Critical strengths
  1. Each scenario has two versions that differ in one private condition and require different valid plans. This design provides a clear test of whether an agent uses the relevant information to adjust its response.

  2. PaCPO uses tentative plans to provide feedback at individual dialogue turns while keeping the total reward unchanged. This offers a practical way to train from final outcomes without manually scoring every utterance.

Critical weaknesses
  1. PaCPO sees 96 training scenarios, while the group baselines see 48, and its extra plan probes and disclosure checks add training cost. Matching dialogue counts alone does not separate the method's benefit from differences in data coverage and total cost.

  2. The ablations do not directly compare PaCPO with using only the intermediate feasibility term in Eq. 6. A control with a comparable reward scale would help show whether paired comparisons add value beyond this simpler signal.

  3. PaCPO and GRPO have the same Feasibility score, so their gap in Verified Success depends on the Integrity check. Appendix K reports 90% agreement between two judges, but their pass rates differ by ten percentage points and the effect on method ranking is not reported.

  4. HICO scenarios are adapted from SOTOPIA, but the paper does not explain whether HICO training and the SOTOPIA transfer test share source scenarios. This makes it hard to judge how far the transfer goes beyond situations already used in training.

  5. The training study uses Qwen3-8B, and the main HICO evaluation also uses Qwen3-8B as the partner. The additional partner settings are useful, but they do not establish whether the HICO gains hold with a different partner model.

Questions for authors
  1. Does PaCPO remain ahead when all methods see the same training scenarios?

  2. How does PaCPO compare with keeping the intermediate feasibility term while removing the paired term?

  3. On the shared outputs in Appendix K, what is each method's Verified Success under each of the two judges? Does the ordering of PPO, GRPO, and PaCPO stay the same?

  4. Are the source scenarios used to build HICO training separate from the fourteen SOTOPIA environments used for the transfer test?

  5. How much of the HICO gain depends on using Qwen3-8B as the partner?

4 comments

Thank you for highlighting the value of paired scenarios and PaCPO’s use of tentative plans for turn-level credit. Your questions about paired credit, source overlap, and dialogue partners helped us clarify when PaCPO’s gains hold.

1. Does PaCPO still lead with the same training scenarios?

We trained PaCPO and all four group-based baselines (G = 4) on the same 96 source scenarios with the same training seed and evaluated all five methods on the same 240 test tasks. PaCPO sampled 384 training dialogues and achieved 28.33% Verified Success. Each baseline sampled 768 training dialogues. Verified Success was 25.00% for GRPO, 22.50% for Dr.GRPO, 22.92% for GiGPO, and 24.17% for GraphGPO. With half as many training dialogues as each baseline, PaCPO outperformed the strongest baseline, GRPO, by 3.33 percentage points.

2. What does the paired term in Eq. (6) add?

We retained the plan probes and within-world feasibility term and set β = 0 to remove paired credit. For both the control and full PaCPO, we tested the default α = 0.25 and an alternative α ≈ 0.3348 calibrated to the reward scale on the training data. With the same development set and tuning budget, both selected α = 0.25. The two values tied for the control, and our prespecified rule selected the smaller one. With only β differing, full PaCPO (β = 0.10) scored higher in both runs. Mean test Verified Success was 26.04% for full PaCPO and 25.21% for the feasibility-only control. These results suggest that cross-world comparisons improve Verified Success beyond what within-world feasibility shaping alone achieves.

3. Does the ranking change with the Integrity judge?

We extended the comparison in Appendix K to all 240 test tasks per method. Qwen3.8-Max and Kimi-K3 evaluated the same dialogue outputs using the same rule-based Feasibility decisions. Each judge assessed whether the dialogue supported the final plan (Integrity). The table reports Verified Success.

JudgeBasePPOGRPOPaCPO
Qwen3.8-Max20.42%25.00%26.25%28.33%
Kimi-K326.67%30.42%30.83%32.92%

Both judges ranked the methods PaCPO > GRPO > PPO > Base. Under Kimi-K3, PaCPO led PPO by 2.50 percentage points and GRPO by 2.08.

4. Do the SOTOPIA transfer environments overlap with HICO?

Matching original SOTOPIA environment IDs revealed one overlap between the 14 transfer environments and HICO’s 96 training sources, free_stuff. During benchmark construction, we screened 884 source records by source ID, scenario family, and alias. We found no prior project use of free_stuff at the time and selected it as one of the 103 sources for new scenarios. Although we rewrote the training task, we counted it as an overlap because it shares the original environment ID.

Excluding free_stuff left 13 environments and 65 environment–character pairs. With GPT-4o as the partner, Goal scores (out of 10) were 7.03 for PaCPO, 6.35 for PPO, and 6.82 for GRPO. In self-play, the corresponding scores were 6.71, 6.17, and 6.58.

Checking all 180 HICO sources revealed one further overlap with the HICO test set. Excluding both environments left 12 environments and 60 environment–character pairs with no source ID shared with HICO. On this subset, PaCPO scored 7.07 with GPT-4o as its partner, compared with 6.35 for PPO and 6.85 for GRPO. We will report the overlap and both filtered evaluations in the revision.

We also evaluated the method with task-specific training. On MultiWOZ, 81.17% of PaCPO’s recommendations satisfied all user constraints (PPO, 80.52%). On CaSiNo, PaCPO reached agreement in 63.95% of negotiations (PPO, 60.47%).

5. How much does the HICO gain depend on the Qwen3-8B partner?

We evaluated the original PaCPO, PPO, and GRPO checkpoints with three additional partners. All combinations used the same 240 HICO test tasks and the same Qwen3.8-Max evaluation protocol. The table reports Verified Success.

Dialogue partnerPaCPOPPOGRPO
Qwen3-8B (original)28.33%25.00%26.25%
Qwen Flash32.92%30.00%31.25%
DeepSeek V4.1 Flash 101768.75%62.08%64.58%
DeepSeek V4 Pro51.25%50.42%46.25%

PaCPO scored higher than both baselines with each additional partner. With DeepSeek V4.1 Flash 1017, its margins were 6.67 percentage points over PPO and 4.17 over GRPO.

Thank you for the detailed replies and for running the new experiments. I still have a few concerns about the new results.

1. The gain over the control is small

The authors compare the full method with a control that keeps only the within world term. The gap between them is a few tasks out of the full test set, and the same reply reports seed variation that is larger than this gap. No interval is given for this comparison, so the data do not show that the paired term helps.

ConditionVerified Success
Full method, average of two runs26.04
Control, average of two runs25.21
Gap0.83

2. One setting, two values

The reply to Reviewer iATJ reports a lower value at zero paired weight in the weight sweep. The reply to me reports a higher value at the same zero weight. Both replies say the other settings are held fixed. The two values differ by more than the paper claims for the paired term itself. The authors should say which run each value comes from, and if the runs differ, they should report both under one setting.

SettingVerified Success
Zero paired weight, weight sweep23.33
Zero paired weight, control comparison25.21

3. Baselines drop when they get more data

The new run gives every method the same scenarios, and the baselines also get more training dialogues than in the original run reported in paper Table 3. More data should not make a method worse. Several baselines do get worse, and one stays exactly the same. This run therefore does not show a stronger method. It shows baselines that moved down while the proposed method kept its old score.

MethodOriginal runNew run
GRPO26.2525.00
Dr.GRPO21.2522.50
GiGPO24.5822.92
GraphGPO24.1724.17
Training dialogues384 for every method768 for baselines, 384 for PaCPO

I would be glad to update my score if these points are addressed.

1. Clarifying the β = 0 results and the contribution of paired credit

Thank you for noting the discrepancy. At α = 0.25 and β = 0, the first two additional runs scored 27.08% and 23.33% on Verified Success. Thus, 23.33% is a single-run result and 25.21% is the two-run mean. We will clarify both passages.

We suspected that using different machines might have contributed to the variation. To investigate this and address your concern about training randomness, we repeated the comparison on one machine with four independent training seeds. Both methods used the same plan probes and within-world feasibility term (α = 0.25).

RunFull PaCPOβ = 0 controlDifference (pp)
Seed 124.58%23.33%+1.25
Seed 223.33%23.75%−0.42
Seed 325.83%22.92%+2.92
Seed 425.00%21.25%+3.75
Mean24.69%22.81%+1.88

Full PaCPO scored higher on three of four seeds, with 237 successes in 960 evaluations versus 219 for the control. The mean gain was 1.88 percentage points, with a 95% bootstrap CI of [+0.42, +3.33] percentage points across training seeds.

We used the six task families from Figure 3 and Section 5.3. Each family comprised 10 test scenarios and 40 tasks per run, for 160 evaluations across four runs.

Task familyFull PaCPOβ = 0 controlDifference (pp)
Set-based consent23.75%25.00%−1.25
Joint scheduling8.75%8.75%0.00
Resource allocation21.25%14.38%+6.88
Shared-resource scheduling4.38%4.38%0.00
Authorization45.63%45.00%+0.63
Bounded quantities44.38%39.38%+5.00
All six families24.69%22.81%+1.88

Resource allocation and bounded quantities gained most (+6.88 and +5.00 percentage points). In resource allocation, Integrity passes rose from 23 to 34, all of which also passed Feasibility. In one anti-bullying campaign task, the participants had to divide 12 poster panels within respective limits of 3–4 and 3–9. PaCPO agreed on a 4/8 split and passed both checks. The control submitted a feasible 3/9 split but failed Integrity after incorrectly claiming that 8 was outside the 3–9 range and then repeating itself.

Set-based consent had two fewer successes (−1.25 percentage points). The failures involved tracking all requirements and knowing when to end the dialogue. In one task, the organizer requested mime, funny faces, and a group photo, while the other participant ruled out the photo. Both methods submitted the same correct agreement, but PaCPO failed Integrity because its dialogue became repetitive.

Overall, paired credit yielded 18 additional verified successes, mainly in resource allocation and bounded quantities. The consent example highlights an opportunity to improve dialogue closure once agents reach a valid agreement.

2. How did additional training affect the baselines?

Thank you for raising this point. PaCPO in Table 3 already covered all 96 training scenarios, so in our previous response we reused its checkpoint and score and trained the four group-based baselines (G = 4) on the same 96 scenarios. As in response for question 1, we suspected that using different machines might have contributed to the variation. We therefore retrained and evaluated PaCPO under the same settings.

MethodCoverage-matched Verified Success
PaCPO27.50%
GRPO25.00%
Dr.GRPO22.50%
GiGPO22.92%
GraphGPO24.17%

With training coverage matched, the retrained PaCPO achieved 27.50% Verified Success, outperforming all four baselines by at least 2.50 percentage points (GRPO, 25.00%). PaCPO scored 0.83 percentage points below its Table 3 result (28.33%), narrowing its lead over GRPO from 3.33 to 2.50 percentage points. The ranking of all five methods remained the same.

We hope these additional analyses address your questions. Thank you again for examining our results so carefully!

Thank you for the detailed replies and for the additional runs. I still have a few concerns about the new results.

1. The reported result is above every run reported later

The paper reports one number for the full method. The latest reply reports four new seeds, and none of them reaches that number. Their average is well below it. The paper value therefore looks like the best of several runs rather than a typical one.

QuantityVerified Success
Paper value28.33
Average of four new seeds24.69
Best of four new seeds25.83

2. The spread between runs is larger than the gain claimed

Both methods move across a wide range in the runs reported so far, and the gain in the new study is smaller than that range. An interval built from the spread across the four seeds includes zero, so the new study does not separate the two methods.

QuantityVerified Success
Lowest full method run23.33
Highest full method run28.33
Lowest control run21.25
Highest control run27.08
Gain claimed1.88

3. The machine explanation does not keep the same size

One reply puts the size of the machine effect below the gain claimed. Another puts it above. If a change of machine can move a result by more than the gain itself, then a single run per method in the main table cannot support the comparison it makes.

ComparisonFirstSecondDifference
Paper value and retrained value28.3327.500.83
Mean of three seeds and mean of four seeds26.8124.692.12
Reviewer_tcwFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces HICO-Bench, which checks whether social agents reach feasible, consistent agreements and whether their conversations support those agreements. It pairs scenarios with identical public information but different private constraints, and proposes PaCPO to assign turn-level credit by comparing tentative plans across these worlds. Experiments report improvements over RL baselines, with additional evaluations in self-play and external tasks.

Critical strengths
  1. Evaluates concrete outcomes of social interaction. Agents must independently submit matching plans that satisfy both parties’ constraints. This reveals coordination failures that convincing dialogue may conceal. Separating Feasibility from Integrity also helps distinguish invalid plans from conversations that fail to support them.
  2. Uses paired scenarios to test whether agents use relevant information. The public setting stays fixed while a private condition changes. Successful agents must adjust their plans accordingly. PaCPO uses these comparisons to redistribute credit across turns while preserving each dialogue’s total return.
  3. Includes broad evaluations and useful implementation details. The experiments cover several RL baselines, self-play, and external tasks. The paper also reports computational costs, cross-judge agreement, and a small human study. These details make the method and its limitations easier to assess.
Critical weaknesses
  1. The gains over strong baselines are small and their reliability is unclear. In Table 3, PaCPO improves Verified Success from GRPO’s 26.25% to 28.33%, equivalent to five additional successes across 240 tasks. The main comparisons and ablations use one training seed. Without repeated training runs and uncertainty estimates accounting for shared scenarios, the reliability of this advantage remains unclear.
  2. The comparisons do not control scenario coverage or computational cost. PPO and PaCPO train on 96 scenarios, while the group-based baselines see only 48. PaCPO also uses additional plan probes and LLM disclosure annotations. Matching dialogue counts therefore leaves training diversity and computational budget as possible explanations for the gains.
  3. The added value of paired-world credit is not fully isolated. The ablations modify the comparison candidates, credit timing, or disclosure gate. However, they omit a simpler baseline that retains plan probes and within-world feasibility shaping while removing the paired-credit term. It remains unclear whether this simpler method would perform similarly.
  4. The advantage relies on a judge-dependent metric. PaCPO and GRPO have identical aggregate Feasibility, while PaCPO leads on Verified Success, which incorporates LLM-judged Integrity. The reported cross-judge agreement does not establish that this advantage survives a change of judge. The human pilot compares PaCPO only with the base model, leaving its advantage over stronger RL baselines unvalidated by humans.
Questions for authors
  1. Does the improvement survive repeated training? Can you repeat the main comparisons and key ablations across training seeds? Please report paired confidence intervals that account for tasks sharing the same underlying scenario. Does PaCPO consistently outperform both PPO and GRPO?
  2. Can the baseline comparisons control coverage and cost? Can you compare methods using the same training scenarios? Please also provide a cost-matched comparison that includes plan probes and disclosure annotations. For GiGPO and GraphGPO, how often does exact-history matching provide repeated states with nonzero step-level advantages?
  3. Is paired credit better than feasibility shaping alone? Can you add a β=0\beta=0 control that retains plan probes and within-world feasibility shaping? Please match the training data and probe budget, with comparable tuning of the shaping strength. This would directly test whether paired-world comparisons add value beyond a simpler intermediate reward.
  4. Does the advantage hold under independent evaluation? Does PaCPO still outperform PPO and GRPO when another judge evaluates Integrity? Please report each method’s Verified Success under both judges. A human comparison against these RL baselines would also help establish whether the measured gains reflect better interactions.
2 comments

Thank you for recognizing the value of assessing Feasibility and Integrity separately. We are grateful for your suggestions on training variability, scenario coverage, computational cost, and judge choice, which helped us evaluate PaCPO’s gains more carefully.

1. Does the gain persist across training seeds?

We trained each method with three seeds, using 384 training dialogues per run, and measured Verified Success on the same 240 test tasks.

MethodSeed 1Seed 2Seed 3Mean
PaCPO28.33%27.50%24.58%26.81%
PPO25.00%23.75%23.33%24.03%
GRPO26.25%27.08%24.17%25.83%

PaCPO scored higher than both baselines on each seed. Giving each seed equal weight, its mean advantage was 2.78 percentage points over PPO and 0.97 over GRPO under this training budget.

2. Scenario coverage and exact-history matching

We trained PaCPO and all four group-based baselines (G = 4) on the same 96 source scenarios with the same training seed and evaluated all five methods on the same 240 test tasks. PaCPO sampled 384 training dialogues and achieved 28.33% Verified Success. Each baseline sampled 768 training dialogues. Verified Success was 25.00% for GRPO, 22.50% for Dr.GRPO, 22.92% for GiGPO, and 24.17% for GraphGPO. With half as many training dialogues as each baseline, PaCPO outperformed the strongest baseline, GRPO, by 3.33 percentage points.

We counted exact-history matches in the original 384-dialogue runs. GiGPO found matches at 192 of 2,265 training steps (8.48%), and GraphGPO at 188 of 2,369 (7.94%). Of the matched steps, 176 (91.67%) had nonzero step-level advantages for GiGPO and 100 (53.19%) for GraphGPO.

Thank you for suggesting that we check how often exact-history matches occur. In HICO’s open-ended, multi-turn dialogues, each method found an identical dialogue history at only about 8% of training steps. This suggests that the exact-history matching we used for GiGPO and GraphGPO may be ill-suited to such tasks, since few dialogue histories recur often enough to estimate step-level advantages. We will discuss this limitation in the revision to help readers interpret both baselines’ performance on HICO.

3. Does paired credit add value beyond within-world feasibility shaping?

We held the training data, training budget, plan probes, and disclosure checks fixed. Full PaCPO (β = 0.10) and the feasibility-only control (β = 0) each selected between α = 0.25 and α ≈ 0.3348 on the same development set. Full PaCPO scored higher at 0.25. The two settings tied for the control, which also selected 0.25 under our prespecified tie-breaking rule. The final configurations therefore differed only in β. Full PaCPO scored higher in both runs, averaging 26.04% test Verified Success versus 25.21% for the control. These results suggest that paired credit improves on within-world feasibility shaping alone.

4. Do the results hold with a second judge and human evaluators?

Both judges evaluated the same 240 archived dialogues per method. We retained the rule-based Feasibility decisions and rescored Integrity. The table reports Verified Success.

JudgeBasePPOGRPO (G = 4)PaCPO
Qwen3.8-Max20.42%25.00%26.25%28.33%
Kimi-K326.67%30.42%30.83%32.92%

Both judges ranked the methods PaCPO > GRPO > PPO > Base. Under Kimi-K3, PaCPO led PPO by 2.50 percentage points and GRPO by 2.08.

Furthermore, two human evaluators reviewed the same paired dialogues from 12 scenarios spanning all six families. Each evaluator assessed 24 PaCPO–PPO pairs and 24 PaCPO–GRPO pairs. In both comparisons, each evaluator preferred PaCPO more often than the baseline.

Human evaluatorBaseline (24 pairs each)Preferred PaCPOPreferred baselineSimilarBoth inadequate
1PPO8664
1GRPO7494
2PPO7683
2GRPO9573

Thank you for the additional information which has explained my doubts, I am appreciative of the work done by all of you and I will keep my rating the same.

Reviewer_iATJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HICO-Bench, a benchmark for evaluating social agents based on verifiable interaction outcomes rather than LLM-judged conversation quality. The benchmark consists of 180 everyday coordination scenarios, including resource allocation, scheduling, and authorization. Each scenario contains two paired worlds that differ in one private condition, requiring agents to reach appropriate agreements through dialogue. The authors evaluate outcomes using rule-based Feasibility checks and LLM-judged Integrity checks. They also propose Paired-Credit Policy Optimization (PaCPO), which uses cross-world comparisons of tentative plans to redistribute terminal rewards across interaction turns. Experiments on 13 LLMs show that high conversation-quality ratings do not necessarily translate to successful outcomes. Training Qwen3-8B with PaCPO also outperforms PPO and GRPO on the proposed benchmark.

Critical strengths

The paper introduces an interesting evaluation setup that measures whether agents can reach feasible agreements based on participants' private conditions. I find this approach interesting compared to the original SOTOPIA, which relies on an LLM judge to evaluate goal achievement based on the conversation history.

Critical weaknesses
  • There is limited explanation of how the benchmark was constructed and its quality was controlled. It is unclear how the authors ensured that the benchmark is sufficiently challenging for recent frontier models.
  • The benchmark also seems relatively easy for recent frontier models, with GPT-6 Astra and Gemini 3.7 Flash achieving over 80% Verified Success. It would be interesting to investigate what types of failures remain for these models.
  • The hyperparameters used for credit assignment in PaCPO are not analyzed.
Questions for authors

Do improvements on this benchmark translate to broader social intelligence, or do they mainly reflect better performance on resource allocation and scheduling tasks?

1 comment

We appreciate your thoughtful feedback on how HICO-Bench evaluates agreements against both parties’ private constraints. Your questions guided our analysis of benchmark difficulty, frontier-model failures, hyperparameter sensitivity, and transfer to broader social interactions.

1. Benchmark construction, quality control, and difficulty

Construction. We adapted SOTOPIA situations into six coordination families, each scenario comprising two worlds that differ in one private condition (Section 3.2, Table 1). We screened 884 SOTOPIA-π source records by ID, scenario family, and alias. Of 465 sources with no recorded prior use, we selected 103 for new scenarios. We also adapted 77 existing scenarios and excluded all 50 sources from the previous test set.

Quality control. We checked each scenario against the original situation, goals, and roles, correcting a buyer–seller reversal in one furniture transaction. In all 180 pairs, the public context and the unaffected role’s initial prompt were identical across worlds. An independent program enumerated 1,189 valid records across 360 worlds, checking scheduling constraints minute by minute where applicable. All 360 reference records failed in the opposite world, and the full sets of valid records were disjoint in every pair.

Difficulty. On the frozen test set (240 tasks, fixed partner), Verified Success ranged from 20.42% to 82.50% across 13 models. GPT-5.6 Luna and Gemini 3.7 Flash had similar Goal ratings (9.61 and 9.54 out of 10), but their Verified Success differed by 30.42 percentage points (52.08% vs. 82.50%).

To better challenge frontier models, we introduce HICO-Bench-Hard, a new subset of the frozen test set. Even the strongest model fails on about 40% of the resulting 48 tasks across 12 scenarios. We will add these results to Section 5.2 after the discussion of Table 2.

ModelHard FeasibilityHard IntegrityHard Verified SuccessFull-test Verified Success
Qwen3.8-27B54.17%41.67%35.42%54.58%
Qwen3.8-Max62.50%54.17%45.83%60.83%
GPT-5.6 Sol66.67%66.67%56.25%76.25%
GPT-6 Astra75.00%68.75%60.42%80.83%

2. Where do the two highest-scoring models fail?

We checked the role cards, transcripts, final records, and evaluation logs for all 88 dialogues scored as failures for GPT-6 Astra and Gemini 3.7 Flash. Authorization and shared-resource scheduling account for one third of the test tasks and 46 of the 88 failures (52.3%), with 23 failures per model across the two families.

The most common failure was misusing information already available in the dialogue (29 cases). In one case, Astra checked the required steps twice, yet its partner incorrectly said one could be skipped. In another, Gemini treated an optional ingredient as mandatory and rejected a feasible plan. In 13 cases, the agents submitted different final records. We also found 20 generation or formatting issues, 3 borderline interpretations of task rules, and 23 possible judge errors, including two cases in which the judge counted changeover time twice.

3. Sensitivity to the credit-assignment weights

In Eq. (6), α weights the feasibility of a tentative plan in its own world. β weights the cross-world comparison once the differing private condition is available in both dialogues. With α fixed at 0.25, setting β to 0, 0.05, 0.10, and 0.20 yielded Verified Success rates of 23.33%, 27.50%, 28.33%, and 27.50%, respectively. With β fixed at 0.10, lowering α from 0.25 to 0.15 reduced Verified Success from 28.33% to 25.00%.

4. Transfer to broader social interaction

We tested direct policy transfer to SOTOPIA role-play. After excluding the environment shared with HICO training, the HICO-trained PaCPO policy achieved a Goal score of 7.03/10 with GPT-4o as its partner, compared with 6.35 for PPO and 6.82 for GRPO. This evaluation used the HICO checkpoints without further training.

We also trained each method on two other interaction tasks. On MultiWOZ, 81.17% of PaCPO’s hotel or restaurant recommendations satisfied all user constraints, compared with 80.52% for PPO, the strongest baseline. On CaSiNo, PaCPO reached agreement in 63.95% of negotiations and achieved 49.37% normalized Nash surplus. PPO scored 60.47% and 45.86%, respectively. These results extend the evaluation to constraint elicitation and bargaining. We plan to test a broader range of interactions in future work.

Comments on the paper

1 comment

We thank all three reviewers for their thoughtful feedback. iATJ highlighted the value of checking agreements against both parties’ private constraints. tcwF noted the distinction between Feasibility and Integrity and the breadth of our evaluations. h35g highlighted how paired worlds test adaptation to private information and how PaCPO uses tentative plans to assign turn-level credit. Their questions motivated the experiments below.

  • Benchmark construction and difficulty. Exhaustive checks confirmed that no plan is valid in both worlds of any of the 180 scenario pairs. Authorization and shared-resource scheduling accounted for 46 of the top two models’ 88 failures. We will add the HICO-Bench-Hard results to Section 5.2 of the revised paper. Even the strongest model fails on about 40% of tasks in this subset.
  • Training and judge robustness. PaCPO outperformed PPO and GRPO on all three training seeds. With the same 96 training scenarios and seed, PaCPO exceeded the best of four baselines (GRPO) by 3.33 percentage points in Verified Success using half as many training dialogues (384 vs. 768). Full PaCPO beat the feasibility-only control in both runs, averaging 26.04% vs. 25.21% Verified Success. Both Integrity judges, Qwen3.8-Max and Kimi-K3, ranked the methods PaCPO > GRPO > PPO > Base.
  • Dialogue partners and transfer. PaCPO outperformed PPO and GRPO with all three additional HICO partners. On 12 SOTOPIA environments with no HICO source overlap, the HICO-trained policy scored 7.07 on Goal with GPT-4o as its partner (GRPO 6.85, PPO 6.35). Task-specific training also improved constraint satisfaction on MultiWOZ and agreement rates on CaSiNo.

We detail the new experiments and their results in our responses below. We will include both in the revised paper and expand our description of benchmark construction and quality control. We appreciate the reviewers’ suggestions and thank the area chair for coordinating the review and discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.