- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors present a benchmark, PraxiScene, to measure open-goal social agency, as agents in interactive scenarios may often require behavior where they decide what situation requires and what more information to inquire about is part of the interaction itself. The authors present a benchmark with stories: Unconditioned Reachable (branching to different story endings) and Conditioned Reachable (intermediate states with hidden information are present that affect future states). The authors create a generator that constructs social situations into executable environments. The authors evaluate 6 different backbones (with ReACT) and find that they differ in how they gather information. Supplying Information, Goal, or Value each has a distinct effect on interaction cost/trajectory burden and agreement on final policy and normative orientation. Successful task completion does not mean agents share a common approach to social situations.
Critical strengths
The framing of the problem is clear and the motivation is generally explained well. The authors have clearly stated that the benchmark is geared towards open-goal social agency (hidden information, state dependent on past actions, and not having an immensely clear objective right from the start).
Appreciated the design of having the Goal x Information x Value (with fixed story) and the different metrics (trajectory, terminal strategy, and value). Most social benchmarks compare agents across different scenarios that have some sort of cumulated score, or don’t isolate so cleanly, so it’s difficult to differentiate where the difference in performance comes from.
The executable environment design allows for clear event graphs with explicit conditions and state updates, allowing for better evaluation and analysis. Since human annotators/evaluators and the agent share the same interface, which makes for a nice and robust/clear way to compare.
Critical weaknesses
The benchmark measures various metrics for the interactions, the experiment results are able to highlight differences between different agents along these metrics. However, there is less of a notion of benchmarking whether these agents are more or less capable at the social interactions, but more so about the behavior policies/behavior profiles of the various agents. If there’s a way to make this clearer, or better represent this in the title/introduction/abstract, this work would be stronger. Or maybe if the authors can provide some sort of cumulative score that can be useful towards indicating generally if some family of agents are better at these tasks than others.
The intro motivation/relevant works section could be stronger if also further motivated by prior works that show that LLMs typically struggle in multi-turn scenarios [1, 2], also more from the agent design/harnesses side (not just the benchmarks side) that attempt to evaluate LLMs on multi-turn benchmarks or that have architectures that attempt to tackle these complicated social dynamics, which will help give a stronger motivation for needing this benchmark and also be other models to benchmark asides from ReACT such as [3,4,5,6].
While there is more information about human behavior/human studies in appendix E5 (one of the links seem broken and redirects to section 5/methods in the main paper instead), would appreciate a couple of notes somewhere on number of participants, basic recruitment qualities, etc. The main text explains that Figure 6 reflects 3 human plays, which seems little (maybe 6 human plays or even more is required to make an accurate human baseline here?)
The work relies on some LLM-as-a-judge for evaluations, could the authors present more details around the model judge? Which model was used/just to make sure that there’s no leakage between the model judge and the models that are evaluated, etc/ if there’s any calibration done.
There are a lot of abbreviations that are present in the results, that make it a little difficult to follow/require more effort to follow. Table 1 has a lot of abbreviations, is it possible to in brackets write a slightly longer abbreviation/a word or two for them, and maybe put arrows to indicate if higher or lower is better? Or perhaps a quick summary table of all the metrics would also be helpful near it, since the authors have already defined all the terms in the earlier section (it just requires a lot of moving up and down to read). The discussion/result paragraphs feature percentages and point changes and confidence intervals together, perhaps there might be a way to put those in the captions or embed into the tables/graphs, so the main text can highlight broad trends (some less numbers) to make results more readable (ie: section 5.1). This will also save space to bring up a basic summary of the limitations in the main text, currently they are all buried in the appendix, and some limitations should be clearly stated in the main text.
[1] Laban, Philippe, et al. "Llms get lost in multi-turn conversation." International Conference on Learning Representations. Vol. 2026. 2026. [2] Wang, Xingyao, et al. "Mint: Evaluating llms in multi-turn interaction with tools and language feedback." International Conference on Learning Representations. Vol. 2024. 2024. [3] Zhou, Xuhui, et al. "Tom-swe: User mental modeling for software engineering agents." arXiv preprint arXiv:2510.21903(2025). [4] Ma, Rachel, et al. "Flexible Agent Alignment with Goal Inference from Open-Ended Dialog." arXiv preprint arXiv:2508.15119 (2025). [5] Liu, Jie, et al. "RTADev: Intention aligned multi-agent framework for software development." Findings of the Association for Computational Linguistics: ACL 2025. 2025. [6] Grand, Gabriel, et al. "Shoot first, ask questions later? building rational agents that explore and act like people." International Conference on Learning Representations. Vol. 2026. 2026.
Questions for authors
Is there a way to make a score from what you have (report a cumulative score) so that there’s a way to report the top family of models? The main text states that value labels are compared with humans (there’s a human agreement evaluation of 90.4% with Value labels), is there a human agreement evaluation done for the Strategy labels? Information about number of human participants, recruitment criteria, etc. Information about the model judge or calibration metrics for the model judge if there are any? The paper mentions: “Interaction length is strongly correlated with information coverage across backbones (r = 0.95 in UR; r = 0.97 in CR), indicating a consistent coupling between engagement and interaction cost.” but is it possible that maybe more actions inherit visit more states? Perhaps more details on how interaction coverage is measured? Why is opencode only evaluated with two models? Is it due to capabilities concern or just cost/time? Goal, information, and value is provided through the prompt. This is less of a concern to other points here but is there any comparison to prompt wording or ordering that would cause any changes in metrics? (a quick ablation-esq test to see if metrics are sensitive to prompts.) Other harnesses that would be good to benchmark?
6 comments
Response to Reviewer TC3y (2/4)
W3. Human Reference and the Three-Play Illustration
We appreciate the request to clarify the human sample and the purpose of Figure 6. Figure 6 illustrates three human trajectories and eight agent runs from a single story; it is not the full human baseline. Figure 5 presents the broader Human–Agent comparison of realized Strategy/Value profiles. The verified human-study records are:
| Human study record | Count |
|---|---|
| Participants | 25 |
| Plays attempted, each with a stored trace | 539 |
| UR / CR plays | 513 / 26 |
| Valid trajectories (stored trace and registered Ending) | 429 |
| Post-ending questionnaire responses | 428 |
Participants were recruited through social media, aged 20–45, and had narrative-game experience ranging from none to regular play. Stories were randomly sampled for each participant. They used the same executable environment and action-resolution interface as the evaluated agents under Open Goal + Stepwise Information + Plain Value. Of 539 attempts, 96 were skipped and 14 ended in a participant quit; one of the 429 completed plays had no questionnaire response. Participants agreed with the displayed terminal Value labels in 387/428 responses (90.4%). This is agreement with displayed labels, not independent blind labeling.
The single-story figure provides a trajectory illustration, while the larger cohort supports the reported behavioral-reference analysis. The human sample is a convenience cohort, with limited CR coverage (26 attempts), rather than a normative definition of the correct social choice. We also acknowledge the incorrect Appendix link noted in the review; the full cohort counts, exclusions, and questionnaire protocol are given in 814B-W1.
W4 / Q2. Judge Models, Calibration, and Evaluation Independence
We thank the reviewer for asking which evaluators were used, whether their model families overlap with the tested agents, and how their outputs were checked. PraxiScene separates semantic interpretation from executable outcome evaluation. Strategy/Value annotations are assigned during benchmark construction, and a runtime resolver maps free-form inputs to registered actions/events. Registered state transitions and Ending conditions determine completion and specified-outcome attainment; there is no LLM-assigned overall social-quality score.
The three-model annotation configuration uses DeepSeek-V4.1-Flash, GLM-4.7, and GPT-4.1-mini. The implementation specifies GPT-4.1 for action grounding and a DeepSeek-based labeler for agent-reported Strategies. Model-family overlap with evaluated backbones is present, including DeepSeek. The construction-stage Ending annotations are fixed across evaluated backbones, while the runtime semantic components operate under their designated evaluator configurations. This avoids per-backbone relabeling of Endings, although grounding and reported-Strategy interpretation remain semantic judgments that can affect the corresponding measurements.
We additionally checked those semantic outputs against human judgments:
| Component | Verification | Result |
|---|---|---|
| Value annotation | Three-model consensus on retained stories; post-ending human questionnaire | UR/CR consensus 99.81%/99.96%; displayed-label agreement 387/428 (90.4%) |
| Strategy annotation | Three-model consensus; two annotators examined 100 cases | UR/CR consensus 96.64%/93.92%; original fine label supported by ≥1 annotator in 91/100, and by both in 54/100 |
| Action grounding | Two annotators examined 60 input–mapping pairs | Inter-annotator agreement 56/60 (93.3%), κ = 0.845; nonempty mappings supported in 43/45 and 41/45 cases |
The human checks assess the correspondence of semantic labels and action mappings to human judgments, rather than calibrating a scalar judge score. The consensus figures describe construction-stage agreement on retained stories, while the Value questionnaire measures acceptance of displayed labels. The independent Strategy and grounding reviews assess the resulting labels and mappings directly. Grounding verification also distinguishes support for nonempty mappings from no-match cases involving unsupported requests (see PZgh-W2 for the full protocol).
Response to Reviewer TC3y (3/4)
W5. Metric Definitions, Directionality, and Results Presentation
We thank the reviewer for suggesting a clearer metric guide and more accessible presentation of the results. We organize the measurements into four groups that characterize how agents enact social decisions, acquire information, and evolve their policies. Specified Goal/Value attainment and realized social-choice profiles provide complementary outcome evidence. The following table summarizes the metrics and their interpretation.
| Evaluation group | Metric | What it measures | Direction |
|---|---|---|---|
| Executability | — Grounding success | Successful executable mappings | ↑ |
| Trajectory Enactment | — Valid-ending completion | Registered Ending reached | ↑ |
| — Action count | Actions taken | — | |
| — Conditional depth | Conditional story depth traversed | — | |
| Information Engagement | — Information coverage | Proportion of story clues acquired | — |
| — Acquisition efficiency | Unique clues per inquiry action | — | |
| Policy Coherence & Dynamics | — Strategy agreement | Reported vs. realized Strategy | — |
| — Strategy shift | Changes after state updates | — | |
| Controlled Outcomes | — Goal attainment | Specified Goal realized | ↑ |
| — Target Value realization | Assigned Value realized | ↑ | |
| — Value steering | Matched change in target realization | ↑ | |
| Social-choice Profiles | Ending / Strategy / Value | Realized social choices | — |
| Strategy / Value distributions | Model-level behavioral tendencies | — |
↑: higher attainment of an explicit criterion; —: no universal performance direction; †: CR-only. Controlled Outcomes apply to the corresponding Goal/Value conditions, while Social-choice Profiles describe autonomous behavior rather than success.
The main scorecard uses the four trajectory-measurement groups, with six ReAct backbones and two OpenCode variants distinguished. Controlled outcomes and social-choice profiles are reported separately. Metric directions, CR-only eligibility, and applicable story cohorts clarify how to read the comparisons without treating all measurements as performance scores.
We also emphasize three findings in the results: similar completion rates can conceal different social choices; Goal specification changes intended-outcome realization much more than overall completion; and Value guidance selectively increases target realization. Detailed estimates and uncertainty are provided with the corresponding comparisons (see PZgh-W1/W3, 814B-W2, and nnJH-W2).
W6. Main-text Limitations and Scope of Conclusions
We appreciate the reviewer's suggestion to bring the principal limitations into the main-text Discussion rather than leaving them in the Appendix. We have added a concise limitations discussion alongside the main findings, covering three points directly relevant to their interpretation:
- Story coverage. PraxiScene evaluates agent decisions within registered, executable story graphs. What-if expansion broadens available options but cannot guarantee every plausible social resolution (nnJH-W1).
- Measurement and human reference. Executable Ending conditions are rule-computed, whereas free-form grounding and Strategy/Value annotations involve semantic judgments assessed through human verification (PZgh-W2). Human trajectories serve as a behavioral reference, with limited CR coverage (26 attempted plays; 814B-W1).
- Experimental generalization. Results cover six ReAct backbones, matched ReAct–OpenCode comparisons for two backbones, a specified NPC configuration, and fixed Goal/Information/Value prompts. They do not establish robustness across alternative NPC models, prompt phrasings, or repeated stochastic runs (TC3y-W2/Q4/Q5).
These limitations define the scope of the reported comparisons: executable and specified-outcome performance under the tested conditions, alongside descriptive profiles of autonomous social decisions.
Response to Reviewer TC3y (1/4)
Thank you for the constructive suggestions on capability assessment, agent baselines, human and evaluator validation, and experimental reporting. We address the comparison criteria, provide the corresponding quantitative evidence, and clarify the scope of the reported evaluations.
W1 / Q1. Capability Assessment and Model Comparison
We appreciate the reviewer's point that a benchmark should help compare agent capabilities, not only describe behavioral differences. PraxiScene provides performance comparisons where success can be explicitly determined, alongside profiles of the social choices agents make without an assigned goal. Grounding, valid-ending completion, and specified Goal/Value attainment have operational criteria; Ending, Strategy/Value, information engagement, and policy dynamics describe complementary aspects of autonomous behavior.
These two views reveal differences that a completion score alone would miss. On the same 1,037 UR stories, DeepSeek and GPT reach valid Endings in 97.78% and 96.82% of runs, respectively, but realize different valid Endings in 52.84% of stories. Under a specified Goal, the comparison is:
DeepSeek has slightly higher observed UR completion, whereas GPT has higher specified-goal attainment in both controlled subsets. Which model is more suitable therefore depends on whether the requirement is executable completion, attainment of a particular social objective, or a particular behavioral profile. An aggregate score would require assigning weights to these different requirements, including dimensions without a universally preferred direction. We retain criterion-specific comparisons so that performance is interpretable without imposing a single ranking over legitimate social choices.
Comparing systems by a single number is useful when a reader must pick one model. Where the construct is multi-dimensional, current evaluation reports the dimensions instead. Pluralistic Leaderboards shows that a single quality score cannot represent users with different behavioral preferences (Haghtalab et al., ICML 2026). A 445-benchmark audit recommends preserving label variability rather than single-point aggregation (Bean et al., NeurIPS 2025, D&B), and social-intelligence benchmarks report several aspects separately (Mou et al., NAACL 2025). Aggregating heterogeneous tasks "reduces the rich information to an overall score that is hard to interpret" (Burnell et al., Science, 2023), which is why HELM reports seven metrics per scenario (Liang et al., TMLR, 2023). The Value layer is the same: 58 Schwartz labels are reported as a profile (Yao et al., ACL 2025; Ren et al., ACL 2024), not as a single score. We report criterion-specific orderings with their eligible populations; a reader who wants one number can apply their own weights.
Related evaluation and measurement questions are addressed in PZgh-W1/W3 and 814B-W2.
W2. Multi-turn Motivation and Agent Architecture Baselines
We thank the reviewer for suggesting closer connections to multi-turn research and broader agent architectures. Studies such as LLMs Get Lost in Multi-Turn Conversation and MINT motivate evaluation across extended interactions. The suggested work on user modeling, goal inference, and iterative agent design is also relevant. We added in our related work section. PraxiScene tests the additional requirement that an agent must decide what social outcome to pursue, while interaction may reveal information and change subsequent options. Consequently, both the underlying backbone and its decision architecture may affect performance.
We evaluated six backbones under the same ReAct harness to compare models while holding the architecture fixed. To examine architecture effects, we additionally compared ReAct and OpenCode using DeepSeek and GPT on matched cohorts of 720 UR and 211 CR stories per backbone. Differences in valid-ending completion are shown below (ReAct − OpenCode, percentage points; 95% CI):
ReAct achieves higher completion for GPT on UR, whereas OpenCode achieves higher completion for DeepSeek on CR; the other confidence intervals include zero. This comparison shows that architecture effects depend on both backbone and story structure, rather than producing a uniform gain. It directly extends the baseline beyond ReAct, although the two evaluated harnesses do not represent the full range of specialized planning, reflection, or search-based agents discussed in the suggested literature. The choice of two OpenCode backbones is explained in TC3y-Q4.