acceptodds
Under review as a conference paper at ICLR 2027

PraxiScene: Benchmarking Agent Social Agency in Generated Open-Goal Interactive Scenarios

Abstract

As AI agents increasingly operate in social environments, they often face situations with no predefined objective and incomplete decision-relevant information. Recent benchmarks have moved toward interactive social behavior, dynamic information, and complex social reasoning, but typically retain a given task, social goal, or inference target. It remains unclear whether social agents retain their performance when deciding what the situation requires and what evidence to seek becomes part of the interaction itself. We formulate this setting as open-goal social agency and introduce PraxiScene, a benchmark combining structured story generation, an interactive environment for action-driven branching narratives, a suite of open-goal social scenes, and trajectory-level evaluation. Its 1248 frozen social situations comprise two story types: one exposes open branching interactions towards different story endings, while the other additionally contains intermediate states with hidden information that reshape subsequent choices. These stories unfold through grounded agent actions and environment responses, producing auditable interaction trajectories. PraxiScene evaluates story completion while characterizing how agents adapt their policies to the situation and available information, and what values are expressed along each trajectory. Matched interventions vary Goal, Information, and Value specification, with humans as a behavioral reference. Across six backbones and two agent systems, supplying Information sharply shortens interaction but reduces CR policy agreement by about 11 percentage points; supplying a Goal instead raises agreement by up to 9 points, while Value guidance selectively increases its targeted realization by 10.8 points.

Primary area:
datasets and benchmarks
Keywords:
multi-agent systems, LLM agents, benchmarks, agent evaluation, human evaluation, value aligment, interactive story generation, social simulation, open-goal social agency

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_TC3yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present a benchmark, PraxiScene, to measure open-goal social agency, as agents in interactive scenarios may often require behavior where they decide what situation requires and what more information to inquire about is part of the interaction itself. The authors present a benchmark with stories: Unconditioned Reachable (branching to different story endings) and Conditioned Reachable (intermediate states with hidden information are present that affect future states). The authors create a generator that constructs social situations into executable environments. The authors evaluate 6 different backbones (with ReACT) and find that they differ in how they gather information. Supplying Information, Goal, or Value each has a distinct effect on interaction cost/trajectory burden and agreement on final policy and normative orientation. Successful task completion does not mean agents share a common approach to social situations.

Critical strengths

The framing of the problem is clear and the motivation is generally explained well. The authors have clearly stated that the benchmark is geared towards open-goal social agency (hidden information, state dependent on past actions, and not having an immensely clear objective right from the start).

Appreciated the design of having the Goal x Information x Value (with fixed story) and the different metrics (trajectory, terminal strategy, and value). Most social benchmarks compare agents across different scenarios that have some sort of cumulated score, or don’t isolate so cleanly, so it’s difficult to differentiate where the difference in performance comes from.

The executable environment design allows for clear event graphs with explicit conditions and state updates, allowing for better evaluation and analysis. Since human annotators/evaluators and the agent share the same interface, which makes for a nice and robust/clear way to compare.

Critical weaknesses

The benchmark measures various metrics for the interactions, the experiment results are able to highlight differences between different agents along these metrics. However, there is less of a notion of benchmarking whether these agents are more or less capable at the social interactions, but more so about the behavior policies/behavior profiles of the various agents. If there’s a way to make this clearer, or better represent this in the title/introduction/abstract, this work would be stronger. Or maybe if the authors can provide some sort of cumulative score that can be useful towards indicating generally if some family of agents are better at these tasks than others.

The intro motivation/relevant works section could be stronger if also further motivated by prior works that show that LLMs typically struggle in multi-turn scenarios [1, 2], also more from the agent design/harnesses side (not just the benchmarks side) that attempt to evaluate LLMs on multi-turn benchmarks or that have architectures that attempt to tackle these complicated social dynamics, which will help give a stronger motivation for needing this benchmark and also be other models to benchmark asides from ReACT such as [3,4,5,6].

While there is more information about human behavior/human studies in appendix E5 (one of the links seem broken and redirects to section 5/methods in the main paper instead), would appreciate a couple of notes somewhere on number of participants, basic recruitment qualities, etc. The main text explains that Figure 6 reflects 3 human plays, which seems little (maybe 6 human plays or even more is required to make an accurate human baseline here?)

The work relies on some LLM-as-a-judge for evaluations, could the authors present more details around the model judge? Which model was used/just to make sure that there’s no leakage between the model judge and the models that are evaluated, etc/ if there’s any calibration done.

There are a lot of abbreviations that are present in the results, that make it a little difficult to follow/require more effort to follow. Table 1 has a lot of abbreviations, is it possible to in brackets write a slightly longer abbreviation/a word or two for them, and maybe put arrows to indicate if higher or lower is better? Or perhaps a quick summary table of all the metrics would also be helpful near it, since the authors have already defined all the terms in the earlier section (it just requires a lot of moving up and down to read). The discussion/result paragraphs feature percentages and point changes and confidence intervals together, perhaps there might be a way to put those in the captions or embed into the tables/graphs, so the main text can highlight broad trends (some less numbers) to make results more readable (ie: section 5.1). This will also save space to bring up a basic summary of the limitations in the main text, currently they are all buried in the appendix, and some limitations should be clearly stated in the main text.

[1] Laban, Philippe, et al. "Llms get lost in multi-turn conversation." International Conference on Learning Representations. Vol. 2026. 2026. [2] Wang, Xingyao, et al. "Mint: Evaluating llms in multi-turn interaction with tools and language feedback." International Conference on Learning Representations. Vol. 2024. 2024. [3] Zhou, Xuhui, et al. "Tom-swe: User mental modeling for software engineering agents." arXiv preprint arXiv:2510.21903(2025). [4] Ma, Rachel, et al. "Flexible Agent Alignment with Goal Inference from Open-Ended Dialog." arXiv preprint arXiv:2508.15119 (2025). [5] Liu, Jie, et al. "RTADev: Intention aligned multi-agent framework for software development." Findings of the Association for Computational Linguistics: ACL 2025. 2025. [6] Grand, Gabriel, et al. "Shoot first, ask questions later? building rational agents that explore and act like people." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

Is there a way to make a score from what you have (report a cumulative score) so that there’s a way to report the top family of models? The main text states that value labels are compared with humans (there’s a human agreement evaluation of 90.4% with Value labels), is there a human agreement evaluation done for the Strategy labels? Information about number of human participants, recruitment criteria, etc. Information about the model judge or calibration metrics for the model judge if there are any? The paper mentions: “Interaction length is strongly correlated with information coverage across backbones (r = 0.95 in UR; r = 0.97 in CR), indicating a consistent coupling between engagement and interaction cost.” but is it possible that maybe more actions inherit visit more states? Perhaps more details on how interaction coverage is measured? Why is opencode only evaluated with two models? Is it due to capabilities concern or just cost/time? Goal, information, and value is provided through the prompt. This is less of a concern to other points here but is there any comparison to prompt wording or ordering that would cause any changes in metrics? (a quick ablation-esq test to see if metrics are sensitive to prompts.) Other harnesses that would be good to benchmark?

6 comments
AuthorsOpenReview ↗

Response to Reviewer TC3y (1/4)

Thank you for the constructive suggestions on capability assessment, agent baselines, human and evaluator validation, and experimental reporting. We address the comparison criteria, provide the corresponding quantitative evidence, and clarify the scope of the reported evaluations.

W1 / Q1. Capability Assessment and Model Comparison

We appreciate the reviewer's point that a benchmark should help compare agent capabilities, not only describe behavioral differences. PraxiScene provides performance comparisons where success can be explicitly determined, alongside profiles of the social choices agents make without an assigned goal. Grounding, valid-ending completion, and specified Goal/Value attainment have operational criteria; Ending, Strategy/Value, information engagement, and policy dynamics describe complementary aspects of autonomous behavior.

These two views reveal differences that a completion score alone would miss. On the same 1,037 UR stories, DeepSeek and GPT reach valid Endings in 97.78% and 96.82% of runs, respectively, but realize different valid Endings in 52.84% of stories. Under a specified Goal, the comparison is:

Evaluation criterionDeepSeekGPT
Open Goal: valid-ending completion, UR97.78%96.82%
Given Goal: Goal attainment, UR63.06%66.11%
Given Goal: Goal attainment, CR46.45%49.29%

DeepSeek has slightly higher observed UR completion, whereas GPT has higher specified-goal attainment in both controlled subsets. Which model is more suitable therefore depends on whether the requirement is executable completion, attainment of a particular social objective, or a particular behavioral profile. An aggregate score would require assigning weights to these different requirements, including dimensions without a universally preferred direction. We retain criterion-specific comparisons so that performance is interpretable without imposing a single ranking over legitimate social choices.

Comparing systems by a single number is useful when a reader must pick one model. Where the construct is multi-dimensional, current evaluation reports the dimensions instead. Pluralistic Leaderboards shows that a single quality score cannot represent users with different behavioral preferences (Haghtalab et al., ICML 2026). A 445-benchmark audit recommends preserving label variability rather than single-point aggregation (Bean et al., NeurIPS 2025, D&B), and social-intelligence benchmarks report several aspects separately (Mou et al., NAACL 2025). Aggregating heterogeneous tasks "reduces the rich information to an overall score that is hard to interpret" (Burnell et al., Science, 2023), which is why HELM reports seven metrics per scenario (Liang et al., TMLR, 2023). The Value layer is the same: 58 Schwartz labels are reported as a profile (Yao et al., ACL 2025; Ren et al., ACL 2024), not as a single score. We report criterion-specific orderings with their eligible populations; a reader who wants one number can apply their own weights.

Related evaluation and measurement questions are addressed in PZgh-W1/W3 and 814B-W2.

W2. Multi-turn Motivation and Agent Architecture Baselines

We thank the reviewer for suggesting closer connections to multi-turn research and broader agent architectures. Studies such as LLMs Get Lost in Multi-Turn Conversation and MINT motivate evaluation across extended interactions. The suggested work on user modeling, goal inference, and iterative agent design is also relevant. We added in our related work section. PraxiScene tests the additional requirement that an agent must decide what social outcome to pursue, while interaction may reveal information and change subsequent options. Consequently, both the underlying backbone and its decision architecture may affect performance.

We evaluated six backbones under the same ReAct harness to compare models while holding the architecture fixed. To examine architecture effects, we additionally compared ReAct and OpenCode using DeepSeek and GPT on matched cohorts of 720 UR and 211 CR stories per backbone. Differences in valid-ending completion are shown below (ReAct − OpenCode, percentage points; 95% CI):

Story structureDeepSeekGPT
UR+1.53 [−0.14, 3.20]+3.89 [1.79, 6.24]
CR−8.53 [−14.35, −1.97]+2.84 [−4.59, 10.73]

ReAct achieves higher completion for GPT on UR, whereas OpenCode achieves higher completion for DeepSeek on CR; the other confidence intervals include zero. This comparison shows that architecture effects depend on both backbone and story structure, rather than producing a uniform gain. It directly extends the baseline beyond ReAct, although the two evaluated harnesses do not represent the full range of specialized planning, reflection, or search-based agents discussed in the suggested literature. The choice of two OpenCode backbones is explained in TC3y-Q4.

AuthorsOpenReview ↗

Response to Reviewer TC3y (2/4)

W3. Human Reference and the Three-Play Illustration

We appreciate the request to clarify the human sample and the purpose of Figure 6. Figure 6 illustrates three human trajectories and eight agent runs from a single story; it is not the full human baseline. Figure 5 presents the broader Human–Agent comparison of realized Strategy/Value profiles. The verified human-study records are:

Human study recordCount
Participants25
Plays attempted, each with a stored trace539
UR / CR plays513 / 26
Valid trajectories (stored trace and registered Ending)429
Post-ending questionnaire responses428

Participants were recruited through social media, aged 20–45, and had narrative-game experience ranging from none to regular play. Stories were randomly sampled for each participant. They used the same executable environment and action-resolution interface as the evaluated agents under Open Goal + Stepwise Information + Plain Value. Of 539 attempts, 96 were skipped and 14 ended in a participant quit; one of the 429 completed plays had no questionnaire response. Participants agreed with the displayed terminal Value labels in 387/428 responses (90.4%). This is agreement with displayed labels, not independent blind labeling.

The single-story figure provides a trajectory illustration, while the larger cohort supports the reported behavioral-reference analysis. The human sample is a convenience cohort, with limited CR coverage (26 attempts), rather than a normative definition of the correct social choice. We also acknowledge the incorrect Appendix link noted in the review; the full cohort counts, exclusions, and questionnaire protocol are given in 814B-W1.

W4 / Q2. Judge Models, Calibration, and Evaluation Independence

We thank the reviewer for asking which evaluators were used, whether their model families overlap with the tested agents, and how their outputs were checked. PraxiScene separates semantic interpretation from executable outcome evaluation. Strategy/Value annotations are assigned during benchmark construction, and a runtime resolver maps free-form inputs to registered actions/events. Registered state transitions and Ending conditions determine completion and specified-outcome attainment; there is no LLM-assigned overall social-quality score.

The three-model annotation configuration uses DeepSeek-V4.1-Flash, GLM-4.7, and GPT-4.1-mini. The implementation specifies GPT-4.1 for action grounding and a DeepSeek-based labeler for agent-reported Strategies. Model-family overlap with evaluated backbones is present, including DeepSeek. The construction-stage Ending annotations are fixed across evaluated backbones, while the runtime semantic components operate under their designated evaluator configurations. This avoids per-backbone relabeling of Endings, although grounding and reported-Strategy interpretation remain semantic judgments that can affect the corresponding measurements.

We additionally checked those semantic outputs against human judgments:

ComponentVerificationResult
Value annotationThree-model consensus on retained stories; post-ending human questionnaireUR/CR consensus 99.81%/99.96%; displayed-label agreement 387/428 (90.4%)
Strategy annotationThree-model consensus; two annotators examined 100 casesUR/CR consensus 96.64%/93.92%; original fine label supported by ≥1 annotator in 91/100, and by both in 54/100
Action groundingTwo annotators examined 60 input–mapping pairsInter-annotator agreement 56/60 (93.3%), κ = 0.845; nonempty mappings supported in 43/45 and 41/45 cases

The human checks assess the correspondence of semantic labels and action mappings to human judgments, rather than calibrating a scalar judge score. The consensus figures describe construction-stage agreement on retained stories, while the Value questionnaire measures acceptance of displayed labels. The independent Strategy and grounding reviews assess the resulting labels and mappings directly. Grounding verification also distinguishes support for nonempty mappings from no-match cases involving unsupported requests (see PZgh-W2 for the full protocol).

AuthorsOpenReview ↗

Response to Reviewer TC3y (3/4)

W5. Metric Definitions, Directionality, and Results Presentation

We thank the reviewer for suggesting a clearer metric guide and more accessible presentation of the results. We organize the measurements into four groups that characterize how agents enact social decisions, acquire information, and evolve their policies. Specified Goal/Value attainment and realized social-choice profiles provide complementary outcome evidence. The following table summarizes the metrics and their interpretation.

Evaluation groupMetricWhat it measuresDirection
ExecutabilityRgrdR_{\mathrm{grd}} — Grounding successSuccessful executable mappings↑
Trajectory EnactmentRendR_{\mathrm{end}} — Valid-ending completionRegistered Ending reached↑
NactN_{\mathrm{act}} — Action countActions taken—
Pdep†P_{\mathrm{dep}}^\dagger — Conditional depthConditional story depth traversed—
Information EngagementICI_C — Information coverageProportion of story clues acquired—
IEI_E — Acquisition efficiencyUnique clues per inquiry action—
Policy Coherence & DynamicsCstrC_{\mathrm{str}} — Strategy agreementReported vs. realized Strategy—
Sstr†S_{\mathrm{str}}^\dagger — Strategy shiftChanges after state updates—
Controlled OutcomesRgoalR_{\mathrm{goal}} — Goal attainmentSpecified Goal realized↑
RtargetR_{\mathrm{target}} — Target Value realizationAssigned Value realized↑
ΔRtarget\Delta R_{\mathrm{target}} — Value steeringMatched change in target realization↑
Social-choice ProfilesEnding / Strategy / ValueRealized social choices—
Strategy / Value distributionsModel-level behavioral tendencies—

↑: higher attainment of an explicit criterion; —: no universal performance direction; †: CR-only. Controlled Outcomes apply to the corresponding Goal/Value conditions, while Social-choice Profiles describe autonomous behavior rather than success.

The main scorecard uses the four trajectory-measurement groups, with six ReAct backbones and two OpenCode variants distinguished. Controlled outcomes and social-choice profiles are reported separately. Metric directions, CR-only eligibility, and applicable story cohorts clarify how to read the comparisons without treating all measurements as performance scores.

We also emphasize three findings in the results: similar completion rates can conceal different social choices; Goal specification changes intended-outcome realization much more than overall completion; and Value guidance selectively increases target realization. Detailed estimates and uncertainty are provided with the corresponding comparisons (see PZgh-W1/W3, 814B-W2, and nnJH-W2).

W6. Main-text Limitations and Scope of Conclusions

We appreciate the reviewer's suggestion to bring the principal limitations into the main-text Discussion rather than leaving them in the Appendix. We have added a concise limitations discussion alongside the main findings, covering three points directly relevant to their interpretation:

  1. Story coverage. PraxiScene evaluates agent decisions within registered, executable story graphs. What-if expansion broadens available options but cannot guarantee every plausible social resolution (nnJH-W1).
  2. Measurement and human reference. Executable Ending conditions are rule-computed, whereas free-form grounding and Strategy/Value annotations involve semantic judgments assessed through human verification (PZgh-W2). Human trajectories serve as a behavioral reference, with limited CR coverage (26 attempted plays; 814B-W1).
  3. Experimental generalization. Results cover six ReAct backbones, matched ReAct–OpenCode comparisons for two backbones, a specified NPC configuration, and fixed Goal/Information/Value prompts. They do not establish robustness across alternative NPC models, prompt phrasings, or repeated stochastic runs (TC3y-W2/Q4/Q5).

These limitations define the scope of the reported comparisons: executable and specified-outcome performance under the tested conditions, alongside descriptive profiles of autonomous social decisions.

Reviewer_nnJHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PraxiScene, a benchmark that measures the so called "open-goal social agency". More specifically, an agent will need to pursue an open social goal by proactively interacting with the environment to collect information or take actions, meanwhile shaping the scenario itself. The authors develop an engine that generates structural stories from value and scene seeds, and that allows free-language interaction which will be mapped to executable semantics; human players can also participate under equal conditions. Based on this engine, PraxiScene includes 1,248 selected social scenarios, of which 211 restrict their goal to be reachable only conditionally; besides the standard open-goal free mode, the benchmark also supports providing the goal, information or value externally. Experiments measure the trajectory, terminal strategy and value of the agents, revealing that LLM-based agents behave differently than humans, and that external hint types affect the actions and strategies differently.

Critical strengths
  1. PraxiScene allows controlled story generation that helps systematically analyzing agent performance under different open-goal social scenarios; meanwhile, the evaluation is not limited to the success rate of the execution, but also considers factors like the coverage of retrieved information or the continuity of strategies. All these turns "open-goal social agency" into a studyable and measurable target.
  2. The authors compare the injection effect of goal, information and value respectively, successfully distinguishing how giving or hiding such hints will change agent behavior.
Critical weaknesses
  1. The structural story design may be too restricted. The execution environment can only (internally) develop and be interacted structurally, and only a limited number of endings are considered, mostly constrained by some kind of commonsense during story generation. While this ensures that the benchmark is operationable, the coverage of the structural story and limited endings is quesionable, and there could be a hidden bias when some rare or counterintuitive situations or endings are not considered as possible when the engine creates the benchmarking story.
    • In Section 4.1, the authors mentioned an "audit and filter" stage that selects candidate stories by quality, diversity and executable verification. Yet this does not complete any missing routes or omitted endings, so the problem remains.
  2. The interference of value is vague. When a story is generated, every character receives its stance and knowledge, which very likely contains some kind of personal value implications. On one hand, if the participant (agent) setting also follows this format, then an external value injection could be redundant or even conflicting. On the other hand, if the personal value is explicitly excluded from the initial hints, participants (either LLM agents or human players) can intrinsically inherent their own values (or some fake values they want to develop) into the story, causing an unmeasured bias.
Questions for authors
  1. How do you ensure that the structured story doesn't miss any rare situation or ending during the story generation?
  2. How do you distinguish between value information from background knowledge, the participant themselves, and the external interference?
5 comments
AuthorsOpenReview ↗

Response to Reviewer nnJH (1/2)

Thank you for highlighting the limitations of finite story coverage and the difficulty of interpreting Value-related behavior. We clarify the coverage of executable choices and the distinct roles of construction, participant, and injected Values.

W1 / Q1. Rare or counterintuitive endings in a finite story graph

We agree that a finite executable graph cannot guarantee coverage of rare or counterintuitive Endings. We state this limit alongside the existing What-if expansion.

Before endings are generated, one process called step2b_what_if reviews structured dialogue and decisions, then appends non-duplicate options. Its prompt targets likely omissions, not exhaustive rare choices.

Construction propertyEvidence
What-if additions per storyTypically 1–5 dialogue options / 0–3 decisions
Endings per storyTypically 6–12
UR endingsOne per decision option; a minority, up to about a quarter, originate from what-if expansion
CR endingsAuthored onto the appended decision subset; a majority originate from what-if expansion
Trigger-reference integrity10,345/10,345 authored endings name an existing trigger

We build executable stories on a grounded causal graph where actions change state through explicit preconditions and effects, rather than unconstrained narrative continuation. Humans and agents play these stories through a shared interface, so the human reference is measured under identical conditions. Input matching no action in this environment is recorded as unresolved, never assigned an invented ending.

Although scoring requires a finite set of endings, this set is not a closed catalogue during authoring. More endings can be added while the story configuration is being constructed, before it is fixed and consolidated. Their trigger conditions must not overlap, because mechanically indistinguishable endings cannot coexist when the runtime cannot determine which one fires. Before freezing, what-if expansion adds plausible missing options, including probable candidate endings, each subject to the same grounding requirements.

Scoring requires these endings to carry Value and Strategy annotations produced through validation, alongside a runtime evaluator that detects story termination. An open end offers neither detectable termination nor a scorable outcome. We therefore exclude free endings invented at play time, because they cannot be grounded, annotated, or scored.

These conditions bound our evaluation to open-goal decision-making within this constructed space, not unrestricted social interaction. The released parametric generator also allows others to extend the registered scenario set. Two annotators checked free-form input grounding on 60 input–mapping pairs, with 93.3% agreement. They supported 43/45 (95.6%) and 41/45 (91.1%) nonempty mappings, respectively (PZgh W2).

We clarify the What-if construction and its coverage limits in Sec. 3, the Appendix, and the Discussion.

AuthorsOpenReview ↗

Response to Reviewer nnJH (2/2)

W2 / Q2. Distinguishing Background, Participant, and Externally Specified Values

We thank the reviewer for raising the important question of how Value information from different sources may interact. In PraxiScene, Construction Value defines the social theme used in story generation, while NPC stances, knowledge, and relationships may convey Value-related cues during interaction. The UR generation protocol does not assign a personal Value to the player, although both human participants and LLM agents may bring their own preferences. Given Value introduces an explicit target instruction, whereas Ending Value annotations characterize the social outcomes actually realized.

We agree that these sources may reinforce or conflict with one another, making their individual contributions difficult to infer from observed behavior. In particular, participants' intrinsic Values are not directly observable, and self-reports alone cannot independently verify them. To examine the influence of external guidance, we conducted matched Plain/Given Value experiments that retain the same StoryConfig, NPC background, executable options, agent backbone, information setting, and interaction budget. Each target Value is selected from stories containing both a reachable target-realizing Ending and an alternative. Only the external Value instruction differs, allowing us to measure its additional effect under the same background conditions and agent configuration.

The paired results show increased realization of the specified Values across both story structures and agent backbones:

StructureBackboneMatched pairsTarget Value realization change (pp, 95% CI)
URDeepSeek674+9.05 [5.64, 12.46]
URGPT677+8.27 [5.17, 11.37]
CRDeepSeek136+13.24 [6.62, 20.59]
CRGPT131+7.63 [0.76, 14.50]

External guidance also changes realized Endings and Strategies. In the target-specific analysis, assigned Value realization increases by an average of 10.8 percentage points, compared with −0.8 points for non-target Values, across 22 UR backbone–target estimates. We additionally analyzed assigned-Value × realized-Value relationships to examine how guidance affects other Value outcomes.

These results show that external Value guidance produces measurable, target-specific changes beyond those observed under the original story context. The matched comparison accounts for existing background cues and baseline model tendencies when estimating this effect, without assuming that participants have no intrinsic preferences. We therefore distinguish the influence of externally specified Values through observable behavioral changes, rather than attributing realized Values to unverifiable internal preferences. This distinction is also relevant to PZgh-W1 and 814B-W2, which address the evaluation of autonomous social decisions.

Thank you again for your careful reading and insightful suggestions. Please let us know if you have any further questions. If the responses resolve your concerns, we would appreciate it if you could reflect them in your rating.

Thank you for the detailed response. The distinction between construction and background values, participant preferences, and externally specified values is helpful. Together with the matched comparisons, this addresses my concern about interpreting the additional effect of value guidance, though it does not fully disentangle the different value sources.

I appreciate the clarification of the what-if expansion and the acknowledgment of the finite environment scope. However, trigger-reference integrity and accurate grounding within the supported action space provide limited evidence about whether plausible alternatives are adequately represented. My remaining concern is how omitted choices might affect the observed behavioral profiles and model comparisons. I'm not asking for exhaustive coverage, but uncertainty about these effects remains material to my assessment.

I will keep my ratings.

Reviewer_814BOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces PraxiScene, a framework for evaluating how an agent understands social situations, acquires information, and decides on actions in the absence of predefined task goals. The dataset consists of 1,248 executable stories, some of which require intermediate conditions to be met before a specific outcome can be reached.

Critical strengths
  1. The problem formulation is meaningful. Incorporating "deciding for oneself what to do" into the evaluation complements existing agent benchmarks, which typically provide predefined task objectives.

  2. The paired intervention design is clear. By varying goals, information, and value-based guidance within the same narrative context, it becomes possible to analyze precisely which aspects of the prompts drive specific behavioral changes.

  3. The paper is easy to follow, with a clear motivation and a well-explained experimental design.

Critical weaknesses

I found the main weakness in the experiment. The human experiments describe the study design, but do not clearly report the number of participants or the number of valid trajectories. Annotation agreement is given as 90.4%, without a complete validation protocol. The main conclusions use these figures as evidence, and a summary figure is not enough.

The volume of information acquired, the rate of strategy shifts, and the consistency between stated intentions and subsequent actions can describe a trajectory. They do not directly show whether the agent identified the key issues, formulated sound objectives, or made the necessary adjustments in response to new information. The paper does not evaluate these abilities separately, and it includes no controlled experiment checking whether the metrics correspond to them.

Questions for authors

please refer to weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer 814B (1/2)

Thank you for your questions about the human study and construct validity. We address the verified human protocol and the correspondence between Social Agency decision requirements and the evidence used to evaluate them.

W1. Human participants, valid trajectories, and the 90.4% agreement protocol

We have verified the human study records and will clarify the protocol and terminal Value-label agreement.

  • Recruitment used social media: 25 participants, aged 20–45, with narrative-game experience ranging from none to regular play.
  • This was a convenience sample without stratification or a representativeness claim. Individual ages were not retained.
  • Each participant received a randomly sampled story set. Participants received compensation and data-policy information and consented.
  • A play is one story attempt. A valid trajectory has a stored trace and a registered ending.
Human study recordsCount
Participants25
Plays attempted (every one with a stored trace)539
UR / CR plays513 / 26
Valid trajectories (stored trace + registered ending)429
Questionnaire responses428

Of the 539 plays, 110 had no ending: 96 participant skips and 14 participant quits. The remaining 429 plays reached endings, but one participant stopped before submitting the questionnaire, leaving 428 responses.

The post-ending questionnaire displayed the ending and Value labels, then requested agreement or disagreement and an optional reason. Participants supplied the judgments. There were no independent annotators.

Value-label statisticResult
Agreement387/428 = 90.4%
Response-level Wilson 95% CI87.3–92.9%
Participant-clustered bootstrap 95% CI83.1–95.6%
Mean Jaccard, assigned versus selected labels0.913
Disagreements where the participant wrote a reason (of 41 disagreements)19/41

The participant-clustered confidence interval accounts for repeated responses within participants. Agreement concerns terminal Value labels, not human–agent agreement or the accuracy of every metric.

Terminal Strategy labels are assigned from a closed 26-label taxonomy to the registered triggering act and were checked by humans in a 100-case audit (PZgh W2). The questionnaire validates Value labels only.

We reported the human-study protocol, sample counts, and exclusions in the Appendix.

AuthorsOpenReview ↗

Response to Reviewer 814B (2/2)

W2. Construct Validity of Social Agency Evaluation

We thank the reviewer for highlighting the need to connect trajectory metrics to the underlying capabilities of Social Agency. We agree that information coverage, Strategy shifts, and report–realization agreement alone do not establish decision quality. In open-goal situations, multiple objectives and courses of action may be reasonable, making it difficult to define a universally correct goal, information request, or Strategy adjustment. PraxiScene therefore combines observations of autonomous behavior with controlled evaluations that provide explicit outcome criteria, rather than over-relying on agents' self-reported reasoning as evidence of correctness.

We examine the three decision requirements through complementary, explicitly scoped measurements. For Goal determination, the reached Ending and its Strategy/Value profile record the social course an agent autonomously realizes; Given Goal adds an independent, specified-outcome attainment criterion. For Information acquisition, ICI_C measures registered-clue coverage and IEI_E measures unique clue gain per active seeking action, while the matched Information conditions test their consequences for interaction and goal attainment. For Action and revision, executable trajectories measure enacted behavior, CstrC_{str} compares stated and realized Strategy, and CR-specific SstrS_{str} records changes following registered state updates. These measures distinguish observable autonomous decisions from performance against specified objectives. The matched results below show why that distinction matters for interpreting system performance.

We further analyzed matched Goal and Information interventions to assess how these decision requirements affect verifiable outcomes:

Intervention and measured outcomeDeepSeekGPT
Given Goal vs. Open Goal: reference-ending realization, UR+26.25 pp+26.39 pp
Given Goal vs. Open Goal: valid-ending completion, UR+0.69 pp+0.83 pp
Provided vs. Stepwise Information: Goal attainment, UR−4.08 pp−7.57 pp
Provided vs. Stepwise Information: Goal attainment, CR+10.90 pp+7.58 pp

Specifying a Goal increases realization of the intended Ending by approximately 26 percentage points, while valid-ending completion changes by less than one point. Thus, successful execution and attainment of a specified social objective capture distinct aspects of performance. Providing information also produces different effects on Goal attainment: decreases in UR (515 matched stories) and increases in CR (211 matched stories). These results show that information availability does not uniformly improve goal-directed performance.

Together, these analyses distinguish what agents autonomously choose and enact from how effectively they achieve specified outcomes. The canonical evaluation characterizes goal selection, information engagement, and policy evolution, while controlled conditions provide explicit performance criteria. This distinction also informs our responses to PZgh-W1/W3 on objective evaluation and model comparison, and nnJH-W2 on the interpretation of agent-reported preferences.

Thank you again for your careful reading and insightful suggestions. Please let us know if you have any further questions. If the responses resolve your concerns, we would appreciate it if you could reflect them in your rating.

Thank you again for your thoughtful comments. We summarize our responses that address your concerns:

  • Human study: We clarified participant recruitment, trajectory accounting, and the interpretation of the 90.4% Value agreement.
  • Value measurement: We clarified the distinction between background Values, participant preferences, and externally specified Values, including the limitations of self-reported preferences.
  • Construct validity: We added controlled Goal and Information comparisons to distinguish observable decision processes from performance against explicit criteria, alongside independent annotation checks.

We hope these clarifications make the relationship between Social Agency and its evaluation clearer. We would appreciate your reconsideration.

Thank you again for your consideration.

Reviewer_PZghOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an evaluation framework for social agents operating under incomplete information. The authors collect scenarios from different social contexts and use LLMs to generate complete scenario information. They then design evaluation metrics and a value taxonomy for settings where agents are not explicitly given their goals or all the information required to complete a task. The paper compares different models, including DeepSeek and GPT, and demonstrates their different behavioral tendencies under this benchmark.

Critical strengths

1.The paper focuses on evaluating agents in more open-ended scenarios, which are common in real-world applications (e.g., in many cases, it is difficult to provide explicit instructions). This is indeed an important task. 2.The benchmark construction pipeline is relatively reasonable: collecting scenarios, generating complete scenario information and workflows based on these scenarios, and designing an evaluation framework. 3.The paper compares evaluation results across multiple settings, including specified/unspecified goals and complete/incomplete information, and reports some useful observations.

Critical weaknesses

1.The evaluation metrics are not sufficiently clear or objective. Although the tasks are fully open-ended, the evaluation criteria lack specificity. For example, in a challenging social situation, both coordinating between the two parties and seeking help from a third party could be reasonable choices, making it difficult to determine which approach is better. 2.The accuracy of the evaluation is unknown. In fully open-ended tasks, LLM biases may have a substantial impact. For example, phenomena such as sycophancy may lead to undeservedly high scores, making it necessary to validate the accuracy of the benchmark's evaluation results. The authors should report the degree of alignment between the benchmark evaluations and human evaluations. 3.There is no overall score. Some evaluation metrics may partially overlap in the dimensions they measure. Without an aggregate score, it is difficult to use this benchmark to compare the capabilities of different models. For example, DeepSeek and GPT differ considerably in their model architectures and capabilities, but their rankings vary across different evaluation dimensions. Consequently, I cannot determine which model is more suitable for open-ended social tasks. One of the most important purposes of a benchmark is to compare the capabilities of different models, but the current version lacks this capability. 4.The NPC mechanism is not described in sufficient detail. Are the NPCs powered by LLMs? If so, which LLM is used? Different NPC models would certainly have different effects on the evaluation results.

Questions for authors

1.What is the underlying mechanism for NPC behavior? 2.How do the authors envision ultimately determining which model is more suitable for open-ended social tasks? 3.How accurate is the proposed evaluation framework?

3 comments
AuthorsOpenReview ↗

Response to Reviewer PZgh (1/2)

Thank you for highlighting the central difficulty of evaluating open-goal social agency.

W1. Objective evaluation when several social choices are valid

We agree that multiple social choices may be valid without being universally ranked. A registered Ending establishes that an agent enacted a social decision, not that it was preferable to alternatives such as reconciliation, disengagement, or seeking help. PraxiScene therefore separates verifiable performance from autonomous social choice.

In the canonical Open Goal + Stepwise Information + Plain Value setting, agents select a course of action, seek information, and act or revise their behavior. UR stories permit multiple initially reachable Endings; CR stories impose history-dependent conditions. Matched Goal and Information conditions test how specified objectives and information availability affect trajectories and outcomes; Given Value tests the effect of an assigned Value orientation.

The three decision requirements are assessed through complementary evidence:

Decision requirementCanonical evaluationControlled evaluation
Goal determinationRealized Ending and Strategy/Value profiles describe the chosen social courseGiven Goal measures specified-outcome attainment through RgoalR_{\mathrm{goal}}
Information acquisitionICI_C and IEI_E describe information engagementMatched Stepwise/Provided Information conditions compare behavior and attainment
Action and revisionRgrdR_{\mathrm{grd}}, RendR_{\mathrm{end}}, NactN_{\mathrm{act}} measure enactment; CstrC_{\mathrm{str}} and CR-only Sstr,PdepS_{\mathrm{str}},P_{\mathrm{dep}} describe correspondence and dynamicsMatched conditions track changes in trajectories and realized choices

Grounding, valid-ending completion, and externally specified Goal/Value attainment (RgoalR_{\mathrm{goal}}, RtargetR_{\mathrm{target}}, ΔRtarget\Delta R_{\mathrm{target}}) have explicit evaluation criteria. Information engagement, action count, Strategy changes, and social-choice profiles describe behavior; they do not independently judge information necessity, revision appropriateness, or the normative quality of a chosen Ending.

On the same 1,037 UR stories, DeepSeek and GPT reach valid Endings in 97.78% and 96.82% of runs but realize different valid Endings in 52.84% of stories. Supplying a Goal increases reference-ending realization by 26.25 and 26.39 percentage points, respectively, while valid-ending completion changes by only 0.69 and 0.83 points. Completion and realized social choice convey different evidence.

W2 / Q3. Evaluation accuracy, human calibration, and judge bias

We agree that LLM consensus alone cannot establish evaluation accuracy. PraxiScene computes executable outcomes from registered events and state transitions; Strategy/Value annotation and free-form action grounding require separate semantic validation. Combined with previous human annotations on values, we conducted two-annotator reviews of sampled 100 Strategy cases and 60 grounding input–mapping pairs, with both annotators examining every case. Reviewers assigned one primary strategy label according to Ending description and provided a judge for each pair of user input and its grounding.

ComponentHuman verificationResults
Value annotationPost-ending questionnaire; 428 responses from 25 participants90.4% agreement (387/428; 95% CI [87.3, 92.9]; participant-clustered CI [83.1, 95.6])
Strategy annotationTwo annotators independently labeled 100 casesOriginal fine label supported by ≥1 annotator in 91% of cases and by both in 54%; Family-level support: 95% / 68%. Individual correspondence: 70% / 75%; inter-annotator agreement: 60% (κ = 0.571)
Action groundingTwo annotators reviewed 60 input–mapping pairsInter-annotator agreement: 93.3% (56/60; κ = 0.845). Nonempty mapping support: 95.6% / 91.1% (43/45 and 41/45)

The human results support the terminal Value annotations and most original Strategy labels, while identifying disagreement in fine-grained Strategy interpretation. For action grounding, both annotators found 14/15 no-match outputs unsuccessful in realizing agent intent. These requests had no corresponding registered event or supported executable action; returning no match was consistent with the environment specification. They reflect the boundary of the supported action space rather than incorrect rejection of executable actions.

The original three-model consensus rates (Value: 99.81%/99.96%; Strategy: 96.64%/93.92%, UR/CR) characterize annotation consistency during benchmark construction. The additional human verification directly examines the resulting labels and action mappings. Registered Ending labels are fixed independently of the tested backbone, and executable outcomes are rule-computed.

AuthorsOpenReview ↗

Response to Reviewer PZgh (2/2)

W3 / Q2. Comparing Social-agent Capabilities without an Aggregate Score

We agree that the original writing did not make the comparison criteria sufficiently clear. PraxiScene supports model comparisons through explicit performance criteria, while separately characterizing the social decisions agents autonomously make. The distinction is important because successfully enacting a social decision and preferring one legitimate social outcome over another require different evaluation criteria.

Specifically, under Open Goal, action grounding and valid-ending completion measure whether an agent can execute a course of action in UR and CR stories. Under controlled conditions, Goal and Value attainment measure whether it can realize a specified outcome. These criteria allow models to be compared under the same decision requirements. Our results illustrate this distinction:

Evaluation criterionDeepSeekGPT
Open Goal, UR valid-ending completion97.78%96.82%
Open Goal, CR valid-ending completion74.88%75.83%
Given Goal + Stepwise Information, UR Goal attainment63.06%66.11%
Given Goal + Stepwise Information, CR Goal attainment46.45%49.29%

DeepSeek has slightly higher observed UR completion, whereas GPT has higher observed Goal attainment in both controlled subsets. Across six ReAct backbones, DeepSeek has the highest UR completion and Gemini the highest CR completion (79.15%). These estimates use their respective eligible story sets. On 438 UR stories where DeepSeek and GPT reach the same Ending, GPT also takes 6.11 fewer actions on average (95% CI [−6.76, −5.46]), providing a matched-outcome comparison of interaction length.

Comparison by a single number is useful when a reader must select one model. Our hesitation concerns what the number would have to mean: it fixes weights over outcomes the benchmark treats as plural, and that weighting is a deployment preference rather than a measured quantity. We therefore report the criterion-specific results above, which a reader can combine under an explicit weighting. The methodological case and current practice are given in our response to TC3y-W1/Q1.

We hope this addresses the reviewer's concern about the comparability of agent capabilities.

W4 / Q1. NPC Mechanism and Its Influence on Evaluation

We appreciate the reviewer's question about NPC behavior and its influence on evaluation. PraxiScene uses a hybrid NPC architecture combining LLM-based interaction with predefined dialogue tasks and event-driven story control. NPCs are powered by GPT-4.1-mini (gpt-4.1-mini-2025-04-14, temperature 0), independently of the evaluated agent backbones.

NPC behavior is governed by the current story state, character-specific knowledge, and predefined interaction requirements. When an NPC has an active task, the task specifies the dialogue content or information to convey. Otherwise, the NPC follows predefined response and behavioral rules included in its prompt. Dialogue generation is further constrained by the NPC's available knowledge and response conditions. This allows natural-language interaction while keeping NPC behavior aligned with the intended story context.

Crucially, NPCs do not independently control story progression or determine evaluation outcomes. Registered events define the conditions for information disclosure, task updates, and state transitions, while the environment executes these transitions and evaluates Ending conditions. Although LLMs support dialogue generation and semantic interpretation, the story structure, executable actions, and outcome criteria are explicitly specified. Compared with fully LLM-controlled simulations, this architecture limits the influence of unconstrained NPC generation on the evaluation process.

We agree that different NPC backbones may affect dialogue realization and consequently influence agent decisions. PraxiScene therefore treats the NPC backbone as a shared environment configuration, rather than varying it with the evaluated agent model. The reported comparisons use the same NPC model and event-execution framework, so observed differences reflect agent performance under a common interaction setting. This design maintains controlled story semantics while retaining the flexibility needed for interactive social scenarios.

Thank you again for your careful reading and insightful suggestions. Please let us know if you have any further questions. If the responses resolve your concerns, we would appreciate it if you could reflect them in your rating.

Thank you again for your careful review. We summarize our responses that address your concerns:

  • Evaluation: We clarified the distinction between valid-ending completion, specified-Goal attainment, and autonomous social choices, with additional model comparisons.
  • Validation: We added human reviews of 100 Strategy cases and 60 action-grounding pairs, and clarified the Value questionnaire and no-match cases.
  • Reproducibility: We provided further details on the annotation models, voting procedures, and NPC configuration.

We hope we have addressed your main concerns and would appreciate your reconsideration of the rating.

Thank you for your time and consideration.

Comments on the paper

1 comment

Dear Reviewers and AC,

As the discussion period is approaching its end, we would like to kindly draw your attention to our responses and follow-up comments. We have made every effort to address the concerns raised and provide additional clarifications and evidence where appropriate.

We would greatly appreciate it if you could take another look at our responses when time permits, and reconsider your assessments in light of the discussion.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.