acceptodds
Under review as a conference paper at ICLR 2027

From Automation to Role-Play: A Population-Level Evaluation of Jev

116 views

Abstract

Can a model designed for fast automation support the role-conditioned decisions needed in a simulated population? We investigate this question in Jev, a newly released System One model that returns structured decisions rather than generated text. Our evaluation proceeds from individual capabilities to population-level behavior, using controlled testbeds to make black-box responses interpretable. PersonaEval and synthetic policy tasks first test role identification and action selection against source-derived or program-computed answers. We then adapt FlockVote into a fixed-population experiment that crosses role representation with shared information. This design distinguishes a change in an initial answer from a change in how the same roles respond to new information. Jev achieves approximately 97% modal-action accuracy on matched policy tasks. Yet, in one historical voting window, converting direct role queries into member-addressed requests changes the aggregate information response by 11.1 percentage points on matched complete records. The corresponding vector interaction remains above a specified tolerance under conservative missing-response bounds. Explicit role instructions and native object inputs leave residual representation effects. Member-level analysis further shows how aligned changes survive aggregation, while cancellation can conceal individual sensitivity. These findings identify a gap between individual decision competence and population-level role-play. They establish concrete validation requirements for using Jev as a shared decision backend before introducing memory, communication, and feedback; the reported probabilities are model responses, not validated human behavioral propensities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
role-playing agents, structured decision models, representation sensitivity, population-level evaluation, agent-based simulation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mtjROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether a model built for fast structured automation can serve as the decision backend for a simulated population, where the same model must make choices on behalf of many heterogeneous roles. Its subject is Jev, a recently released commercial model that returns a selected alternative and a probability vector over supplied options rather than generated text, and its premise is that automation and role-play can share an output format without sharing an objective, so adopting one for the other involves a transfer the interface itself does not validate. It proposes a two-level evaluation: individual capability tests of role identification against source-derived labels and of action selection against program-computed answers, followed by a population experiment that crosses how a role is represented, either queried directly or addressed through a member record, with what information it receives, using two historical windows over a fixed synthetic panel. The central estimand is the interaction between these two factors, which distinguishes a representation change that merely shifts all answers by a constant from one that alters how the same role responds to new information, and the analysis is carried out at member, subgroup, and overall levels with conservative finite-panel bounds for unavailable responses. It is tested with additional controls for explicit role binding, serialized versus native object inputs, co-presented profiles, question count, ordering, and co-member exchanges, alongside a generative model used as an efficiency-oriented reference rather than an architectural control.

Critical strengths
  • The interaction estimand is well motivated and precisely targeted. Querying the same fixed profiles under both representations at both ends of an information window separates a representation change that shifts all answers uniformly from one that alters how a role updates on new evidence, a distinction that matters for simulation use and that a single-condition agreement check cannot detect.

  • The paper is unusually disciplined about what its measurements establish. It bounds missing responses conservatively rather than imputing them, declares several of its own comparisons unresolved when those bounds straddle the tolerance, and labels the generative model as an efficiency reference rather than an architectural control. That restraint makes the surviving claims considerably more credible.

Critical weaknesses
  • The entire empirical object is one hosted commercial model on a mutable alias, which the paper concedes may not return identical responses on repeated calls, with no architectural control. The conclusions therefore attach to a product at a moment in time rather than to models or methods, and the striking slope contrast in Figure 2 is uninterpretable by the paper's own standard.

  • Incomplete coverage stops a large share of the planned experiment from concluding, and the subsets that do conclude may be selected by the effect itself. Since a missing response may depend on which representation was used, the retained complete cases could be biased toward the measured interaction, a possibility the paper does not test. With no inferential statistics anywhere, only the one headline interaction carries a real guarantee.

Questions for authors
  • Response availability may itself depend on the representation, which would make the complete-case subsets endogenous to the effect you measure. What are the per-condition missing-response rates for the direct and member-addressed forms, and does the retained complete-case subset differ systematically from the dropped members on profile attributes or state?
Reviewer_34HaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary
  • The authors are interested in whether Jev, a model that was released on September 15, could be used for agent-based simulations, and seeks to answer this by evaluating its ability to make choices based on an individual persona that it is given.
  • They directly compare Jev to one other model (Deepseek Flash) on two roleplaying-based tasks (action selection from an explicit policy and persona identification, adapted from PersonaEval). They then check both models’ response to new information when simulating a voter persona in the 2024 American Presidential Election (a setting adapted from FlockVote).
  • While Jev seems to have moderate base capabilities needed for role-playing based tasks, the authors find mixed results in its application to agent-based models of the 2024 election; specifically, they note a significant difference in information response based on the format of the persona record.
Critical strengths

None

Critical weaknesses
  • The authors do not offer a compelling explanation of why Jev would be expected to be better or worse than LLMs on their evaluations. Additionally, because the architecture and training process of Jev are not publicly documented, and the authors only compare Jev to one other model, the paper does not contribute any insights into what might make a model better or worse at such tasks.
  • It would be better to frame the paper as a general benchmark than a Jev-specific analysis given this, but the benchmarking part of the paper appears to be a narrow extension of the PersonaEval and FlockVote work that the authors build off of. Additionally, the main result seems to be downstream of models' sensitivity to prompt formatting, which has been extensively studied (e.g. by the Sclar work cited). As such, the current paper is not ready for acceptance as either a benchmarking paper or an analysis of Jev.
  • The paper is incredibly dense and difficult to read. New technical terms are frequently invented without sufficient definition and only used sparingly (e.g. “member-addressed request”, “inherited binding”, “same-slot exchange”). Additionally, results are frequently presented in quick succession without any further analysis, making the conclusions a reader is supposed to take away from this paper incredibly hard to keep track of (e.g. “The reference model provides a better-covered example. Its 432 complete members change by .04239 on average, while the aggregate changes by .00294. The full-panel upper bound, .01507, is below .02. Yet Wisconsin’s upper bound is .08615, so overall agreement does not establish agreement for every state. At the next information condition, even the overall upper bound rises to .02976. These are limits of the agreement claim, not proof of a subgroup error or a paired backend ranking.”).
Questions for authors
  • Why are missing response rates from Jev so high (e.g. Table 8 seems to suggest that only around 55-60% members have complete records from Jev), and what kinds of prompts are causing so many missing responses?
  • Why is 0.02 used as the tolerance value? It’s not explained anywhere in the paper - are conclusions robust to choice of tolerance value?
Reviewer_cUn8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper evaluates whether Jev, a fast structured decision-making model can simulate heterogeneous agents in a population. It leverages personaeval, synthetic policy tasks, voting simulations, with Jev achieving 97% accuracy on individual policy changes. The responses, however, significantly changes when roles are represented differently. The authors claim that the main findings the how strong individual decision performance does not guarantee consistent population level role-playing behavior.

Critical strengths
  • The broader research question is interesting in that it addresses whether we can feasibly utilise LLMs to simulate population.
  • The evaluation is careful to distinguish multilevel aggregation analysis, such as individual, subgroup and population-level sensitivity.
Critical weaknesses
  • This seems to be an AI generated paper. I found it very hard to read, with heavy presentation issues. The paper's writing is hard to unpack, with many different AI-specific phrasings as well. I think the work may not benefit the community if published the ICLR, given how its degree of unintelligibility.
  • I think this paper also doesn't really meet the high bar of ICLR in terms of the rigor of the experiments, nor the experimental findings. It simply is focused on evaluating Jev's proficiency in specific simulation settings, rather than contributing anything new with respect to the methodology or evaluation benchmarks etc. So, in this vein, its contribution is hard to recognise.
  • Building on the seperate point, the experiments focus on just this one model (Jev), and only 2 datasets, and yet definitively claims that strong individual decision performance does not guarantee consistent population-level role-playing behavior. Rigor aside, the central finding seems rather trivial as well.
Questions for authors

Weaknesses are specified above. Additional questions below:

  • Can the authors expand the work by evaluating more models and more datasets? To provide a larger contribution for population simulation?
  • Can the authors expand the experimental setups and analysis to produce more findings that can benefit the community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.