- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents E2Sim, a framework for long-horizon human behavior simulation that couples evolving actor states with a shared environment, actor-specific observations, and delayed consequences. It also introduces longitudinal diagnostics for consistency, event progression, collaboration, and non-degeneracy. Evaluation includes one 60-day run per method against four direct-generation baselines and 14-day continuation ablations. The clearest observed gains are reduced repetition and higher meaningful-event density; consistency results are mixed.
Critical strengths
-
The framework addresses an important challenge: preserving persistent consequences and differences in actors’ knowledge across extended interactions. Its explicit treatment of shared-world updates and observation-dependent state changes provides a useful design for longitudinal simulation.
-
The evaluation recognizes that locally coherent behavior can still become repetitive or inactive. Assessing non-degeneracy alongside consistency is consequential because conventional plausibility scores may overlook this failure mode.
Critical weaknesses
-
Each method is evaluated only once. The main comparison uses one 60-day run per method from a single initial scenario. The bootstrap intervals reflect variation across actors, not independent runs. It is therefore unclear whether the reported gains would persist with different random seeds or scenario configurations.
-
The baseline comparison is limited. All four baselines are direct-generation variants. The paper does not compare against existing simulation frameworks with persistent memory or environment state. It also does not report comparable inference costs, leaving unclear how much of the improvement comes from the proposed design versus additional model calls.
-
The evaluation needs stronger validation. Most semantic judgments come from an LLM evaluator, without human agreement measurements. The perturbation tests cover selected collaboration errors but provide limited validation for the other metrics. In addition, the IDC procedure sometimes exposes the response when judging whether a state change was expected, which may bias the assessment.
-
Event extraction may affect the main findings. The extractor links 1,276 actions for E2Sim, compared with 1,906–2,298 for the baselines. Event-quality scores thus cover different subsets of the trajectories. Meaningful-event density also depends on how actions are grouped into threads. The paper does not establish whether its conclusions hold under alternative grouping and abstention decisions.
-
The ablations do not test several central mechanisms. They examine outcome sampling, external events, and goal creation, but not shared-world persistence, actor-specific visibility, or delayed consequences. The experiments therefore do not show what these mechanisms individually contribute.
-
The practical value remains unclear. E2Sim reduces repetition and increases meaningful-event density, but does not consistently improve dynamic consistency or collaboration. The exploratory QA study shows that the generated history supports questions, but does not establish whether E2Sim is more useful than other simulators for training or evaluating assistants.
Questions for authors
I would ask the authors to primarily respond to the points raised in Critical Weaknesses, as these most strongly determine my assessment. Beyond those points.
4 comments
Response to Reviewer gPV6, Part 2
Response to Weakness 3
The reviewer asks for stronger validation of the LLM evaluator, and points out that the IDC judge could see the response when forming its expectation. Following this point, we made a minor change to IDC, which checks whether changes in an actor’s state are supported by its history. The judge now (L324–327, L730–733). E²Sim scores 93.36, above Direct (82.49) and Guided (91.57) but below Per-Actor (97.22) and Summary (98.24). Where state stays the same, it matches them (3.94 vs. 3.90–3.93 out of 4).
We made a similar minor change to ECD, which checks whether decisions involving other actors respect dependencies. It now scores sampled decisions that name another actor, using both actors’ records, so it no longer depends on extracted threads (Weakness 4). It also flags repeated requests, overly long waits, and unsupported claims (L337–338, L758–762). E²Sim scores 87.85 vs. 72.22–81.03.
Table R4. Controlled checks of the ECD judge.
| Judge prompt | Injected errors caught | Long waits caught | Harmless edits not penalized |
|---|---|---|---|
| Revised | 9 / 10 valid | 4 / 4 | 3 / 3 |
These checks show that the judge catches injected errors, not that it agrees with people. We did not repeat such checks for revised IDC. The exact-repetition rate in our ablations is computed directly from the text, without a judge. The near-duplicate rate is judged by the evaluator. For human judgment, 19 participants compared anonymous short segments from E²Sim and each baseline in a blinded study, choosing the one more consistent with the history, the one with clearer development, and optionally the more believable one (Table R5).
Table R5. Blinded reading study with 19 participants (E²Sim / baseline / no clear difference).
| E²Sim vs. | More consistent with history | Clearer development | More believable (optional) |
|---|---|---|---|
| Direct | 12 / 7 / 0 | 14 / 3 / 2 | 13 / 5 / 1 |
| Guided | 10 / 6 / 3 | 15 / 4 / 0 | 12 / 7 / 0 |
| Per-Actor | 11 / 5 / 3 | 13 / 6 / 0 | 7 / 7 / 5 |
| Summary | 9 / 8 / 2 | 13 / 2 / 4 | 13 / 2 / 4 |
| Total | 42 / 26 / 8 | 55 / 15 / 6 | 45 / 21 / 10 |
Across 76 judgments (19 participants × 4 pairs), readers more often picked E²Sim for clearer development against every baseline. On consistency, they did not favor the baselines, including Per-Actor and Summary, which score higher on IDC. The study is small, so we treat it as a sanity check on the automatic metrics rather than a full human evaluation.
Response to Weakness 4
We agree that event grouping differs across methods, and will state that event-quality scores cover different action subsets (1,276 vs. 1,906–2,298). We asked the extractor to split 30 sampled actions per method into events twice with the original prompt (A and B), and once each with instructions for coarser and finer splits (Table R6).
Table R6. Segmentation of 30 sampled actions per method.
| Method | Events found (A) | Coarser | Finer | Same split in A and B | Overlap of split points ↑ |
|---|---|---|---|---|---|
| Direct | 97 | 83 | 110 | 8 / 30 | 0.46 |
| Guided | 98 | 78 | 104 | 10 / 30 | 0.53 |
| Per-Actor | 100 | 77 | 139 | 8 / 30 | 0.42 |
| Summary | 85 | 73 | 107 | 11 / 30 | 0.58 |
| E²Sim | 59 | 40 | 89 | 12 / 30 | 0.65 |
How actions are grouped varies for every method. We did not recompute meaningful-event density under these alternatives, so we report it only as a supporting measure and base the collaboration claim on revised ECD, which does not group actions into threads.
Response to Weakness 5
The reviewer asks what shared-world persistence, visibility, and delayed consequences each contribute. Our ablations target random selection, external events and new Stage Goals (Table R7), and we evaluate these three mechanisms differently. Shared-world persistence, visibility and delayed consequences operate on the same shared record (Figure 4), and the baselines lack all three, so the baseline gap reflects their combined effect. We therefore evaluate them jointly, through what they should produce. On handling what others must deliver, E²Sim scores 74.3 vs. 34.3–47.8 for the baselines, and on building on earlier steps 83.6 vs. 37.3–58.2 (sub-scores of the revised ECD). In the Concordia comparison, E²Sim keeps a bank sign-off pending for days, while Concordia resolves it in one action. Runs restarted from day 13 keep the unfinished tasks from the shared record (Table R2). We will state in the paper that these three mechanisms are evaluated jointly.
Table R7. Ablations over days 61–74, rerun from one day-60 state (one run per setting).
| Measure | Full E²Sim | No random selection | No external events | No new Stage Goals |
|---|---|---|---|---|
| Actions that did not go as planned | 49% | 19% | 46% | 47% |
| Responses to outside events per day | 1.14 | 1.86 | 0.00 | 0.93 |
| Time with no active project | 7% | 13% | 11% | 26% |
| Exact repeated actions | 2.0% | 8.0% | 3.3% | 5.6% |
Response to Reviewer gPV6, Part 3
Response to Weakness 6
We agree that the practical value for assistants is the key question. E²Sim improves collaboration and reduces repetition, while IDC is mixed. Our QA study (Table R8) shows that the trajectories require long-horizon memory, but not that they improve an assistant or are more useful than other simulators' outputs. We will narrow this claim. The 80 questions come from the 60-day trajectory: half can be answered from one record, and half need information across long periods or several actors. In the notes condition, the model reads the records in order, keeps its own compressed notes, and answers from those notes plus the last three days.
Table R8. QA accuracy on 80 four-option questions about the E²Sim history (chance 25%).
| Reader model | Full record ↑ | Own notes + last 3 days ↑ |
|---|---|---|
| GPT-5.6 Luna | 92.1% | 51.7% |
| GPT-6 Luna | 96.7% | 50.0% |
Response Follow-up
Dear reviewer, thank you again for your detailed review. As the discussion period is coming to an end, we would like to kindly check whether our response has addressed your concerns. We would be happy to answer any further questions.
Response to Reviewer gPV6, Part 1
Thank you for the careful review and for recognizing that E²Sim addresses persistent consequences and differences in what actors know, that its shared-world updates and observation-dependent state changes are a useful design, and that assessing non-degeneracy alongside consistency matters because plausibility scores can miss repetitive or inactive trajectories. Our main responses:
We have revised the paper to address your comments. Changes are marked in blue, and we cite line numbers in the revised paper below.
Response to Weakness 1:
We agree that with one run per method, the actor-bootstrap intervals do not measure run variance. A full 60-day rerun of every method exceeded our budget. Instead, we restarted E²Sim from two points of the original run, the initial scenario (day 0) and a mid-run state (day 13), and generated three new 10-day trajectories from each. Restarting mid-run also tests whether stability depends on starting from the initial scenario. The revised metrics need seven days of history, so the scored windows are short. Run 2 from day 0 uses a later code version, and the original run is shown for reference only.
Table R1. Repeated runs from two starting points (same scenario and model).
Runs from both starting points stay within IDC 92.8–98.3 and ECD 88.7–91.4. These windows are not comparable to the full-run results, and IDC excluded 3/80 and 18/400 invalid judge outputs. We did not rerun the baselines or test other scenarios and models. The IDC spread across these reruns (about 5.5 points) is larger than the IDC gaps among E²Sim, Guided, Per-Actor and Summary, so we do not read those gaps as differences. The ECD spread (about 2.7 points) is smaller than E²Sim's ECD lead over the strongest baseline (6.8 points).
Reading the day-13 runs, each keeps the unfinished tasks from the shared history, but they end differently: the open item carries over, and its outcome is not scripted (Table R2).
Table R2. A day-13 open item in three new runs: a study whose evaluation jobs had produced no results.
Schedule changes also come with recorded reasons: in two runs, a confirmed Tuesday accessibility test moves to Thursday after a failed sign-in or a missing auditor confirmation.
Response to Weakness 2:
To separate the effect of the design from that of extra model calls, we compared with Concordia (v2.4.0, its planning agent and a game master built from its own components), using the same model, scenario and ten actors. In a traced case, Concordia resolved a bank sign-off and launch in one action, while E²Sim kept them open.
Table R3. Usage over seven simulated days.
E²Sim makes far more calls than the prompting baselines (4,069 vs. 7–71) and uses more tokens (34.6M vs. 0.17–0.32M), so part of its gain may come from additional computation. Costs use reference prices, not invoices.