acceptodds
Under review as a conference paper at ICLR 2027

SimLife: Pattern Understanding for Long-Horizon Human-Agent Partnership

Abstract

Understanding humans over long horizons requires agents to infer not only what people need in the moment, but also how routines form, why they repeat, and when they change. We introduce SimLife, a scalable platform for simulating long-term household life with rich visual observations, ground-truth action logs, and synthetic dialogues with audio. Built on SimLife, SimLife-BP evaluates long-context pattern understanding: the ability to infer latent behavioral rules from weeks or months of everyday observations. The benchmark contains 106 episodes averaging 15.49 hours and 38.57 in-game days, and 1,439 question-answer pairs. Each task probes direct, counterfactual, noisy, and inverse reasoning under different levels of rule hints. Evaluating frontier models and architectures, we find that current models often achieve surface-level prediction without comprehensive rule understanding, rely on frequency-based heuristics rather than if-then reasoning over evidence, and struggle to adapt when behavioral patterns change. These findings suggest that long-context pattern understanding remains a major bottleneck for future embodied agents, while SimLife opens a broader space for studying memory, personalization, adaptation, and planning in everyday human-AI interaction.

Primary area:
datasets and benchmarks
Keywords:
Long Context Learning, Long Video Understanding, Life Simulation, Pattern Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KmyBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether models can infer and adapt human behavioral patterns from long-term observations, motivated by the needs of persistent human–agent interaction and the limitations of retrieval-focused evaluations. Its main contributions are SimLife, a household simulation and data-generation platform built on The Sims 4, and SimLife-BP, a benchmark comprising 106 episodes and 1,439 questions covering multiple reasoning types and hint levels. Results suggest that tested models often rely on surface heuristics, apply rules inconsistently, and struggle with pattern changes.

The work offers an interesting evaluation perspective, substantial infrastructure, and generally clear presentation. Its rule-based construction and hypothesis-elimination procedure provide a useful foundation for controlled evaluation. However, Tested model coverage is limited, the practical connection between recognizing behavioral patterns and providing useful assistance remains untested by the prediction-focused tasks.

Overall, the paper presents a promising benchmark, but stronger validation and experimental controls would strengthen its broader conclusions.

Critical strengths
  1. The paper takes a relatively novel perspective on evaluating models’ understanding of human behavioral patterns over extended periods. Its emphasis on discovering latent rules, applying them consistently across different reasoning tasks, and adapting to behavioral changes provides a valuable direction for long-context evaluation.

  2. The paper develops SimLife, a platform built on The Sims 4 for constructing long video sequences of household activities with synchronized observations and annotations. The pipeline translates predefined behavioral rules into activity constraints and assembles matching recorded days into sequences. This constitutes a substantial infrastructure contribution and provides a controllable testbed for studying long-term behavioral understanding.

  3. The paper is well organized and clearly written.

Critical weaknesses
  1. Coverage of strong proprietary models is limited. The reported coverage is concentrated on Flash/Lite variants. Additional SOTA reasoning models would make it easier to assess whether the benchmark exposes limitations that persist across leading systems. Without broader coverage, the generality of the conclusions about current models remains uncertain. Expanding this comparison would strengthen the evidence for the benchmark’s diagnostic value.

  2. The connection to useful long-term assistance remains underexplored. The benchmark evaluates inference and application of behavioral rules through prediction and counterfactual questions, but it does not directly assess whether this understanding enables an agent to provide useful assistance. Given the paper’s framing around long-horizon human–agent partnership, an additional evaluation of assistance decisions—for example, choosing an appropriate intervention based on an inferred routine—would strengthen its practical relevance. The current experiments support conclusions about behavioral pattern understanding; the broader connection to effective partnership remains to be demonstrated.

Questions for authors
  1. Beyond constraint-based matching, how extensively were the selected video sequences independently checked against the predefined behavioral rules? Could you report the verification coverage and any observed rule–video mismatch rate?

  2. How are the rendered questions and their answer labels verified to ensure there is a unique correct answer, particularly for inverse counterfactual questions?

  3. Could you evaluate additional leading reasoning models from multiple providers? How do they perform on SimLife-BP, and do the reported limitations in rule inference and adaptation persist across these models?

2 comments
AuthorsOpenReview ↗

Response to Reviewer KmyB

We thank the reviewer for the thoughtful feedback and for recognizing the novelty of evaluating long-term behavioral pattern understanding, particularly rule inference, application, and adaptation. We also appreciate the acknowledgment of our substantial platform contribution, controllable rule-first generation pipeline, and clear presentation.

W1. Coverage of proprietary models We respectfully disagree that our evaluation lacks sufficient coverage to demonstrate the benchmark's diagnostic value. We have already evaluated three strong proprietary models, alongside multiple open-source models, and observed substantial limitations in their ability to infer and adapt to long-term behavioral patterns.

While evaluating additional leading reasoning models (e.g. other model families) would further strengthen the generality of our findings, we believe the current results already provide sufficient evidence of the challenges posed by SimLife-BP. Given our computational and financial constraints, we prioritize a systematic analysis of these limitations over expanding model coverage. We acknowledge that our conclusions do not necessarily generalize to all leading reasoning models.

W2. Connection to useful long-term assistance We agree that evaluating the usefulness of inferred behavioral patterns for long-term assistance is an important next step. However, we believe that establishing a controllable platform and systematically understanding models' capabilities and limitations in long-term behavioral reasoning are fundamental steps before moving toward more complex human–agent interactions and decision-making over ultra-long horizons.

Evaluating effective ultra-long-term assistance requires carefully defining the tasks, interaction scenarios, expected agent behaviors, and evaluation metrics, which constitutes a substantial research problem in its own right. Rather than introducing an oversimplified assistance task (e.g. asking the model “should you serve coffee today?”) that may not adequately capture the complexity of long-term human–agent partnership, we chose to focus on establishing and evaluating the underlying behavioral understanding capabilities. We see assistance-oriented evaluation as a natural and important direction for future work, but it is beyond the scope of this paper.

Q1. Verification of video sequences Beyond constraint-based matching, we checked 71 video chains out of 106, comprising 2,863 days. We found that 197 days (6.9%) contained one visibility mismatch, where an event recorded in the logs was either not visible or visible for only about one second, making it easy to miss at 0.5 FPS. These discrepancies mainly arise from the limited temporal precision of game ticks, which can cause delays in updating character positions and facing directions, which are used in visibility calculation. This mainly affects V+L and V+A variants. We are continuing to verify and improve the dataset to minimize such issues.

Q2. Question and label verification Answer labels are determined by the predefined deterministic rules rather than LLM judgments. For inverse counterfactuals, we select an alternative Day Condition Set whose outcome is opposite to the observed one, and construct the two options by negating the relevant condition statement while keeping all other conditions fixed.

For example, consider a deterministic pattern in which B occurs whenever A occurs. Suppose both A and B occurred on the observed day. We ask, "If B had not occurred, which statement would be more likely to be true?" with the options "A occurred" and "A did not occur." Under this hypothetical premise, only "A did not occur" is consistent with the rule. Uniqueness is guaranteed by construction, as we only select conditions whose negation would contradict the hypothetical outcome. We will clarify these construction constraints in Appendix C.2.

Q3. Additional reasoning models Please refer to our response to W1 regarding model coverage.

I appreciate the authors' reponse, I will maintain the rating.

Reviewer_U3qCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents SIMLIFE, a long-term household-life simulator built on The Sims 4, and SIMLIFE-BP, a benchmark of 106 episodes and 1,439 binary questions that tests whether models can infer deterministic if-then behavioral rules from ultra-long multimodal observations. Episodes are assembled rule-first from pre-recorded days. Each task combines four question types (direct, counterfactual, noisy, inverse) under three hint levels, and query points are set by hypothesis elimination. Across vision, audio, and text inputs, models are near chance in the multimodal settings. Most models track frequency heuristics rather than hypothesis elimination, and accuracy drops after pattern shifts.

Critical strengths
  1. A clear and meaningful problem, backed by a substantial platform that would benefit the community if open-sourced. The paper clearly distinguishes long-context retrieval from cross-temporal latent rule inference. The latter is a capability that existing long-video and long-context benchmarks, which mostly target localization or retrieval-style QA, rarely cover, and it bears directly on long-term companion assistants, embodied agents, and personalization.

  2. A rigorous rule-first generation pipeline, with question design that effectively exposes shortcuts. Defining the rule first and then assembling video chains from a pool of pre-recorded days makes the ground-truth labels verifiable. The four question types, combined with the All-Correct (AC) metric, distinguish answering forward questions correctly from genuinely mastering the rule.

Critical weaknesses
  1. The multimodal setting barely discriminates between models and conflates perception with reasoning. Under V+L and V+A, every model lands at 48–55% Acc, with most differences inside the confidence interval. All of these settings pass through the same low-resolution Socratic diary pipeline, and Appendix concedes that the diaries often lack the deciding evidence. The results therefore mainly reflect the recall of the summarization pipeline rather than the models' reasoning. The paper neither evaluates a model with native long-video input nor reports how often the diaries capture target and condition events. Nearly all informative conclusions thus come from the text-only L/L-N settings, where the task differs little from existing long-context induction benchmarks

  2. Missing reference baselines make the absolute numbers hard to interpret. With four binary questions per task, chance-level AC is about 6.25% (12.5% for Hist tasks). Table 1 does not list this, and the 7.6–14.7% AC observed under V+L is not far above it. In addition, the frequency baseline and hypothesis-elimination solver already implemented for RQ2 are absent from Table 1. Under partial hints with L-N input, a symbolic hypothesis-elimination solver should score near-perfectly by construction. Including it would show that the gap stems from LLM reasoning rather than from the task being unsolvable.

  3. Missing reference baselines make the absolute numbers hard to interpret. With four binary questions per task, chance-level AC is about 6.25%. Table 1 does not list this, and the 7.6–14.7% AC observed under V+L is not far above it. In addition, the frequency baseline and hypothesis-elimination solver already implemented for RQ2 are absent from Table 1. Under partial hints with L-N input, a symbolic hypothesis-elimination solver should score near-perfectly by construction. Including it would show that the gap stems from LLM reasoning rather than from the task being unsolvable.

  4. Label uniqueness for inverse counterfactuals is questionable. Under an OR rule, a premise of "the activity occurred" can be consistent with multiple condition combinations. In the chess example, if the day is a weekend, both "played violin yesterday" and "did not play violin yesterday" are compatible with chess being played. Appendix C.2 only says the answer "follows from matching the premised outcome against M(d_inv)" and does not explain how such multi-solution cases are excluded. The phrasing "which statement is more likely to be true" is also ambiguous under deterministic rules. Notably, several L-N models score systematically below chance on this question type (e.g., Gemini-3 at 42.2% with full hints, Qwen3.5-9B at 41.9% with no hints), which usually signals a problem with the items or labels. The authors should report human-validated agreement for these questions.

Questions for authors
  1. How many distinct phenomena (rules) do the 106 episodes correspond to? When the same rule is instantiated as multiple chains with different start dates or trigger samples, are these treated as independent samples in the statistics?
  2. In the V+L setting, what is the recall of target activities and condition events in the generated diaries? Could you add an end-to-end result with native video input to separate the perception bottleneck from the reasoning bottleneck?
  3. How is label uniqueness ensured for inverse counterfactual questions? Are there items where multiple condition combinations are compatible with the premise? Were these items validated by humans, and what is the agreement rate? How do you explain several L-N models scoring systematically below 50% on this question type?
  4. Why does Gemini-3-flash-preview perform worse under L-N than under L across the board? Could the L-N filtering remove context that this model relies on (e.g., cues outside day boundaries), or change the prompt format in a way that matters?
  5. In RQ2, what are the partial correlations of each model with the frequency and hypothesis-elimination baselines after controlling for ground truth? Do the conclusions still hold when comparing only on days where the baselines disagree?
3 comments
AuthorsOpenReview ↗

Response to Reviewer U3qC

We thank the reviewer for the thoughtful feedback and for recognizing the importance of studying cross-temporal latent rule inference, which bears directly on long-term companion assistants, embodied agents, and personalization. We also appreciate the acknowledgment of our rule-first generation pipeline and question design, particularly their roles in ensuring verifiable ground truth and evaluating genuine rule understanding.

W1. Multimodal evaluation and the perception bottleneck We agree that multimodal performance is strongly affected by the quality of the generated summaries. However, this is also part of the capability we aim to evaluate: over weeks of observation, can a model identify and remember the details needed to infer behavioral patterns? The near-chance results and limited differences between models highlight the difficulty of this process.

Regarding native long-video evaluation, at 0.5 FPS and 224 × 224 resolution, a full SimLife episode would require approximately 683K visual tokens on average and 2.13M for the longest episode with Qwen3-VL, even before adding dialogue or questions. This exceeds its native 256K context window, and even a 1M-token extension would be insufficient for the longest episode. More aggressive sampling risks missing brief but critical events. Native video mode also imposes stricter duration constraints (e.g., Gemini-3-Flash supports roughly one hour, while most of our episodes exceed this duration). We additionally tested Qwen3-VL using native video mode and found that it produced substantially worse summaries than processing sampled frames.

We therefore adopt a Socratic protocol, following prior work (Zeng et al., 2023; Zhang et al., 2024), where each model processes every window up to the question and answers from its own summaries. This provides coverage across the full timeline and a controlled comparison without designing model-specific memory or retrieval systems.

The L and L-N settings further isolate reasoning performance by providing the event evidence directly. Although they remove the visual perception challenge, they keep the same long-horizon behavioral structure, including distributed observations, contextual dependencies, and evolving patterns, allowing us to investigate how evidence quality affects rule understanding. We would also appreciate references to the text-based rule-induction benchmarks the reviewer has in mind, so that we can make more direct comparisons and further clarify our novelty.

Zeng et al. Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language. ICLR. 2023.

Zhang et al. A Simple LLM Framework for Long-Range Video Question-Answering. EMNLP. 2024.

W2 & 3. Reference baselines and interpretation of AC Thank you for the suggestion. The chance-level AC is 7.7% across our benchmark, rather than 6.25%, because some tasks contain three instead of four questions. We will include this reference in Table 1. We also computed a majority-class baseline, which achieves 56.8% Acc / 12.3% AC overall, with per-question-type accuracies of 51.5% (Direct), 57.2% (CF), 54.6% (Noise-CF), and 63.2% (Inverse-CF). We also identified an answer imbalance in Inverse-CF and will fix it in the final revision.

The frequency and hypothesis-elimination baselines in RQ2 are designed for day-by-day direct predictions, rather than the four question types jointly evaluated in RQ1, so they are not directly applicable to Table 1. More importantly, the symbolic hypothesis-elimination solver mentioned by the reviewer is already the foundation of our task construction, as described in Appendix D. We insert tasks only after hypothesis elimination guarantees that all remaining hypotheses agree on the ground-truth predictions. Therefore, applying the same procedure as a baseline would yield 100% accuracy by construction under Partial and Full Hints. It serves as an answerability guarantee rather than an independent baseline for evaluating model performance.

W4. Label uniqueness and inverse counterfactual reasoning We thank the reviewer for raising this concern. We agree that Appendix C.2 does not sufficiently explain two important construction constraints that ensure label uniqueness.

First, the alternative Day Condition Set dinvd_{\mathrm{inv}} is selected only when its outcome is opposite to the observed outcome, i.e., M(dinv)≠M(dobs)M(d_{\mathrm{inv}}) \neq M(d_{\mathrm{obs}}). Otherwise, no inverse-counterfactual question is generated. Second, the two options are constructed from the conditions that differ between dobsd_{\mathrm{obs}} and dinvd_{\mathrm{inv}}, while all other conditions preserve their observed values. Therefore, the distractor reconstructs dobsd_{\mathrm{obs}} and contradicts the counterfactual premise, whereas the correct option reconstructs dinvd_{\mathrm{inv}} and is consistent with it. For the reviewer's OR-rule example, if the hypothetical premise is that chess was played, the observed day must have been one when chess was not played, meaning it was not a weekend and violin was not played the previous day. Since the weekend condition remains fixed, only the option stating that violin was played the previous day is consistent with the hypothetical premise. We also double-checked the labels and model reasoning traces. Our checks found the labels consistent with the rules, while some reasoning traces suggest that models struggle to evaluate counterfactual conditions that contradict the observed history.

Q1. Distinct Phenomena The 106 episodes correspond to 106 distinct phenomena, one-to-one. No rule is instantiated more than once with a different start date or trigger sample, though it is doable. If character identity and specific weekdays are abstracted away, 102 distinct rule templates remain; the four near-duplicate pairs differ only in which character performs the activity or on which weekday.

Q2. Recall in V+L diaries and native video input. We thank the reviewer for this suggestion, which sharpens our claim about where the bottleneck lies.

We measured how often the generated diaries mention the target activity and the condition events, attributed to the correct character, on days when the event actually occurred. Ground truth comes from the visually logged game actions, and we used every history day visible to the QA stage, i.e. every day before the query day, across all episodes. Mentions were identified by keyword matching; for computer-based activities (e.g., writing or gaming at a computer) we also accepted generic descriptions such as "working on a laptop", from which the activity can be partially inferred. Recall is limited across all models: target-activity recall ranges from 17.7% (Gemma-4-26B-A4B-it) to 39.8% (Gemini-3-flash-preview), and condition-event recall ranges from 14.7% to 35.6%.

Adding the per-window fine-grained action lists, which the QA stage does not receive, raises recall by up to 12 points (1–12 across models), yet recall remains below 46% for every model. This suggests that, while some evidence is lost during summarization, a substantial portion of the relevant events is not captured at the perception stage. Together with the large gain from V+L to L, which replaces perception with ground-truth logs, these results are consistent with our interpretation that perception is a major bottleneck for V+L performance.

Q3. Label uniqueness of inverse-counterfactual (ICF) questions. In our generator, the ICF premise inverts the observed outcome of the query day. The correct option is the condition assignment from a different observed day whose outcome matches the premise. The generator verifies that the option preserving the actual conditions is inconsistent with the premise. It does not exhaustively enumerate all condition combinations compatible with it, and it does not have to.

On the sub-50% scores: these are systematic, not random. By design, the correct ICF answer always contradicts what actually happened on the query day. Inspection of reasoning traces shows that models frequently treat the observed conditions as fixed, e.g. "even if she had not played the piano, the date remains Monday", which leads them to pick the factual side. This reflects a failure to reason abductively from an outcome back to its likely cause, which is precisely the ability ICF is designed to probe.

Q4. Gemini-3-flash-preview under L-N vs L. The degradation is confined to inverse-counterfactual questions (61.4% → 42.2% under Full Hints). Direct and noisy-counterfactual accuracy actually improved (+4.6 and +11.0 points). The cause is the same factual-anchoring behaviour discussed in Q3. In the compact L-N context, the query day's weekday and observed events are highly salient. Gemini-3-flash-preview also adapts its reasoning length to input length: median reasoning tokens on ICF questions fall from about 2,450 to about 140. Reasoning traces that settle the question from today's facts alone ("today is Thursday, so … regardless") rise from 7.6% to 28.6% of ICF responses.

Q5. Partial correlations and baseline-disagreement days.

We thank the reviewer for this control. Ground truth (GT) is strongly correlated with the hypothesis-elimination (HE) baseline (ρ = 0.79). A model's correlation with HE may therefore simply reflect its accuracy. We report partial Spearman correlations controlling for GT, with 95% CIs from a chain-level bootstrap:

Modelρ(Freq | GT)ρ(HE | GT)Δ (HE − Freq)
Qwen3.5-35B-A3B23.713.6−10.1 [−16.4, −3.7]
Gemma-4-26B-A4B-it25.18.8−16.3 [−23.5, −7.1]
Gemini-2.5-flash33.720.4−13.3 [−18.3, −7.0]
Gemini-3-flash-preview18.817.4−1.4 [−7.7, 5.0]
Human6.047.7+41.7 [9.7, 66.5]
  • Frequency reliance holds. For Qwen, Gemma and Gemini-2.5-flash, the frequency association remains significantly stronger than HE after controlling for GT. On the 939 days where the two baselines disagree, these models show no tendency to follow HE.
  • The Gemini-3-flash-preview claim needs revision. Its higher raw HE correlation is largely explained by its higher accuracy. Given GT, it is equally associated with both baselines. On disagreement days, its tilt toward HE (+18.0 pts) vanishes once GT is held fixed (+1.7, CI [−8.4, 11.6]).
  • Humans remain clearly HE-aligned under both controls.

We will revise the paper accordingly. Gemini-3-flash-preview is less frequency-driven and better aligned with GT than the other models, but it shows no evidence of hypothesis elimination beyond its accuracy. Humans are the only source with a clear hypothesis-elimination signature.

Reviewer_zWLaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary
  • Simlife is a pipeline built on the sims 4. it records 466 simulated days from a in-game household robot’s egocentric view
  • adds synthesised and spatialised dialogue, and then works “rule-first”: it specifies a deterministic latent rule, either temporal or trigger-based with at-most 2 conditions merged by OR/AND. It then concatenates pre-recorded days that satisfy the rule’s per day constraints
  • the resulting benchmark simlife-bp has 106 episodes (86 static patterns, 20 "dynamic" where the rule changes mid-episode). episodes average ~38.6 in-game days (~15.5 hrs of video) and go up to 120 days. in total there are 381 tasks / 1,439 binary questions
  • they evaluate open (qwen3-vl, qwen3.5, gemma 4) and proprietary (gemini 2.5/3 flash) models, plus two agentic baselines (videotree, m3-agent). there are 4 input settings: video + dialogue transcripts (caption each window, then answer over the captions), video + raw audio, raw text logs, and oracle-filtered “noiseless” logs
  • main findings: with video input most models are near chance (~48–55% accuracy; I have doubts about this raised later). performance goes up as the input gets cleaner. the best result is gemma-4-31b on noiseless logs with full hints (74.6% accuracy, 36.2% all-correct rate, i.e. all questions in a task right). counterfactual questions are much easier than inverse counterfactuals, which stay near chance even with oracle logs
Critical strengths
  • the rule-first generation is the main contribution. the authors specify the latent rule first and then assemble pre-recorded days that satisfy it, instead of recording and annotating afterwards. this makes the per-day labels correct by construction and gives a full truth table (the "day condition set") for every episode, which is what makes counterfactual questions possible at all. it is also a reusable recipe for building controlled long-horizon benchmarks beyond this paper

  • the four question types are coupled per task and scored with the all-correct rate. this guards against a model scoring well on forward prediction alone without actually having the rule. the resulting gap between counterfactual and inverse counterfactual accuracy is an informative finding in itself

  • the evaluation goes beyond final accuracy. day-by-day belief trajectories compared against interpretable baselines (frequency, last-day, hypothesis elimination) and humans is a nice probing approach. the reasoning-trace analysis in app. it gives concrete failure modes (misattributing activities to the wrong sim, treating "today" as the previous day, not updating after a change) that are useful for the community

  • Paper is quite detailed about the pipeline and as far as I can tell documents generation, dialogue synthesis, annotation tools and prompts in the appendix.

Critical weaknesses

W1. Vision results are testing a captioning bottleneck and not really testing long video understanding

  • Every V+L model runs through a socratic pipeline. Frames are sampled at 0.5 fps at 224×224, and each 4-hour game window is compressed to about 180 words. Phase 2 seems to never see the frames themselves.
  • So the near chance V+L performance mainly just measures whether the captioner happened to record the target and trigger events.
  • the framing in the paper of an ultra long multimodal benchmark isn’t really supported by native long-video eval.
  • Can the authors run any of the latest models natively on a subset?
  • what fraction of target and trigger occurrences in the ground-truth logs appear in the diaries

W2. Answerability guarantee is defined on GT labels, not on what is observable

  • compounded with the above problem. in V+L, an off camera target or trigger maybe neither visible nor mentioned in the dialogue

W3. Is the benchmark’s "long-horizonness" actually established

  • It seems based on table 18, the weekday rules converge in ~7 days while episodes average 38 days (?)
  • There is no ablation that gives the model only the last 'n' days. If a 2-wk suffix matches full context accuracy then the long-context claim isn’t relevant
  • relatedly, depending on the truncation settings, it could remove the evidence the convergence guarantee relies on

W4. Inverse Couterfactual accuranct sits at or below chance, even with oracle logs. For example, Gemini 3 flash L-N with full hints scores 42.2 much below 50%

  • I am not sure this translates to a reasoning failure as much as it does a semantical mismatch:
  • the correct option contradicts the factually observed condition, i.e., the items require a backtracking reading of the counterfactual, which is known to be a non-default interpretation for humans (Sloman & Lagnado, 2005; Rips, 2010), and could be so for LLMs (https://aclanthology.org/2023.acl-short.70/, https://aclanthology.org/2024.naacl-long.102/) where counterfactual reasoning is fragile if not non-default.
  • can the authors report (i) the faction of inverse counterfactual items whose correct option contradics the observed Day Condition set, (ii) accuracy split by that property, (iii) human accuracy on the inverse counterfactual items, (iv) the answer-position distributions?

W5. The noise-condition selection test is described as n independence test, but requiring all four contingency cells to be non-empty only rules out XX as a deterministic single variable predictor of A.

  • it doesn’t actually establish statistical independence (sample counts: 10, 1, 1, 18; odds ratio=180)
  • Based on Fig 3. it seems the reasoning is expected to solve the N-CF items by checking for correlation in the observed data, a strongly associated X would make the 'unchanged' label contradict sound inference especially under the "No Hints" case, where X is not excluded by the hypothesis space (rightly so).

Refs:

Sloman, S. A., & Lagnado, D. A. (2005). Do we "do"? Cognitive Science, 29(1), 5–39.

Rips LJ. Two causal theories of counterfactual conditionals. Cogn Sci. 2010 Mar;34(2):175-221.

Questions for authors
  1. Will the dataset, generation code, mods and eval harness be released? What are the terms of redistribution for Sims 4 recordings?

  2. The CIs treat all ~1400 questions are independent, is that right? I understand the questions to be clustered within the tasks and tasks within the 106 episodes. Can the authors report episode level bootstrap CIs (resample with replacement), chance-level all correct rate, majority class baselines per question type?

  3. The benchmark is built around several design factors but I think results are only reported in aggregate (unless I missed something of course). Is it possible to break down the performance by:

  • "vision-only" vs dialogue required episodes
  • temporal vs activity conditions
  • episode length?
  1. Tab 14 says partial hints convey "condition categories + logic" although the rendered examples re the hypothesis space in Appx D suggest the logic is maybe not given. Can the authors clarify which is it and whether the hint phrasing was ablated? especially given Appx I.

  2. How is the opposite statement built for the inverse counterfactual reasoning and is a unique answer guaranteed?

7 comments
AuthorsOpenReview ↗

Response to Reviewer zWLa

We thank the reviewer for the detailed and thoughtful feedback. We appreciate their recognition of three contributions in particular. Our rule-first approach makes it possible to construct long episodes with known behavioral patterns, the four question types test whether a model can apply the same rule consistently, and the day-by-day analysis reveals how models and humans update their predictions as evidence accumulates.

W1. What the vision baseline measures. If we understand the reviewer correctly, “native long-video evaluation” refers to either sampling frames at a fixed rate and answering questions directly within a single context window, or using the model's native “video” mode to process the entire video rather than a sequence of frames. Prior work has adopted the former approach in evaluating models on 1H-VideoQA (Gemini Team, 2024), Video-MME (Fu et al., 2025), and LVBench (Wang et al., 2025).

However, at 0.5 FPS and 224 × 224 resolution, a full SimLife episode would require approximately 683K visual tokens on average and 2.13M for the longest episode with Qwen3-VL, even before adding dialogue or questions. The average already exceeds its native 256K context window, and even a 1M-token extension would be insufficient for the longest episode. More aggressive frame sampling to fit these histories into a single context window risks missing brief but critical events that distinguish one behavioral rule from another. Meanwhile, native video mode imposes stricter constraints on video duration (e.g., Gemini-3-Flash has a roughly one-hour limit, while most of our episodes exceed this duration). We also experimented with the Qwen3-VL series using its native video mode and found that it produced substantially worse summaries than providing the video as a sequence of sampled frames.

We therefore use a Socratic protocol, following prior work on egocentric video QA (Zeng et al., 2023) and long-range video QA (LLoVi, Zhang et al., 2024). Each model processes every window up to the question at the same sampling rate and answers from its own summaries. This gives us a controlled comparison across tasks without designing a separate memory or retrieval system for each model. We agree that performance depends on which events survive summarization. This is part of the capability the benchmark measures: over weeks of observation, can a model record the details it will later need to infer a routine? Our log-based evaluations then test how well the models infer the rule when the event evidence is supplied directly.

Gemini Team. Gemini 1.5: Unlocking Multimodal Understanding across Millions of Tokens of Context. arXiv preprint arXiv:2403.05530. 2024.

Fu et al. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. CVPR. 2025.

Wang et al. LVBench: An Extreme Long Video Understanding Benchmark. ICCV. 2025.

Zeng et al. Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language. ICLR. 2023.

Zhang et al. A Simple LLM Framework for Long-Range Video Question-Answering. EMNLP. 2024.

W2. Observability of the evidence. The answerability guarantee is defined over the ground-truth history, and observability is part of the history. As described in §3.1 and Appendix E.2 (“Invisible activity completion,” p. 32), when a relevant activity occurs off camera or is occluded, we add a reference to it in same-day dialogue. For example, after exercising off camera, Father Sim might say, “I just finished my workout.” Appendix E.4 describes our manual spot checks, and Appendix F.3 (p. 35 and Fig. 8) explains how annotators inspect the video to check activity visibility when placing questions.

W3. How much history is needed? We may have been unclear about the distinction between episode length and the observation history available at each question. The seven days in Table 18 refer to a simplified weekday rule, while more complex rules and combinations of conditions can require longer histories to distinguish. Furthermore, we include days after convergence to ensure that every Day Condition Set occurs and can be queried (Appendix D.2). The 38.6 days represent the average length of a complete episode, not the history required for every question. In practice, we select query points as close to convergence as possible, once the required Day Condition Sets have been observed, rather than unnecessarily extending the observation history.

Regarding the proposed last-n-days ablation, we do not believe that arbitrarily truncating the history would provide a meaningful comparison. Our task construction explicitly guarantees that the accumulated observations are sufficient to determine a unique answer under Partial and Full Hints. Removing earlier observations can invalidate this guarantee, leaving multiple hypotheses consistent with the remaining history. In such cases, lower accuracy could simply reflect insufficient evidence rather than a failure of long-horizon reasoning, while similar accuracy would not necessarily indicate that earlier observations were unnecessary, particularly when model performance is already near chance.

A meaningful truncation comparison would first need to verify that the truncated history still provides sufficient evidence for each question. Since our questions are already placed near the earliest point at which this guarantee is satisfied, we do not consider an unconditional fixed-window ablation, such as watching only the last two weeks, an appropriate evaluation of the benchmark's long-horizon requirements.

W4. Interpretation of inverse counterfactuals. We thank the reviewer for raising this point. To clarify how we construct inverse counterfactual questions, consider a deterministic pattern in which B occurs whenever A occurs. Suppose both A and B occurred on the test day. We ask, “If B had not occurred, which statement would be more likely to be true?” The two options are “A occurred” and “A did not occur.” By design, the if-clause always poses the opposite of the observed outcome, and we use the subjunctive (“had not occurred”) to signal that the options should be evaluated in that hypothetical scenario. We agree that this backtracking interpretation is non-default and can be challenging. Reasoning backward from a hypothetical outcome to a condition consistent with the learned rule is precisely the capability we aim to test.

However, on closer inspection, we identified a shortcut in the current design. We intentionally flipped the observed outcome, so the answer is always the opposite of the observed condition. In the previous example, the answer is always “A did not occur.” This introduces a shortcut because we can achieve 100% accuracy by blindly selecting the option that contradicts the observed history. Although no training is involved in this work, we will revise the setting to improve the usability of our benchmark. Instead of using two opposite options (e.g., “A occurred” and “A did not occur”), we will use a noisy counterfactual distractor alongside the correct option (e.g., “C did not occur” and “A did not occur”), where C was observed to occur. This ensures that no shortcut is possible. We will update this in our final revision.

W5. What the four-cell criterion establishes. Thank you for pointing this out. We agree that requiring all four cells of a contingency table to be nonempty does not guarantee statistical independence. However, our goal is to evaluate whether models can identify and reason about the underlying deterministic rules, rather than their ability to infer stochastic associations.

To further investigate this concern and assess how frequently strong associations occur in practice, we conducted a post-hoc analysis of the empirical association between the noise condition XX and the target activity AA, measured by Δ=∣P(X∣A)−P(X∣¬A)∣\Delta = |P(X\mid A)-P(X\mid \neg A)|. The distribution across all N-CF questions is shown below:

Difference (Δ\Delta)0–10%10–25%25–50%50–75%75–100%
Percentage28%30%28%11%3%

While we acknowledge that strong incidental associations can make the No Hints setting more challenging, our analysis shows that extreme imbalances (≥75 percentage points) occur in only 3% of the current questions. Nevertheless, we agree that such cases could introduce unintended ambiguity. We will update the dataset by resampling more balanced noise conditions, aiming to minimize the empirical association between the noise condition and the target activity as much as possible. We will also clarify that our selection criterion does not establish statistical independence.

Q1. Dataset and code release We plan to release the generation code and evaluation harness, which have already been included in the supplementary materials. The redistribution of Sims 4 recordings will be subject to EA's content policy and applicable licensing restrictions, which we will clarify in the release documentation.

Q2. Confidence intervals and baselines As all questions are binary, chance accuracy is 50%, with an expected chance All-Correct rate of 7.7% (95% range: 5.2–10.5%). The majority-class baseline scores 51.5% (Direct), 57.2% (CF), 54.6% (Noise-CF), and 63.2% (Inverse-CF), yielding 56.8% Acc / 12.3% AC overall. Although we balanced answer positions 50/50 during generation, the final LLM-based rewriting step placed 63.2% of correct answers in option A. This leads to slight deviation in other three question types, but severe in Inverse-CF questions. We will fix this in the final revision.

Q3. Performance breakdowns Thank you for the suggestion. We will add a breakdown table with episode-level bootstrap CIs and chance/majority baselines. We focus on language-only settings here for direct comparability.

  • Vision-only vs. dialogue-required (29 vs. 77 episodes): Vision-only episodes contain only physical action histories, without dialogue. Models perform better on vision-only episodes. L-N Gemma-4-31B (Full) scores 79.1 [74.7, 83.4] vs. 73.4 [70.0, 76.7], and L Qwen3.5-35B scores 65.0 vs. 60.0. Restricting to the 86 pattern episodes (29 vs. 57) to exclude dynamic episodes widens the gaps to 79.1 vs. 69.2 and 65.0 vs. 56.2, suggesting that integrating dialogue and action evidence is more challenging than tracking actions alone.
  • Temporal vs. activity conditions (44 temporal-only, 10 activity-only, 32 mixed pattern episodes): Activity-only patterns are the hardest with noisy logs. L Gemini-2.5-Flash scores 61.1 / 38.8 / 57.9, and L Gemini-3-Flash scores 71.0 / 53.8 / 67.3, respectively. The gap largely closes with noiseless logs (L-N Gemma-4-31B: 73.3 / 70.0 / 72.6), suggesting that tracking event dependencies under noise is the main challenge. The activity-only subset is small (80 questions), with wide CIs (approximately ±10 points).
  • Episode length: Accuracy declines with more observed history before the query. L-N Qwen3.5-35B drops from 78.5 (≤10 days) to 53.9 (>40 days), L-N Gemma-4-31B from 78.5 to 63.2, and L Gemini-3-Flash from 75.6 to 53.9. Grouping by total episode length shows a milder decline, mainly for episodes ≥45 days.

Q4. Partial hints and logical operators We apologize for the inaccurate description in Table 14. This was a typo, and the correct description should be "condition categories" only. Neither Partial nor Full Hints explicitly specify the logical operators. We use "and" in the prompt only as a linguistic connector between conditions. We will correct the description accordingly.

Q5. Inverse counterfactual options and uniqueness The opposite statement is constructed by negating the condition statement selected from an alternative Day Condition Set, with the labeled answer determined by the ground-truth rule. Uniqueness is guaranteed by construction, as we only select conditions whose negation would lead to an incorrect answer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.