- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates whether models can infer and adapt human behavioral patterns from long-term observations, motivated by the needs of persistent human–agent interaction and the limitations of retrieval-focused evaluations. Its main contributions are SimLife, a household simulation and data-generation platform built on The Sims 4, and SimLife-BP, a benchmark comprising 106 episodes and 1,439 questions covering multiple reasoning types and hint levels. Results suggest that tested models often rely on surface heuristics, apply rules inconsistently, and struggle with pattern changes.
The work offers an interesting evaluation perspective, substantial infrastructure, and generally clear presentation. Its rule-based construction and hypothesis-elimination procedure provide a useful foundation for controlled evaluation. However, Tested model coverage is limited, the practical connection between recognizing behavioral patterns and providing useful assistance remains untested by the prediction-focused tasks.
Overall, the paper presents a promising benchmark, but stronger validation and experimental controls would strengthen its broader conclusions.
Critical strengths
-
The paper takes a relatively novel perspective on evaluating models’ understanding of human behavioral patterns over extended periods. Its emphasis on discovering latent rules, applying them consistently across different reasoning tasks, and adapting to behavioral changes provides a valuable direction for long-context evaluation.
-
The paper develops SimLife, a platform built on The Sims 4 for constructing long video sequences of household activities with synchronized observations and annotations. The pipeline translates predefined behavioral rules into activity constraints and assembles matching recorded days into sequences. This constitutes a substantial infrastructure contribution and provides a controllable testbed for studying long-term behavioral understanding.
-
The paper is well organized and clearly written.
Critical weaknesses
-
Coverage of strong proprietary models is limited. The reported coverage is concentrated on Flash/Lite variants. Additional SOTA reasoning models would make it easier to assess whether the benchmark exposes limitations that persist across leading systems. Without broader coverage, the generality of the conclusions about current models remains uncertain. Expanding this comparison would strengthen the evidence for the benchmark’s diagnostic value.
-
The connection to useful long-term assistance remains underexplored. The benchmark evaluates inference and application of behavioral rules through prediction and counterfactual questions, but it does not directly assess whether this understanding enables an agent to provide useful assistance. Given the paper’s framing around long-horizon human–agent partnership, an additional evaluation of assistance decisions—for example, choosing an appropriate intervention based on an inferred routine—would strengthen its practical relevance. The current experiments support conclusions about behavioral pattern understanding; the broader connection to effective partnership remains to be demonstrated.
Questions for authors
-
Beyond constraint-based matching, how extensively were the selected video sequences independently checked against the predefined behavioral rules? Could you report the verification coverage and any observed rule–video mismatch rate?
-
How are the rendered questions and their answer labels verified to ensure there is a unique correct answer, particularly for inverse counterfactual questions?
-
Could you evaluate additional leading reasoning models from multiple providers? How do they perform on SimLife-BP, and do the reported limitations in rule inference and adaptation persist across these models?
2 comments
I appreciate the authors' reponse, I will maintain the rating.
Response to Reviewer KmyB
We thank the reviewer for the thoughtful feedback and for recognizing the novelty of evaluating long-term behavioral pattern understanding, particularly rule inference, application, and adaptation. We also appreciate the acknowledgment of our substantial platform contribution, controllable rule-first generation pipeline, and clear presentation.
W1. Coverage of proprietary models We respectfully disagree that our evaluation lacks sufficient coverage to demonstrate the benchmark's diagnostic value. We have already evaluated three strong proprietary models, alongside multiple open-source models, and observed substantial limitations in their ability to infer and adapt to long-term behavioral patterns.
While evaluating additional leading reasoning models (e.g. other model families) would further strengthen the generality of our findings, we believe the current results already provide sufficient evidence of the challenges posed by SimLife-BP. Given our computational and financial constraints, we prioritize a systematic analysis of these limitations over expanding model coverage. We acknowledge that our conclusions do not necessarily generalize to all leading reasoning models.
W2. Connection to useful long-term assistance We agree that evaluating the usefulness of inferred behavioral patterns for long-term assistance is an important next step. However, we believe that establishing a controllable platform and systematically understanding models' capabilities and limitations in long-term behavioral reasoning are fundamental steps before moving toward more complex human–agent interactions and decision-making over ultra-long horizons.
Evaluating effective ultra-long-term assistance requires carefully defining the tasks, interaction scenarios, expected agent behaviors, and evaluation metrics, which constitutes a substantial research problem in its own right. Rather than introducing an oversimplified assistance task (e.g. asking the model “should you serve coffee today?”) that may not adequately capture the complexity of long-term human–agent partnership, we chose to focus on establishing and evaluating the underlying behavioral understanding capabilities. We see assistance-oriented evaluation as a natural and important direction for future work, but it is beyond the scope of this paper.
Q1. Verification of video sequences Beyond constraint-based matching, we checked 71 video chains out of 106, comprising 2,863 days. We found that 197 days (6.9%) contained one visibility mismatch, where an event recorded in the logs was either not visible or visible for only about one second, making it easy to miss at 0.5 FPS. These discrepancies mainly arise from the limited temporal precision of game ticks, which can cause delays in updating character positions and facing directions, which are used in visibility calculation. This mainly affects V+L and V+A variants. We are continuing to verify and improve the dataset to minimize such issues.
Q2. Question and label verification Answer labels are determined by the predefined deterministic rules rather than LLM judgments. For inverse counterfactuals, we select an alternative Day Condition Set whose outcome is opposite to the observed one, and construct the two options by negating the relevant condition statement while keeping all other conditions fixed.
For example, consider a deterministic pattern in which B occurs whenever A occurs. Suppose both A and B occurred on the observed day. We ask, "If B had not occurred, which statement would be more likely to be true?" with the options "A occurred" and "A did not occur." Under this hypothetical premise, only "A did not occur" is consistent with the rule. Uniqueness is guaranteed by construction, as we only select conditions whose negation would contradict the hypothetical outcome. We will clarify these construction constraints in Appendix C.2.
Q3. Additional reasoning models Please refer to our response to W1 regarding model coverage.