- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 1
Read review
Paper summary
The paper proposes a resident-level evaluation protocol for evaluating LLM-derived behavioral trait assignments in agent-based evacuation simulations. It shows that simulations can predict similar aggregate outcomes (total death count) even when assigning risk to different residents, and that the proposed checks can detect author-injected score corruptions.
Critical strengths
-
The paper provides good observations that a simulation may look promising when looking at the aggregate outcomes alone (e.g., total death count), but the simulation may not be accurate at the individual resident level. The motivation is intuitive: inaccurate or biased risk assignments could lead to suboptimal/bias/unfair planning decisions, although the manuscript does not clearly establish the connection to real-world consequences.
-
The paper reiterates some insightful findings in LLM research and demonstrates their relevance to Evacuation agent-based models. For example, the paper suggests that cross-model agreement does not necessarily mean validity, and direct prompting can be highly sensitive to wording.
Critical weaknesses
-
The experiments evaluate one injected failure error at a time, with each error closely aligned with one or more proposed checks. While this is useful for unit testing, it may not be robust to real-world scenarios where multiple errors can coexist. Evaluating on combined errors or independently designed held-out errors would help demonstrate the robustness of the proposed protocol.
-
Although the exact metrics and decision rules appear to be introduced in this paper, the individual checks mostly build upon ideas from areas such as ABM validation and simulation sensitivity analysis. The contribution seems to be primarily on combining these checks, rather than introducing new validation principles.
Questions for authors
-
Would the same protocol apply if the resident scores are produced by other methods, e.g., a regression model, survey imputation, or handwritten rules? If so, what aspects of the protocol (e.g., C3) are specifically designed to address potential failures introduced by LLM-derived scores?
-
The manuscript states that the proposed checks "establish consistency with evidence rather than individual accuracy," but also claims that they identify traits that can be “trusted”. Can the authors elaborate more on the level of validity supported by the checks, and what “trusted” means? Does passing all checks only indicate that the selected evidence and diagnostics found no inconsistency, or is it used to justify using the resulting resident-level assignments?
-
When one or more checks fail, what are the implications, and how should practitioners determine what component to revise, e.g., the LLM-derived scores, the profile-to-score mapping, the score-to-behavior channel, or the underlying simulator? Can failure patterns across multiple checks reliably localize the problem, or do they only indicate that the end-to-end pipeline is inconsistent with the available evidence?
-
The manuscript describes warning reliance as a trait that can be “trusted,” yet no score source passes all four C2 patterns under the stated rule. Can the authors clarify the basis for this conclusion? Are some checks considered more informative/important than others? How should one interpret potentially conflicting results across C1–C5 when determining whether a trait is trustworthy?
Thanks for the substantial revision. I think the new mobility and robustness analyses largely address my concern about whether the age-related failure could instead come from a missing simulator mechanism. My main concern, however, is still the role and necessity of the LLM component. The revision makes the positioning clearer, but I still see the protocol mainly as a combination of existing validation tools applied to a new evaluation target. Similarly, saying that LLMs provide broader coverage does not fully answer why the LLM component is needed or preferable to simpler alternatives.
Model scale is not my main concern, but I still think a larger-model comparison would be useful. The cited earlier work does not show that the same failures would necessarily persist for current larger models, especially since recent results suggest that scaling effects in social/persona simulation are mixed and task-dependent. Ziems et al. (2026) find substantial gains from scaling on many social-simulation tasks, while Lutz et al. (2025) show that larger models are not consistently better for persona simulation. Overall, the revision strengthens the paper, but it does not change my main concerns about the necessity of the LLM component and the generalizability of the findings across model scales, so I plan to keep my score unchanged.
References
Ziems, Caleb, et al. “Will Scaling Improve Social Simulation with LLMs?” arXiv preprint arXiv:2607.02464, 2026.
Lutz, Marlene, et al. “The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models.” Findings of EMNLP, 2025.