- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the retrospective interpretation of emergent risk events in multi-agent simulations. It combines counterfactual replay with causal-order-constrained Shapley values to attribute reference-relative terminal risk to individual agent–time actions. These scores are then aggregated into five metrics describing when contributions arise, which agents contribute, and which behaviors are involved. The framework is evaluated in economic, financial, and social-network simulations, alongside controlled games.
Critical strengths
-
The paper addresses a relevant problem: connecting system-level outcomes to individual actions in multi-agent simulations. Organizing the analysis around “when, who, and what” makes the intended explanatory outputs clear.
-
The framework coherently connects simulator replay, action-level attribution, and aggregation. The authors explicitly define contributions relative to a reference and characterize the mathematical properties of the allocation.
-
The controlled games and appendix analyses provide useful checks on attribution recovery, sampling precision, reference sensitivity, precedence assumptions, and computational cost.
Critical weaknesses
-
General applicability requires further justification. The framework standardizes attribution once the risk functional, reference actions, precedence structure, and behavior categories are specified, but offers limited guidance on choosing and validating them in a new setting. In particular, treating each agent–time action as a separate player enables fine-grained attribution while requiring a dependency structure that supports this resolution. Constructing and validating that structure may involve substantial domain expertise. The appendix provides useful sensitivity analyses, but does not sufficiently establish how practitioners can obtain reliable specifications or assess the consequences of plausible misspecifications.
-
Trajectory boundaries and recurring risk events are insufficiently addressed. Attribution starts from the beginning of the recorded trajectory, which becomes costly for long-running simulations. The paper does not adequately discuss how to select a starting point or how conditioning on a later state changes the interpretation. When risk exhibits recurring peaks, it is also unclear how the framework should distinguish separate episodes and how the attributed drivers change when different event endpoints or starting points are selected. Experiments on longer trajectories, with varying starting points and repeated risk events, would help establish the method’s applicability and computational feasibility.
-
The relationship to conventional domain-specific analyses remains unclear. The paper should compare the proposed attribution with relevant domain-specific scoring methods under the same setting and information access. In the financial setting, for example, measures based on trading activity, order flow, or price impact could serve as alternative agent-level scores where appropriate. Comparing their rankings and compatible aggregate summaries would help clarify what additional information the Shapley-based attribution provides. To establish explanatory or intervention value, the evaluation should also use independently defined objectives and report computational cost, rather than comparing only the resulting summary metrics.
-
Presentation needs improvement. Some results are separated from their discussion: Table 2 appears on page 5, while its main discussion is on page 8. Dense tables provide little visual emphasis on the entries supporting the main findings. The exposition also requires frequent cross-referencing between definitions, scenario-specific assumptions, and appendix details, making the method and its interpretation difficult to follow.
Questions for authors
-
Improve the presentation by placing tables near their discussion, highlighting findings relevant to the text, and clearly distinguishing the general framework from its scenario-specific requirements.
-
As a future direction, Data Shapley in One Training Run (Wang et al., ICLR 2025) may offer useful inspiration: reformulate terminal-risk attribution as local risk contributions computed along the realized simulation trajectory and accumulated or summarized over windows. This would change the attribution target, but could potentially reduce repeated replay costs and support analysis of longer-running systems.
Overall Assessment
The framework is coherent and potentially useful, but its general applicability and practical advantage remain insufficiently established. My main concern is whether the additional modeling and computational costs yield meaningful explanatory benefits over simpler alternatives. Clearer guidance on problem specification, evidence on longer trajectories, and comparisons with domain-specific methods would substantially strengthen the paper.
Reference: Wang, J. (T.), Mittal, P., Song, D., and Jia, R. Data Shapley in One Training Run. ICLR 2025.