acceptodds
Withdrawn from ICLR 2027

Interpreting Emergent Extreme Events in Multi-Agent Systems

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model-powered multi-agent systems are powerful tools for simulating complex human-like systems. Their interactions often produce extreme events whose origins remain obscured by the black box of emergence. Interpreting these events is critical for system safety. To our knowledge, this paper proposes the first unified framework for retrospectively interpreting emergent extreme-event risk across time, agents, and behaviors in LLM-powered social simulations, answering three questions: When does the event originate? Who drives it? And what behaviors contribute to it? Specifically, we adapt the causal-order-constrained Shapley value to attribute reference-relative extreme-event risk to each agent action at each time step, i.e., scoring the action by its marginal contribution in replay. We then aggregate the attribution scores over time, agents, and behaviors to quantify the risk contribution of each dimension. Finally, we design metrics from these contribution scores to characterize extreme events. Experiments across diverse multi-agent system scenarios (economic, financial, and social) demonstrate how our framework characterizes extreme events through their temporal, agent, and behavioral contributions.

Primary area:
interpretability and explainable AI
Keywords:
Multi-agent systems, Large language models, Extreme events, Risk attribution, Asymmetric Shapley values, Social simulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vd5bOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the retrospective interpretation of emergent risk events in multi-agent simulations. It combines counterfactual replay with causal-order-constrained Shapley values to attribute reference-relative terminal risk to individual agent–time actions. These scores are then aggregated into five metrics describing when contributions arise, which agents contribute, and which behaviors are involved. The framework is evaluated in economic, financial, and social-network simulations, alongside controlled games.

Critical strengths
  1. The paper addresses a relevant problem: connecting system-level outcomes to individual actions in multi-agent simulations. Organizing the analysis around “when, who, and what” makes the intended explanatory outputs clear.

  2. The framework coherently connects simulator replay, action-level attribution, and aggregation. The authors explicitly define contributions relative to a reference and characterize the mathematical properties of the allocation.

  3. The controlled games and appendix analyses provide useful checks on attribution recovery, sampling precision, reference sensitivity, precedence assumptions, and computational cost.

Critical weaknesses
  1. General applicability requires further justification. The framework standardizes attribution once the risk functional, reference actions, precedence structure, and behavior categories are specified, but offers limited guidance on choosing and validating them in a new setting. In particular, treating each agent–time action as a separate player enables fine-grained attribution while requiring a dependency structure that supports this resolution. Constructing and validating that structure may involve substantial domain expertise. The appendix provides useful sensitivity analyses, but does not sufficiently establish how practitioners can obtain reliable specifications or assess the consequences of plausible misspecifications.

  2. Trajectory boundaries and recurring risk events are insufficiently addressed. Attribution starts from the beginning of the recorded trajectory, which becomes costly for long-running simulations. The paper does not adequately discuss how to select a starting point or how conditioning on a later state changes the interpretation. When risk exhibits recurring peaks, it is also unclear how the framework should distinguish separate episodes and how the attributed drivers change when different event endpoints or starting points are selected. Experiments on longer trajectories, with varying starting points and repeated risk events, would help establish the method’s applicability and computational feasibility.

  3. The relationship to conventional domain-specific analyses remains unclear. The paper should compare the proposed attribution with relevant domain-specific scoring methods under the same setting and information access. In the financial setting, for example, measures based on trading activity, order flow, or price impact could serve as alternative agent-level scores where appropriate. Comparing their rankings and compatible aggregate summaries would help clarify what additional information the Shapley-based attribution provides. To establish explanatory or intervention value, the evaluation should also use independently defined objectives and report computational cost, rather than comparing only the resulting summary metrics.

  4. Presentation needs improvement. Some results are separated from their discussion: Table 2 appears on page 5, while its main discussion is on page 8. Dense tables provide little visual emphasis on the entries supporting the main findings. The exposition also requires frequent cross-referencing between definitions, scenario-specific assumptions, and appendix details, making the method and its interpretation difficult to follow.

Questions for authors
  1. Improve the presentation by placing tables near their discussion, highlighting findings relevant to the text, and clearly distinguishing the general framework from its scenario-specific requirements.

  2. As a future direction, Data Shapley in One Training Run (Wang et al., ICLR 2025) may offer useful inspiration: reformulate terminal-risk attribution as local risk contributions computed along the realized simulation trajectory and accumulated or summarized over windows. This would change the attribution target, but could potentially reduce repeated replay costs and support analysis of longer-running systems.

Overall Assessment

The framework is coherent and potentially useful, but its general applicability and practical advantage remain insufficiently established. My main concern is whether the additional modeling and computational costs yield meaningful explanatory benefits over simpler alternatives. Clearer guidance on problem specification, evidence on longer trajectories, and comparisons with domain-specific methods would substantially strengthen the paper.

Reference: Wang, J. (T.), Mittal, P., Song, D., and Jia, R. Data Shapley in One Training Run. ICLR 2025.

Reviewer_RnENOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for retrospectively interpreting extreme events in LLM-based multi-agent systems. The method adapts asymmetric Shapley values with causal precedence constraints to attribute event risk to individual agent actions. These attributions are aggregated across temporal, agent, and behavioral dimensions, yielding five interpretable metrics. The framework is evaluated on economic, financial, and social simulations, together with synthetic intervention-based benchmarks and comparisons against several attribution baselines.

Critical strengths
  1. A unified and interpretable attribution framework. The paper provides a systematic way to analyze when extreme events develop, which agents contribute most, and which behaviors are responsible. This is a meaningful problem for understanding complex multi-agent systems.

  2. Principled methodology with controlled validation. The use of causal-order-constrained Shapley values provides a mathematically grounded attribution mechanism. The synthetic intervention benchmarks and comparisons with standard Shapley values, leave-one-out attribution, and LLM-based methods provide useful evidence of its effectiveness under controlled assumptions.

Critical weaknesses
  1. Limited causal validity of counterfactual replay. The framework replays recorded agent actions under modified system states without allowing downstream LLM agents to reconsider their decisions. However, in interactive multi-agent systems, changing an earlier action may substantially alter subsequent agent behaviors. Consequently, the resulting Shapley scores may not accurately reflect the actual causal contributions of actions to extreme events.

  2. Insufficient validation in realistic multi-agent environments. Although the synthetic benchmarks demonstrate strong attribution recovery, they rely on predefined dependency structures and additive risk accumulation, which closely match the method's assumptions. The experiments on realistic LLM-based simulations mainly characterize attribution patterns rather than independently verifying whether the identified agents and behaviors are genuinely responsible for the extreme events. This limits the evidence supporting the framework's practical explanatory value.

Questions for authors
  1. Since counterfactual replay keeps downstream agent decisions fixed, how can the authors establish that the resulting attributions reflect actual causal influence rather than artifacts of the replay procedure? Have the authors compared their results with counterfactual rollouts in which downstream agents are allowed to respond to changed states?

  2. The intervention-based benchmark uses relatively simple risk-accumulation games with known dependency structures. Can the authors provide evidence that high-attribution agents or actions in the realistic LLM simulations also have a greater impact on extreme-event outcomes when directly intervened upon?

Reviewer_ZYwNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a post-hoc framework for interpreting extreme events in LLM-powered social simulations. It attributes the reference-relative terminal risk of a recorded event to individual agent actions using an asymmetric (causal-order-constrained) Shapley value on a counterfactual replay game, estimated by monte carlo order sampling. The action-level scores are aggregated over time, agents, and behavior categories into five metrics answering “when, who, and what.” The framework is applied to 142 trajectories from EconAgent, TwinMarket, and SocialNetwork across five LLM groups, yielding five insights. The revised version adds formal properties, Monte Carlo convergence studies, synthetic intervention-grounded validation, and sensitivity analyses to the event subset, latency threshold, reference actions, and precedence structure.

Critical strengths
  1. The problem is important and under-explored. Explaining how recorded actions give rise to emergent risk in LLM-based social simulations is relevant to AI safety. The when, who, what decomposition, built on an efficiency-satisfying attribution, makes the three aggregation levels consistent. The novelty lies mainly in the problem formulation and the replay-game application, since the attribution rule itself adapts asymmetric shapley values.

  2. The paper is rigorous and candid. It scopes its theoretical claims (no uniqueness claim, symmetry only under constraint-preserving relabeling), reports where its own precision criteria fail, and adds sensitivity analyses for references and precedence structure. Reporting these openly makes the evidence easier to evaluate.

Critical weaknesses
  1. Conclusions depend strongly on the reference, and this dependence is not resolved. In the TwinMarket sensitivity study (Table 6), alternative references yield rankings almost uncorrelated with the default (cosine 0.03–0.11), and some even yield negative excess risk. The main text responds by redefining the "safe reference" as a prescribed comparison. This is coherent, but it leaves open which Table 2 findings would survive a different reference. No ablation or economic justification is given for the EconAgent reference.

  2. The "Black Swan" framing is not matched by the event selection. Events are the raw-risk maximum of each trajectory with a timing filter, not fat-tailed outliers or extreme-impact cases, and two EconAgent events have nonpositive excess risk. In addition, model versions and decoding parameters for the five LLM groups were not retained, and cross-model differences are neither analysed nor statistically tested.

  3. The effectiveness evaluation is largely circular. In the intervention-grounded benchmark (§4.2, App. D.10), the target is defined as the stage-local increment, which the paper itself shows equals the ASV marginal on every legal prefix. ASV's exact recovery is therefore guaranteed by construction, and the benchmark does not show that the attributions are faithful in the actual simulators.

Minor: Section 3 mixes definitions, sampling verification and result tables/figures, and the text is dense. Metric names such as "risk concentration" are computed on absolute contributions, which can mislead.

Questions for authors
  1. Which findings in Table 2 and I1–I5 hold under alternative references in all three scenarios? Please include an EconAgent ablation reporting action-ranking Spearman and the five metrics.

  2. What are GagG_{ag}, CagC_{ag}, ZagZ_{ag} and GbeG_{be} under null models (shuffled attributions, random-sign contributions)? Do I2–I4 exceed them?

  3. In TwinMarket replays, what fraction of retained actions become infeasible, non-executed or semantically changed, and how far do replay states diverge from factual states? Does the attribution change if such actions are treated as no-ops, or if the LLM agents are re-queried in the modified states instead of reusing recorded decisions?

  4. How do the selected extreme events differ from non-extreme trajectory peaks under the same metrics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.