- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes an new approach to attributing individual agent contributions in LLM-based multi-agent systems. The proposed method formulates attribution as a Semantic Cooperative Game, which does not rely on counterfactual executions. Instead, it constructs a semantic hypergraph from an observed execution trajectory. This hypergraph captures how different agents generate, preserve, and transform task-relevant information throughout the interaction. To quantify individual contributions, the method introduces the Semantic Shapley Value (SSV) and an algorithm that identifies minimal semantic support sets, applies Boolean absorption, and computes agent contributions using multilinear coefficients. The paper also establishes the equivalence between SSV and the classical Shapley value under specified conditions. The proposed method is evaluated through two main experiments. The first compares SSV against baseline Shapley attribution, leave-one-out attribution, and holistic LLM as a judge evaluation on HealthBench cases. The second investigates the alignment between SSV-based attributions and perturbation-induced performance degradation across four multi-agent task settings.
Critical strengths
-
The authors provide a well-motivated discussion of the limitations of counterfactual-based contribution attribution methods in LLM-based multi-agent systems, highlighting how counterfactual interventions can alter downstream interactions, introduce instability, and incur substantial computational costs.
-
The proposed approach estimates agent contributions by leveraging the semantic structure of an existing execution trajectory. This enables contribution attribution without requiring counterfactual reruns, offering an efficient alternative that relies on the interactions between agents.
Critical weaknesses
-
The proposed SSV appears to distribute the Möbius coefficients of the induced semantic game using the established Shapley dividend formula in [1]. This may suggest that the main novelty is in constructing the semantic value function, not in the allocation rule itself.
-
In addition, the theoretical justification for the claimed extension remains unclear. Specifically, while Eq. (6) follows from the standard Shapley allocation identity once Eq. (5) holds, the equivalence between the induced semantic value function and the classical coalition value function in Eq. (5) does not appear to be independently established.
-
While the paper provides several experiments evaluating reduction consistency, structural diagnosis, and analyzer sensitivity, I think the scalability of the proposed method is still not sufficiently evaluated. In particular, the experiments are limited to workflows with at most five agents. The authors should evaluate how SLIC scales to larger and more complex MAS. It also remains unclear how the computational cost of constructing the semantic graph and computing Boolean supports grows with the number of agents and the complexity of their interactions.
-
The conditions of "set-based value reachability," "full observability," and "no masking or order dependence" are not formally defined, making it difficult to assess when the claimed equivalence holds.
[1] Harsanyi, J. C. (1963). A simplified bargaining model for the n-person cooperative game. International economic review (Philadelphia), 4(2), 194–220.
Questions for authors
-
Could the authors clarify the main theoretical novelty of the proposed SSV relative to the existing Harsanyi-Shapley framework [1]?
-
Could the authors clarify how the conditions of “set-based value reachability”, “full observability”, and “no masking or order dependence” ensure that the semantic value function captures the values of alternative coalitions whose behavior is not observed, given that it constructed from a single realized trajectory?
3.Could the authors provide an analysis of how the computational cost of SLIC grows with the number of agents and the complexity of their interactions?
- Could the authors provide an ablation study to evaluate the contribution of Boolean support computation? For example, how does SLIC compare with a simpler approach that directly assigns contributions based on the reconstructed semantic graph, without the Boolean support logic?