acceptodds
Withdrawn from ICLR 2027

Who Bears How Much Blame? Semantic Cooperative Games for Behavioral Accountability and Incentive Alignment in Agentic AI Systems

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM-based multi-agent systems form judgments and decisions through sequences of semantic transformations. Failures in cross-provider workflows can impose financial losses, and the allocation of responsibility can shape providers' future configuration choices. Execution logs identify the agents and messages involved, but not which local semantic behaviors affected the outcome.We propose a behavior-level accountability framework built on Semantic Cooperative Games (SCG). SCG traces backward from outcomes to recover local semantic nodes, provenance relations, generation links, and the agents that execute them. We model these links as semantic behaviors. BCAST performs counterfactual replay over behavior subsets, estimates their effects from terminal outcomes, and assigns values to behaviors and their output nodes without manual responsibility labels. These values support Behavioral Shapley and other allocation rules. Explicitly specifying the target behavior set, reference replacement, propagation rule, and outcome evaluation also places existing message-, step-, component-, and agent-level counterfactual methods in a common theoretical framework.To connect retrospective accountability with prospective incentive design, we use Behavioral Shapley allocations as penalties and study participants' choices among semantic modes, the configurations that generate behaviors. Under stable behavioral effects and a common cancellation reference, we bound the excess expected task loss of any pure-strategy equilibrium relative to the optimum. When each participant has a mode whose advantage exceeds its variation across contexts, minimizing the proposed penalties induces a unique equilibrium that achieves the lowest expected task loss among candidate configurations. Because objective ground-truth labels for behavioral responsibility are generally unavailable, we design two complementary empirical evaluations. The first examines semantic interventions, downstream replay, and the direction of behavioral effects in LLM workflows, assessing whether system replay can provide reliable automated evaluation signals for fine-grained behavioral attribution. The second tests whether penalty-minimizing mode choices improve task outcomes. The framework connects semantic explanations of completed executions, outcome responsibility allocation, and incentive evaluation for future deployments.

Primary area:
interpretability and explainable AI
Keywords:
Large language models, multi-agent systems, behavioral accountability, causal attribution, counterfactual reasoning, semantic interventions, Shapley value, incentive design

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_UGBvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Behavioral Counterfactual Attribution of Semantic Transformations (BCAST), a framework based on Semantic Cooperative Games (SCG) that attributes multi-agent system outcomes to individual semantic behaviors. By combining counterfactual replay with game-theoretic allocation, BCAST quantifies each behavior's contribution to overall task loss and demonstrates how Shapley-based penalties can align individual agent incentives with system performance.

Critical strengths

This work introduces a unified framework for fine-grained behavioral attribution in multi-agent systems, connecting low-level semantic transformations to high-level task outcomes. It also provides a theoretical analysis demonstrating how Shapley-based responsibility allocation enables incentive alignment.

Critical weaknesses
  1. Limited novelty and insufficient distinction from prior work. BCAST largely extends SCG by replacing semantic support-based attribution with outcome-based valuation through counterfactual replay and standard Shapley allocation. At the agent, message, or step level, the framework also overlaps with existing attribution and intervention methods, such as [1,2]. The authors do not convincingly establish BCAST's methodological novelty or demonstrate the advantages of fine-grained semantic attribution through direct comparisons.

  2. Unclear validity of counterfactual replay assumptions. BCAST assumes that generation modes remain invariant under counterfactual semantic inputs, but the reliability of such replays is insufficiently validated, particularly when modifying individual behaviors within a message. Errors in semantic identification, intervention, or downstream regeneration could affect attribution scores. The effect-stability assumption underlying the incentive guarantees also lacks empirical justification.

  3. Unclear experimental setup and insufficient validation. Appendix F appears to provide the primary empirical evaluation, but key experimental details and the interpretation of evaluation metrics are insufficiently explained. For example, Table 2 reports task-level factual accuracy without clearly establishing its relevance to attribution quality, while Section F.2 introduces the semantic analyzer without adequately explaining its role or validating its extraction accuracy. Furthermore, each task includes only 50 trajectories, with 20 held out for validation. Direct comparisons with existing attribution methods or independently annotated responsibility labels are needed to establish the framework's effectiveness.

  4. Limited empirical support for incentive alignment. The incentive analysis relies on a constructed three-agent procurement task with three predefined behavioral modes. This controlled setting provides limited evidence of generalizability to realistic multi-agent workflows. Moreover, the effect-stability assumptions underlying the theoretical guarantees are not empirically examined.

  5. Significant presentation issues and excessive reliance on the appendix. The main text focuses primarily on conceptual and theoretical formulations, while nearly all experimental setups, empirical analysis, and case studies are relegated to the appendix. Consequently, the main manuscript provides insufficient empirical evidence to substantiate its central claims, making it difficult to assess the paper's contributions without consulting the appendix. This is particularly concerning given that ICLR guidelines [3] do not require reviewers to read the appendix.

Reference


[1] Chen et al. The Trace Is the State: Exact Credit Assignment for LLM Agent Teams, arXiv 2026

[2] Ma et al. DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems, ICLR 2026

[3] https://iclr.cc/Conferences/2026/AuthorGuide

Questions for authors

See weakness

Reviewer_JVqcOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper extends prior work on Semantic Cooperative Games (Jiang et al., 2026) and proposes a method to assign blame when a multi-agent LLM workflow a costly failure. From what I understood, it breaks an execution trace into “semantic behaviors” (individual operations that introduce or transform meaning) and estimates each behavior’s effect on the final outcome. It allocates a change in task loss with a Shapley-style rule, so harmful steps get positive blame while corrective steps get negative blame. The paper then treats these allocations as penalties and proves a result under a strong stability assumption. The experiments are mostly in an Appendix.

I am not at all familiar with the closest related work, so I did not fully follow parts of the paper – I have indicated this in my low confidence. I did not have time to read the Appendices, which is unfortunate for the authors because it appears that much of the important content is not in the main paper. I strongly recommend the authors consider a major rewrite of the paper to address this issue.

Critical strengths

The paper targets a real-world application where a potentially cross-provider agent workflow causes a financial loss and logs show who acted but not whose judgment contributed in various ways to the error.

The paper separates a realized “behavior” (what an agent actually did in this run) from the “mode” that generates it (model, prompt, and tool settings that persist across inputs). This clarifies what replay-based attribution estimates. From my understanding, this is one of the main contributions of the work.

Critical weaknesses

For me, the paper’s exposition is a major weakness that could be grounds for rejection on its own. I found this paper very hard to read. Most of the content sits in the appendices, e.g., the main text’s entire empirical section is a single paragraph. The first two-plus pages are dense abstract discussion about behaviors, modes, references, and propagation, with no concrete worked example until page 6. Figure 1 adds nothing as it is a generic hypergraph of nodes, conveying no task content and no insight beyond what a graph looks like. A paper whose main body cannot be understood or evaluated without its appendices is problematic.

Another weakness is that the incentive theorem appears to hold only when each provider’s effect on the loss is roughly independent of what the others do, which is close to assuming the problem away. It is also not clear what is new here as Shapley-based cost sharing is already known to yield potential games with pure equilibria, and the paper does not position its result against that game theory literature.

As far as I can tell, the method is never compared against simpler alternatives such as leave-one-out, agent- or step-level attribution, or existing failure-attribution methods. Nor is there evidence that humans agree that the flagged behaviors are the real causes of failure. The reported agreement numbers seem to come from checking the method against its own replays, which shows internal consistency rather than the fact that the blame assignments are correct or more useful than other approaches.

Questions for authors

Some other comments/questions follow:

Another instance of the writing issue is reflected by the abstract, which is long and poorly written. It mentions several undefined terms such as semantic nodes, provenance relations, generation links, semantic modes, etc. and introduces the acronym BCAST without ever expanding or explaining it. It put me off immediately and I never recovered to appreciate the paper. I recommend making it more compact and understandable.

The paper looks like a direct follow-up to SCG that replaces support-based agent credit with replay-based, signed behavior-level blame. The main difference seems to be applying existing counterfactual-replay attribution at SCG’s finer granularity. Is that right?

The claim to unify prior methods amounts to relabeling them in the paper’s terminology, without yielding any new result, comparison, or insights. Perhaps I am missing something here.

Why is there the word “paper” after every reference?

To be honest, I am a bit surprised that all experiments are in an Appendix.

Reviewer_oqaMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a behavior-level accountability framework for LLM-based multi-agent systems, building upon Semantic Cooperative Games (SCG). The proposed BCAST method estimates the effects of individual semantic behaviors on task outcomes through counterfactual replay and uses Behavioral Shapley values to allocate responsibility among behaviors and agents. The paper further investigates whether Shapley-based penalties can align individual agents' mode-selection incentives with overall system performance, providing theoretical guarantees under specific assumptions.

Critical strengths
  1. The paper extends responsibility attribution from the agent or step level to individual semantic behaviors. Combining semantic dependency structures with counterfactual replay provides a potentially useful framework for distinguishing loss-increasing behaviors from corrective behaviors that reduce downstream loss. This is a meaningful problem formulation for understanding failures in multi-agent systems.

  2. The paper attempts to connect retrospective behavioral attribution with prospective incentive design. The theoretical analysis provides a bound on the performance gap between equilibrium and optimal configurations under explicit assumptions. This connection is conceptually interesting and could motivate further research on accountability mechanisms for multi-agent collaboration.

Critical weaknesses
  1. The main paper presents almost no quantitative experimental results, relegating nearly all evaluations to the appendix. Even there, BCAST lacks systematic comparisons with strong baselines and independent ground-truth responsibility labels. The reported 93.6% agreement is dominated by zero-effect cases (1,060/1,181), while the validation relies on counterfactual replay records from the same evaluation framework. Moreover, Shapley estimates use at most two sampled backgrounds per behavior, raising concerns about their reliability.

  2. The incentive experiment relies on a single constructed three-agent procurement task with manually designed Careful, Shortcut, and Distorter modes. Finding that Careful achieves the optimal outcome is therefore unsurprising and provides limited evidence of generalizability. Furthermore, key theoretical assumptions, including effect stability and a common cancellation reference, are not empirically validated in realistic multi-agent environments. Consequently, the experimental evidence is insufficient to support the paper's broader incentive-alignment claims.

Questions for authors
  1. How can the authors demonstrate that BCAST produces accurate behavioral responsibility assignments beyond consistency with its own counterfactual replay evaluations? Are there independently annotated responsibility benchmarks or quantitative comparisons with existing agent-level, step-level, or counterfactual attribution methods that support the claimed advantages?

  2. Given that the main experiment uses at most two sampled backgrounds per behavior, how reliable are the estimated Behavioral Shapley values? Can the authors provide a more comprehensive convergence analysis, including estimation uncertainty, sensitivity to sampling budgets, and the effects of excluding non-executable counterfactual combinations?

  3. Can the authors provide evidence that the proposed incentive mechanism improves overall performance in realistic multi-agent tasks beyond the constructed procurement scenario? More specifically, how sensitive are the equilibrium results to violations of the effect-stability assumption, and are the corresponding stability parameters empirically small in representative LLM workflows?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.