acceptodds
Under review as a conference paper at ICLR 2027

Attribution Is a Propagation Problem: End-to-End Causal Evidence Attribution in Multi-Agent Systems

Abstract

Large language models are increasingly used in hierarchical multi-agent systems, where external evidence can be transformed and passed through several agents before affecting a final decision. This raises a basic attribution question: does evidence that appears important when it first enters the system also matter to the final outcome? We study this question using controlled evidence-removal interventions and use TradingAgents, a 12-agent trading system, as our testbed. For each removable evidence item, we compare its sensitivity at the agent that first processes it, at a shared downstream checkpoint, and at the final decision. Across three LLM backends, local sensitivity is only weakly aligned with final causal influence, while downstream sensitivity is substantially more aligned. We introduce Propagation-Aware Replay (PAR), which uses intermediate propagation states to decide which evidence items receive full end-to-end causal verification. Our study covers 25 ticker-date cases, with the main evaluation on 15 cases and 2,346 candidate-model observations. At the evaluated operating point, PAR retains 79.53% of evidence with measured final causal influence while reducing projected structural verification cost by 20.70%. These results support treating attribution in hierarchical multi-agent systems as an end-to-end propagation problem rather than a purely local one.

Primary area:
interpretability and explainable AI
Keywords:
multi-agent systems, causal attribution, evidence attribution, LLM agents, counterfactual reasoning, interpretability, explainability, agentic systems

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_1oE5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies external-evidence attribution in hierarchical multi-agent systems. Using TradingAgents, the authors remove individual news or social-media items, replay the affected downstream workflow, and compare changes at the local analyst, Research Manager, and final trading decision. Final influence is defined as the absolute mean change in numerically encoded trading actions across three paired runs. The proposed PAR method uses two intermediate screening stages to select candidates for final verification. Evaluation on fifteen ticker-date cases and three LLM backends reports stronger alignment between downstream sensitivity and final influence than between local sensitivity and final influence, together with 79.53% recall and a projected 20.70% reduction in structural verification cost.

Critical strengths
  1. The attribution question is practically relevant. In hierarchical multi-agent systems, an individual agent's response to evidence need not reflect the final decision's dependence on that evidence. Item-level removal followed by downstream replay provides a direct intervention for studying this distinction.

  2. Several limitations are disclosed clearly. The authors acknowledge differences between checkpoint scores, the retrospective nature of the evaluation, and the distinction between projected structural cost and measured execution savings. These qualifications help delimit what the experiments establish.

Critical weaknesses
  1. The central empirical conclusion is confounded by the measurement design and does not isolate the role of propagation. Local G1G_1 sensitivity uses character-sequence dissimilarity, whereas downstream G2G_2 sensitivity prioritizes changes in an investment rating closely related to the final BUY/HOLD/SELL outcome. The comparison therefore changes checkpoint location, output semantics, and scoring rule simultaneously. It establishes that the chosen downstream score is better aligned than the chosen local text score, but does not separate propagation from the advantage of a more outcome-aligned metric. The discussion acknowledges this limitation, yet no consistent semantic scoring or comparable control resolves it. The final reference score also averages signed action changes before taking the absolute value, using only three runs. Opposing decision changes can consequently cancel to zero. The resulting final-important set captures a particular directional-average effect, whose stability and sensitivity as a screening reference are insufficiently validated.

  2. PAR's practical utility is not established through adequate comparisons or independent execution. The main ablations use different retention levels and computational budgets, preventing a meaningful comparison with equal-budget random screening or G2-only screening. PAR verifies approximately 69.91% of candidates while still missing 20.47% of final-important observations. Notably, 199 misses occur at the local screening stage, despite the paper's central finding that local sensitivity is weakly aligned with final influence; candidates discarded there cannot be recovered downstream. The reported 20.70% saving is reconstructed retrospectively from exhaustive trajectories, without native PAR execution measuring calls, tokens, or runtime. Evaluation is also confined to fifteen cases in one system, with some cases inspected during development. Independent validation and actual screened execution are essential evidence for the current effectiveness and efficiency claims, rather than peripheral future extensions. The present results do not establish a reliable attribution method with demonstrated practical computational advantages.

Questions for authors
  1. How much of the downstream alignment advantage remains when G1G_1 and G2G_2 use a consistent semantic scoring criterion? Can the authors separate the contribution of checkpoint location from that of the scoring rule, rather than comparing character dissimilarity with investment-rating changes?

  2. At the same total structural cost, how does PAR compare with random screening and G2-only screening? Please provide budget-recall curves and establish whether the savings from local screening justify its irreversible false negatives.

  3. How sensitive is the final-important set to the number of seeds and the effect definition? Do the main correlation and screening conclusions persist under decision-change probability, mean absolute action change, and the current absolute mean signed change? What additional state or execution randomness explains the appendix case with identical saved handoffs but different downstream outcomes?

  4. Can PAR be executed as an actual screened workflow on cases not inspected during development, with calls, tokens, runtime, and recall reported? How closely does the structural projection approximate execution cost when loops, retries, and repair behavior are included?

Reviewer_EU2fOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether an external evidence item’s local sensitivity predicts its influence on the final decision of a multi-agent system. It constructs an evidence-removal evaluation in TradingAgents and proposes Propagation-Aware Replay (PAR), which screens candidates at local and downstream checkpoints before final verification. The contribution is an empirical attribution study and an inference-stage screening method. Across 15 ticker–date cases and three model backends, PAR reports 79.53% recall with a 20.70% reduction in projected structural cost.

Critical strengths

1 Following individual external evidence items to the final system decision is a useful interpretability objective. The paper examines a concrete limitation of relying on the immediate response of the first agent. 2 The exhaustive evidence-removal results across three model backends provide a useful reference for assessing screening coverage within the chosen TradingAgents configuration.

Critical weaknesses

1.The evaluation is too narrow to support general conclusions about attribution in multi-agent systems.

The study uses only one trading-agent architecture. Changing the underlying language model does not test whether the findings survive changes in agent connectivity, communication, or decision aggregation. This is particularly consequential because PAR depends on a shared downstream checkpoint whose output is already closely aligned with the final decision. The observed advantage may therefore reflect the structure of this particular pipeline rather than a general property of multi-agent attribution.

The evaluation also relies on a small, concentrated collection of stock–date scenarios, with repeated stocks and nearby dates. Numerous removal interventions within these scenarios do not provide equivalent diversity or independent evidence. Limited repetitions and the absence of uncertainty estimates leave the stability of the attribution rankings and screening gains insufficiently characterized. Moreover, some evaluation scenarios were inspected during development, so the reported results do not establish robustness on an untouched cohort. Together, these limitations substantially weaken both the study’s generality and the empirical validation of PAR.

2.The central finding and screening method are not sufficiently established as substantive advances.

Efficient removal-based attribution and computation reuse have already been studied (AttriBoT, ICLR 2025; HiveMind, AAAI 2026), as has causal-effect propagation through other agents (Agent-Specific Effects, ICML 2024). These works address different attribution targets, but the submission does not sufficiently establish an additional methodological contribution beyond changing the intervention unit and combining familiar screening and replay principles.

The main empirical comparison also changes the scoring rule together with checkpoint depth: local textual change is compared against downstream investment-rating change. Consequently, stronger downstream alignment does not isolate a propagation effect. Furthermore, the proposed method first prunes using the same local signal that the study finds weakly informative. Such pruning could still be useful as a cheap filter, but its benefit requires a matched-budget comparison. The current ablations do not establish whether this stage improves the cost–coverage trade-off over downstream-only screening or simple sampling followed by complete verification. The paper therefore leaves both the interpretation of its central observation and the necessity of its proposed design insufficiently supported.

Questions for authors
  1. Relative to published methods (AttriBoT, ICLR 2025; HiveMind, AAAI 2026; Agent-Specific Effects, ICML 2024), does PAR offer sufficient methodological novelty beyond changing the intervention unit and combining existing screening and replay principles?

  2. Do the findings generalize across different agent architectures and aggregation mechanisms, rather than only different language models within TradingAgents?

  3. At matched computational budgets, how does PAR compare with uniform sampling followed by complete verification and downstream-only screening, including uncertainty in the differences?

  4. Does the downstream advantage persist when local and downstream checkpoints use comparable scoring rules, rather than comparing local textual changes against downstream investment-rating changes?

  5. Given the evaluation’s concentration in AAPL and closely spaced dates, do the conclusions remain stable when these cases are excluded or additional stocks and time periods are evaluated?

  6. Which evaluation cases influenced method development or parameter selection, and what results remain on entirely untouched cases?

  7. How were stocks, dates, and evidence items selected, including any filtering for human-perceived decision clarity, and was selection fixed before inspecting model outputs or attribution results?

  8. How stable are the findings across independent cases and repeated runs, with uncertainty estimates accounting for observations that share the same case?

  9. How does the cost–coverage trade-off change across screening budgets, and is the reported operating point representative?

Reviewer_KZUhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper asks whether evidence that looks important where it first enters a hierarchical multi-agent system also matters for the final decision. In TradingAgents, it removes each social-media post or news item in turn and replays the system, comparing sensitivity at the analyst that first reads the item (G1), at the downstream Research Manager (G2), and at the final decision. Over 15 cases, 782 items and three 8B backends, G1 is essentially uncorrelated with final influence (mean within-case Spearman ρ between −0.102 and 0.075), while G2 is much more aligned (ρ from 0.463 to 0.760). PAR uses the two checkpoints to choose which items receive full verification. At one operating point it recalls 79.53% of the items with non-zero final influence and reduces projected structural cost by 20.70%, a projection from exhaustive trajectories rather than a native run.

Critical strengths

(1) The unit of analysis is well chosen: a single external evidence item followed from entry to the final decision, with exhaustive removal replay as the reference (2,346 removal trajectories per model, l.322). The G2-over-G1 pattern holds across all three backends and most cases (Table 1; l.377–378, l.404–405). The paper is also candid about its limits, including that cost is projected rather than measured (l.286–287), that some evaluation cases were inspected during development (l.311–313), and that G1 and G2 produce different kinds of output (l.466–469).

Critical weaknesses

(1) PAR, the paper's method contribution, does about as well as random selection at the same cost under the paper's own accounting. Exhaustive verification costs 63,342 structural stage slots and PAR costs 50,232, i.e., 79.30% (App. D). Verifying a random 79.3% of items fully costs the same and has an expected recall of 79.3%; PAR's recall is 79.53%, and per backend the difference from random ranges from about −2.9 to +2.2 points. App. D's per-model totals seem to charge G2 replay for all G1 survivors in every run, whereas Algorithm 1 screens each run separately (l.255); by my count, per-run charging could not bring PAR's cost much below 77%, so its advantage would stay within about 3 points.

Table 6 shows why. The G1 stage behaves like random dropping: its recall matches its retention rate (Llama: 92.37% recall with 720/782 = 92.07% of items kept; pooled: 88.29% with 88.06% kept), and its precision is 72.65% against a base rate of 72.46%. The two other settings in Table 6 do not change this: G1-only falls slightly below matched-cost random, and G2-only (in effect b1 = 1) is about 2 points above it. Because the first stage carries no measurable information about final influence, a different operating point is unlikely to fix the method.

(2) The descriptive finding, which carries the paper's conclusion that attribution should be treated as a propagation problem (l.26–28), confounds checkpoint position with the type of measurement and has no noise floor. G1 is measured as character-level change in a free-text analyst report and G2 mainly as a change in a five-level rating (l.324–330). A rating closer to the decision can be expected to track the decision more closely, and the authors themselves note that the stronger agreement at G2 may not be caused only by propagation (l.466–469).

Separately, for 1,700 of the 2,346 item–backend observations (72.46%), removing the item shifts the final decision in at least one of the three seeded runs without the shifts cancelling (Eq. 1). In a 12-agent system, that rate raises the question of how often the decision changes after any small edit of the input, whatever the item. Decoding is at temperature 0 (l.316), but the paper reports neither how often the clean decision agrees across the three seeds nor how often it flips under a placebo edit, so it is hard to tell how much of the measured final influence belongs to the evidence item. With one system and 15 cases, these controls matter for every conclusion in the paper.

Questions for authors

(1) Using the existing trajectories, could you report a cost–recall curve over (b1, b2) together with matched-cost random selection, and clarify whether App. D charges G2 replay per run as in Algorithm 1? Survivor sets are reconstructed from exhaustive trajectories, so this needs no new model calls. If some setting clearly beats random, my assessment of the method would change.

(2) How often does the clean decision agree across the three seeds, and what is the flip rate under a placebo edit (for example swapping in an item from an unrelated ticker or date, or reordering the items)? How do the main results change under a stricter threshold on final influence?

(3) With the same type of measurement at both checkpoints (for example text dissimilarity at G2, which the paper already computes as a tie-breaker, or a rating-type score at the analyst), does G1 remain weakly aligned?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.