- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies external-evidence attribution in hierarchical multi-agent systems. Using TradingAgents, the authors remove individual news or social-media items, replay the affected downstream workflow, and compare changes at the local analyst, Research Manager, and final trading decision. Final influence is defined as the absolute mean change in numerically encoded trading actions across three paired runs. The proposed PAR method uses two intermediate screening stages to select candidates for final verification. Evaluation on fifteen ticker-date cases and three LLM backends reports stronger alignment between downstream sensitivity and final influence than between local sensitivity and final influence, together with 79.53% recall and a projected 20.70% reduction in structural verification cost.
Critical strengths
-
The attribution question is practically relevant. In hierarchical multi-agent systems, an individual agent's response to evidence need not reflect the final decision's dependence on that evidence. Item-level removal followed by downstream replay provides a direct intervention for studying this distinction.
-
Several limitations are disclosed clearly. The authors acknowledge differences between checkpoint scores, the retrospective nature of the evaluation, and the distinction between projected structural cost and measured execution savings. These qualifications help delimit what the experiments establish.
Critical weaknesses
-
The central empirical conclusion is confounded by the measurement design and does not isolate the role of propagation. Local sensitivity uses character-sequence dissimilarity, whereas downstream sensitivity prioritizes changes in an investment rating closely related to the final BUY/HOLD/SELL outcome. The comparison therefore changes checkpoint location, output semantics, and scoring rule simultaneously. It establishes that the chosen downstream score is better aligned than the chosen local text score, but does not separate propagation from the advantage of a more outcome-aligned metric. The discussion acknowledges this limitation, yet no consistent semantic scoring or comparable control resolves it. The final reference score also averages signed action changes before taking the absolute value, using only three runs. Opposing decision changes can consequently cancel to zero. The resulting final-important set captures a particular directional-average effect, whose stability and sensitivity as a screening reference are insufficiently validated.
-
PAR's practical utility is not established through adequate comparisons or independent execution. The main ablations use different retention levels and computational budgets, preventing a meaningful comparison with equal-budget random screening or G2-only screening. PAR verifies approximately 69.91% of candidates while still missing 20.47% of final-important observations. Notably, 199 misses occur at the local screening stage, despite the paper's central finding that local sensitivity is weakly aligned with final influence; candidates discarded there cannot be recovered downstream. The reported 20.70% saving is reconstructed retrospectively from exhaustive trajectories, without native PAR execution measuring calls, tokens, or runtime. Evaluation is also confined to fifteen cases in one system, with some cases inspected during development. Independent validation and actual screened execution are essential evidence for the current effectiveness and efficiency claims, rather than peripheral future extensions. The present results do not establish a reliable attribution method with demonstrated practical computational advantages.
Questions for authors
-
How much of the downstream alignment advantage remains when and use a consistent semantic scoring criterion? Can the authors separate the contribution of checkpoint location from that of the scoring rule, rather than comparing character dissimilarity with investment-rating changes?
-
At the same total structural cost, how does PAR compare with random screening and G2-only screening? Please provide budget-recall curves and establish whether the savings from local screening justify its irreversible false negatives.
-
How sensitive is the final-important set to the number of seeds and the effect definition? Do the main correlation and screening conclusions persist under decision-change probability, mean absolute action change, and the current absolute mean signed change? What additional state or execution randomness explains the appendix case with identical saved handoffs but different downstream outcomes?
-
Can PAR be executed as an actual screened workflow on cases not inspected during development, with calls, tokens, runtime, and recall reported? How closely does the structural projection approximate execution cost when loops, retries, and repair behavior are included?