- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduce MAS-Gate, a unified node-centric quantification that provides a common formulation for the distinct objectives of the MAS lifecycle. MAS-Gate evaluates the role of an agent from two dimensions: within-node utility and inter-node strength. Based on this, MAS-Gate constructs node influence scores and applies them to three scenarios. Experiments demonstrate improved orchestration and generalization, effective trained runtime intervention, and gains in assisted attribution across most evaluated settings, supporting the use of a shared quantification mechanism for these distinct objectives.
Critical strengths
1.While existing studies often focus on multi-agent workflow design, runtime intervention, or post-hoc error attribution separately, MAS-Gate attempts to connect these three stages through a unified node quantification framework. 2.MAS-Gate evaluates the role of agents from two dimensions: Within-node Utility and Inter-node Strength, which provides a unified quantitative index for analyzing the positive contributions and negative influences in multi-agent cooperation. 3.Experimental results demonstrate the improvement of MAS-Gate over existing baselines in orchestration and failure localization across interactive environments, multi-hop question answering, general reasoning, and failure attribution.
Critical weaknesses
1.MAS-Gate multiplies the internal utility of nodes by the influence strength between nodes to construct the node influence score. This design is intuitive, but it still relies on a heuristic combination. 2.Although MAS-Gate brings workflow design, runtime intervention, and post-hoc error attribution into a unified framework, the three phases are currently still optimized independently. The paper also lists cross-stage feedback and joint optimization as future research directions in the conclusion. 3.184 execution traces of the Who&When dataset and 137 single target test samples selected from the AEGIS dataset are used in the experiment. This provides preliminary evidence to validate the method, but experiments on joint failures caused by multiple nodes and on the interactions of multiple error sources remain insufficient.
Questions for authors
1.Please clarify why the node impact is defined as , and why multiplication better reflects the actual contribution of the agent. 2.The paper uses an NLI model to estimate the semantic entailment probability between the outputs of an upstream agent and a downstream agent, using it to measure how much upstream information is absorbed by downstream agents. How can it prove that the downstream agent actually uses the information of the upstream agent? 3.The three modules of the method are trained independently, so what additional advantages does the MAS lifecycle framework bring besides sharing the same scoring mechanism?
8 comments
Dear Reviewer dZ97,
We sincerely appreciate your time and effort in reviewing our paper and providing valuable comments. Your insights have been instrumental in refining our work.
To address your concerns, we have provided additional clarifications and experiments to enhance the clarity of our contributions. We hope these responses effectively resolve any ambiguities, provide a more comprehensive understanding of our work, and encourage you to reconsider your rating.
If you have any further questions or require additional clarification, we would be happy to discuss them. We truly appreciate your time and consideration and look forward to your feedback.
Best regards,
The Authors
Dear Reviewer dZ97,
Thank you again for your time and constructive feedback.
With about one day remaining in the author–reviewer discussion period, we would greatly appreciate it if you could take a moment to review our rebuttal and revised manuscript. We have addressed your concerns through additional analyses, experiments, and clarifications.
We would be happy to clarify any remaining questions and welcome your further feedback.
Thank you for your time and consideration.
Best regards,
The Authors
Response to Reviewer dZ97
[W1, Q1] Rationale for multiplicative impact
This multiplicative composition follows the distinct roles of the two quantities: utility determines direction, whereas strength scales magnitude. Accordingly, I=U(1+K) preserves the sign of U and retains local utility when K=0. We therefore adopt this formulation to combine them. The controlled composition ablation (Multiplicative vs Additive) in Appendix D.1.1 supports the effectivess of multiplication design.
[W2, Q3] Benefits of joint lifecycle optimization
Following your suggestion, we add joint design/runtime training, alternating Orchestrator updates from edited execution outcomes with Editor updates on the current workflows. Each policy is evaluated under its original experimental setting for direct comparison of each model: the Orchestrator runs without an Editor, and the Editor uses the original frozen Qwen3-8B Orchestrator for both separate and joint training comparisons. Both policies improve, demonstrating benefits from coordinated training beyond sharing a metric. We have incorporated this experiment into Section 4.2.3 and Appendix D.2.2 of the revised paper.
[W3] Localization of multiple error sources
We extend evaluation to 463 AEGIS trajectories containing multiple faulty agents. Direct and Guided use Qwen3-30B-A3B-Instruct-2507 and gold references, with a guidance budget of k=4. Accuracy requires an exact match of the predicted and annotated agent sets. Guided improves overall accuracy and every target-count group, extending the effectiveness of MAS-Gate to multi-source localization. We have incorporated these results into Appendix D.3.3 of the revised paper.
[Q2] Human validation of semantic support
As for the validation of R, we complement Figure 1(b)'s qualitative analysis with human validation on 50 HotpotQA trajectories. Three blinded annotators (postgraduates in AI, don't know our paper before) achieve Krippendorff's α>0.8 for both quantities, indicating high agreement and reliable annotations. The correlations with mean human ratings confirm alignment with the intended semantic roles of R. Together, the case analysis and human evaluation yield complementary qualitative and quantitative evidence for the shared quantification used across the three lifecycle applications. We have incorporated these results into Section 3.4 and Appendices D.1.3 and D.1.5 of the revised paper.