- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Agentic Intellectual Property (AIP), a multi-agent framework that aims to improve long-horizon tool use through contribution recognition and peer review. Agents generate Action, Evidence, or Synthesis contributions, which undergo peer review before being selectively incorporated into a shared ledger to inform a Final Judge. Experiments on BFCL and ToolHop demonstrate improved accuracy over existing single-agent and multi-agent baselines.
Critical strengths
-
Practical and relevant problem. The paper addresses an important challenge in long-horizon multi-agent tool use by exploring contribution recognition, peer review, and selective information sharing.
-
Illustrative case studies. The paper provides concrete examples of how agent contributions can help correct errors and inform subsequent tool-use decisions, offering useful insights into the proposed framework's behavior.
Critical weaknesses
-
Insufficient methodological justification. While AIP is motivated by contribution recognition and attribution, it remains unclear how faithfully peer-assigned credit reflects each agent's actual contribution to the outcome. The paper lacks direct empirical validation of this attribution mechanism. Moreover, despite the peer-review process, the final decision ultimately depends on the Final Judge, which can independently modify or disregard accepted contributions. This raises questions about how much the proposed credit mechanism influences the final decision. More broadly, AIP shares similarities with existing critique-and-aggregation frameworks [1,2,3], and its methodological contribution is not sufficiently justified.
-
Insufficient empirical analysis. While the authors evaluate AIP on BFCL and ToolHop, the experimental evaluation has several limitations:
-
Marginal improvements, computational budgets, and generalizability. Some improvements over the strongest baselines are relatively small (e.g., 40.1% vs. 39.7% on ToolHop), yet the main results lack confidence intervals or significance tests to establish their robustness. Moreover, it is unclear whether the comparisons are fair in terms of computational budgets. For example, AIP requires up to 50 model calls per decision state, but no computation-matched comparisons are provided to determine whether the improvements persist under comparable inference budgets for different baselines. Finally, given that AIP is proposed as a general multi-agent collaboration framework, the authors do not sufficiently justify why the evaluation is restricted to two tool-use benchmarks.
-
Missing ablation analysis and controlled comparisons. The individual contributions of Action, Evidence, and Synthesis remain unclear, as the Action-only ablation removes both Evidence and Synthesis simultaneously. Moreover, AIP employs seven contributors and a separate Final Judge, while the baselines adopt different agent configurations. More controlled comparisons are needed to isolate the effects of individual components and establish whether the improvements arise from the proposed contribution-centered mechanism rather than differences in agent configurations.
-
Reference
- Liang, Tian, et al. "Encouraging divergent thinking in large language models through multi-agent debate." EMNLP 2024
- Wang, Junlin, et al. "Mixture-of-agents enhances large language model capabilities." ICLR 2025
- Qian, Chen, et al. "Scaling large language model-based multi-agent collaboration." ICLR 2025
Questions for authors
See weakness
Gentle Reminder: ICLR Rebuttal Discussion Deadline – November 18
Dear Reviewers,
Thank you very much for your time, thoughtful reviews, and valuable contributions to the evaluation process. Thank you to the reviewers who have already interacted and provided additional feedback to the authors!
The authors have carefully addressed your comments and concerns, and the discussion period will conclude on November 18. We would greatly appreciate it if you could take a few moments to review their responses and results, engage in the discussion, and indicate whether their clarifications or additional evidence affect your assessment.
Your continued engagement is essential to ensuring a rigorous, constructive, and fair evaluation process.
Thank you again for your time, expertise, and dedication!
Best regards, Area Chair