- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces SharedIntentBench - a benchmark for agents coordinating on behalf of two independent principals.
The information regarding conditions of the acceptance is private to each agent and in some task families one principal's requirements can be updated mid-negotiation, making the previously issued authorizations outdated. The agreement is only valid if the terms satisfy both principals and the authorizations are up to date. The benchmark also evaluates the cases of agents prematurely abandoning the negotiations when an agreement is possible.
Critical strengths
I think this is a strong benchmark with a range of models of different sizes evaluated across multiple task families covering different possible coordination failures and varying levels of difficulty.
I appreciate that the benchmark includes infeasible tasks for which no-deal is the correct outcome. It is also good that the authors verify task solvability and provide scripted valid solutions for all tasks.
The finding that different scoring rules can change model rankings is very interesting. I also appreciate the distinction between different failure modes which provide better insights into agent coordination failures than a single metric of failure/success.
Critical weaknesses
-
I found some of the writing in the paper quite dry and difficult to follow. Especially the "Discussion and Limitations" section reads like a list of points rather than a discussion. I think the paper would benefit from more careful editing.
-
The benchmark assumes that any mandate version update invalidates all previously issued authorizations, even if the agreed terms are still acceptable. This requirement seems unnecessarily strict. Could the authors discuss how this design choice affects the generalizability of the benchmark, or investigate different authorization policies where only approvals affected by the update become invalid?
Questions for authors
-
Section 4 states that unparsable agents' replies consume steps. Among the 'step cap' failure, what percentage do the unparsable replies take? It would be great to be able to distinguish between formatting issues and coordination capability failures.
-
How sensitive are the success rates and model rankings reported to the step cap restriction? Is it possible that weaker models would perform substantially better if given more steps?