acceptodds
Withdrawn from ICLR 2027

Do Collaboration Metrics Track Multi-Agent Success? A Cross-Benchmark Measurement Audit

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-agent LLM systems are increasingly deployed on complex, long-horizon tasks, but are usually evaluated through final outcomes or structural scale, neither of which tells us whether the collaboration itself was effective. Successful trajectories can contain poor coordination, and failed trajectories can still involve useful information exchange. We therefore investigate whether trajectory-level collaboration metrics provide stable signals of task success across domains, models, and system designs. We evaluate five metrics of collaboration quality, namely a holistic judged communication score, judged redundancy, judged downstream usage, judged contradiction rate, and semantic evolutionary distance, across four multi-agent benchmarks spanning algorithmic coordination, software engineering, and enterprise workflows. The study covers five language models and 18 model-benchmark configurations. Four of the five metrics fail to hold up. The narrower per-event judged measures survive multiplicity correction in at most six of 18 configurations, while evolutionary distance changes sign across settings and even within a single benchmark, so neither can be read as a directional quality score. The fifth, a holistic communication score, is associated with task success in 14 of 18 configurations (–), every one surviving correction, but its rubric partly references task progress. We next ask whether the same signal transfers to a different outcome. In a secondary analysis of 1,500 trajectories from three privacy-constrained benchmarks, no general collaboration metric survives multiplicity correction as a predictor of information exposure, and this non-detection holds whether exposure is measured across the whole trajectory or restricted to failures that never reach the final output, so the privacy analysis can rule out large associations while smaller effects may still have gone undetected. Under this instrument, the measures that are cleanly separable from the task-success outcome carry little signal that transfers across benchmarks, and the one measure that does carry signal is not cleanly separable from it. Collaboration metrics should therefore be validated for a specified outcome and setting rather than treated as general indicators of collaboration quality.

Primary area:
datasets and benchmarks
Keywords:
multi-agent LLM systems, collaboration metrics in multi-agent LLM systems, agentic AI, AI evaluation metrics

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_P8wuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits whether trajectory-level collaboration measures are associated with multi-agent task success across settings. It screens 25 candidates, retains five measures, and studies 18 model–benchmark configurations across four primary benchmarks and five models. Communication score has the most consistent reported task-success association, whereas the other measures have fewer corrected detections or changing signs. The paper examines rubric overlap and judge dependence through rescoring controls. A separate analysis of 1,500 trajectories from three privacy benchmarks finds no multiplicity-corrected association between general collaboration measures and leakage, and shows that many observed leaks are absent from final outputs. The intended contribution is an empirical measurement audit and guidance on outcome- and context-specific metric validation.

Critical strengths
  1. The study makes a useful measurement question concrete and provides meaningful sensitivity checks. The candidate screen and analytical-role separation distinguish process measures, task outcomes, and robustness outcomes (§3.2; Table 2, p.12). The primary analysis reports per-metric samples, sentinel exclusions, Spearman checks, and both within-metric and global Holm correction (Appendices A.1–A.3). Particularly valuable are the communication-score controls: removing task-progress wording leaves 11 of 18 corrected associations versus 14 originally, while a second judge yields 13 (Tables 6–7, pp.19–21). These results support an informative, qualified finding rather than an unexamined correlation with an outcome-overlapping rubric.

  2. The privacy analysis supplies a practically relevant outcome distinction. Of 259 trajectories with observed leakage, 206 lack leakage in the final output (79.5%; §4.2 and Table 8). This supports inspecting internal channels in the evaluated settings. The analysis separates completion from leakage, uses scenario/model fixed effects and scenario-clustered inference, and includes strict multi-agent and hidden-leak sensitivity checks. Its explicit power discussion, including the 7.3-percentage-point median minimum detectable effect for leakage, appropriately cautions against interpreting non-detection as equivalence (Appendices B.4–B.11).

Critical weaknesses
  1. The paper does not yet establish an unambiguous mapping from the stated metrics to the measurement procedure. Section 3.3 (p.5) defines redundancy and downstream usage as shares of eligible events, and contradiction as the share of events participating in contradictions unresolved at trajectory end. However, the prompts described as verbatim in Appendix A.6.1 ask for fractions of information within a message for the first two measures, and counts of contradictory assertions and total assertions for contradiction (pp.17–18). The contradiction prompt does not specify excluding resolved contradictions. These are materially different units and constructs unless an additional transformation is supplied. Eligibility is also unclear: §3.3 says a trajectory with no “relied-upon successor” contributes no observation, whereas Appendix A.1 (p.13) describes absence of eligible later outputs. A valid trajectory with successors but zero downstream use should be distinguishable from an undefined measure. This is a reporting ambiguity, not a demonstrated implementation bug, but it affects three of the five audited measures and therefore the central interpretation.

  2. The primary inference does not sufficiently support the strength of the cross-context failure conclusion. Section 3.6 (p.7) explicitly acknowledges non-independent observations within primary cells and optimistic standard errors; Appendix A.3 states that scenario identifiers are unavailable. Nevertheless, the main summary relies on unclustered correlation tests and counts of Holm survivors. Holm correction does not repair invalid individual-test uncertainty. Furthermore, different rejection decisions do not establish different underlying associations: the event-level measures have expected signs in 14/18, 16/18, and 16/18 configurations, while sample sizes range from 45 to 213 (Tables 1, 3–4). The paper discusses unequal power and appropriately bounds its privacy nulls, but the primary claim that four measures “fail to hold up” needs dependence-aware uncertainty and a clearer distinction between heterogeneity, weak association, and insufficient precision. The stronger secondary analysis and the progress-blind control do not resolve this primary inference problem.

Questions for authors
  1. Clarification: What exact prompts, aggregation formulas, and eligibility rules produced the three event-level columns? Please reconcile §3.3 with Appendix A.6.1 and show a short worked example containing a resolved contradiction and an unused message. Are trajectories with eligible successors but zero downstream use retained as zero?
  2. Clarification and limited validation: How many distinct scenarios and repeated runs occur within each primary benchmark–model cell? Please distinguish repetition within a cell from scenario pairing across models. If within-cell repetitions are present, can the main and progress-blind correlations be reanalysed with scenario-level resampling or an appropriate clustered procedure, and the resulting Holm decisions reported?
  3. Limited reanalysis or claim clarification: Which conclusions about cross-context instability remain after accounting for uncertainty in differences between associations, rather than comparing significance decisions? An analysis of the existing data, or narrower descriptive claims where identification is unavailable, would materially affect the assessment.
Reviewer_Poa2OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors analyze how existing communication metrics for multi-agent systems correlate with task success in different settings and conclude that most of them are unreliable across predictors of success.

Critical strengths
  • Relevant problem statement.
  • There was at least some rigour in judging noise and attempting to eliminate noise.
Critical weaknesses
  • The paper is written in a way that makes it very hard to follow. In addition to this being a reason for a low score in itself, it gives me low confidence that I can adequately evaluate its claims and experiments. I suspect a large fraction of this paper was LLM generated or heavily LLM assisted. While this is not inherently problematic, in this case it results in the paper often being very unclear and difficult to understand. I listed many examples below. I recommend the authors spend more time on improving clarity and communication of their research.

  • The abstract mentions things like “per-event judged measures”, “multiplicity correction”, “evolutionary distance changing signs” or “information exposure” without ever introducing them.

  • This paper has ZERO plots! The main text has one single table. All the other results are in text. Please help out the reader a bit by visualizing your findings!

  • 3.3: the individual scores are poorly explained.

    • L 223: what does the judge score for the communication score? Is the rating C \in {1,2,3,4,5} related to the C1, C2,...C5 in 3.2? No, right, since 3.2 explains the exclusion criteria? If not, I strongly recommend renaming this as it causes confusion.
    • Then in l 224 you state “the three event level rates”: which event level rates? This wasn’t clear from context.
    • For redundancy rate: Is it “num of events that repeat information”/m-1 (bc one is missing?)? I think it would be helpful to be more precise here so that the reader has to guess less. Also, how do you classify repetition?
    • L 228: “This is the most restrictive denominator, since a trajectory in which no event has a relied-upon successor contributes no observation at all, which is why downstream usage has the smallest analysed samples in Section 4.1.” I don’t understand this sentence. What “downstream usage”? I’d recommend not referring to information that comes in the future as the paper is read linearly. We don’t know what’s done in Section 4.1 when reading 3.3.
    • How do you measure whether a message relies materially on smth? Same for the contradiction rate? With a judge? If so, you need to specify that. Edit: it’s mentioned in 3.4. I find that confusing, it should at least be mentioned that they are evaluated by an LLM judge. You also mention a judge for the Communication Score.
  • Some evidence of other very unclear things across chapters:

    • L 253 “First, the communication rubric partially distinguishes its middle bands by whether the task progresses” What middle bands? The communication score is never explained?
    • L 257 “We therefore read communication score as a diagnostic of process rather than an independent cause of success,” what does this mean?
    • L 265 “Second, all four judged measures come from a single batched judgment procedure” No idea what that means.
Questions for authors
  • Do you look at reasoning traces as well? Or what does “complete trajectory” entail? As far as I can tell from l. 218 it’s really only inter-agent messages? Might be good to mention this earlier.
  • L 203: what does “mapped to exactly one process dimension” (C4) mean?
Reviewer_mQZoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is about whether trajectory-level collaboration metrics provide stable signals of task success across domains, models, and system designs. It emphasizes that task success does not equate to a high-quality collaborative process. It evaluates five metrics across four multi-agent benchmarks and 18 model-benchmark configurations, followed by a secondary analysis of 1,500 trajectories from three privacy benchmarks to examine whether these metrics are associated with information exposure. Communication score shows the most consistent association with task success, although its rubric partly references task progress. The other metrics show less consistent or setting-dependent associations, and no association between the general collaboration metrics and privacy leakage survives multiple-testing correction.

Critical strengths

I like the idea of this paper on focusing the quality of collaboration in MAS. The success of a task alone cannot tell us whether agents did well or exchanged useful information in their work.

  1. This paper has a clearly defined research question and practical significance: It investigates multi-agent systems from the perspective of their task scores, communication volumes, and collaboration quality, as these three concepts are often confused with each other. It attempts to provide clarity on these three measures and to demonstrate how they can be used to avoid making the assumption that a high task score automatically indicates that a system is collaborating effectively.
  2. Cross-benchmark evaluation and robustness checks have substantial value: This paper covers a wide range of domains and models rather than using the metrics on a single task, and it reports results including Holm correction, Spearman analysis, results from a second judge.
Critical weaknesses
  1. The metric definitions are not reconciled with the evaluation prompts: three of the five have unclear mappings between their definitions and evaluation prompts. Section 3.3 defines redundancy rate and downstream usage as proportions of events, while Appendix A.6.1 asks for proportions of the information within individual messages. Furthermore, the paper does not explain how trajectory-level aggregation is performed.
  2. The headline claims are not supported by the analysis: The negative claim (whether a metric holds up) is judged by how many cells survive per-metric Holm correction (with n as low as 45). The associations for the three event-level rates are small but consistent: pooling the paper’s own Table 3–4 coefficients (random effects on Fisher z), I obtain redundancy r = −0.17 [−0.27, −0.08], downstream usage +0.13 [+0.07, +0.19], and contradiction −0.20 [−0.27, −0.12] (intervals are optimistic because cells share scenarios), and the expected sign holds in 14, 16, and 16 of the 18 cells. If every cell were to share one true correlation of the above values then the reported n would yield only about 3, 1, and 5 Holm survivors (at or below the observed 6, 3, and 6) and thus survival counts cannot distinguish a uniformly small effect from an unstable one. The positive claim is also problematic. The judge reads each trajectory to its end (Sec. 3.4) and the primary files lack scenario identifiers (App. A.3) so communication score’s association cannot be separated from outcome visibility or scenario difficulty. The progress-blind check removes rubric wording, not the outcome information in the transcript. Where scenario and model fixed effects are used (e.g. in the privacy testbed) no metric’s association with completion survives correction (smallest adjusted p = .131, App. B.4) so the RQ3 contrast changes the outcome and the estimand at the same time. These two claims are the paper’s contribution and neither is established.
Questions for authors
  1. Section 3.3 defines redundancy rate as a fraction of events, while in Appendix A.6.1 the percentage of redundant information within each message is requested. How are the message-level scores for redundancy rate and downstream usage rate combined into a score for each trajectory?
  2. How do you check whether a contradiction remains unresolved at the end of a trajectory, i.e., does the prompt in Appendix A.6.1 only count contradictions without checking whether they are later resolved?
  3. The paper does not report human labels. Can you report agreement with a human-annotated subsample or second judge for redundancy, downstream usage, and contradiction rate?
  4. What is the rule used to eliminate the extra downstream-usage trajectories (27 out of 119 in Silo-Bench/GPT 5.4 Mini)? Are the trajectories that are eliminated different from the ones that are kept in terms of completion? Do the 21–26% of AgentCollabBench trajectories that are dropped because of judge errors (App. A.1) differ in any way from the ones that are kept?
  5. Is it possible to recover the scenario identifiers for the primary benchmarks in order to estimate the communication-score association within scenarios? Does the association survive re-scoring with the final output masked?
Reviewer_ec2pOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The tasks are:

  • AgentCollabBench: general multi-agent tasks scored by final outcome and milestone progress.
  • Silo-Bench: algorithmic coordination tasks where each agent holds part of the information and the team should combine it.
  • EntCollabBench: enterprise workflows where role-specialized agents delegate and hand off work, scored by execution traces and database changes.
  • CooperBench: coding tasks where agents implement interdependent parts of the same software.
  • PAC-BENCH: coordination tasks where agents must complete the work without revealing restricted information.
  • AgentSocialBench: tasks where agents act on behalf of different users in a social network while protecting each user’s private information.
  • AgentLeak: short two-agent tasks instrumented to detect private data leaking through internal channels.

It is unclear to me after reading the paper what is the structure of the agent team here. The paper suggests that the benchmarks come with their own harnesses but it looks like there is more than one harness that some of these benchmarks offer and I could not understand which one the authors have used.

The paper uses 5 metrics:

  • Communication score: an LLM judge rates the overall quality of the agents’ communication on a 1-5 scale.
  • Redundancy rate: the share of messages that repeat earlier content without adding new information.
  • Downstream usage: the share of messages that later messages actually build on.
  • Contradiction rate: the share of messages involved in a contradiction that is still unresolved at the end.
  • Evolutionary distance: the average change in meaning between consecutive messages, measured with text embeddings.
Critical strengths

S1. Measuring how well agents collaborate, beyond whether the task succeeds, is an interesting and timely problem.

S2. The paper runs a large set of experiments across four benchmarks and five models, and adds robustness checks such as a second judge and a rubric without task-progress wording.

Critical weaknesses

W1. Only 2 out of the 5 metrics actually come from other papers. The other metrics are invented by the authors themselves (marked “Op.” in Table 2, with no cited source), who then show that they do not work . It should not be difficult to invent a metric that does not work. Therefore, the paper should acknowledge that these are not standard metrics in the field. The paper introduces these three metrics and then shows that they do not work, which makes it unclear whether “Four of the five metrics fail to hold up.” statement is meaningful at all.

W2. The paper doesn’t describe the agent team structures. The paper suggests that all of their benchmarks supply the team structure. But at least four of them (AgentCollabBench, Silo-Bench, CooperBench and AgentSocialBench) can be run with different topologies, protocols, team sizes or settings. The paper never says which ones it used. However, AgentCollabBench’s own authors found that topology explains a large share of their results. So we know the results are highly dependent on the communication structure, which is not reported in the paper.

W3. Building on W2, even more significant is the fact that RQ1 of the paper asks about generalization across interaction structures, but the paper never describes the interaction structure used in any of the benchmarks. Furthermore, it attempts to answer this question with a methodology that does not make sense. The authors measure the metrics computed in each benchmark, but each benchmark also has its own interaction structure. Thus, neither the task nor the interaction structure is an independent variable. Each task is run with its own interaction structure. It is statistically impossible to extract any meaningful answer to RQ1 from the experiments they conducted. The methodology is fundamentally flawed.

Questions for authors

can you clarify the soundness of your experimental methodology?

What is your justification for not reporting the collaboration structures while your RQ1 is about identifying the effect of the collaboration structures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.