acceptodds
Under review as a conference paper at ICLR 2027

Whose Outcome Is the Team's Score? Auditing Aggregation and Scored-Object Binding in Multi-Agent Evaluation

Abstract

Multi-agent systems increasingly coordinate multiple agents to solve complex tasks, yet their evaluation often compresses several agent executions into a single team score. Interpreting such a score requires two mappings that are rarely made explicit: how agent-level outcomes are aggregated into a team outcome, and which execution state or artifact is actually consumed by the grader. We formalize these as agent-to-team aggregation and scored-object binding, and introduce an executable aggregation receipt that makes this measurement path auditable from execution evidence. We apply the framework to a released multi-agent evaluation harness using source-code audits, controlled interventions, and model-driven runs. We then test whether the same audit transfers to an independently developed multi-agent framework. We find that nominally separate agents can become coupled through shared mutable state and that a reported team score can depend entirely on a single agent. In controlled interventions, changing the score-bound agent changes the reported result in all 24 paired comparisons, whereas changing either of the other agents changes none. We further show that unspecified aggregation creates a deterministic ambiguity whose width equals the agent-disagreement rate. On the independent framework, a source-pinned native adapter reconstructs the executed decision path while revealing a mismatch between the declared consensus semantics and the recorded accounting. Finally, we repair the harness by isolating agent states, explicitly aggregating all agent outcomes, and preserving scored-object lineage. Our results show that reliable multi-agent evaluation requires an auditable path from individual agent outcomes to the object actually scored.

Primary area:
datasets and benchmarks
Keywords:
multi-agent evaluation, benchmark methodology, measurement validity, aggregation, LLM agents, evaluation harness, reproducibility, auditing, SWE-bench

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qMQDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the measurement validity of multi-agent evaluation, with particular attention to how individual outcomes are aggregated into team-level outcomes and which state or artifact is actually evaluated by the scorer. Through source-code audits and controlled experiments, the authors trace the scoring procedures and propose an executable “aggregation receipt” for reconstructing and verifying how scores are computed. Across all 24 paired comparisons in the PlanCraft experiments, modifying the first worker’s outcome changed the score, whereas modifying the outcome of either of the other two workers had no effect. Taken together with the repair experiments on SWE-bench and an independent audit of MALLM, these results suggest that the implemented scoring procedures may not align with team-level interpretations of the reported outcomes. However, they do not establish that historical leaderboard scores or rankings were incorrect.

Critical strengths

1.Controlled interventions directly reveal which worker outcomes affect the score. The PlanCraft experiments vary a single worker’s success or failure while holding the other workers’ outcomes fixed, and are complemented by synthetic text replacement experiments. Compared with source-code analysis alone or correlations between scores, this design provides stronger evidence for the conclusion that “only the bound worker’s outcome can reach the scorer.” These controlled interventions provide the strongest empirical justification for accepting the paper.

2.The paper makes scoring interpretable through verifiable execution relationships and identifies concrete semantic issues in an independent framework. By jointly examining agent outcomes, aggregation rules, and the provenance of the objects being scored, the paper enables the scoring process to be reconstructed and verified. The native MALLM adapter reconstructs the answer path and identifies discrepancies between MALLM’s declared consensus semantics and its actual outcome accounting. These findings demonstrate that the paper’s contribution extends beyond describing known implementation behavior in the original framework.

Critical weaknesses

1.The independent validation of aggregation receipts and the method’s added value are not yet sufficiently demonstrated. The generic checker returned “unresolved” for 216 of the 220 MALLM episodes, while the main audit findings depended on a newly introduced framework-specific adapter. Although this is consistent with the authors’ design, it indicates that applying the method to another framework still requires additional semantic modeling. The paper does not yet provide a complete example of an aggregation receipt, pseudocode for the key checks, or explicit results from the claimed validation controls. These omissions make it difficult to determine which capabilities can be reused across frameworks and how the checker avoids reproducing errors in the implementation being audited. Consequently, the evidence supporting the approach as a general-purpose audit method remains limited.

2.The claim that all four predictions were confirmed in the independent-framework experiment requires clarification. The statement that “all four predictions were confirmed” is not explicitly reconciled with the results reported in Appendix E: “at least 9/10 expected, but 8 violations and 2 cases with no output observed.” The main text also applies different counting criteria to execution reconstruction, label consistency, and semantic violations, without clearly explaining how these criteria relate to the four predictions. The authors should report the preregistered criteria for each prediction alongside the episode-level classifications and explicitly distinguish predictions that were met, those that were not met, and those that could not be assessed. Without this information, readers cannot readily verify whether the evidence supports the method’s applicability to another framework.

Questions for authors

1.What provides the basis for the checker’s independent semantic judgments? Please provide a complete example of an aggregation receipt and pseudocode for the key aggregation and scored-object binding checks. For MALLM, how does the checker distinguish between reproducing source-code behavior and independently evaluating compliance with the declared semantics? Please also report the results of each validation control mentioned in Section 4: declaration modification, missing observations, source-code substitution, and ineffective interventions.

2.Were all four preregistered MALLM predictions met? Please report the preregistered pass criteria and observed results for each prediction, and explain how “at least 9/10 violations” is reconciled with “8 violations and 2 cases with no utterance.” At the time of preregistration, were episodes with no output classified as failures, exclusions, or unresolved cases? Was the criterion for handling these episodes specified in advance, and how was it applied when concluding that “all four predictions were confirmed”?

3.Can the episode counting criteria and the scope of the conclusions drawn from MALLM be clarified? For each of the four experimental groups, please report the counts of episodes with answers, unscored episodes, episodes with no utterance, label-consistent episodes, and episodes with semantic violations. Table 5 marks scored-object lineage as uncheckable for the native adapter, whereas the main text reports successful answer-path reconstruction in 214/214 cases. Does this mean that the answer content can be reconstructed, but its attribution to a specific agent remains unverified? If so, which team-level conclusions cannot yet be established?

Reviewer_aBtKOpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that multi-agent team scores require recording how agent outcomes are aggregated and which object is scored. In an existing multi-agent evaluation harness, it shows that parallel workers share a Docker container and that the score depends only on worker 1. It proposes a logging receipt and a harness fix, tested on 40 SWE-bench instances and 220 MALLM episodes.

Critical strengths
  • The topic of measurement validity is important. The survey of the SWE-bench leaderboard in the introduction is an interesting starting point (but underused by the paper).
Critical weaknesses
  • The technical contributions are very thin and limited. The contributions read more as a bug report than a scientific study. For example, the paper supports its main argument with specific class names from one implementation. The core problem is not a general measurement validity issue as advertised. Stripped of the overly complicated wording, the core finding is that in one harness, three parallel workers share a Docker container and the score is read from worker 1 only. This behaviour is already documented in the implementation, so it is not novel either. In addition, the paper advertises its findings as general observation of multi-agents setups, but they are really specific to the narrow case of parallel runs of a single task.
  • The paper writing is poor and obscures what the paper does. The vocabulary used is overly complicated to describe simple ideas, and obfuscates what the paper is really doing instead of clearly presenting it. Overall, the paper contains many traces of heavy LLM usage, including sentences that resemble LLM meta-comment. I suspect that the human involvement was very limited and most of the work was generated by an LLM.
  • The experimental settings are limited too: tiny sample size, harness specific to one or two benchmarks, fragmented experimental setups across RQs (no common setting).
Questions for authors

Minor weakness: The survey of the SWE-bench leaderboard mentioned in the introduction is not properly introduced: these new results are only reported in the introduction, and not mentioned in the main part of the paper, and key details that are needed to interpret the results are only available in appendix.

Reviewer_brQZOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies the reliability of team scores in multi-agent benchmarks. It focuses on how the results from different agents are combined into a team score, and which agent output or execution state is actually used for evaluation. The authors propose an Executable Aggregation Receipt to check whether the aggregation rules, execution records, and scored objects are consistent.

The authors test these issues through controlled experiments on SWE-bench and PlanCraft, and also implement a fix for the original harness. They further evaluate their approach on MALLM, an independent multi-agent framework.

Critical strengths

S1. I think the paper addresses a meaningful problem. Many multi-agent benchmarks report a single team score without clearly explaining how agent outputs are combined or which agent is actually evaluated. This makes the results harder to understand and reproduce.

S2. I like the controlled experiments, especially the PlanCraft results. Changing Worker 1 outcome affects the final score in all 24 comparisons, while changing the other workers has no effect. The authors also fix the original harness and test on MALLM, which provides further evidence for their findings.

Critical weaknesses

W1. Previous work like ClaimReceipt (Zhu & Chang, 2026) has already explored evaluation auditing using execution evidence and automated checkers. While this paper combines Agent-to-Team Aggregation and Scored-Object Binding, I think there is some overlap with existing methods. A direct comparison on the same tasks would help clarify what new value this approach brings.

W2. The model-driven experiments only cover 40 SWE-bench tasks and use a different model interface from the original study. Also, the generic checker returns Unresolved for 216 of 220 MALLM episodes, requiring a framework-specific adapter. I would like to see more discussion of how much extra work is needed and how much checking logic can be reused across frameworks.

W3. I found the paper quite dense, with many implementation details but few helpful visualizations. The two existing figures are too simple, and I think clearer system diagrams / result plots would make the paper much easier to understand.

Questions for authors

Q1. I would like to understand what new technical advantages or verification capabilities Aggregation Receipt offers compared to existing methods like ClaimReceipt.

Q2. Since MALLM requires a framework-specific adapter, could the authors clarify how much additional work is needed to apply this method to other multi-agent frameworks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.