- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines the measurement validity of multi-agent evaluation, with particular attention to how individual outcomes are aggregated into team-level outcomes and which state or artifact is actually evaluated by the scorer. Through source-code audits and controlled experiments, the authors trace the scoring procedures and propose an executable “aggregation receipt” for reconstructing and verifying how scores are computed. Across all 24 paired comparisons in the PlanCraft experiments, modifying the first worker’s outcome changed the score, whereas modifying the outcome of either of the other two workers had no effect. Taken together with the repair experiments on SWE-bench and an independent audit of MALLM, these results suggest that the implemented scoring procedures may not align with team-level interpretations of the reported outcomes. However, they do not establish that historical leaderboard scores or rankings were incorrect.
Critical strengths
1.Controlled interventions directly reveal which worker outcomes affect the score. The PlanCraft experiments vary a single worker’s success or failure while holding the other workers’ outcomes fixed, and are complemented by synthetic text replacement experiments. Compared with source-code analysis alone or correlations between scores, this design provides stronger evidence for the conclusion that “only the bound worker’s outcome can reach the scorer.” These controlled interventions provide the strongest empirical justification for accepting the paper.
2.The paper makes scoring interpretable through verifiable execution relationships and identifies concrete semantic issues in an independent framework. By jointly examining agent outcomes, aggregation rules, and the provenance of the objects being scored, the paper enables the scoring process to be reconstructed and verified. The native MALLM adapter reconstructs the answer path and identifies discrepancies between MALLM’s declared consensus semantics and its actual outcome accounting. These findings demonstrate that the paper’s contribution extends beyond describing known implementation behavior in the original framework.
Critical weaknesses
1.The independent validation of aggregation receipts and the method’s added value are not yet sufficiently demonstrated. The generic checker returned “unresolved” for 216 of the 220 MALLM episodes, while the main audit findings depended on a newly introduced framework-specific adapter. Although this is consistent with the authors’ design, it indicates that applying the method to another framework still requires additional semantic modeling. The paper does not yet provide a complete example of an aggregation receipt, pseudocode for the key checks, or explicit results from the claimed validation controls. These omissions make it difficult to determine which capabilities can be reused across frameworks and how the checker avoids reproducing errors in the implementation being audited. Consequently, the evidence supporting the approach as a general-purpose audit method remains limited.
2.The claim that all four predictions were confirmed in the independent-framework experiment requires clarification. The statement that “all four predictions were confirmed” is not explicitly reconciled with the results reported in Appendix E: “at least 9/10 expected, but 8 violations and 2 cases with no output observed.” The main text also applies different counting criteria to execution reconstruction, label consistency, and semantic violations, without clearly explaining how these criteria relate to the four predictions. The authors should report the preregistered criteria for each prediction alongside the episode-level classifications and explicitly distinguish predictions that were met, those that were not met, and those that could not be assessed. Without this information, readers cannot readily verify whether the evidence supports the method’s applicability to another framework.
Questions for authors
1.What provides the basis for the checker’s independent semantic judgments? Please provide a complete example of an aggregation receipt and pseudocode for the key aggregation and scored-object binding checks. For MALLM, how does the checker distinguish between reproducing source-code behavior and independently evaluating compliance with the declared semantics? Please also report the results of each validation control mentioned in Section 4: declaration modification, missing observations, source-code substitution, and ineffective interventions.
2.Were all four preregistered MALLM predictions met? Please report the preregistered pass criteria and observed results for each prediction, and explain how “at least 9/10 violations” is reconciled with “8 violations and 2 cases with no utterance.” At the time of preregistration, were episodes with no output classified as failures, exclusions, or unresolved cases? Was the criterion for handling these episodes specified in advance, and how was it applied when concluding that “all four predictions were confirmed”?
3.Can the episode counting criteria and the scope of the conclusions drawn from MALLM be clarified? For each of the four experimental groups, please report the counts of episodes with answers, unscored episodes, episodes with no utterance, label-consistent episodes, and episodes with semantic violations. Table 5 marks scored-object lineage as uncheckable for the native adapter, whereas the main text reports successful answer-path reconstruction in 214/214 cases. Does this mean that the answer content can be reconstructed, but its attribution to a specific agent remains unverified? If so, which team-level conclusions cannot yet be established?