- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper audits whether trajectory-level collaboration measures are associated with multi-agent task success across settings. It screens 25 candidates, retains five measures, and studies 18 model–benchmark configurations across four primary benchmarks and five models. Communication score has the most consistent reported task-success association, whereas the other measures have fewer corrected detections or changing signs. The paper examines rubric overlap and judge dependence through rescoring controls. A separate analysis of 1,500 trajectories from three privacy benchmarks finds no multiplicity-corrected association between general collaboration measures and leakage, and shows that many observed leaks are absent from final outputs. The intended contribution is an empirical measurement audit and guidance on outcome- and context-specific metric validation.
Critical strengths
-
The study makes a useful measurement question concrete and provides meaningful sensitivity checks. The candidate screen and analytical-role separation distinguish process measures, task outcomes, and robustness outcomes (§3.2; Table 2, p.12). The primary analysis reports per-metric samples, sentinel exclusions, Spearman checks, and both within-metric and global Holm correction (Appendices A.1–A.3). Particularly valuable are the communication-score controls: removing task-progress wording leaves 11 of 18 corrected associations versus 14 originally, while a second judge yields 13 (Tables 6–7, pp.19–21). These results support an informative, qualified finding rather than an unexamined correlation with an outcome-overlapping rubric.
-
The privacy analysis supplies a practically relevant outcome distinction. Of 259 trajectories with observed leakage, 206 lack leakage in the final output (79.5%; §4.2 and Table 8). This supports inspecting internal channels in the evaluated settings. The analysis separates completion from leakage, uses scenario/model fixed effects and scenario-clustered inference, and includes strict multi-agent and hidden-leak sensitivity checks. Its explicit power discussion, including the 7.3-percentage-point median minimum detectable effect for leakage, appropriately cautions against interpreting non-detection as equivalence (Appendices B.4–B.11).
Critical weaknesses
-
The paper does not yet establish an unambiguous mapping from the stated metrics to the measurement procedure. Section 3.3 (p.5) defines redundancy and downstream usage as shares of eligible events, and contradiction as the share of events participating in contradictions unresolved at trajectory end. However, the prompts described as verbatim in Appendix A.6.1 ask for fractions of information within a message for the first two measures, and counts of contradictory assertions and total assertions for contradiction (pp.17–18). The contradiction prompt does not specify excluding resolved contradictions. These are materially different units and constructs unless an additional transformation is supplied. Eligibility is also unclear: §3.3 says a trajectory with no “relied-upon successor” contributes no observation, whereas Appendix A.1 (p.13) describes absence of eligible later outputs. A valid trajectory with successors but zero downstream use should be distinguishable from an undefined measure. This is a reporting ambiguity, not a demonstrated implementation bug, but it affects three of the five audited measures and therefore the central interpretation.
-
The primary inference does not sufficiently support the strength of the cross-context failure conclusion. Section 3.6 (p.7) explicitly acknowledges non-independent observations within primary cells and optimistic standard errors; Appendix A.3 states that scenario identifiers are unavailable. Nevertheless, the main summary relies on unclustered correlation tests and counts of Holm survivors. Holm correction does not repair invalid individual-test uncertainty. Furthermore, different rejection decisions do not establish different underlying associations: the event-level measures have expected signs in 14/18, 16/18, and 16/18 configurations, while sample sizes range from 45 to 213 (Tables 1, 3–4). The paper discusses unequal power and appropriately bounds its privacy nulls, but the primary claim that four measures “fail to hold up” needs dependence-aware uncertainty and a clearer distinction between heterogeneity, weak association, and insufficient precision. The stronger secondary analysis and the progress-blind control do not resolve this primary inference problem.
Questions for authors
- Clarification: What exact prompts, aggregation formulas, and eligibility rules produced the three event-level columns? Please reconcile §3.3 with Appendix A.6.1 and show a short worked example containing a resolved contradiction and an unused message. Are trajectories with eligible successors but zero downstream use retained as zero?
- Clarification and limited validation: How many distinct scenarios and repeated runs occur within each primary benchmark–model cell? Please distinguish repetition within a cell from scenario pairing across models. If within-cell repetitions are present, can the main and progress-blind correlations be reanalysed with scenario-level resampling or an appropriate clustered procedure, and the resulting Holm decisions reported?
- Limited reanalysis or claim clarification: Which conclusions about cross-context instability remain after accounting for uncertainty in differences between associations, rather than comparing significance decisions? An analysis of the existing data, or narrower descriptive claims where identification is unavailable, would materially affect the assessment.