- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Paper Summary
The paper studies how task specifications degrade when relayed across LLM agents. Using an 8-hop paraphrase chain with a fixed executor, the authors propose CommAudit. CommAudit labels each atomic constraint as lost, mutated, weakened, or added relative to the original specification. It then measures two quantities:
- Exposure: how often each type of change occurs.
- Impact: the executor's accuracy gain when a change type is repaired, compared against a matched stylistic control.
Across three benchmarks and four communication agents, relays degrade accuracy, mostly in the first few hops. Repairing the annotated changes recovers accuracy on BigCodeBench. The most frequent change type differs from the most harmful in 6 of 12 settings. A formal result states that change counts predict impact only if impact is constant among relays with identical counts.
Critical strengths
Critical Strengths
- The repair-versus-matched-control design is a clean way to separate consequential constraint changes from surface rewording.
- Supporting checks are reasonable. Repair closes 89% of the accuracy gap between agents, the effect persists when the executor is swapped, and annotation typing shows good agreement (κ=0.849).
Critical weaknesses
Critical Weaknesses
-
The insights are limited. The headline finding, that change frequency differs from change impact, is close to self-evident. In practice it reduces to "additions are frequent but mostly harmless, while losses are rarer but harmful," which matches intuition. The theory adds little. Proposition 1 restates the data-processing inequality, and Proposition 2 is essentially definitional: counts predict outcomes if and only if equal counts imply equal outcomes.
-
The analysis offers little actionable guidance. The paper proposes no mitigation, and the design implications in Appendix I are generic. As an analysis paper, it does not show how its findings should change the design of multi-agent communication.
-
The "6 of 12" result is fragile. Many impact estimates are around ±0.02, and their confidence intervals include zero. Taking the argmax over such noisy, near-zero values is unreliable, and only 4 of the 6 disagreements are robust (p_mis ≥ 0.89). The within-fiber variance test faces a similar issue: it uses a binary outcome over small fibers, so the measured variance largely reflects noise.
-
The repair intervention may be confounded. The repair model sees the original specification x0, while the control model does not. Repair could therefore restore more than the annotated changes, and repaired accuracy nearly returns to the original baseline A0. The paper does not verify that repairs touch only the listed changes.
-
The setting is artificial and one model plays several roles.
- Eight hops of free paraphrasing rarely occur in real multi-agent systems.
- DeepSeek V4-Flash serves as annotator, repair model, control model, and executor.
- Change detection itself is not validated.
- The paper overlaps substantially with prior work on iterated distortion and delegation corruption (Mohamed et al., 2025; Laban et al., 2026).
Questions for authors
Questions For Authors
- Beyond "frequency ≠ impact," what non-obvious insight does CommAudit reveal? How should practitioners change their system design based on these results?
- Do repaired specifications differ from the hop-8 relay only at the annotated changes? Please report a human or automatic check.
- Do the findings hold in a realistic delegation setting, such as a planner handing a task to a coder, rather than repeated free paraphrasing?
Dear Reviewer 9YFQ,
Thank you again for your thoughtful and constructive feedback. As the rebuttal period will end soon, we would like to briefly follow up on our responses to your questions. If any points remain unclear or require further clarification, we would be happy to address them before the discussion period ends.
Best regards,
The Authors