- Rating
- 2
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates how multiple coding agents coordinate when they build a repository together. This paper proposes AECP, a protocol in which agents communicate only through structured artifacts that the execution harness acts on. Evaluated on three Python benchmarks, AECP improves average test pass rate by 28.2%, with reduced average wall time.
Critical strengths
-
AECP is a concrete and checkable design. Instead of relying on agents to act on messages, AECP defines explicit rules triggered by tool calls.
-
Evaluation shows consistent gains over a single agent. AECP beats Single in 14 of 15 settings and matches or beats Team-Integrator in all 15 while being faster than both team baselines. The leader-worker allocation study and adaptive team sizing give practical findings on when a team is worth its cost.
Critical weaknesses
- The experiments do not show which parts of AECP improve quality, and the main comparison is against a weak baseline.
The controlled baseline is weak. Team-Freedom is presented as "the controlled comparison for AECP as a whole". However, it scores below Single in 12 of 15 settings, and sometimes far below. For example, for Opus-4.8, it scores 0.25 vs. 0.41 on CodeProjectEval and 0.52 vs. 0.71 on Doc2Repo. A team that does much worse than one agent with the same model suggests a weak implementation, for example, uncontrolled concurrent edits in the shared workspace. So the 28.2% in the abstract mostly measures that weakness.
There are also smaller gains against the other baselines. The more relevant gains are +0.050 over Single and +0.041 over Team-Integrator. The second is at most 0.02 in 9 of 15 settings and is driven by GLM-5.2, where Team-Integrator collapses (0.42 and 0.35).
The ablation study on quality is also missing. RQ2 adds the mechanisms one at a time only for token usage, on one model and one benchmark. Its first step also bundles the shared workspace with Knowledge Sharing. However, this paper never shows how much of the score gain comes from Knowledge Sharing, the stubs, the AST checker, or the stale-consumer gating.
Exclusivity is also untested. There is no configuration keeps the artifacts and harness rules while still allowing messages. Exclusivity is the idea in the paper's title and the basis of RQ4, but no studies or evidence support the claimed exclusivity.
The paper's main claims are that artifact-exclusive communication improves quality and efficiency. However, current experiments and results is only for this particular harness and show that it beats a weak messaging team and modestly beats a single agent. They do not show which parts matter, or whether exclusivity matters. So current experiments and results cannot effectively support the paper's main claims.
- The security test only shows that AECP blocks attacks sent as messages, which follows directly from AECP having no message channel. However, attacks sent through AECP's own Knowledge Artifacts, which reach other agents directly, are never tested.
In RQ4, the compromised worker tries to relay instructions as messages, which AECP does not provide. So 0% reach follows directly from the design. The authors say so in §5.4, but the abstract presents "from 95% to 0%" and "from 40% to 0%" as a finding.
However, as written in the paper, Knowledge Artifacts are free-text claims of up to 280 characters (section C.1). The harness inserts them into other workers' observations when those workers read the matching code, and also forwards them to workers owning related modules (section C.1). The paper's own live example in Listing 3 is an instruction to pin a dependency version:
- A witness may contain only shape-level operations such as imports, calls, construction, and attribute checks (C.1). A malicious claim with a passing witness would therefore still be delivered to other workers by the harness, labeled witness-checked, and witness-checked findings make up 88% of the findings used.
- Workers also share one workspace, so a compromised worker can write the malicious change directly into the code.
This is important because the abstract and conclusion present blocking malicious instructions as a strength of AECP. But the test only removes messages. AECP still places one agent's text into another agent's context, at the moment that agent reads the related code, and this path is not tested. So the results and evidence cannot support the paper's claims here.
Questions for authors
-
Could you show the evidence to support your main claims? For example, a score-level ablation on the three benchmarks, for at least one strong and one open model with three runs each, such as: (a) Team-Freedom + Knowledge Sharing, (b) + Contract Sharing, with the importable stubs, the static contract checker and stale-consumer synchronization separated if possible, (c) full AECP with messages also allowed, and other necessary settings to validate the findings and your main claims in the paper?
-
Why does Team-Freedom fall below Single in 12 of 15 settings? How would a stronger message-based team, for example one with enforced file ownership or an existing agent-team system, change the comparison?
-
Could you please show the evidence to support your security claims? For example, the quick way could be to run the same 20 payload families through Knowledge Artifacts with a passing shape-level witness and through direct edits in the shared workspace, and show reach and adoption?
-
How does AECP compare with a compute-matched single agent, for example one given the same token budget for a self-review and testing pass? Please also clarify the turn budgets: 150 turns for Single versus 150 per worker plus the leader for the teams.
-
How reliable are the LLM labels behind Figure 1(c) and the 36% usage figure, in particular beyond the 50 confirmed flagged events?