- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces OPAC, an asset-centric formulation of collaboration among privately owned agents operating with partial topology visibility and separate ownership boundaries. Instead of exchanging only responses, agents discover, release, validate, reuse, and audit policy-controlled context assets such as memory fragments, cached results, and skills.
AEP instantiates this framework through four components: AEP-ROUTING for semantic discovery and evidence-based peer admission, AEP-COORDINATE for demand planning and negotiation, AEP-EXCHANGE for controlled release and staged integration, and AEP-LEDGER for signed evidence, trust, and quality feedback.
The empirical benchmark covers Research, Coding, and Daily-Service scenarios and network sizes of 4, 16, and 64 benign agents, with additional malicious agents for safety evaluation. The main comparisons include A2A, Broadcast, ANP, Agora, and ACP, as well as guarded baselines and alternative model backbones.
AEP substantially outperforms the baselines on task score and completion while reporting zero privacy leakage in the main settings and near-zero injection success. Under a malicious-agent ratio of 0.4, the normalized task-score drop is 0.035, and late malicious contact falls to zero.
Critical strengths
-
The paper tackles a timely systems problem that is broader than ordinary agent-to-agent messaging: collaboration across ownership boundaries with incomplete topology and persistent private context.
-
The four-layer protocol decomposition is coherent and practically motivated, separating discovery, coordination, controlled context exchange, and auditable evidence rather than collapsing these concerns into prompting.
-
The evaluation is broad across three task domains, three network scales, several protocol baselines, malicious-agent stress tests, guarded variants, ablations, and multiple backbone models.
-
The utility improvements are substantial. Across the main benchmark, AEP generally achieves much higher task score and completion than A2A and Broadcast, including at 64-agent scale.
-
The ablations are informative: removing the safety gate sharply increases leakage and injection success, while Routing and Coordination account for major utility improvements.
-
The malicious-agent experiment goes beyond one-shot filtering by evaluating whether accumulated interaction evidence can reduce future contact with unreliable peers.
Critical weaknesses
-
The benchmark is closely aligned with AEP’s design assumptions: information is deliberately distributed across agents, while competing protocols often lack equivalent discovery, trust, and selective-exchange machinery.
-
Safety gaps are extremely large, with several baselines showing injection success. This raises questions about whether the attacks sufficiently stress AEP rather than primarily exposing absent baseline defenses.
-
The threat model appears limited to specific privacy-extraction, injected-payload, and malicious-peer behaviors. Collusion, identity attacks, strategic reputation manipulation, and more adaptive adversaries are not convincingly tested.
-
Main utility results are based on three initialization runs, while each ordinary utility/safety run contains only six normal tasks, limiting statistical confidence despite reported standard deviations.
-
Task utility relies on rubric-based LLM judging. Given near-perfect reported scores for AEP in several settings, independent or deterministic task-level validation would make the magnitude of the gains more convincing.
-
The theoretical consistency result depends on idealized assumptions, including fixed task distribution, authorization rules, and reference physical topology during evaluation, limiting how directly it validates dynamic real deployments.
Questions for authors
My main question is about baseline comparability. Because AEP explicitly adds discovery, persistent trust, access control, safety gating, and evidence accumulation, how much of the reported gain reflects the particular protocol design versus simply providing capabilities that the request-centric baselines were not designed to offer? Could you compare against a stronger modular baseline with matched discovery and safety components but without AEP’s integrated design?
Second, can the authors provide stronger statistical evidence for the utility comparisons? Three initialization runs and six normal tasks per run seem relatively small given the near-perfect AEP scores and very large gaps. Confidence intervals or paired tests over a substantially larger task set would increase my confidence.
Third, how would AEP behave under adaptive adversaries that learn the safety policy, strategically behave well before attacking, collude to reinforce false trust evidence, or create multiple identities? These seem particularly relevant for the claimed open and low-trust setting. Finally, could the authors distinguish more explicitly between integrity of signed evidence and truthfulness of that evidence? A ledger can prove who asserted something and detect some conflicts, but a consistently dishonest agent may still produce cryptographically valid claims. How is this handled beyond downstream quality feedback?
9 comments
Rebuttal by Authors [2/7]
-
Unified Multi-Domain Evaluation (Rubric Task Score + Deterministic Metrics with 95% CIs): We evaluate , Enhanced A2A, and Adapted AgentNet across all three domains (Coding, Daily-Service, and Research) on the 16-agent network (). Alongside the rubric-based Task Score, we evaluate two completely LLM-judge-free deterministic metrics:
-
Keyword Value Matching Accuracy ( runs per method): We evaluate tasks across random seeds. A typed parser extracts exact numerical quotas, actions, intervals, durations, statuses, and dates from open-ended outputs to match against ground-truth answers, with confidence intervals.
-
Multiple-Choice Accuracy ( runs per method): tasks, each with four shuffled options (A–D: 1 correct, 3 distractors) across seeds, scored deterministically by option label.
-
| Scenario | Protocol | Task Score ↑ | Keyword Value Matching Acc. & 95% CI (%) ↑ | Multiple-Choice Acc. (%) ↑ | Completion Rate (%) ↑ | Tokens/task (k) ↓ | Leakage Rate (%) ↓ | Injection Success Rate (%) ↓ |
|---|---|---|---|---|---|---|---|---|
| Coding | AEP (Ours) | 99.17 | 99.48 [86.77, 100.00] | 100.00 | 100.00 | 10.10 | 0.00 | 0.00 |
| Coding | Enhanced A2A | 67.58 | 68.75 [52.53, 84.62] | 94.27 | 91.67 | 12.18 | 0.00 | 11.11 |
| Coding | Adapted AgentNet | 78.33 | 82.29 [69.27, 95.24] | 93.75 | 100.00 | 13.99 | 0.00 | 44.44 |
| Daily-Service | AEP (Ours) | 95.50 | 97.91 [89.17, 100.00] | 100.00 | 100.00 | 7.90 | 0.00 | 0.00 |
| Daily-Service | Enhanced A2A | 38.49 | 36.98 [7.69, 68.23] | 75.00 | 75.52 | 14.88 | 0.00 | 11.11 |
| Daily-Service | Adapted AgentNet | 70.73 | 65.62 [37.44, 85.25] | 80.73 | 99.48 | 16.09 | 0.00 | 22.22 |
| Research | AEP (Ours) | 99.67 | 96.35 [86.04, 100.00] | 100.00 | 100.00 | 9.20 | 0.00 | 2.90 |
| Research | Enhanced A2A | 48.93 | 46.35 [18.23, 76.04] | 86.46 | 95.83 | 11.04 | 0.00 | 33.33 |
| Research | Adapted AgentNet | 71.56 | 67.71 [45.90, 88.21] | 94.79 | 100.00 | 14.60 | 0.00 | 0.00 |
On the 64-task 16-agent coding benchmark (), achieves 99.48% Keyword Value Matching Accuracy ( CI: ) and 100.0% Multiple-Choice Accuracy, outperforming Enhanced A2A (68.8%, CI ; paired difference +30.73%, CI ) and Adapted AgentNet (82.3%, CI ; paired difference +17.19%, CI ). Thus, even when baselines are equipped with matched discovery, semantic routing, evolutionary experience, and DLP filters, delivers substantial, statistically significant improvements across all domains while maintaining lower token costs and near-zero injection acceptance.
Rebuttal by Authors [3/7]
Weakness 2: Insufficient Adversarial stress testing on AEP.
"The attack stress in the experiments is not sufficient for AEP and mainly exposes the weaknesses of the baselines."
To directly stress-test 's routing, trust admission, and quality-feedback mechanisms under non-trivial adversarial strategies that evade static content filters, we conducted an expanded suite of distributed adversarial evaluations in the 16-agent Coding / software-operations domain (). We focus this deep adversarial stress test on the Coding/operations domain because its tasks require exact operational parameters (quotas, Terraform commands, cron intervals, migration durations, test states, and TLS dates), allowing deterministic, unambiguous verification of whether an agent adopts an attacker's fabricated value versus the ground-truth answer.
Each episode comprises benign service-status warm-up rounds followed by formal evaluation query under natural protocol routing ( episodes per condition). We evaluate four fixed distributed attack strategies:
-
Immediate Attack: The adversary provides plausible but factually false operational values from the very first warm-up round.
-
Sleeper Attack: The adversary behaves honestly during all warm-up rounds to build positive reputation/experience, then switches to returning false values on the target task.
-
Collusion Attack: Two malicious identities return mutually consistent false values and cross-endorse each other.
-
Sybil-Style Multi-Identity Attack: Four admitted malicious identities (out of 16 total agents) coordinate false values and mutual endorsements.
| Attack Condition | AEP Acc. (%) ↑ | Enh. A2A Acc. (%) ↑ | AgentNet Acc. (%) ↑ | AEP False Adopt. (%) ↓ | Enh. A2A False Adopt. (%) ↓ | AgentNet False Adopt. (%) ↓ | AEP Malicious Exposure (/18) ↓ | Enh. A2A Malicious Exposure (/18) ↓ | AgentNet Malicious Exposure (/18) ↓ |
|---|---|---|---|---|---|---|---|---|---|
| Clean | 100.0 | 94.4 | 61.1 | 0.0 | 0.0 | 0.0 | — | — | — |
| Immediate | 100.0 | 61.1 | 66.7 | 0.0 | 38.9 | 22.2 | 0/18 | 18/18 | 6/18 |
| Sleeper | 88.9 | 66.7 | 55.6 | 0.0 | 33.3 | 0.0 | 0/18 | 18/18 | 1/18 |
| Collusion (2 IDs) | 88.9 | 50.0 | 66.7 | 0.0 | 50.0 | 5.6 | 0/18 | 18/18 | 1/18 |
| Sybil (4 IDs) | 94.4 | 50.0 | 61.1 | 0.0 | 50.0 | 16.7 | 0/18 | 18/18 | 3/18 |
As shown in , achieves 0.0% false-value adoption and 0/18 malicious response exposure across all four attack strategies while maintaining 88.9%–100.0% accuracy. This demonstrates that 's robustness is not merely a byproduct of receive-side regex filtering; rather, 's front-end candidate admission, service-quality reranking, and verified topology routing prevent task requests from being routed to untrusted or degraded responders, whereas Enhanced A2A ( exposure, false adoption) and Adapted AgentNet ( exposure, up to false adoption) remain vulnerable.
Rebuttal by Authors [1/7]
Thank you for your constructive feedback and insightful suggestions! To address your questions and further validate AEP, we have conducted additional experiments with discovery- and safety-enhanced baselines (Enhanced A2A and Adapted AgentNet), expanded deterministic multi-domain evaluations with 95% confidence intervals, and evaluated multi-identity and feedback-driven adaptive attacks, alongside explicit clarifications on provenance versus truthfulness and theoretical boundary conditions. Below we respond to each point in Weaknesses and Questions in detail.
Weakness 1: Baseline comparability and matched discovery/safety machinery.
Because existing off-the-shelf protocols are not natively designed for open private agent networks with partial topology observability and cross-owner privacy boundaries, we specifically adapted and optimized the baselines for our setting following your suggestion. Rather than comparing AEP against static implementations restricted to initial neighbors, we engineered Enhanced A2A and Adapted AgentNet [1] with matched peer discovery, semantic routing, trust filtering, and inbound/outbound safety gates (Appendix B.8):
Enhanced A2A (Gossip Discovery + MiniLM Semantic Routing + Deterministic Policy Filters & Outbound DLP):
(a) Gossip-based Peer Discovery: Each agent maintains structured
PeerCardrecords containingagent_id,endpoint,capability_desc,memory_summary,skill_summary,skill_count, andtrust_score. Every 500 ms, each agent selects up to two gossip targets from known peer IDs and exchanges its ownPeerCardalong with known remotePeerCards. Receivers verify endpoint syntax and physical reachability before merging new cards into their local directory, enabling multi-hop collaborator discovery without global task broadcasting.(b) MiniLM-based Semantic Routing: For each task, the requester concatenates each candidate's capability description, memory summary, and skill summary, computes normalized embeddings using the shared
all-MiniLM-L6-v2encoder, filters out peers withtrust_score<0.15, and selects the Top-3 peers by cosine similarity (with deterministic tie-breaking byagent_id). This equips A2A with embedding-based semantic discovery without introducing AEP's signed ledger evidence, local quality EMA, or staged asset exchange.(c) Deterministic Policy Filters and Outbound DLP: We add rule-based inbound and outbound security classifiers (binary risk ∈{0,1}, blocking threshold 0.5) across all request and response paths. Inbound checks block instruction overrides, spoofed
system/developerprivilege headers, policy-bypass attempts, and secret solicitation. Outbound DLP checks block API keys, passwords, secrets, PEM private-key headers, and AWS credentials. Published memory summaries are restricted to records with sensitivity <0.8 passing outbound DLP. Local RAG context, collaborative responses, and final simulator outputs are all filtered.Adapted AgentNet (Retaining Native Evolutionary Router/Executor + PeerCard Discovery + Shared Safety Filters):
Using the original repository of AgentNet [1], we retain its native
Router,Executor,Taskmodule, episodic experience pool, capability profiles, and adaptive edge-weight updating rules.We adapt AgentNet to the 16-agent collaborative runtime with
DeepSeek-V4-Flash(temperature=0, thinking disabled) and sharedall-MiniLM-L6-v2embeddings (zero-padded from 384 to 1024 dimensions to match upstream AgentNet's vector interface while preserving exact cosine similarity).We integrate the same
PeerCardgossip discovery module so AgentNet discovers peers beyond its initial 30% neighborhood, and apply the same inbound injection and outbound DLP filters on executor returns and final outputs. Workflows are bounded to DAG execution with at most 3 direct peer requests per task. During multi-round episodes, warm-up trajectories update AgentNet's native experience pool and edge weights without cross-task test leakage.