- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EvoProtocol, which represents each inter-agent communication protocol as text (an output schema, a set of forwarded harness-state entries and a message template) and evolves it jointly with the agent prompts inside GEPA, guided by local feedback built from what the receiving agent does after each message. It is evaluated on five MAS-PromptBench tasks with a four-agent sequential workflow and on BrowseComp-Plus with a planner-executor workflow, against prompt-only GEPA under raw-trace, agent-output and PACT protocols, with cost-aware optimization and ablations. The paper reports an average of 75.3 on MAS-PromptBench against 72.9 for the best prompt-only baseline, and 78.0 on BrowseComp-Plus against 72.5 with PACT, with a cost-aware variant at 78.5 and $0.47 per execution.
Critical strengths
-
Treating the communication protocol as an explicit, textual optimization variable is a clean idea that plugs directly into existing textual optimizers, and the local feedback that ties each message to the receiving agent's next actions is a sensible way to give the optimizer something more specific than the final score. It matters to my assessment because it opens a component that prompt optimization leaves fixed, and Table 1 gives a reasonable motivation, since the best fixed protocol changes from task to task.
-
The per-execution cost reporting is unusually careful. Table 3 breaks each method down into planner and executor tokens and dollars, and the totals reproduce from the prices in Table 7, which lets a reader weigh accuracy against deployment cost directly.
Critical weaknesses
-
My main concern is that the comparison with the prompt-only baselines is not quite like for like. Section 4.1 states that the methods "differ only in the set of optimized variables", yet the protocol modules receive diagnostics the prompts never see, such as the per-hand-off "Answer trail" (Table 10), and on BrowseComp-Plus only EvoProtocol can forward the hand-designed harness state of Table 5, at over 25 times the default baseline's executor input tokens (Table 3). The search is also matched only in executions: EvoProtocol proposes revisions for more modules in each iteration, and the optimizer's own token cost is not reported for any method. This matters because the central claim is that optimizing the protocol is what helps, and at present the gain could also come from the extra information, compute and search.
-
The evidence for several of the headline margins is thin. Every result comes from one optimization run, with no per-seed results, confidence intervals or significance tests, and the MAS-PromptBench test-set sizes are never stated, while the claim of winning on "all five tasks" rests on margins of 0.3 points on HotpotQA and 0.7 on BFCL (Table 1). The 5.5 point BrowseComp-Plus gain over PACT is averaged over two evaluation seeds on 100 questions. None of this shows the gains are absent, but I cannot yet tell which comparisons would survive sampling and run-to-run variation.
Questions for authors
-
[Experiment] How does prompt-only GEPA perform when given the same protocol diagnostics as EvoProtocol, how does EvoProtocol perform with global feedback only, and on BrowseComp-Plus, how does a fixed protocol that forwards the Table 5 harness state compare at a matched executor cost? If the gains hold under these controls, I would be glad to raise my score.
-
[Experiment] Could you report per-seed results with confidence intervals for Tables 1 and 2, repeat the optimization over a few seeds if feasible, and state the test-set size of each MAS-PromptBench task?
-
[Clarification] Could you report each method's search cost (optimizer calls, tokens and dollars) separately from the per-execution cost, with a break-even number of deployed runs against accuracy-only PACT, where cost-aware EvoProtocol saves about $0.09 per execution (Table 3)? It would also help to know how sensitive the cost-aware results are to the 80% accuracy threshold of Section 3.3.
-
[Experiment] In Table 4(a), EvoProtocol's prompts with the raw trace match or exceed the full system on MATH, GPQA and BFCL. What happens when the optimized protocols are paired with the GEPA-baseline prompts, and how does the sequential variant of Table 4(b) do with a full prompt budget?
-
[Clarification] Table 10 does not cover BFCL, and the 11 interface checks and the optimizer prompts are not given. Could these be added to the appendix, and do you plan to release the code?
6 comments
W1.2 and Q1.2 and on BrowseComp-Plus, how does a fixed protocol that forwards the Table 5 harness state compare at a matched executor cost? Q3.2 It would also help to know how sensitive the cost-aware results are to the 80% accuracy threshold of Section 3.3.
We compare methods by the total cost per rollout rather than the executor cost alone. In multi-agent systems that combine a large and a small model, the cost per token differs greatly between agents (25 times in our setting, Table 7), so the total cost depends on how information is divided between them. EvoProtocol discovers such a division through search rather than by hand design. It forwards more harness states to the cheap executor, which increases the executor input but reduces the planner cost to about one third (\0.268 for accuracy-only PACT). However, we agree that the original submission lacked a baseline that forwards the harness state to the executor with a fixed protocol. We therefore added a fixed protocol that forwards all harness states, optimized with both accuracy-only and cost-aware objectives. We also report both the highest-accuracy candidate (“Acc”) and the candidate selected by the 80% rule (“Cost”) to show the sensitivity to the threshold. In the Pareto column of the table, “Acc” and “Cost” denote these two candidates, respectively, and “Acc, Cost” indicates that both select the same candidate. As summarized in the table, forwarding all harness states achieves the highest accuracy (81.0%) at the highest cost (\0.467 vs. 75.5% at \0.467 vs. \0.145 per rollout is larger than that against accuracy-only PACT. These results show that an efficient harness cannot be obtained simply by forwarding more information to the executor; it requires selecting which information to forward, which EvoProtocol learns through search. Based on these results, we will revise the manuscript as follows:
- We will add the results on all harness states in Table 2
- Report both the highest-accuracy candidate and the candidate selected by the 80% rule for cost-aware optimization in Table 3, including the protocol with all harness states.
- We will revise the discussion of the BrowseComp-Plus results in Section 4.3 accordingly, shifting the focus from accuracy alone to the accuracy–cost trade-off.
| Setting | Pareto | Prompt | Protocol | Harness State | Acc (%) on BC-Plus | Acc (%) [trial1, trial2] | \$/rollout |
|---|---|---|---|---|---|---|---|
| GEPA | Acc | Evolved | Fixed | None | 62.0 | [63.0, 61.0] | 0.172 |
| GEPA + Cost | Acc, Cost | Evolved | Fixed | None | 62.5 | [63.0, 62.0] | 0.251 |
| PACT | Acc | Evolved | Fixed | Human | 72.5 | [74.0, 71.0] | 0.555 |
| PACT + Cost | Acc | Evolved | Fixed | Human | 78.5 | [80.0, 77.0] | 0.612 |
| PACT + Cost | Cost | Evolved | Fixed | Human | 59.0 | [58.0, 60.0] | 0.239 |
| EvoProtocol | Acc | Evolved | Evolved | Optimized | 78.0 | [77.0, 79.0] | 0.796 |
| EvoProtocol + Cost | Acc, Cost | Evolved | Evolved | Optimized | 78.5 | [76.0, 81.0] | 0.467 |
| All Harness State | Acc | Evolved | Fixed | All | 81.0 | [81.0, 81.0] | 0.915 |
| All Harness State + Cost | Acc | Evolved | Fixed | All | 75.5 | [74.0, 77.0] | 0.727 |
| All Harness State + Cost | Cost | Evolved | Fixed | All | 68.0 | [69.0, 67.0] | 0.319 |
W1.3 and Q3.1 Could you report each method's search cost (optimizer calls, tokens and dollars) separately from the per-execution cost, with a break-even number of deployed runs against accuracy-only PACT, where cost-aware EvoProtocol saves about \$0.09 per execution (Table 3)?
Since the optimizer cost was not recorded in the original experiments, we re-ran two optimization iterations for each method, one starting from the seed and one starting from the final optimized state, to measure the per-iteration search cost. The table below separates this cost into rollout generation and optimizer calls. For EvoProtocol, we report the cost-aware variant to compare it against accuracy-only PACT. The search cost is dominated by rollout generation, and the additional optimizer calls of EvoProtocol cost less than \211.5 for PACT and \0.09, the additional search cost is recovered after about 110 runs. Based on these results, we will revise the manuscript as follows:
- Add the search cost of each method, separated into rollout generation and optimizer calls, to the appendix.
| Method | Iteration | Rollouts | Roll in | Roll cached | Roll out | Roll $ | Opt calls | Opt in | Opt out | Opt $ | Total $ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GEPA | Seed | 50 | 56,371,912 | 13,553,408 | 2,447,217 | 15.668 | 2 | 157,928 | 5,678 | 0.960 | 16.628 |
| PACT | Seed | 50 | 50,678,215 | 12,114,048 | 1,717,449 | 14.720 | 2 | 146,208 | 5,363 | 0.892 | 15.612 |
| EvoProtocol + Cost | Seed | 50 | 82,573,782 | 11,836,672 | 2,288,990 | 21.122 | 4 | 200,944 | 7,561 | 1.232 | 22.354 |
| GEPA | Optimized | 50 | 17,078,191 | 4,237,952 | 1,289,754 | 6.783 | 2 | 139,630 | 9,002 | 0.968 | 7.752 |
| PACT | Optimized | 50 | 26,717,577 | 7,889,024 | 1,953,300 | 19.584 | 2 | 147,974 | 8,619 | 0.998 | 20.583 |
| EvoProtocol + Cost | Optimized | 50 | 41,105,490 | 6,543,360 | 2,286,116 | 14.209 | 4 | 137,063 | 10,550 | 1.002 | 15.211 |
W1.4 the central claim is that optimizing the protocol is what helps, and at present the gain could also come from the extra information, compute and search.
Given the controlled experiments (see W1.1, W1.2, and W1.3), the gains of EvoProtocol cannot be explained by the additional information, compute, or search alone. Providing the protocol diagnostics to prompt-only GEPA does not improve its accuracy (W1.1). Forwarding all harness states improves accuracy but at a substantially higher cost, whereas EvoProtocol learns which information to forward and to whom, achieving a better accuracy–cost trade-off (W1.2). Moreover, expanding the search space to protocols adds only a marginal optimizer cost (less than \1 per iteration). In addition, EvoProtocol is estimated to cost \220.9, only about 4.5% more than accuracy-only PACT (\$211.5), and this additional cost is recovered after about 110 deployed runs thanks to its lower per-execution cost (W1.3). Instead, the gains stem from co-optimizing prompts and protocols, with segment-level feedback guiding the protocol search, as supported by the ablations (see W1.1).
We thank the reviewer for their feedback and insightful suggestions. We address each of the reviewer’s concerns individually below.
We added two experiments that compare prompt-only GEPA using protocol feedback in EvoProtocol and EvoProtocol with global feedback only. As summarized in the table, giving the protocol diagnostics to prompt-only GEPA does not improve its accuracy as observed in EvoProtocol. In addition, removing segment-level feedback from EvoProtocol lowers the average accuracy from 75.3 to 73.5. These results show that the gain comes neither from the modified information alone nor from expanding the search space to protocols alone, but from the joint optimization of prompts and protocols, each with feedback tailored to it.