- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PACO, an edge-centric communication optimization method for LLM-based multi-agent systems. Instead of optimizing each agent prompt independently, PACO treats each directed communication edge as an optimization unit consisting of upstream sending instructions, downstream receiving instructions, and a shared communication convention. The method keeps the model weights, agent roles, information access, and workflow topology fixed. A discounted bandit selects which edge to refine, and an LLM-based proposer performs two rounds of paired revision using execution feedback. Candidate updates are evaluated through staged end-to-end workflow execution under a fixed optimization-token budget. Experiments are conducted on a fixed four-agent workflow using HotpotQA for optimization and evaluation, with transfer to DROP and MuSiQue-Ans. PACO outperforms several communication and prompt-optimization baselines, including GEPA, while using similar optimization cost. Ablations suggest that paired endpoint refinement and adaptive edge selection are useful.
Critical strengths
- I think the paper picks a sensible unit of optimization. The distributed-averaging example in the introduction is simple, but it makes the point: providing the correct intermediate value is not enough if the next agent lacks the information or instructions to use it. Grouping the sender instructions, receiver instructions, and shared convention around an edge is a more specific proposal than generally 'improving communication.'
- The comparison is reasonably controlled. The workflow and model weights are kept fixed while PACO is compared against prompt and communication optimizers, including GEPA and MASPOB. On HotpotQA, the reported Joint F1 is 61.26 for PACO versus 59.27 for GEPA at similar optimization and deployment cost. Freezing the learned configuration before testing on DROP and MuSiQue-Ans is also a useful check; this is not a fresh optimization for each target dataset.
- I found the message example in Table 5 worth including. It shows a concrete handoff in which making the transaction direction explicit avoids ambiguity downstream. This kind of small execution trace helps explain what is actually being changed by the optimizer, which is otherwise difficult to see from final F1 scores alone.
Critical weaknesses
- The results are all from a fixed four-agent chain. That setting is appropriate for studying handoffs, but it also makes the edge-level formulation easier than it might be in a graph where an agent receives several messages or sends to multiple downstream agents. Paired updates could interact through a shared receiver. The paper makes fairly broad claims about multi-agent collaboration; I would be more comfortable with a narrower claim or at least one test on a branching workflow.
- I am less certain about the size of the gain specifically attributable to paired refinement. Table 3 reports 49.35 Joint F1 for PACO and 48.42 for independent updates at the 5M-token budget, which is a modest gap. The main PACO result uses optimizer seed 0, whereas some search comparisons report averages over three runs. The appendix notes a small range across repeated PACO runs but does not provide comparable means and uncertainty for the key pairwise comparisons. With the HotpotQA margin over GEPA also below two Joint F1 points, I would like clearer evidence that these differences are consistent rather than seed-specific.
Questions for authors
- What happens when one agent has two incoming edges and the corresponding communication packages are optimized separately? If you have a branching-workflow experiment, even a small one, it would be more informative to me than adding another chain-based QA dataset.
- Could you report PACO versus independent updates and PACO versus GEPA over matched optimization seeds, with means and variability? The 0.93-point Table 3 gap is the comparison I would most like to see repeated.
6 comments
Response to Question 1
1. Shared-receiver interactions and a branching-workflow experiment
Concern (summary): The original experiments use a four-agent chain, while communication packages may interact when an agent receives multiple messages. What happens when incoming-edge packages are optimized separately, and does PACO remain effective in a branching workflow?
Response: (PART 2/2)
Direct test with three incoming edges. We additionally evaluate a four-role hierarchical MAS inspired by MASPO. Three parallel agents—Answer Proposer, Evidence Scout, and Bridge Resolver—independently examine the same complete source evidence and send messages to one Answer Verifier. The Verifier receives the question and their three messages, without raw evidence. It therefore combines three incoming communication packages within a single receiver call. Detailed role definitions, initialization, message assembly, and constraints are provided in our response to Reviewer 74Th (Weakness 2).
All agents use gpt-5.6-luna with low reasoning effort. The optimized methods start from the same hierarchical configuration under the original 25M-token optimization budget, with roles, topology, evidence access, and output constraints fixed. Configurations learned on HotpotQA are frozen for evaluation on DROP and MuSiQue-Ans, without target-specific optimization. For each optimized method, we independently repeat optimization and evaluation five times; Vanilla and PACT are each evaluated five times. Results are mean ± std, where std is the population standard deviation (denominator N = 5). Token costs are in millions (M).
HotpotQA: quality metrics and token costs.
| Method | Joint F1 | Joint EM | Answer F1 | Answer EM | Support F1 | Support EM | Dep. tokens (M) | Opt. tokens (M) |
|---|---|---|---|---|---|---|---|---|
| Vanilla | 48.05 ± 0.37 | 21.18 ± 0.45 | 54.86 ± 0.39 | 34.80 ± 0.66 | 69.52 ± 0.35 | 45.40 ± 0.34 | 6.34 ± 0.003 | 0 |
| PACT | 46.74 ± 0.62 | 12.98 ± 0.36 | 57.97 ± 1.17 | 33.83 ± 0.60 | 72.25 ± 0.70 | 36.35 ± 0.76 | 6.73 ± 0.04 | 0 |
| MASPO | 46.46 ± 1.79 | 20.60 ± 0.95 | 53.37 ± 1.53 | 34.60 ± 0.53 | 67.89 ± 0.89 | 43.37 ± 0.43 | 6.34 ± 0.02 | 24.36 ± 0.11 |
| MASPOB | 50.31 ± 2.13 | 22.98 ± 1.46 | 59.18 ± 2.42 | 39.17 ± 2.24 | 70.01 ± 1.24 | 45.03 ± 0.60 | 6.43 ± 0.02 | 24.62 ± 0.08 |
| GEPA | 60.59 ± 1.18 | 28.88 ± 1.65 | 69.16 ± 1.24 | 46.83 ± 1.87 | 79.14 ± 0.22 | 52.88 ± 0.22 | 7.53 ± 0.10 | 24.55 ± 0.31 |
| PACO | 62.73 ± 0.76 | 30.33 ± 0.87 | 72.43 ± 0.71 | 51.40 ± 0.82 | 79.42 ± 0.40 | 51.78 ± 0.91 | 6.90 ± 0.32 | 24.37 ± 0.40 |
Frozen cross-dataset evaluation.
| Dataset | Method | Answer F1 | Answer EM | Dep. tokens (M) |
|---|---|---|---|---|
| DROP | Vanilla | 30.03 ± 0.41 | 7.50 ± 0.57 | 1.36 ± 0.002 |
| DROP | PACT | 26.55 ± 0.49 | 6.35 ± 0.90 | 1.64 ± 0.05 |
| DROP | MASPO | 28.13 ± 2.48 | 6.45 ± 1.21 | 1.36 ± 0.12 |
| DROP | MASPOB | 36.08 ± 3.73 | 13.60 ± 3.17 | 1.41 ± 0.03 |
| DROP | GEPA | 40.57 ± 0.83 | 13.30 ± 0.58 | 1.91 ± 0.05 |
| DROP | PACO | 48.09 ± 1.32 | 20.00 ± 1.28 | 1.64 ± 0.15 |
| MuSiQue-Ans | Vanilla | 26.60 ± 0.45 | 15.35 ± 0.51 | 4.49 ± 0.002 |
| MuSiQue-Ans | PACT | 25.52 ± 1.20 | 11.55 ± 0.78 | 4.64 ± 0.06 |
| MuSiQue-Ans | MASPO | 26.24 ± 2.31 | 16.40 ± 0.93 | 4.48 ± 0.15 |
| MuSiQue-Ans | MASPOB | 31.49 ± 0.92 | 20.75 ± 0.76 | 4.53 ± 0.02 |
| MuSiQue-Ans | GEPA | 44.28 ± 1.19 | 30.85 ± 1.85 | 5.12 ± 0.07 |
| MuSiQue-Ans | PACO | 48.62 ± 2.57 | 34.25 ± 1.86 | 4.79 ± 0.17 |
With three branches feeding a shared receiver, PACO achieves the highest mean scores among the evaluated methods on five of the six HotpotQA quality metrics, including Joint F1/EM and Answer F1/EM. Its mean advantage over GEPA is 2.14 Joint F1 points on HotpotQA, with 7.52 and 4.34 Answer F1 points on DROP and MuSiQue-Ans, respectively. These results extend the empirical evidence from sequential handoffs to a parallel workflow with shared evidence and multiple incoming edges.
Response to Question 2
2. Repeated optimization for PACO versus Independent updates and GEPA
Concern (summary): The 0.93-point Table 3 advantage over Independent updates and the approximately two-point main advantage over GEPA come from single optimization runs, while some other comparisons use multiple runs. Matched optimization seeds, means, and variability are needed for these two key comparisons.
Response: (PART 1/2)
We have now repeated both comparisons using five independent optimization runs per optimized method, with optimizer seeds 0–4, including the original seed-0 run. Each run repeats optimization from the common initial workflow and then evaluates its resulting configuration. The model, workflow, data splits, and evaluation conditions are fixed within each comparison.
Paired-refinement ablation (Table 3; 5M-token budget). All four variants use the same 5M-token optimization budget. Independent updates can optimize the same sender instructions, receiver instructions, and shared convention as PACO, but changes only one behavioral component per refinement step while holding the other two fixed. PACO allows coordinated changes within an edge package. These restrictions apply to both ordinary and cooperative steps, with the remaining search procedure unchanged.
The table reports mean ± std over the five independent optimization-and-evaluation runs. Token costs are in millions (M).
| Variant | Joint F1 | Answer F1 | Opt. tokens (M) |
|---|---|---|---|
| PACO | 49.66 ± 0.95 | 59.33 ± 0.77 | 4.61 ± 0.05 |
| Upstream only | 47.16 ± 1.07 | 56.24 ± 0.89 | 4.67 ± 0.06 |
| Downstream only | 46.41 ± 1.21 | 55.92 ± 0.74 | 4.70 ± 0.08 |
| Independent updates | 48.58 ± 0.76 | 58.29 ± 0.63 | 4.85 ± 0.06 |
Across five independent optimization runs, PACO's mean scores are 1.08 points higher in Joint F1 and 1.04 points higher in Answer F1 than those of Independent updates. The revised Table 3 will report these results, which support a modest benefit from coordinated refinement under the same budget.
Response to Question 1
Thank you for your review and for the specific questions about shared receivers and optimization variability. We address these two points below.
1. Shared-receiver interactions and a branching-workflow experiment
Response: (PART 1/2)
How incoming-edge updates are evaluated. A shared receiver's communication prompt contains the receiving blocks associated with its incoming edges. In an ordinary single-edge step, PACO may revise the selected edge's sender instructions, receiver instructions, and shared convention, while keeping the other edge packages fixed. The receiver still processes all incoming messages together. Updating one package can therefore change how it uses information from the other messages, even though their instruction blocks remain unchanged.
Candidate updates are assessed through complete workflow executions, with candidate and parent task scores compared on the same examples at each evaluation stage. The observed gain therefore includes the effects of the update on the shared receiver and the final output. Edge-scheduling histories are also specific to the full configuration, since an edge's benefit can change when the surrounding packages change. When an ordinary step achieves a new best score on the full internal validation set, PACO additionally triggers a cooperative step that can revise multiple packages. Thus, single-edge editing does not assume independent effects across edges; candidate changes are evaluated in their existing communication context.