- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores the necessity of multi-agent collaboration and the optimal allocation of the degree of collaboration. PLASMAS employs a shared language interface to characterize tasks, agent roles, and communication topologies. The system utilizes demonstration data—filtered during the execution process—to initialize a topology generator, which is then aligned based on preferences derived from task success rates and the costs associated with token consumption. Experiments compare accuracy against inference costs across several benchmarks, with further analysis provided on the allocation of nodes and edges for groups with varying collaborative requirements.
Critical strengths
- It is practically significant to jointly decide whether to engage in collaboration and to determine the level of investment in such collaboration; by considering both execution outcomes and cost preferences, the training process can be aligned with this objective.
- Experimental results demonstrate that, compared to the baseline methods evaluated, this approach achieves a superior trade-off between accuracy and cost.
Critical weaknesses
- Modifications to the backbone network, output representations, and supervision methods make it impossible for these control groups to effectively isolate the "semantic bottleneck" claimed in the paper. Since the ablation studies are conducted within a unified language-interface framework, further control groups—in which the supervision method is held constant—are required to disentangle the effects of interface design from those of supervision quality.
- Differences in the allocation of nodes and edges do not constitute proof that the chosen topology is superior. The study lacks control groups that employ simple routing mechanisms or focus solely on controlling the size of the team; consequently, the independent contribution of "learned connectivity" remains unclear.
Questions for authors
1.Can the conventional encoder–graph-predictor pipeline be trained using the same execution-selected supervision and effectiveness/cost preferences? Without such a control, which evidence distinguishes an interface bottleneck from differences in supervision quality? 2.Under matched roles and inference budgets, how well do a single-agent/expert router and a team-size-only predictor with fixed connectivity perform? Can additional executions or alternative anchors establish the stability of the demand groups and allocation gaps? 3.Adding the chosen-sequence NLL term slightly improves accuracy while increasing token cost. Is the effect stable across repeated training and execution runs, and does the benefit justify the extra cost?
3 comments
Weakness 2 and Question 2:
First, the performance of the single-agent/expert router and the team-size-only predictor with fixed connectivity on MATH (test set sized 5000) is shown below.
| Method | Accuracy (%) ↑ | Token Cost ↓ |
|---|---|---|
| Binary Router (Single / Expert) | 81.28 | 3,626 |
| Team-size Predictor (Fixed Connectivity) | 82.50 | 3,383 |
| PLASMAS (Ours) | 84.02 | 3175 |
For the binary router, we use the easy and collaboration-required labels defined in Section 2.1, with 250 training problems from each group. For the team-size predictor, we manually design graphs with 1 to 6 nodes and select 100 problems that perform best with each corresponding team size for training.
During supervised training, the binary router achieves a best training classification accuracy of 75% and a validation classification accuracy of 70% for the labels. For the team-size predictor, the best validation classification accuracy is only 36%, with a mean absolute error of 1.46 nodes.
These results show that the routers themselves do not learn the classification tasks well. This is also consistent with our semantic bottleneck phenomenon: embeddings produced by text encoders are difficult to map to easy and collaboration-required categories using a binary classifier, or to the appropriate team sizes using a multi-class classifier.
Second, we clarify that the demand groups are used only for evaluating topology sensitivity. They do not affect model training or the accuracy and token consumption reported in the main results. We acknowledge that additional executions or alternative expert anchors may change the classification of individual problems. However, our main finding concerns whether PLASMAS maintains a positive topology allocation gap under different grouping protocols, rather than whether the group assignments remain unchanged.
To examine this, we conducted two additional sensitivity analyses on the same fixed set of 400 problems: (1) five additional executions of both original anchors, and (2) ten executions of an alternative expert topology. For the alternative-anchor comparison, we use matched execution seeds and keep the single-agent outcomes unchanged. We apply the original grouping criteria in all settings, retaining unclassified problems without resampling to balance group sizes.
| Grouping protocol | Easy / Collaboration N | Node ΔS [95% CI] | Edge ΔS [95% CI] |
|---|---|---|---|
| Original 5 executions | 100 / 100 | +1.12 [0.67, 1.57] | +1.47 [0.86, 2.08] |
| Original + 5 executions | 89 / 73 | +1.02 [0.50, 1.53] | +1.40 [0.68, 2.12] |
| Alternative expert, 10 executions | 104 / 50 | +0.83 [0.43, 1.24] | +1.17 [0.60, 1.73] |
Although the group assignments and sample sizes change across different protocols, PLASMAS consistently maintains positive node- and edge-level allocation gaps, with all 95% confidence intervals excluding zero. These results suggest that while the exact demand classification depends on execution outcomes and anchor selection, our central observation—that PLASMAS allocates more agents and connections to problems requiring collaboration—remains robust across the tested settings.
Question 3:
We agree that the improvement from the chosen-sequence NLL term is relatively small. To examine its stability, we repeated training with five different random seeds, comparing DPO with and without NLL, both starting from the same Initialization checkpoint. Each of the resulting ten DPO models was then evaluated using three shared execution seeds. The results are shown below.
| Method | Accuracy (%) ↑ | Token Cost ↓ |
|---|---|---|
| Initialization only | 83.06 | 3,626 |
| DPO without NLL | 83.80±0.22 | 3,248.29 ± 168.14 |
| DPO with NLL | 83.95±0.22 | 3,247.44 ± 151.50 |
Across repeated runs, adding NLL improves average accuracy by 0.16 percentage points and token cost remains nearly the same. These results suggest that the NLL term provides a small positive benefit under our current training setup, although its contribution remains modest. We therefore view NLL as an auxiliary objective rather than a major source of PLASMAS's performance gains.
We thank the reviewer for the opinion, and we are sorry to response late because the MAS execution of the additional experiments takes time.
Weakness 1 and Question 1:
Yes. We added a conventional MiniLM–GCN pipeline (the same architecture as AGP) as a control baseline, trained using the same execution-selected topology targets and fixed effectiveness–cost preference pairs, with identical training and validation splits. Supervised learning minimizes node- and edge-level binary cross-entropy, while DPO uses the joint Bernoulli likelihood of the proposed topology.
The accuracy and token costs on MATH(test set sized 5000) are reported below. As shown, even with this two-stage optimization, the conventional MiniLM–GCN pipeline achieves only marginal improvements.
Specifically, when trained solely with supervised learning, MiniLM–GCN achieves only a modest improvement over the single-agent baseline, with relatively low token consumption. We hypothesize that the model primarily learns common node activation patterns from the training set instead of effectively leveraging the semantic information encoded by MiniLM to guide topology generation. The limited node-level sensitivity (ΔS = 0.10) is consistent with this hypothesis, indicating little difference in agent allocation between Easy and Collaboration-Required problems. This observation also aligns with the results in Table 2 of our paper, where the high Holm-adjusted p-values indicate that the differences in topology allocation between the two groups are not statistically significant.
Similarly, adding DPO improves accuracy from 80.43% to 82.40%, but substantially increases token consumption from 1,322 to 3,559, while node-level sensitivity increases only slightly to 0.23, compared with 1.12 for PLASMAS. This suggests that although DPO encourages more collaboration, the conventional pipeline still struggles to distinguish task-specific collaboration demands and allocate agents accordingly.
Together, these findings suggest that conventional pipelines struggle to translate fine-grained collaboration-relevant semantics into adaptive topology decisions, providing further evidence consistent with the semantic bottleneck identified in our work.