acceptodds
Under review as a conference paper at ICLR 2027

PLASMAS: How Much Collaboration Do You Need in Multi-Agent System?

Abstract

Semantically similar tasks can differ substantially in whether and how much multi-agent collaboration they require, making task-specific topology generation critical for efficient multi-agent systems (MAS). Existing graph-based generators typically encode tasks into intermediate representations before predicting topologies with a separate graph model. We show that these pipelines exhibit limited sensitivity to fine-grained collaboration demands, with only marginal gains from stronger encoders or graph backbones, suggesting a semantic bottleneck between task understanding and topology generation. To address this limitation, we propose PLASMAS, a preference-aligned topology generator following the language-based graph foundation model paradigm. PLASMAS jointly models task semantics, agent roles, and topology outputs within a shared language interface. Utility-aware pairwise preferences guide direct preference optimization based on task effectiveness and execution cost. Experiments show that PLASMAS better distinguishes collaboration demands among semantically similar tasks, allocating additional collaboration where beneficial and reducing unnecessary coordination on easy tasks while maintaining strong task performance. Our code is available at https://anonymous.4open.science/r/PLASMAS-F526/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent systems, topology learning, graph structure learning, task-adaptive communication, agent collaboration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r2jiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the necessity of multi-agent collaboration and the optimal allocation of the degree of collaboration. PLASMAS employs a shared language interface to characterize tasks, agent roles, and communication topologies. The system utilizes demonstration data—filtered during the execution process—to initialize a topology generator, which is then aligned based on preferences derived from task success rates and the costs associated with token consumption. Experiments compare accuracy against inference costs across several benchmarks, with further analysis provided on the allocation of nodes and edges for groups with varying collaborative requirements.

Critical strengths
  1. It is practically significant to jointly decide whether to engage in collaboration and to determine the level of investment in such collaboration; by considering both execution outcomes and cost preferences, the training process can be aligned with this objective.
  2. Experimental results demonstrate that, compared to the baseline methods evaluated, this approach achieves a superior trade-off between accuracy and cost.
Critical weaknesses
  1. Modifications to the backbone network, output representations, and supervision methods make it impossible for these control groups to effectively isolate the "semantic bottleneck" claimed in the paper. Since the ablation studies are conducted within a unified language-interface framework, further control groups—in which the supervision method is held constant—are required to disentangle the effects of interface design from those of supervision quality.
  2. Differences in the allocation of nodes and edges do not constitute proof that the chosen topology is superior. The study lacks control groups that employ simple routing mechanisms or focus solely on controlling the size of the team; consequently, the independent contribution of "learned connectivity" remains unclear.
Questions for authors

1.Can the conventional encoder–graph-predictor pipeline be trained using the same execution-selected supervision and effectiveness/cost preferences? Without such a control, which evidence distinguishes an interface bottleneck from differences in supervision quality? 2.Under matched roles and inference budgets, how well do a single-agent/expert router and a team-size-only predictor with fixed connectivity perform? Can additional executions or alternative anchors establish the stability of the demand groups and allocation gaps? 3.Adding the chosen-sequence NLL term slightly improves accuracy while increasing token cost. Is the effect stable across repeated training and execution runs, and does the benefit justify the extra cost?

3 comments
AuthorsOpenReview ↗

We thank the reviewer for the opinion, and we are sorry to response late because the MAS execution of the additional experiments takes time.

Weakness 1 and Question 1:

Yes. We added a conventional MiniLM–GCN pipeline (the same architecture as AGP) as a control baseline, trained using the same execution-selected topology targets and fixed effectiveness–cost preference pairs, with identical training and validation splits. Supervised learning minimizes node- and edge-level binary cross-entropy, while DPO uses the joint Bernoulli likelihood of the proposed topology.

The accuracy and token costs on MATH(test set sized 5000) are reported below. As shown, even with this two-stage optimization, the conventional MiniLM–GCN pipeline achieves only marginal improvements.

MethodAccuracy (%) ↑Token Cost ↓Node ΔS
Single-Agent78.98977-
MiniLM-GCN (SL)80.4313220.10
MiniLM-GCN (SL+DPO)82.4035590.23
PLASMAS (Ours)84.0231751.12

Specifically, when trained solely with supervised learning, MiniLM–GCN achieves only a modest improvement over the single-agent baseline, with relatively low token consumption. We hypothesize that the model primarily learns common node activation patterns from the training set instead of effectively leveraging the semantic information encoded by MiniLM to guide topology generation. The limited node-level sensitivity (ΔS = 0.10) is consistent with this hypothesis, indicating little difference in agent allocation between Easy and Collaboration-Required problems. This observation also aligns with the results in Table 2 of our paper, where the high Holm-adjusted p-values indicate that the differences in topology allocation between the two groups are not statistically significant.

Similarly, adding DPO improves accuracy from 80.43% to 82.40%, but substantially increases token consumption from 1,322 to 3,559, while node-level sensitivity increases only slightly to 0.23, compared with 1.12 for PLASMAS. This suggests that although DPO encourages more collaboration, the conventional pipeline still struggles to distinguish task-specific collaboration demands and allocate agents accordingly.

Together, these findings suggest that conventional pipelines struggle to translate fine-grained collaboration-relevant semantics into adaptive topology decisions, providing further evidence consistent with the semantic bottleneck identified in our work.

Weakness 2 and Question 2:

First, the performance of the single-agent/expert router and the team-size-only predictor with fixed connectivity on MATH (test set sized 5000) is shown below.

MethodAccuracy (%) ↑Token Cost ↓
Binary Router (Single / Expert)81.283,626
Team-size Predictor (Fixed Connectivity)82.503,383
PLASMAS (Ours)84.023175

For the binary router, we use the easy and collaboration-required labels defined in Section 2.1, with 250 training problems from each group. For the team-size predictor, we manually design graphs with 1 to 6 nodes and select 100 problems that perform best with each corresponding team size for training.

During supervised training, the binary router achieves a best training classification accuracy of 75% and a validation classification accuracy of 70% for the labels. For the team-size predictor, the best validation classification accuracy is only 36%, with a mean absolute error of 1.46 nodes.

These results show that the routers themselves do not learn the classification tasks well. This is also consistent with our semantic bottleneck phenomenon: embeddings produced by text encoders are difficult to map to easy and collaboration-required categories using a binary classifier, or to the appropriate team sizes using a multi-class classifier.

Second, we clarify that the demand groups are used only for evaluating topology sensitivity. They do not affect model training or the accuracy and token consumption reported in the main results. We acknowledge that additional executions or alternative expert anchors may change the classification of individual problems. However, our main finding concerns whether PLASMAS maintains a positive topology allocation gap under different grouping protocols, rather than whether the group assignments remain unchanged.

To examine this, we conducted two additional sensitivity analyses on the same fixed set of 400 problems: (1) five additional executions of both original anchors, and (2) ten executions of an alternative expert topology. For the alternative-anchor comparison, we use matched execution seeds and keep the single-agent outcomes unchanged. We apply the original grouping criteria in all settings, retaining unclassified problems without resampling to balance group sizes.

Grouping protocolEasy / Collaboration NNode ΔS [95% CI]Edge ΔS [95% CI]
Original 5 executions100 / 100+1.12 [0.67, 1.57]+1.47 [0.86, 2.08]
Original + 5 executions89 / 73+1.02 [0.50, 1.53]+1.40 [0.68, 2.12]
Alternative expert, 10 executions104 / 50+0.83 [0.43, 1.24]+1.17 [0.60, 1.73]

Although the group assignments and sample sizes change across different protocols, PLASMAS consistently maintains positive node- and edge-level allocation gaps, with all 95% confidence intervals excluding zero. These results suggest that while the exact demand classification depends on execution outcomes and anchor selection, our central observation—that PLASMAS allocates more agents and connections to problems requiring collaboration—remains robust across the tested settings.

Question 3:

We agree that the improvement from the chosen-sequence NLL term is relatively small. To examine its stability, we repeated training with five different random seeds, comparing DPO with and without NLL, both starting from the same Initialization checkpoint. Each of the resulting ten DPO models was then evaluated using three shared execution seeds. The results are shown below.

MethodAccuracy (%) ↑Token Cost ↓
Initialization only83.063,626
DPO without NLL83.80±0.223,248.29 ± 168.14
DPO with NLL83.95±0.223,247.44 ± 151.50

Across repeated runs, adding NLL improves average accuracy by 0.16 percentage points and token cost remains nearly the same. These results suggest that the NLL term provides a small positive benefit under our current training setup, although its contribution remains modest. We therefore view NLL as an auxiliary objective rather than a major source of PLASMAS's performance gains.

Reviewer_pf96OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies task-specific communication topology generation for LLM-based multi-agent systems, focusing on whether a generator can distinguish semantically similar tasks that differ in how much collaboration they need. Doing a pilot study on MATH and a sensitivity analysis of AGP and GTD, the authors argue that the standard text-encoder–graph-learner pipeline suffers from a semantic bottleneck that stronger encoders or graph backbones do not fix. They then propose PLASMAS, which uses a language model to generate serialized topologies directly from the task and role descriptions. The generator is first initialized by supervised learning on execution-selected demonstrations and then aligned with weighted DPO plus a chosen-sequence NLL term on preference pairs built from task effectiveness and token cost. With execution agents on GSM8K, MATH-500, MultiArith, SVAMP and MMLU, PLASMAS achieves the best or near-best accuracy at substantially lower token cost than the multi-agent baselines, and is the only method with a significantly positive Topology Sensitivity Gap at both node and edge level.

Critical strengths
  1. The paper frames the problem more pointedly than most topology-generation work. It studies "whether collaboration is needed at all" rather than "which topology", and builds a reproducible protocol for it. This protocol, and the insensitivity it reveals in AGP/GTD, is valuable to the community independently of the proposed method.

  2. The experiments show that the method improves accuracy while reducing inference cost. On MATH-500 PLASMAS achieves higher accuracy than all baselines with fewer tokens than most multi-agent baselines. On the remaining benchmarks its accuracy is comparable to the baselines at noticeably lower token cost than GTD/MaAS.

Critical weaknesses
  1. The "semantic bottleneck" attribution is not isolated from differences in supervision, and the sensitivity comparison is partly circular. PLASMAS differs from AGP/GTD in interface and in training labels selected as the cheapest correct topology and a candidate pool that directly includes singleton and expert anchor. Table 4 shows that SFT initialization alone already yields most of the sensitivity. The control keeps AGP’s original supervision, so there is no experiment training an encoder–graph pipeline on the same D_demo, and one cannot tell whether the language interface or the better-targeted supervision is responsible. And the expert anchor used to define the collaboration-required stratum is itself one of PLASMAS’s candidate templates, which the baselines never see, inflating the comparison in Figure 3.

  2. Baseline fidelity and statistical reliability in Table 3 are insufficient for the size of the claimed gap. Most baselines are reimplemented by the authors, and GPTSwarm and AGP fall below the single-agent Vanilla baseline on MATH-500, suggesting under-tuning. Whether baselines were trained on the same corpora and execution model, and whether token accounting is consistent, is not stated. The ablation accuracy (84.02) does not match Table 3 (84.6), and the ablation benchmark is unspecified. And the total training cost (on the order of 100 full multi-agent executions per training problem) is not reported, and the method relies on a fixed role pool and hand-crafted templates with no test of generalization to a new role pool.

  3. Some component claims are overstated. The chosen-sequence NLL term yields only a marginal accuracy gain while increasing token cost, which is indistinguishable from noise without variance. The pilot study’s gap in ranking agreement between nearest neighbours and random problems is small, reflects a property of MiniLM embeddings rather than the behaviour of any topology generator.

Questions for authors
  1. Please provide a control that separates the effect of the language interface from that of the cost-aware supervision.

  2. How were the baselines trained and evaluated, and how much do the main numbers move across runs?

  3. What did training actually cost in total, and how much of the result depends on the fixed role pool and the hand-crafted templates?

  4. Which part of the pipeline do you consider the technical contribution beyond applying existing GFM and DPO techniques to MAS?

3 comments
AuthorsOpenReview ↗

We thank the reviewer for the opinion, and we are sorry to response late because the MAS execution of the additional experiments takes time.

Weakness 1 and Question 1:

We added a matched-supervision control using AGP’s MiniLM–GCN pipeline. Both pipelines use the same candidate pool, the same supervised demonstrations (DdemoD_{\mathrm{demo}}), the same effectiveness–cost preference pairs, and the same train/validation splits. We apply the same two-stage optimization protocol, supervised learning followed by DPO, using the joint Bernoulli topology likelihood for MiniLM–GCN (AGP). Results on the 5,000-problem MATH test set are below.

MethodAccuracy (%) ↑Token Cost ↓Node ΔS ↑
Single-Agent78.98977—
MiniLM–GCN (SL)80.431,3220.10
MiniLM–GCN (SL+DPO)82.403,5590.23
PLASMAS84.023,1751.12

Supervised learning improves accuracy by only 1.45 percentage points over the single-agent baseline. The demonstrations activate 2.26 nodes on average, while MiniLM–GCN predicts a mean soft node count of 2.30. At a 0.55 activation threshold, 74.7% of its graphs are single agent, even reaching 100% in one paticular run. This tendency toward nearly fixed allocation recurs across runs and is consistent with the low Node ΔS (0.10).

Figure 1(b) shows that problems with similar text-encoder embeddings can differ in difficulty. A binary classifier trained on these embeddings achieves 75% training accuracy and 70% validation accuracy, suggesting limited recoverability of collaboration-relevant information. We hypothesize that weak semantic signals make adaptive allocation difficult to learn, allowing the graph model to settle into a local optimum that reflects the training set’s average activation level. This difficulty in translating problem semantics into adaptive topology decisions motivates our notion of a semantic bottleneck.

Adding DPO improves accuracy but substantially increases token cost, while Node ΔS reaches only 0.23, compared with 1.12 for PLASMAS. We agree that initialization accounts for most of PLASMAS’s sensitivity in Table 4. However, the same demonstrations yield a Node ΔS of only 0.10 in MiniLM–GCN. Thus, matching the candidate pool, supervision, and optimization protocol is insufficient for the tested conventional pipeline to achieve comparable adaptation to task-specific collaboration demands.

The expert anchor defines the evaluation groups by identifying problems that benefit from additional collaboration. These group labels are used only for evaluation and do not enter training. Figure 3 compares the generated topologies across 200 held-out problems, comprising 100 Easy and 100 Collaboration-Required problems, to evaluate whether the methods allocate more nodes and edges to problems requiring collaboration. Topology generation is unrestricted by the evaluation anchor.

We acknowledge that using a candidate template as the evaluation anchor may cause your concern. To assess this dependence, we repeated the analysis using an alternative expert topology which is not in the candidate anchor pool for PLASMAS. PLASMAS retains positive allocation gaps for both nodes (ΔS = 0.828, 95% CI [0.427, 1.241]) and edges (ΔS = 1.170, 95% CI [0.586, 1.731]). These results support the robustness of PLASMAS’s adaptive allocation to the tested change in evaluation anchor. The alternative expert is:

  A["agent_0: Problem analyst"] --> C["agent_2: Primary solver"]
  B["agent_1: Strategy planner "] --> D["agent_3: Alternative solver "]
  C --> E["agent_4: Symbolic/proof verifier "]
  D --> E
  C --> F["agent_5: Finalizer "]
  D --> F
  E --> F

Question 2:

We reproduced the baselines using their official implementations wherever available. For methods with released checkpoints, such as AGP, we used those checkpoints directly. For benchmarks covered by official data-preparation scripts, we followed the provided procedures. For benchmarks requiring additional adaptation, we constructed optimization data from the same MATH training split used by PLASMAS, following each baseline’s prescribed sample budget and optimization procedure. We don't think training data and process should be identical across the baselines because these are part of their methods. But evaluation problems, execution models, and token accounting are aligned as followed.

We revised the evaluation protocol to use a common held-out set for each benchmark, excluding the union of examples used to train or optimize any compared method. We then evaluated all methods over five execution runs and report the updated means and standard deviations in the table below. All agent calls use the same Qwen3-4B model with temperature 0.7. Token cost includes input and output tokens across all agent calls and inter-agent communication during execution after topology generation.

During verification, we identified and corrected an error in the answer-scoring procedure that affected all previously reported baseline results on MATH-500. This is the reason why several baselines are seemed under-tuning. The updated results use the corrected scoring procedure consistently across methods. Due to the time limit, we present here the most important baselines and the MMLU is tested on the first 153 problems of the official validation set of MMLU. We acknowledge that the original scores overstated PLASMAS’s accuracy advantage on MATH-500 and will revise Table 3 and the corresponding claims accordingly.

MethodGSM8KMATH-500MultiArithSVAMPMMLU (val 153)
Acc. ↑Tokens ↓Acc. ↑Tokens ↓Acc. ↑Tokens ↓Acc. ↑Tokens ↓Acc. ↑Tokens ↓
Vanilla85.55±0.37227.48±0.7674.84±0.54513.18±1.1196.90±0.00156.69±0.6591.20±0.20156.74±0.4269.67±2.51228.84±1.01
GPTSwarm89.52±0.283136±33.9778.36±1.122446.60±26.3296.56±0.003587.74±17.0788.04±0.901856.61±7.7465.49±0.728033.07±176.84
G-Designer89.57±0.544003.74±47.2677.09±1.057343.26±65.1396.05±0.743511.46±51.1293.29±0.373377.21±54.2075.03±3.1115893.36±275.81
MaAS91.28±0.493360.91±84.4279.96±1.2110258.29±772.0696.52±0.603383.75±83.3694.18±0.632670.83±95.3669.28±2.813612.77±200.98
AGP89.16±0.415608.67±32.1182.32±0.467716.55±42.6996.14±0.394899.09±21.7793.60±0.414883.22±16.8274.38±3.8510073.81±118.49
GTD91.01±0.423511.56±24.7782.72±0.418161.73±91.1997.04±0.172054.56±28.9693.55±0.212422.14±32.3175.53±2.674024.24±56.94
OFAMAS88.42±0.733925.02±23.7878.08±1.386623.97±77.2196.43±0.452969.93±63.7592.95±0.302987.19±49.5274.51±2.404264.26±226.07
PLASMAS (Ours)91.27±0.48846.36±5.8483.44±0.783742.74±25.2096.97±0.13560.53±1.3994.03±0.38653.65±3.5374.72±2.27652.86±25.48

Most metrics show modest variation across runs, with means generally close to the previously reported values, except for the corrected MATH-500 results. MMLU shows somewhat larger fluctuations, likely reflecting the smaller evaluation set of 153 validation questions used due to time constraints.

Across five execution runs, PLASMAS consistently maintains competitive accuracy across all five benchmarks while substantially reducing execution token costs relative to most multi-agent baselines. These repeated evaluations support the same accuracy–efficiency trade-off observed in our original experiments, showing that PLASMAS can achieve strong task performance with substantially less communication and inference overhead.

Question 3:

For data construction, we use 1,000 training problems from MATH and 1,000 from MMLU-Pro. Constructing the Initialization demonstrations and DPO preference pairs requires an average of 220 individual LLM calls per problem, totaling approximately 440,000 calls, with a maximum output length of 2,048 tokens per call. Using vLLM on four RTX 3090 GPUs with 16 concurrent requests per GPU, generation takes approximately 10 hours per dataset, totaling 40 GPU-hours per dataset.

Initialization requires approximately 1.3 hours on one RTX 3090 and an estimated 2.81–3.00×10172.81–3.00 × 10^{17} FLOPs. DPO requires approximately 6.1 hours on one RTX 3090 and 1.38–1.47×10181.38–1.47 × 10^{18} FLOPs. The combined cost of data construction and one complete training run of the final model is therefore approximately 47.4 RTX 3090 GPU-hours.

The role pool and candidate templates play an important role because they determine the collaboration strategies available during data construction. Effective learning requires these candidates to reveal differences in collaboration needs across problems. If a role pool leads to single agent solutions or full-team collaboration for nearly all problems, the resulting demonstrations and preference pairs provide limited signals for learning adaptive allocation.

Our study focuses on learning task-dependent allocation within a given role pool and template collection. The original role pool and templates were used without benchmark-specific tuning. To examine sensitivity to the role pool, we replaced the original six-role pool with AGP’s six-role pool, regenerated the training data, and retrained PLASMAS. Both configurations were evaluated on the full 5,000-problem MATH test set.

Role poolAccuracy (%) ↑Token Cost ↓
Original (6)84.023175
AGP's (6)83.682677

Using AGP’s six-role pool, PLASMAS achieves 83.68% accuracy and token consumption of 2,677. This suggests that its favorable accuracy–cost trade-off extends to another role pool, although role composition affects the resulting balance.

Role capabilities, candidate templates, and the supervision derived from their execution interact, making their individual contributions difficult to isolate. Together with our baseline comparisons, these results support the effectiveness of PLASMAS with suitable role pools under the tested settings. Broader robustness across arbitrary role pools and template collections remains to be established.

Weakness 2:

Most concerns raised in Weakness 2 are addressed in our responses to Questions 2 and 3. We additionally clarify that the ablation accuracy of 84.02% was measured on the full 5,000-problem MATH test set, whereas the originally reported 84.6% in Table 3 was measured on MATH-500. These values therefore correspond to different evaluation sets. The ablation test problems were excluded from training and training-data construction. We will explicitly identify the evaluation set and sample count for each experiment in the revision.

Question 4:

We build on existing GFM and DPO techniques. Our contribution is a language-based topology-generation interface jointly designed with execution-derived supervision to capture fine-grained collaboration requirements among problems of the same type. We construct execution-derived supervision by selecting the lowest-cost correct topology, including singleton solutions, and refine this allocation through effectiveness–cost preferences. This connects problem semantics to both whether collaboration is needed and how much is sufficient. The topology interface, target construction, and optimization are designed jointly for this purpose. Our matched-supervision experiment further shows that the same training signals yield substantially lower allocation sensitivity in the AGP pipeline, supporting the effectiveness of the proposed integrated design.

Reviewer_JGUkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PLASMAS, a topology generator for multi-agent communication. The paper claims that existing pipelines exhibit limited sensitivity to fine-grained collaboration demands, suggesting a ‘semantic bottleneck’ between task understanding and topology generation. PLASMAS jointly models task semantics, agent roles, and topology outputs within a shared language interface. The paper evaluates PLASMAS on five benchmarks, reporting favorable accuracy-efficiency tradeoffs and improved sensitivity to task-specific collaboration requirements.

Critical strengths
  1. The paper identifies an important practical problem that semantically similar problems can require different levels of collaboration.
  2. The empirical results on MATH-500 are strong.
  3. The method seems to achieve a favourable accuracy-efficiency tradeoff.
Critical weaknesses
  1. Evidence of the Semantic Bottleneck: The qualitative and quantitative evidence provided for semantic bottlenecks is poor. A graph generator predicting the expert MAS for both the questions does not demonstrate a semantic bottleneck, as the expert MAS is a valid graph for both the problems. The pipeline might be capable of outputting more complex graphs for more difficult problems, and could just be generating a complex graph for the simpler problems. This would be wasteful, but would not be a case of semantic bottleneck. Similar reasoning holds for the quantitative results. Semantic bottleneck is a plausible cause, but these experiments do not evidence it.
  2. Insufficient Baselines: It is unclear whether the baselines are trained to minimise cooperation for easy problems. The paper should also be comparing PLAMAS against straightforward baselines, like a classifier trained to predict whether the problem is difficult or easy and then choosing a set graph expert graph or a sparse/single-node graph, respectively.
  3. Unclear Benefits: Table 4 shows that the ablation w/o Chosen NLL obtains accuracy close to the full PLASMAS, while requiring fewer tokens. This ablated model might be providing superior accuracy-efficiency tradeoff.

Smaller Formatting Issue: The paper doesn’t respect the paper formatting instructions. Table captions and numbers should appear before the table.

Questions for authors
  1. Could you please provide stronger evidence that the limitations of existing graph-based methods arise from the semantic bottleneck, i.e. the gap between text encoding and graph prediction, rather than simply from differences in supervision or optimisation?
  2. Can you please compare PLASMAS with a baseline classifier trained to predict whether the problem is difficult or easy and then choosing a set graph expert graph or a sparse/single-node graph, respectively?
  3. What are the accuracies and token-consumption for PLASMAS, ablations and the baselines for the easy and difficult problems separately?
5 comments
AuthorsOpenReview ↗

We thank the reviewer for the opinion, and we are sorry to response late because the MAS execution of the additional experiments takes time.

Weekness 1 and Question 1:

First, we use semantic bottleneck to describe the broader phenomenon that conventional text-encoder–graph-model pipelines struggle to capture and respond to task-specific collaboration demands. In our setting, an effective model should select a topology that solves the task with minimal token cost. If a model generates unnecessarily complex graphs for simple problems, it fails to adapt its topology to task requirement. We view this as a manifestation of the semantic bottleneck, where task-relevant semantic information(e.g. hard or easy) is not effectively translated into topology decisions. Our training objectives explicitly address this issue: the initialization stage favors lower-cost topologies among equally effective candidates, while the DPO stage incorporates execution cost into preference selection.

Second, we conducted two additional experiments to further investigate the semantic bottleneck.

We first tested whether MiniLM embeddings could distinguish collaboration requirements using an attention-pooling layer and a two-layer MLP. The classifier achieved 75% training accuracy and 70% validation accuracy, suggesting that collaboration-relevant information is difficult to extract reliably from the encoded representations.

We then trained a conventional MiniLM–GCN pipeline using the same supervised training data and DPO preference pairs as PLASMAS, to control for differences in supervision and optimization.

MethodAccuracy (%) ↑Token Cost ↓Node ΔS
Single-Agent78.98977-
MiniLM-GCN (SL)80.4313220.10
MiniLM-GCN (SL+DPO)82.4035590.23
PLASMAS (Ours)84.0231751.12

With supervised learning alone, MiniLM–GCN achieves only a small accuracy improvement over the single-agent baseline, with token consumption remaining relatively low and limited node-level sensitivity (ΔS = 0.10). Adding DPO improves accuracy but substantially increases token cost, while the node-level sensitivity gap remains at 0.23, compared with 1.12 for PLASMAS. This suggests that although preference optimization encourages more collaboration, the conventional pipeline still struggles to allocate agents according to task-specific collaboration demands.

Together with the encoder and graph-backbone scaling experiments in Table 2, these results suggest that the limited adaptability of conventional pipelines cannot be explained solely by differences in supervision, optimization, or model capacity. As demonstrated by OFA-MAS, conventional pipelines can generate different topologies for tasks with substantially different semantics. However, distinguishing fine-grained collaboration demands among semantically similar problems is much more challenging. We therefore believe that the main limitation lies in the difficulty of capturing and utilizing these fine-grained, collaboration-relevant semantics.

Weekness 2 and Question 2:

First, we would like to clarify that the existing baselines already consider efficiency in their designs. Specifically, AGP encourages sparse agent participation through pruning and sparsity regularization, while GTD explicitly considers communication cost alongside task performance in its optimization objective. Therefore, we believe these methods are reasonable baselines for evaluating the accuracy–efficiency trade-off. As shown in Table 3, across all five benchmarks, PLASMAS substantially reduces token consumption while maintaining competitive task performance compared with these baselines.

Second, we provide additional results for the straightforward baselines as suggested. Specifically, we implement a binary router that selects between a single-agent topology and the fixed expert anchor, and a team-size predictor that selects the number of agents with fixed connectivity. Their performance on the MATH test set (5,000 problems) is shown below. The results suggest that fine-grained topology generation provides benefits beyond simple binary routing or team-size prediction.

MethodAccuracy (%) ↑Token Cost ↓
Binary Router (Single / Expert)81.283,626
Team-size Predictor (Fixed Connectivity)82.503,383
PLASMAS (Ours)84.023,175

For the binary router, we use the easy and collaboration-required labels defined in Section 2.1, with 250 training problems from each group. For the team-size predictor, we manually design graphs with 1 to 6 nodes and select 100 problems that perform best with each corresponding team size for training.

During supervised training, the binary router achieves a best training classification accuracy of 75% and a validation classification accuracy of 70% for the labels. For the team-size predictor, the best validation classification accuracy is only 36%, with a mean absolute error of 1.46 nodes. These results suggest that predicting collaboration requirements or appropriate team sizes directly from text embeddings remains challenging, which is consistent with our semantic bottleneck phenomenon.

Weakness 3:

We agree that the improvement from the chosen-sequence NLL term is relatively small. To examine its stability, we repeated training with five different random seeds, comparing DPO with and without NLL, both starting from the same Initialization checkpoint. Each of the resulting ten DPO models was then evaluated using three shared execution seeds. The results are shown below.

MethodAccuracy (%) ↑Token Cost ↓
Initialization only83.063,626
DPO without NLL83.80±0.223,248.29 ± 168.14
DPO with NLL83.95±0.223,247.44 ± 151.50

Across repeated runs, adding NLL improves average accuracy by 0.16 percentage points and token cost remains nearly the same. These results suggest that the NLL term provides a small positive benefit under our current training setup, although its contribution remains modest. We therefore view NLL as an auxiliary objective rather than a major source of PLASMAS's performance gains.

Question 3:

We report accuracy and token consumption separately for Easy and Collaboration-Required problems, following the definitions in Section 2.1. Since these labels are derived from repeated executions of the Single-Agent and Expert MAS anchors rather than provided by the MATH benchmark, we conduct this analysis on our labeled subset of 200 problems (100 per group), with all methods evaluated on the same problems.

MethodEasy Accuracy (%) ↑Easy Token Cost ↓Hard Accuracy (%) ↑Hard Token Cost ↓
AGP974,602696,698
GTD984,771656,816
OFAMAS775488486785
Initialization only962,244734,408
w/o Chosen NLL982,207734,878
PLASMAS982,118744,956

On Easy problems, PLASMAS achieves 98% accuracy with only 2,118 tokens, reducing token consumption by over 50% compared with AGP and GTD while maintaining comparable accuracy. On Collaboration-Required problems, PLASMAS allocates more tokens and achieves the highest accuracy (74%), while still consuming fewer tokens than the graph-based baselines.

Compared with Initialization Only, preference alignment reduces token consumption on Easy problems while improving accuracy. On Collaboration-Required problems, it allocates more resources and achieves a small accuracy improvement. Removing the chosen-sequence NLL term has only a minor effect, consistent with our findings in the response to Weakness 3.

Comments on the paper

1 comment

Dear Area Chair,

As the rebuttal period is nearing its end, we would appreciate your consideration of our responses and additional experiments. We have not yet received follow-up from several reviewers and would welcome any remaining questions.

The reviewers' main concern is that PLASMAS’s improvements might arise from its training data and optimization procedure, leaving the proposed semantic bottleneck insufficiently established. To address this, we trained AGP’s conventional MiniLM–GCN pipeline using the same candidate pool, supervised demonstrations, effectiveness–cost preference pairs, and two-stage training protocol as PLASMAS. We also evaluated simpler classifiers and a random-routing control. Results on the full 5,000-problem MATH test set are shown below.

MethodNode ΔSAccuracy on MATH test setToken Cost
AGP0.0082.877824
MiniLM-GCN (SL)0.1080.431322
MiniLM-GCN (SL+DPO)0.2382.403559
Binary Router (Single / Expert)0.2481.283,626
Team-size Predictor (Fixed Connectivity)0.2082.503,383
Random Avg. 50.0082.713440
PLASMAS (Ours)1.1284.023175

The additional controls show Node ΔS values close to zero, indicating little additional node allocation to Collaboration-Required problems compared with Easy problems. Their aggregate performance is also comparable to randomly assigning singleton or expert-anchor solutions. These findings suggest that optimization primarily learns overall resource-allocation preferences, with limited sensitivity to individual problems’ collaboration requirements.

This difficulty in learning the mapping from problem semantics to appropriate collaboration structures motivates our notion of a semantic bottleneck. The matched-supervision control shows that the observed difference persists when training data and optimization protocols are aligned. We therefore believe that the tested encoder–graph architecture has difficulty extracting and using collaboration-relevant semantic information, and that differences in supervision alone cannot explain the results.

We have also provided additional experiments and detailed responses to the reviewers’ other concerns. If time permits, we would appreciate your help encouraging further discussion of whether these updates address the outstanding concerns. We remain available to provide concise clarifications.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.