- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the joint optimization of role prompts and execution models in heterogeneous LLM-based multi-agent systems. The authors propose UniMASO, a framework that jointly optimizes task quality and deployment cost through local prompt–model search, multi-level feedback, cost-aware Pareto search, topology-aware global coordination, and trajectory memory. The empirical study covers five benchmarks and five multi-agent execution topologies, and includes ablation studies of each major component as well as several parameter analyses.
In the centralized setting, compared with DeepSeek-V4-Pro + MASPO, UniMASO improves the average task score from 69.79 to 70.40 while reducing the weighted token deployment cost of task agents by approximately 85.2%. This combination of a modest gain in average performance and a substantial reduction in deployment cost constitutes a practically meaningful quality–cost trade-off.
Critical strengths
-
The problem addressed is of practical importance, and the motivation for joint optimization is well founded. An agent's behavior is determined by its role prompt but is also bounded by the capabilities of the underlying execution model. By treating these two variables jointly rather than in isolation, the paper more fully captures the coupling among role requirements, model capabilities, and inference cost, which is directly relevant to the deployment of heterogeneous multi-agent systems.
-
The method is internally coherent and explicitly accounts for inter-agent interactions. Integrating local, downstream, and global feedback avoids evaluating a modification solely on the basis of a single role's performance. Trajectory memory allows prior optimization experience to be reused, while system-level coordination manages the mutual compatibility of local updates. These design choices align well with the dependency structure of multi-agent systems and with the stated quality–cost objective.
-
The empirical study is substantial and examines the method along multiple dimensions. The evaluation spans five diverse benchmarks and five execution topologies, and is complemented by ablations of the main components, hyperparameter analyses, memory transfer experiments, and an analysis of model assignment behavior. The evaluation is reasonably comprehensive, and the reported results indicate a consistent deployment cost advantage across settings.
Critical weaknesses
-
The presentation of the method could be further strengthened. Algorithm 1 outlines the overall optimization procedure, and the appendix documents the optimizer and evaluator configurations as well as the invocation templates. However, the rules governing candidate retention and configuration updates, namely those embodied by the operators , , and , are insufficiently specified (see Question 1). Since these rules determine both the search trajectory and the final configuration, the current presentation makes the algorithm difficult to understand precisely and to reproduce faithfully.
-
The experimental comparison with model routing methods remains insufficient. MasRouter is discussed in the related work, but no direct empirical comparison is reported. Because all existing baselines are prompt optimization methods operating with a fixed, homogeneous model, the current evidence does not establish that joint optimization outperforms optimizing the two components separately.
-
The applicability of the method to broader model pools remains to be established (minor limitation). The five execution models do span different capability and cost tiers, but they are drawn exclusively from the Qwen and DeepSeek model families. Consequently, the existing evidence characterizes the method's behavior only within this particular model pool. Its sensitivity to model pool composition, and its effectiveness with other model families or with mixed pools that include commercial API models, lack direct empirical support.
Questions for authors
-
Could the authors provide definitions for the main operators in Algorithm 1? Specifically, (i) how does (line 15) truncate the non-dominated candidate set when its size exceeds the beam width; (ii) what criterion does (line 18) use to determine the locally optimal configuration; and (iii) what is the explicit form of in Eq. (6), and how is the corresponding optimization problem solved?
-
The paper does not appear to state whether the experiments were independently repeated. How many independent runs were conducted for each setting, and are the reported values averages over these runs? It would be helpful if the authors could report the mean together with the standard deviation or standard error across independent runs to substantiate the stability of the reported results.
-
Could the authors provide a comparison with model routing methods such as MasRouter under comparable model pools, topologies, datasets, and search budgets? If such a comparison is infeasible, which specific differences in experimental setup preclude it?
-
How sensitive is the method to the composition of the model pool? Are there results on other model families or commercial API models, or any further analysis that delineates the method's scope of applicability?
6 comments
Official Comment to Reviewer YWfx (2/3)
Q2. Were the experiments independently repeated? How many independent runs were conducted for each setting? Are the reported values means over these runs? Reporting means and standard deviations or standard errors would help establish result stability.
The main tables in the original manuscript report one independent optimization run for each setting. In the supplementary experiments, we conducted three independent runs for both UniMASO and the DeepSeek-V4-Pro + MASPO baseline under the centralized topology, covering all five benchmarks. Both methods used random seeds 42, 1024, and 2048. Other topologies retain the single-run results from the original manuscript. Each run starts from the same initial configuration, reinitializes the beam and trajectory memory, and independently executes the complete optimization process. We then evaluate the final configuration selected in that run using the same test set and evaluation protocol. The table lists the task scores and deployment costs for each run, with the final row reporting the mean ± sample standard deviation across the three seeds.
| Method | Setting | AGIEval Acc. | Cost | GPQA Acc. | Cost | BFCL Acc. | Cost | LiveCodeBench pass@1 | Cost | HotpotQA EM | Cost | Average Score | Cost |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MASPO | Seed 42 | 78.12 | 249.6 | 80.30 | 115.2 | 92.60 | 216.6 | 35.11 | 124.0 | 62.80 | 336.0 | 69.79 | 208.28 |
| MASPO | Seed 1024 | 77.34 | 251.4 | 81.31 | 111.8 | 93.00 | 222.1 | 34.35 | 123.5 | 63.00 | 330.6 | 69.80 | 207.88 |
| MASPO | Seed 2048 | 78.91 | 243.0 | 81.31 | 120.7 | 92.60 | 213.5 | 35.88 | 119.9 | 63.00 | 327.7 | 70.34 | 204.95 |
| MASPO | Mean | 78.12 ± 0.79 | 248.0 ± 4.4 | 80.97 ± 0.58 | 115.9 ± 4.5 | 92.73 ± 0.23 | 217.4 ± 4.4 | 35.11 ± 0.77 | 122.5 ± 2.2 | 62.93 ± 0.12 | 331.4 ± 4.2 | 69.98 ± 0.31 | 207.04 ± 1.82 |
| UniMASO | Seed 42 | 82.42 | 56.9 | 78.28 | 33.0 | 92.40 | 12.2 | 35.11 | 13.3 | 63.80 | 39.3 | 70.40 | 30.92 |
| UniMASO | Seed 1024 | 83.20 | 59.4 | 77.78 | 32.8 | 92.20 | 12.7 | 35.88 | 13.9 | 63.60 | 40.1 | 70.53 | 31.79 |
| UniMASO | Seed 2048 | 81.64 | 58.5 | 77.27 | 33.6 | 92.20 | 11.8 | 35.88 | 12.8 | 63.80 | 39.1 | 70.16 | 31.16 |
| UniMASO | Mean | 82.42 ± 0.78 | 58.3 ± 1.3 | 77.78 ± 0.51 | 33.1 ± 0.4 | 92.27 ± 0.12 | 12.2 ± 0.5 | 35.62 ± 0.44 | 13.3 ± 0.6 | 63.73 ± 0.12 | 39.5 ± 0.5 | 70.36 ± 0.19 | 31.29 ± 0.45 |
Taking the macro-average of the five task scores within each run and then computing statistics across the three seeds gives a mean task score of 70.36 ± 0.19 and a mean deployment cost of 31.29 ± 0.45M for UniMASO. These results show that, under the centralized topology evaluated, the overall task performance and deployment cost obtained with the three random seeds are relatively close. We have added the three-seed stability results to Appendix G.1.
Official Comment to Reviewer YWfx (3/3)
Q3. Can the authors compare UniMASO with model-routing methods such as MasRouter under comparable model pools, topologies, datasets, and search budgets? If comparison is infeasible, which experimental differences prevent it?
Thank you for suggesting the addition of a model-routing baseline. The main experiments in this paper focus on system-configuration optimization before MAS deployment. We therefore prioritized prompt-optimization methods that also optimize before deployment and reuse the resulting configuration on subsequent samples. These methods optimize role prompts under a given model assignment, whereas UniMASO jointly determines each role’s prompt and execution model at the same stage. MAS model-routing methods, represented by MasRouter, focus on per-sample model selection at runtime: the system selects an execution model for a sample only after receiving its specific input. Given this task difference, the original manuscript discussed model-routing methods in related work. Following the reviewer’s suggestion, we added a direct comparison with MasRouter to examine the performance of pre-deployment configurations relative to per-sample routing under the same topology, model pool, and test data.
We use the setting in MasRouter’s original paper, in which its router is attached to an existing multi-agent system. Both methods use the same centralized topology, role definitions, initial prompts, execution-model pool, and data splits, as well as the same task metrics and deployment-cost coefficients, to ensure a fair comparison. During evaluation, MasRouter routes each input, whereas UniMASO uses the prompts and model assignment selected before deployment for all evaluation samples.
| Method | AGIEval Acc. | Cost | GPQA Acc. | Cost | BFCL Acc. | Cost | LiveCodeBench pass@1 | Cost | HotpotQA EM | Cost | Average Score | Cost |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MasRouter | 74.22 | 58.2 | 71.72 | 29.4 | 92.00 | 20.5 | 38.17 | 21.9 | 58.40 | 37.1 | 66.90 | 33.42 |
| UniMASO | 82.42 | 56.9 | 78.28 | 33.0 | 92.40 | 12.2 | 35.11 | 13.3 | 63.80 | 39.3 | 70.40 | 30.92 |
As shown in the table, UniMASO achieves a higher average score than MasRouter under the fixed topology and model pool described above. This indicates that, on the evaluated tasks, a configuration obtained through pre-deployment joint optimization can be reused across samples while maintaining a quality advantage over per-sample routing. We have added this comparison to Appendix G.2.
Q4. How sensitive is the method to model-pool composition? Are there results with other model families or commercial API models, or further analysis delineating the method's scope?
Thank you for raising the question about model-pool composition and the scope of the method. UniMASO jointly optimizes each role’s prompt and execution model at the system level, screening candidates based on task performance, their effects on downstream collaboration, and call cost. The process requires neither access to model weights or gradients nor changes to model architecture. Therefore, any model can be included in the same search procedure as long as it supports the task interface of the corresponding role, its outputs can be evaluated, and its call cost can be accounted for. This design also applies to models available only through commercial APIs.
| Setting | AGIEval Acc. | Cost | GPQA Acc. | Cost | BFCL Acc. | Cost | LiveCodeBench pass@1 | Cost | HotpotQA EM | Cost | Average Score | Cost |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Default model pool | 82.42 | 56.9 | 78.28 | 33.0 | 92.40 | 12.2 | 35.11 | 13.3 | 63.80 | 39.3 | 70.40 | 30.92 |
| Expanded model pool | 83.20 | 60.7 | 78.79 | 37.4 | 92.00 | 12.9 | 35.88 | 17.1 | 64.00 | 40.9 | 70.77 | 33.80 |
We added GLM-5 and Grok-4.7 to the original model pool, which consists of Qwen3.5-4B, Qwen3.5-9B, Qwen3.5-27B, DeepSeek-V4-Flash, and DeepSeek-V4-Pro, and reran UniMASO on all five benchmarks under the centralized topology. As shown in the table, the task scores across the five benchmarks and the average deployment cost remain stable under this expansion. With the two model families included, UniMASO continues to use the same search and evaluation rules, assessing the match between each model and role prompt based on actual execution feedback without changing the optimization method for the internal implementation of new models. We have added the model-pool analysis to Appendix G.3.
Official Comment to Reviewer YWfx (1/3)
Thank you for highlighting the need to clarify these operators. We have added formal definitions of these operators in Appendix H.
(i) Candidate truncation. Candidate a dominates b if Ui(a)≥Ui(b) and Ci(a)≤Ci(b), with at least one strict inequality. We remove dominated candidates first. If more than B nondominated candidates remain,
Select_Bretains the B candidates with the highest cumulative search scores; otherwise, it retains all nondominated candidates. Candidates from subsequent Pareto fronts are not added to fill the beam.(ii) Local-best selection. For a candidate configuration zi′, its utility and cost-adjusted score are
Ui(zi′)=αqiloc(zi′)+βqilook(zi′)+γqiglob(zi′),si(zi′)=Ui(zi′)ηi(zi′)−λ−21,
where ηi(zi′)=Ci(zi′)/Ci(zi) is the measured cost relative to the current configuration. A candidate is eligible for expansion when si(zi′)>0. Its score contributes to the cumulative search-path score, and
BESTselects the retained state with the highest cumulative path score.(iii) Balance term and consensus assignment. Let N be the number of roles, K the size of the global model pool, and ni the number of models feasible for role i. For an assignment M=(m1,…,mN), let ji be the rank of mi in the predefined ascending capability order of feasible models, starting at one. Let Li and Hi count feasible models with costs strictly lower and strictly higher than that of mi, respectively. The balance term is
Ωbal(M)=(1−K1)∑i=1N[max{ni−1,1}ji−1+21(1−(niLi−Hi)2)].
The proposal support is Ai(m)=πi(m)(1−Hˉi), where πi(m) is the fraction of focal views proposing m for role i. The Shannon entropy Hˉi is normalized by log∣Mi∣, where Mi is the set of feasible models for role i; it is set to zero when ∣Mi∣=1. Let F contain complete assignments satisfying model availability, role-sharing, and routing constraints. The consensus assignment maximizes
Mcon=argmaxM∈F[∑i=1NAi(mi)+Ωbal(M)].