acceptodds
Under review as a conference paper at ICLR 2027

Share First, Specialize Later: Temporal Parameter Sharing in Multi-Agent Reinforcement Learning

Abstract

In multi-agent reinforcement learning (MARL) behavioral diversity among agents is crucial when a task requires them to pursue distinct goals, cover different areas, or take on asymmetric roles. Existing methods induce this diversity by modifying the training objective or the policy architecture, often at the cost of extra structural and hyperparameter complexity. We instead induce diversity through training dynamics alone, via temporal parameter sharing: agents first pre-train a single shared, homogeneous policy, then split into independently trained, heterogeneous policies initialized from it. We formalize the choice of how long to pre-train homogeneously as a bilevel optimization problem. Since its non-convex lower level cannot be differentiated through, we solve it with Bayesian optimization over the switch point — an approach we call BOSS (Bayesian Optimization for Shared-to-Split training). Across four multi-agent tasks (navigation, dispersion, predator-prey, and sampling) spanning three MARL algorithms (IPPO, MADDPG, IDDPG), BOSS matches or even exceeds DiCo, a state-of-the-art diversity-control baseline evaluated at its own best published configuration without any change to the policy architecture. The variation in the switch point found by BOSS tracks how symmetric each task’s underlying coordination problem is, positioning the duration of homogeneous pre-training as an interpretable, previously underappreciated diagnostic of task structure rather than a hyperparameter to tune away.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, parameter sharing, bilevel optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rLjnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes temporal parameter sharing: agents initially train a shared policy, then initialize independent policies from it and specialize. BOSS selects the switch point using Gaussian-process Bayesian optimization with three initial evaluations and five Expected Improvement queries. Experiments use VMAS tasks and several actor-critic backbones, and interpret the preferred sharing duration as a diagnostic of task structure.

Critical strengths
  1. The paper offers a useful temporal perspective on parameter sharing. Treating the duration of sharing as an explicit design variable provides a conceptually appealing way to study the tension between shared learning and agent specialization. This perspective highlights that the appropriate sharing regime may change over the course of training, making the training schedule itself an important component of MARL design.

  2. The shared-to-independent transition has a clear learning rationale. Shared pretraining allows experience from multiple agents to contribute to a common initialization, while subsequent independent updates permit agent-specific adaptation. This creates a coherent mechanism for transferring common competencies into specialized policies, rather than requiring each agent to acquire those competencies independently from the beginning.

  3. The improvement over baselines is compelling. Dispersion provides a substantive positive result. Navigation also suggests a useful improvement in convergence speed.

Critical weaknesses
  1. The predator-prey comparison is confounded. Appendix C states that BOSS trains against a continuously learning prey, whereas the DiCo prey is frozen after 3M frames. Predator returns under different opponent-learning processes do not isolate the benefit of temporal sharing. This directly weakens the claim of outperforming DiCo on every main task.

  2. The sharing schedule is restrictive. All agents transition simultaneously and irreversibly from full sharing to fully independent training. This excludes potentially useful intermediate structures, such as retaining shared representations while specializing decision layers, splitting different agent groups at different times, or continuing knowledge sharing after specialization. The simplicity is attractive, but the suitability of this particular schedule requires stronger justification.

  3. Switching is determined by an offline training clock rather than the current learning state. BOSS selects a fixed switch point through complete training evaluations, without conditioning the transition on convergence, gradient conflict, or emerging specialization. The same frame count may correspond to different learning stages across seeds. Consequently, the method does not directly address online adaptation of sharing, and its applicability to a new task entails additional schedule-selection costs.

Questions for authors
  1. Could the global switch point be generalized to agent-specific switching times? Agents with different roles or learning speeds may benefit from different durations of shared training. Would it be feasible to assign each agent a switch time, allowing it to leave the shared policy and specialize while the remaining agents continue sharing?

  2. The caption of Fig. 3 (line 394-395) mentions that FuPS and NoPS are single-seed. Why don't all methods use the same multiple seeds?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer rLjn

Thank you for your thoughtful and constructive review. Several additional experiments are running (see our general response); here we answer the points that do not depend on them.

W1: Predator-prey comparison. We agree that the different prey-training protocols confound this comparison. We are retraining all methods under a single matched protocol and will report the results.

W2 and Q1: Agent-specific switch points. Thank you for this suggestion. Agent-specific switching fits our formulation directly: the scalar switch point s becomes a vector (s₁, …, sₙ), where agent i leaves the shared policy at sᵢ while the others continue sharing. The same black-box search applies, although its cost grows with the number of agents; group-wise switching keeps the dimension low. We use a single global switch point in this paper because it makes the duration of sharing directly comparable across tasks, which is what reveals its strong task dependence (0.5% of the budget in navigation vs. 21% in dispersion). We will add a discussion of agent-specific and group-wise switching to Section 6.

W3: Switching on the learning state. The best switch point differs by two orders of magnitude across tasks: switching almost immediately is best when agents pursue individually distinct goals (navigation), while a longer shared phase pays off when agents must break symmetry over shared targets (dispersion). BOSS selects it per task from the outcomes of full training runs rather than fixing it by hand. We agree, however, that within a single run the switch is not conditioned on that run's state. This is deliberate: our objective is the final return after fine-tuning (Eq. 7), which is observable only at the end of training, whereas learning-state signals such as shared-phase convergence or gradient conflict are proxies whose relation to final performance is not known in advance. A learning state trigger nonetheless fits the same formulation: switching when, for example, the gradient similarity between agents falls below a threshold c turns the upper-level search over s into a one-dimensional search over c. We will add this discussion, together with the selection cost on new tasks, to the Limitations.

Q2: Seeds. MARL training is computationally expensive, and we initially allocated our budget to the multi-seed comparison between DiCo and BOSS. We agree that all methods should be compared on the same seeds, and we are now running FuPS, NoPS, DiCo and BOSS with at least five seeds on all three main tasks. We will post the results when they are ready.

AuthorsOpenReview ↗

Follow-up to Reviewer rLjn

Thank you again for your review. The additional experiments are complete, and the revised PDF marks changes in blue. All four methods now use 10 seeds on all three tasks, with the search seed excluded from the BOSS runs.

W1: Predator-prey confound. All methods are now retrained under one protocol: the prey learns for the first 3M frames and is then frozen. Only the way the predators share parameters differs (Appendix A.3). Final rewards over 10 seeds are BOSS 1.0986, FuPS 0.9949, NoPS 0.6921 and DiCo 0.6398. A late switch (86.5% of the budget) is best. This replaces the submission’s finding that never splitting was best, which came from the mismatched protocol. BOSS learns as fast as FuPS and much faster than DiCo and NoPS, which do not reach 75% of the best final reward within the budget. BOSS is also the most stable across seeds (CV 0.08, against 0.13–0.59 for the other methods; Table 1). Returns are measured against each run’s own prey. That prey is stationary after 3M frames in every run, but before then it co-adapts with the predators it faces.

We have also revised the claim you mentioned. With held-out seeds, BOSS and DiCo are on par on dispersion (0.0334 vs. 0.0335), while BOSS learns faster between about 4M and 10M frames. The paper now says BOSS “matches or exceeds” DiCo: a tie on dispersion, +115% on navigation and +72% on predator-prey.

W2 / Q1: Restrictive schedule and agent-specific switching. The revised paper now addresses this concern in Section 6, paragraph “Restricted switching schedule”. That paragraph explains how agent-specific switching (a vector of switch points s₁, …, sₙ) and group-wise switching fit our formulation, and what each costs. Supporting evidence for the single global switch is in Appendix A.5 (Figure 7): after the switch, BOSS reaches the same final diversity (SND) as NoPS, so the shared phase does not limit agent specialization.

W3: Training clock vs. learning state. The revised paper now addresses this concern in Section 6, paragraphs “Switching on the learning state” and “Cost of schedule selection”. The first explains why we switch on a training clock and how a learning-state trigger, such as a threshold on gradient similarity between agents, fits the same one-dimensional search. The second reports the search cost for a new task (about 29.2 GPU-hours for dispersion, with checkpoint reuse included). On variation across seeds: each switch point was chosen on one seed and holds up on 10 held-out seeds, with narrow seed bands (Figures 4 and 6, Table 1).

Q2: Seeds. All methods now use the same 10 seeds. We removed the single-seed sampling task, so every claim rests on the three 10-seed tasks.

We hope these revisions address your concerns, and we would be grateful if you would consider updating your assessment.

Reviewer_rLjnOpenReview ↗

Thanks for the rebuttal.

Thank you for the additional experiments and revisions. I appreciate the authors' efforts to address my concerns.

However, I believe these limitations are particularly important, as heterogeneous agent roles, different learning speeds, and evolving training dynamics are common in MARL scenarios. The proposed method relies on a single, irreversible, globally synchronized switching point determined by a fixed training budget, which may significantly limit its applicability to more general multi-agent settings.

More importantly, I find the core idea of temporal parameter sharing promising. It is precisely because of this potential that I feel simply relegating these fundamental concerns to the Limitations section is somewhat premature. I believe that further addressing these issues, rather than leaving them entirely to future work, could substantially strengthen the method and potentially turn this into a much more impactful contribution. It would be unfortunate if such a promising idea were not developed further.

Therefore, despite the improvements made during the rebuttal, I am inclined to maintain my current rating. Nevertheless, I strongly encourage the authors to continue pursuing this direction, particularly by developing more flexible and adaptive switching mechanisms.

AuthorsOpenReview ↗

Reply to Reviewer rLjn

Thank you for your encouragement. We appreciate your recognition of temporal parameter sharing as a promising idea, and we will keep developing the method in this direction. We are grateful for your time and constructive feedback throughout the review.

Reviewer_zSFSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the trade-off between parameter sharing and policy specialization in multi-agent reinforcement learning. It proposes temporal parameter sharing, a two-stage procedure in which agents first train a shared policy and then initialize independently trained policies from the shared parameters. The authors formulate the training procedure as a bilevel optimization problem and introduce BOSS, which uses Bayesian optimization to select the switching point. Experiments on three VMAS tasks—dispersion, navigation, and predator-prey—with additional sampling results in the appendix compare BOSS against DiCo and, where available, full-sharing and no-sharing baselines. The reported results suggest that suitable switching schedules can improve performance or convergence speed. The authors further interpret the task-dependent switching points as indicators of the balance between shared behavior and specialization.

Critical strengths
  1. A simple and practically useful approach to balancing sharing and specialization. The proposed two-stage procedure requires neither additional diversity rewards nor specialized policy architectures and can be integrated with both on-policy and off-policy MARL algorithms. This simplicity is consequential because it makes the approach accessible as a training strategy and provides a clear framework for investigating when parameter sharing helps or hinders learning.

  2. Promising empirical evidence that the sharing schedule matters. In dispersion, BOSS achieves a three-seed mean return of 0.0419 compared with 0.0335 for DiCo, an improvement of approximately 25%. The navigation learning curves also suggest a convergence-speed benefit from a brief shared phase. These results provide meaningful evidence that choosing when to specialize can improve learning, making the proposed research direction practically relevant.

Critical weaknesses
  1. The empirical comparisons do not establish a sufficiently reliable advantage. The switch-point searches and available FuPS/NoPS controls use single-seed evaluations, while only the selected BOSS schedule and DiCo receive three-seed evaluation on the main tasks. Furthermore, Appendix C states that DiCo freezes the prey after 3M frames, whereas BOSS trains against a continuously learning prey. This difference confounds the predator-prey comparison. Together, these limitations make it difficult to determine whether the reported improvements reflect a reproducible benefit of temporal sharing, which directly weakens the paper’s central empirical claim.

  2. The added value of Bayesian optimization remains unproven. BOSS is not compared with random search, grid search, or fixed switching schedules under an equivalent computational budget. In navigation, Table 4 shows that the best candidate was already the first point in the random initialization phase; subsequent BO queries did not improve its observed return. Although the experiments suggest that switching time matters, they do not establish that BO selects it more efficiently than simpler alternatives. This distinction is consequential because BO is a central component of the proposed method and its claimed practical contribution.

Questions for authors
  1. Clarification of the predator-prey comparison. Appendix C states that DiCo freezes the prey after 3M frames, while BOSS trains against a continuously learning prey. Are the reported returns measured against each method’s own training opponent or a common evaluation opponent? How should the reported advantage be interpreted given this difference?

  2. Clarification and evidence on statistical robustness. Does the three-seed evaluation include the seed used to select the switching point? Can the authors provide multi-seed comparisons with FuPS and NoPS, particularly for navigation’s convergence-speed advantage and predator-prey’s preference for full sharing? These comparisons would help establish whether the principal conclusions hold beyond the reported individual control runs.

  3. Evidence for the contribution of Bayesian optimization. Under the same eight-run search budget, does BOSS outperform random search or a uniform grid in the quality of the selected schedule? Since navigation’s best candidate was already in the random initial design, what evidence demonstrates a benefit from the subsequent BO queries? Please also report the actual search cost and clarify whether shared-prefix checkpoint reuse was implemented.

  4. Evidence of generality in more challenging environments. The experiments focus on small-scale VMAS tasks. Do the authors have additional results on more challenging benchmarks, such as SMAC or SMACv2, showing that temporal parameter sharing remains beneficial with larger teams and more demanding coordination requirements? Such evidence would materially affect my assessment of the method’s generality. If these results are unavailable, please clarify how broadly the current findings are intended to apply.

6 comments
AuthorsOpenReview ↗

Reply to Reviewer zSFS

Thank you for your detailed and constructive review. Several additional experiments are running (see our general response); here we answer what we can address now.

Q1: Predator-prey comparison. The reported returns are measured against each method's own training opponent. We agree that the different prey-training protocols confound this comparison, and we are retraining all methods under a single matched protocol.

Q2: Statistical robustness. The three evaluation seeds included the seed used for the switch- point search. We plan to refine our evaluation by leaving out the seed that is used for the switch- point search. We are now running FuPS, NoPS, DiCo and BOSS with at least five seeds on dispersion, navigation and predator-prey, which we will report shortly, including for navigation’s convergence speed and for predator-prey.

Q3: Contribution of Bayesian optimization. We are comparing BO with random search and a uniform grid under the same eight-run budget. On search cost: each search used eight full training runs, in total about 29.2 GPU-hours on an NVIDIA L40S for dispersion, the most expensive task. Shared-prefix checkpoint reuse was implemented in the reported experiments, and the reported cost already reflects it. We will correct the sentence in Section 6 that lists this reuse as future work.

Q4: More challenging environments. We do not currently have results on SMAC or SMACv2. Our current evidence covers four VMAS tasks with three algorithms (IPPO, MADDPG and IDDPG), spanning on-policy and off-policy learning. We will state this scope explicitly in the revision and identify larger-scale benchmarks such as SMACv2 as an important next step.

Thank you for clarifying the evaluation protocol, the search cost, and the use of shared-prefix checkpoints. I also appreciate the plan to state the scope of the evidence explicitly, and I would be comfortable with a revision that limits its empirical claims to the VMAS settings evaluated here.

Multi-seed evaluation is standard practice in this field, since final performance in multi-agent reinforcement learning can vary substantially across random seeds. For navigation, I would particularly value learning curves showing variability across fresh training seeds: the reported advantage over NoPS is primarily in learning speed, and the final returns are close. For predator–prey, please specify how the evaluation opponent is chosen under the matched protocol. If each method still learns its own prey policy, evaluating the predators against a common prey policy or against a fixed set of opponents would help separate predator performance from differences in opponent difficulty.

For the eight-run search comparison, I would like to see the returns of the schedules selected by BO, random search, and uniform grid search, evaluated on the same seeds held out from schedule selection. This would help assess both the benefit of choosing when to specialize and the added value of using BO to make that choice.

More importantly, VMAS is a simple grid-world environment with a small number of agents. Without results on more demanding benchmarks such as SMAC or SMACv2, it is difficult to establish that the method is effective for MARL more broadly.

I would be happy to revisit my assessment once these results are available.

AuthorsOpenReview ↗

Reply to Reviewer zSFS

Thank you for the detailed follow-up and for being open to revisiting your assessment. We address your points in the following.

1. Navigation: variability across fresh seeds.

This is now in Figure 4: all four methods over 10 training seeds with ±1 std bands, with the search seed excluded from the BOSS runs. As you note, final returns for BOSS and NoPS are close (0.0303 vs. 0.0302). The differences are in the curves:

  • BOSS is ahead of NoPS early and stays ahead until the two merge at about 6M frames.
  • BOSS's seed-to-seed variation is much smaller than NoPS's and DiCo's, especially early on.
  • Table 1 reports frames to 50% and 75% of final reward, and the coefficient of variation across seeds.

We now describe the navigation benefit as faster and steadier learning, not a higher final return.

2. Predator-prey: evaluation opponent.

  • Under the matched protocol (Appendix A.3), every method trains the predators against a prey that learns for the first 3M frames and is then frozen. Only the predators' parameter sharing differs.
  • The reported return is the predators' reward against the prey of the same run, not against a single common prey policy.
  • After 3M frames the opponent is stationary in every run. Before that, the prey co-adapts with whichever predator scheme it faces, so opponent difficulty may still differ across methods.

We plan to evaluate the final predators of all methods, over 10 seeds, against a common fixed set of frozen prey checkpoints, and add it to the paper. Until then, the predator-prey result should be read as holding under the matched protocol, not as a common-opponent comparison.

3. BO vs. grid vs. random search: selected schedules on held-out seeds. Thank you for this suggestion. We understand the goal as separating two questions: whether choosing when to specialize helps, and whether BO adds value in making that choice. We believe the newly added Appendix A.6 (Figure 8) and Section 5.2 (Figure 4) already speak to both, though not in exactly the form requested.

Does choosing when to specialize matter?

  • Figure 8 (right) plots every evaluated switch point on navigation against its final return, pooled over 5 searches per method. Return falls from about 0.030 for switch points in roughly the first 0.8M frames to about 0.007 for the latest ones, so the choice of switch point changes the outcome several-fold.
  • Figure 4 evaluates the selected switch point on 10 seeds that exclude the search seed, against both extremes: BOSS reaches 0.0303, versus 0.0054 for never splitting (FuPS) and 0.0302 for splitting at frame 0 (NoPS), with faster and steadier learning than NoPS.

Does BO add value over grid and random search under the same 8-run budget?

  • Figure 8 (left) shows that BO's mean return over its evaluated points is higher than grid search on 5 of 5 seeds and than random search on 3 of 5. Figure 8 (right) shows that 12 of its 40 points fall in the high-reward region, against 7 for random search and 5 for grid search.
  • On navigation, grid search reaches the high-reward region only because its first point sits at the start of the range, and random search would miss it in about 43% of 8-point searches (a back-of-envelope estimate). Navigation is thus a favorable case for the baselines, as its optimum lies at the start of the range, so we do not claim that BO finds a better single point here.

The advantage of BO is how effectively it explores the high reward region where the optimum is more likely to reside. It concentrates its queries in the high-reward region (12 of 40 points, against 7 and 5) while still exploring unseen regions. This matters because the best switch point varies widely across tasks (0.5% of the budget on navigation, 21% on dispersion, 86.5% on predator-prey). BO adapts its queries to each task, whereas a grid fixes its resolution and positions in advance and random search leaves the outcome to chance.

If our newly added experiments do not fully address your concern, we are happy to run further experiments as you suggest, such as re-running the schedule selected by each search on held-out seeds.

4. Scope and more demanding benchmarks. Thank you for suggesting SMAC and SMACv2. We consider them an important next step for testing our approach. We agree that our tasks use small teams (at most four agents) and that our experiments do not establish that temporal parameter sharing remains beneficial with larger teams or more demanding coordination requirements, as in SMAC or SMACv2. We do not have SMAC/SMACv2 results and cannot obtain reliable ones within the discussion period. As you suggested, we restrict our claims to the VMAS settings evaluated. The "Scope of the evidence" paragraph in Section 6 now states this, and names evaluating BOSS on SMAC and SMACv2 as an important next step, which we intend to pursue.

Thank you again for the review. We are happy to answer any further questions.

Reviewer_4EstOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a two-stage MARL framework that first learns common behaviors through parameter sharing and then independently fine-tunes agent-specific policies. Bayesian optimization is used to select the switching point, encouraging agent specialization without auxiliary diversity rewards or architectural modifications.

Critical strengths
  1. Novel perspective on parameter sharing. The paper reframes the homogeneity–heterogeneity trade-off in MARL as a temporal decision, offering a new perspective on balancing shared learning and agent specialization.
  2. Simplicity and portability. The framework requires neither architectural modifications nor auxiliary rewards and can be readily integrated into MARL algorithms supporting shared and independent parameter training.
Critical weaknesses
  1. Limited statistical reliability. MARL training and switch-point selection are inherently stochastic, yet the main comparisons use only three seeds, with some experiments relying on a single seed. The substantial variation observed in Predator-Prey further weakens the reliability of the reported results.
  2. Insufficient evidence of agent specialization. Although the method aims to promote agent specialization, experiments primarily report task returns. Without quantitative measures of policy divergence, role differentiation, or trajectory diversity, the claimed specialization remains insufficiently validated.
  3. Insufficient validation of Bayesian optimization. The paper does not compare Bayesian optimization against simpler alternatives, such as random or grid search, under matched computational budgets. Thus, the relative efficiency and necessity of Bayesian optimization remain unclear.
Questions for authors
  1. Could the authors increase the number of random seeds (e.g., to at least five) and report mean performance with standard deviations or confidence intervals, particularly for switch-point selection and the Predator-Prey results?
  2. Could the authors provide quantitative analyses of behavioral diversity, such as policy divergence, trajectory differences, or role specialization, to verify whether the performance improvements arise from effective agent specialization?
  3. Could the authors compare Bayesian optimization with random search and grid search under identical training budgets to establish its relative search efficiency and benefits?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer 4Est

Thank you for your helpful feedback. All three of your questions are being addressed by experiments now in progress (see our general response).

Q1: Seeds. We are running FuPS, NoPS, DiCo and BOSS with at least five seeds on all three main tasks, and will report shortly, including for predator-prey and for the switch-point selection.

Q2: Behavioral diversity. We are measuring System Neural Diversity (SND; Bettini et al., 2024a), the average pairwise Wasserstein distance between agents' action distributions, for all methods over training. This shows directly whether, and when, agents diverge after the switch.

Q3: BO vs. random and grid search. We are comparing BO with random search and a uniform grid under the same eight-run budget. We will post the results here as soon as they are ready.

Thank you for the response and additional experiments. However, I believe the current response does not fully address my concerns. In particular, the advantage of Bayesian optimization (BO) remains insufficiently demonstrated. As the authors acknowledge, all three search methods can identify good solutions, and the results do not establish a clear and consistent advantage of BO over simpler alternatives across tasks.

Additionally, I share Reviewer zSFS's concern regarding the lack of experiments on SMAC/SMACv2. SMAC is a well-established and widely used benchmark in MARL, and without evaluation on such benchmarks, it remains difficult to assess the generalizability of the proposed method to more challenging multi-agent scenarios.

Therefore, I have decided to maintain my original rating.

AuthorsOpenReview ↗

Reply to Reviewer 4Est

We hope the 10-seed results and the SND analysis address your first two concerns, on statistical reliability and agent specialization.

1. Advantage of Bayesian optimization. You are right that on navigation all three searches can identify a good solution. We would like to explain why we do not use "which search finds the best point on one task" as our measure of benefit.

Our measurement standard. We consider BO advantageous because it is adaptive across tasks, not because it clearly beats the other searches on any single task. The best switch point can lie at the beginning of the range (0.5% of the budget on navigation), in the middle (21% on dispersion), or near the end (86.5% on predator-prey). A search procedure has to work whichever case it meets. We therefore measure how much of a limited search budget each method spends in the high-reward region, where the optimum is most likely to lie. We think this is more reliable than asking whether one run on one task happens to land on the optimum, which also depends on luck.

Bayesian optimization. On navigation (Appendix A.6, Figure 8), BO places 12 of 40 evaluated points in the high-reward region, against 7 for random search and 5 for grid search, while still exploring unseen regions. This is not luck but the design of BO: the surrogate model and the Expected Improvement criterion direct each new query toward regions that look promising, so BO concentrates its queries where the optimum is more likely to reside and adapts this to the task.

Grid search. A grid uses the same strategy on every task. Only 5 of its 40 points, one per 8-run search, fall in the high-reward region. It finds a good point on navigation only because the optimum happens to sit at the start of the range, and this luck hides both its rigidity and its low coverage of the high-reward region. If the optimum were in the middle or at the end of the range, as in dispersion and predator-prey, a grid of this resolution could easily miss it.

Random search. It leaves the outcome to chance. The high-reward region covers about 10% of the range, so an 8-point search misses it entirely about 43% of the time (a back-of-envelope estimate).

We do not have time to run this comparison on the other tasks within the discussion period. We plan to do so in future work, on dispersion and predator-prey, where the optimum is not at the start of the range. Based on the design of BO and on the results above, we expect BO to explore the high-reward region more than the other two methods on these tasks as well, and we will report the outcome whichever way it turns out.

2. SMAC / SMACv2. We acknowledge that testing our approach on SMAC and SMACv2 is an important and valuable next step, and we thank you for the suggestion. At the same time, using VMAS alone does not undercut the validity of our comparison with DiCo (Bettini et al., ICML 2024), the state of the art in diversity control for MARL and our main baseline, which relies on VMAS alone for its evaluation. Our work targets the same setting as DiCo: controlling behavioral diversity in small teams of up to five agents, evaluated on VMAS. DiCo's published results and diversity targets are tuned for VMAS, so evaluating both methods there allows a direct comparison against DiCo at its best published settings. A comparison on a different simulator would require re-implementing and re-tuning DiCo outside the setting it was designed and validated for. The revised paper states this scope explicitly and names larger-scale benchmarks such as SMAC and SMACv2 as an important next step (Section 6, "Scope of the evidence").

Thank you again for the review.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response: Summary of additional experiments in progress

We thank all reviewers for their constructive feedback. Several concerns are shared across reviews, so we summarize here the experiments now running. We will post the results here and in the revised PDF before the end of the discussion period.

1. Statistical reliability (all reviewers): FuPS, NoPS, DiCo and BOSS with at least five seeds on dispersion, navigation and predator-prey.

2. Predator-prey comparison (zSFS, rLjn): all methods retrained under a single, matched prey-training protocol.

3. Value of Bayesian optimization (4Est, zSFS): BO compared with random search and an uniform grid under the same eight-run budget.

4. Agent specialization (4Est): System Neural Diversity (SND) of each method over training, before and after the switch.

AuthorsOpenReview ↗

Status update/Timeline

  • About 120 runs are going on GPU batch: 4 methods × 10 seeds on each of 3 tasks.
  • The BO vs. random/grid search comparison will start soon.
  • I expect results in 2–3 days and will post the second round then.
AuthorsOpenReview ↗

1. Statistical Reliability (all reviewers): Navigation (Part 1)

We thank all reviewers for suggesting more seeds for a more reliable evaluation. We reran the navigation experiment with 10 seeds for all methods (BOSS, DiCo, NoPS, FuPS), without changing any settings (same switch point for BOSS, same target SND for DiCo, and so on). For BOSS, we excluded the seed used for the switch-point search. The results are in the updated Figure 4 (Section 5.2).

Figure 4 shows that BOSS and NoPS reach almost the same final reward (0.0303 vs. 0.0302). This is about twice DiCo's (0.0141) and more than five times FuPS's (0.0054). BOSS learns slightly faster than NoPS early in training, and its variation across seeds is much smaller than that of NoPS and DiCo, especially early on. Note that the bands for BOSS and FuPS are thin because their learning is very stable across seeds. BOSS thus exhibits the combination of NoPS’s high final reward and FuPS’s learning stability across seeds.

AuthorsOpenReview ↗

General response (update): additional experiments completed

We thank you all for your constructive feedback and for your patience while we ran these experiments. All four items from our earlier general response are now complete. The results are in the revised PDF, with new or changed text in blue. Your remaining individual points are answered in the replies to each of you.

Results for the four planned items

1. Statistical reliability (all reviewers). FuPS, NoPS, DiCo and BOSS now each use 10 seeds on dispersion, navigation and predator-prey. The seed used for the BO search is excluded from the BOSS runs.

TaskBOSSDiCoNoPSFuPS
Navigation0.03030.01410.03020.0054
Dispersion0.03340.03350.03100.0107
Predator-prey1.09860.63980.69210.9949

Mean final reward over 10 seeds. BOSS matches or exceeds DiCo on every task: a tie on dispersion, +115% on navigation and +72% on predator-prey. We have revised the text to say "matches or exceeds". On dispersion, the original submission reported a 25% gain over DiCo from 3 seeds that included the search seed; with 10 held-out seeds, the two are on par, while BOSS learns faster (Table 1).

Revised or added: dispersion in Section 5.1 (Figure 3), navigation in Section 5.2 (Figure 4), predator-prey in Appendix A.3 (Figure 6), and the cross-task summary in Section 5.3 (Table 1).

2. Predator-prey comparison (zSFS, rLjn). All methods now run under one protocol: each method changes only how the predators share parameters, and the prey is trained in the same way in all runs (it learns for the first 3M frames, then is frozen). Under this protocol, a late switch (86.5% of the budget) is best, 10% above full sharing. This replaces the submission's finding that never splitting was best, which came from the mismatched protocol.

Revised or added: Appendix A.3 (Figure 6 and the new BO search table, Table 2), and the updated summary in Section 5.3.

3. Value of Bayesian optimization (4Est, zSFS). We compared BO with grid search and random search under the same eight-run budget, on navigation with 5 seeds per method (40 evaluated points each).

  • BO places 12 of its 40 points in the high-reward region, against 7 for random search and 5 for grid search. Its mean return over the evaluated points is higher than grid search on 5 of 5 seeds and than random search on 3 of 5.
  • Grid search is rigid: its spacing (about 1.2M frames) is wider than the high-reward region (about 0.8M), so it can place at most one point there. Random search is unreliable: since the high-reward region covers about 10% of the range, an eight-point search would miss it entirely about 43% of the time.
  • On navigation all three searches can find a good point, so we read the result as better use of a fixed budget. The switch points used in the main results were chosen on one seed and evaluated on 10 others.

Revised or added: Appendix A.6 (Figure 8). The BO search tables for all tasks are now in query order (Tables 2, 3 and 4).

4. Agent specialization (4Est). We measured System Neural Diversity (SND) on navigation over 10 seeds.

  • BOSS's SND is zero at the switch, then rises to about 3, in step with its reward gain.
  • NoPS follows a similar path. DiCo with target 1.2 holds its target. DiCo with target 3 is unstable and ends with negative reward.
  • SND measures policy divergence rather than roles, so the evidence is correlational.

Revised or added: Appendix A.5 (Figure 7). The DiCo run with target 3 also appears in Figure 4 (Section 5.2).

Additional results and changes

  • Learning speed and stability. The cross-task table now compares all four methods on learning speed (frames to reach 50% and 75% of the best final reward) and stability across seeds (coefficient of variation of the final reward). BOSS learns faster than DiCo and NoPS on predator-prey and dispersion, ties NoPS on navigation at 50% and is slightly ahead at 75%, and is much steadier than DiCo on navigation and predator-prey. On dispersion all trained methods vary similarly, so we make no stability claim there. Revised or added: Section 5.3, Table 1.
  • Sampling task removed. Its results were single-seed and lacked FuPS and NoPS controls, and we ran out of time to bring it to the 10-seed standard within the limited discussion period. Our claims now rest on the three tasks evaluated over 10 seeds. Revised or added: Section 5 (setup) and Appendix A.2 (task list).
  • New section. Section 6 (Limitations and Future Work) states the scope of our evidence and discusses the cost of schedule selection, agent-specific switching, and learning-state triggers. Revised or added: Section 6.

Thank you again for helping us strengthen the paper. We hope these additions address the concerns you shared, and we are glad to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.