- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes temporal parameter sharing: agents initially train a shared policy, then initialize independent policies from it and specialize. BOSS selects the switch point using Gaussian-process Bayesian optimization with three initial evaluations and five Expected Improvement queries. Experiments use VMAS tasks and several actor-critic backbones, and interpret the preferred sharing duration as a diagnostic of task structure.
Critical strengths
-
The paper offers a useful temporal perspective on parameter sharing. Treating the duration of sharing as an explicit design variable provides a conceptually appealing way to study the tension between shared learning and agent specialization. This perspective highlights that the appropriate sharing regime may change over the course of training, making the training schedule itself an important component of MARL design.
-
The shared-to-independent transition has a clear learning rationale. Shared pretraining allows experience from multiple agents to contribute to a common initialization, while subsequent independent updates permit agent-specific adaptation. This creates a coherent mechanism for transferring common competencies into specialized policies, rather than requiring each agent to acquire those competencies independently from the beginning.
-
The improvement over baselines is compelling. Dispersion provides a substantive positive result. Navigation also suggests a useful improvement in convergence speed.
Critical weaknesses
-
The predator-prey comparison is confounded. Appendix C states that BOSS trains against a continuously learning prey, whereas the DiCo prey is frozen after 3M frames. Predator returns under different opponent-learning processes do not isolate the benefit of temporal sharing. This directly weakens the claim of outperforming DiCo on every main task.
-
The sharing schedule is restrictive. All agents transition simultaneously and irreversibly from full sharing to fully independent training. This excludes potentially useful intermediate structures, such as retaining shared representations while specializing decision layers, splitting different agent groups at different times, or continuing knowledge sharing after specialization. The simplicity is attractive, but the suitability of this particular schedule requires stronger justification.
-
Switching is determined by an offline training clock rather than the current learning state. BOSS selects a fixed switch point through complete training evaluations, without conditioning the transition on convergence, gradient conflict, or emerging specialization. The same frame count may correspond to different learning stages across seeds. Consequently, the method does not directly address online adaptation of sharing, and its applicability to a new task entails additional schedule-selection costs.
Questions for authors
-
Could the global switch point be generalized to agent-specific switching times? Agents with different roles or learning speeds may benefit from different durations of shared training. Would it be feasible to assign each agent a switch time, allowing it to leave the shared policy and specialize while the remaining agents continue sharing?
-
The caption of Fig. 3 (line 394-395) mentions that FuPS and NoPS are single-seed. Why don't all methods use the same multiple seeds?
4 comments
Follow-up to Reviewer rLjn
Thank you again for your review. The additional experiments are complete, and the revised PDF marks changes in blue. All four methods now use 10 seeds on all three tasks, with the search seed excluded from the BOSS runs.
W1: Predator-prey confound. All methods are now retrained under one protocol: the prey learns for the first 3M frames and is then frozen. Only the way the predators share parameters differs (Appendix A.3). Final rewards over 10 seeds are BOSS 1.0986, FuPS 0.9949, NoPS 0.6921 and DiCo 0.6398. A late switch (86.5% of the budget) is best. This replaces the submission’s finding that never splitting was best, which came from the mismatched protocol. BOSS learns as fast as FuPS and much faster than DiCo and NoPS, which do not reach 75% of the best final reward within the budget. BOSS is also the most stable across seeds (CV 0.08, against 0.13–0.59 for the other methods; Table 1). Returns are measured against each run’s own prey. That prey is stationary after 3M frames in every run, but before then it co-adapts with the predators it faces.
We have also revised the claim you mentioned. With held-out seeds, BOSS and DiCo are on par on dispersion (0.0334 vs. 0.0335), while BOSS learns faster between about 4M and 10M frames. The paper now says BOSS “matches or exceeds” DiCo: a tie on dispersion, +115% on navigation and +72% on predator-prey.
W2 / Q1: Restrictive schedule and agent-specific switching. The revised paper now addresses this concern in Section 6, paragraph “Restricted switching schedule”. That paragraph explains how agent-specific switching (a vector of switch points s₁, …, sₙ) and group-wise switching fit our formulation, and what each costs. Supporting evidence for the single global switch is in Appendix A.5 (Figure 7): after the switch, BOSS reaches the same final diversity (SND) as NoPS, so the shared phase does not limit agent specialization.
W3: Training clock vs. learning state. The revised paper now addresses this concern in Section 6, paragraphs “Switching on the learning state” and “Cost of schedule selection”. The first explains why we switch on a training clock and how a learning-state trigger, such as a threshold on gradient similarity between agents, fits the same one-dimensional search. The second reports the search cost for a new task (about 29.2 GPU-hours for dispersion, with checkpoint reuse included). On variation across seeds: each switch point was chosen on one seed and holds up on 10 held-out seeds, with narrow seed bands (Figures 4 and 6, Table 1).
Q2: Seeds. All methods now use the same 10 seeds. We removed the single-seed sampling task, so every claim rests on the three 10-seed tasks.
We hope these revisions address your concerns, and we would be grateful if you would consider updating your assessment.
Thanks for the rebuttal.
Thank you for the additional experiments and revisions. I appreciate the authors' efforts to address my concerns.
However, I believe these limitations are particularly important, as heterogeneous agent roles, different learning speeds, and evolving training dynamics are common in MARL scenarios. The proposed method relies on a single, irreversible, globally synchronized switching point determined by a fixed training budget, which may significantly limit its applicability to more general multi-agent settings.
More importantly, I find the core idea of temporal parameter sharing promising. It is precisely because of this potential that I feel simply relegating these fundamental concerns to the Limitations section is somewhat premature. I believe that further addressing these issues, rather than leaving them entirely to future work, could substantially strengthen the method and potentially turn this into a much more impactful contribution. It would be unfortunate if such a promising idea were not developed further.
Therefore, despite the improvements made during the rebuttal, I am inclined to maintain my current rating. Nevertheless, I strongly encourage the authors to continue pursuing this direction, particularly by developing more flexible and adaptive switching mechanisms.
Reply to Reviewer rLjn
Thank you for your encouragement. We appreciate your recognition of temporal parameter sharing as a promising idea, and we will keep developing the method in this direction. We are grateful for your time and constructive feedback throughout the review.
Reply to Reviewer rLjn
Thank you for your thoughtful and constructive review. Several additional experiments are running (see our general response); here we answer the points that do not depend on them.
W1: Predator-prey comparison. We agree that the different prey-training protocols confound this comparison. We are retraining all methods under a single matched protocol and will report the results.
W2 and Q1: Agent-specific switch points. Thank you for this suggestion. Agent-specific switching fits our formulation directly: the scalar switch point s becomes a vector (s₁, …, sₙ), where agent i leaves the shared policy at sᵢ while the others continue sharing. The same black-box search applies, although its cost grows with the number of agents; group-wise switching keeps the dimension low. We use a single global switch point in this paper because it makes the duration of sharing directly comparable across tasks, which is what reveals its strong task dependence (0.5% of the budget in navigation vs. 21% in dispersion). We will add a discussion of agent-specific and group-wise switching to Section 6.
W3: Switching on the learning state. The best switch point differs by two orders of magnitude across tasks: switching almost immediately is best when agents pursue individually distinct goals (navigation), while a longer shared phase pays off when agents must break symmetry over shared targets (dispersion). BOSS selects it per task from the outcomes of full training runs rather than fixing it by hand. We agree, however, that within a single run the switch is not conditioned on that run's state. This is deliberate: our objective is the final return after fine-tuning (Eq. 7), which is observable only at the end of training, whereas learning-state signals such as shared-phase convergence or gradient conflict are proxies whose relation to final performance is not known in advance. A learning state trigger nonetheless fits the same formulation: switching when, for example, the gradient similarity between agents falls below a threshold c turns the upper-level search over s into a one-dimensional search over c. We will add this discussion, together with the selection cost on new tasks, to the Limitations.
Q2: Seeds. MARL training is computationally expensive, and we initially allocated our budget to the multi-seed comparison between DiCo and BOSS. We agree that all methods should be compared on the same seeds, and we are now running FuPS, NoPS, DiCo and BOSS with at least five seeds on all three main tasks. We will post the results when they are ready.