- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ALLOTS, an adaptive multi-agent reinforcement learning (MARL) framework for tasks with variable observation and action spaces induced by dynamically changing entity sets. Specifically, ALLOTS introduces a permutation-equivariant set-to-set actor that generates entity-aligned actions for varying numbers of entities. It further develops a set-structured critic that combines entity-level value contributions with a global value term and performs TD bootstrapping using the next-step active entity set. Additionally, an entity-level advantage weighting mechanism is introduced to guide actor updates. Experiments on FOgym, Modified MPE, and MultiDrone demonstrate that ALLOTS generally outperforms several MARL baselines under different entity churn intensities.
Critical strengths
-
This paper addresses variable observation and action spaces caused by dynamic entity participation, which is an important challenge in practical multi-agent systems but is not directly supported by many conventional MARL algorithms.
-
The permutation-equivariant set-to-set actor provides a straightforward way to accommodate varying numbers of entities while maintaining entity-action correspondence, without requiring architecture modifications when the entity set changes.
-
The set-structured critic explicitly encodes entity-level state-action pairs and combines entity-level and global value representations, providing a flexible architecture for value estimation under changing entity configurations.
-
The proposed method achieves promising results across three application domains and different entity churn intensities. The ablation experiments also provide evidence of the effectiveness of the proposed components.
Critical weaknesses
-
The main architecture largely builds upon existing permutation-equivariant networks, Deep Sets, and conventional actor-critic learning. The proposed TD bootstrapping mechanism also follows the standard Bellman update using the next state. Consequently, the methodological contributions beyond integrating existing techniques for variable-sized entity sets are incremental.
-
The proposed entity-level advantage is calculated by subtracting the mean entity value contribution from each entity's value score. However, this baseline cancels out under softmax normalization, making the resulting weights equivalent to directly applying softmax to the entity-level values. Moreover, a larger entity value does not necessarily indicate a greater contribution to the global return or a greater need for policy improvement. If you employ the counterfactual inference here, the reviewer may agree with the argument of contribution.
-
The centralized critic combines entity-level contributions with a global value term, whereas the actor objective only maximizes a weighted sum of entity-level values. Consequently, the actor update does not directly incorporate the action gradients of the global value term, which may limit its ability to capture complex inter-entity coordination.
-
Although the paper emphasizes adaptation without retraining, the experiments mainly evaluate performance under different entity churn intensities. It remains unclear whether ALLOTS generalizes to previously unseen entity counts, entity configurations, or churn distributions without additional policy updates.
-
Most baselines are conventional MARL algorithms adapted through padding and masking. Comparisons with stronger set-based or permutation-equivariant actor-critic architectures (e.g., [1] and [2]) would help determine whether the observed improvements arise from the proposed learning mechanisms or primarily from the set-structured network architecture.
References:
[1] PIC: permutation invariant critic for multi-agent deep reinforcement learning[C]//Conference on robot learning. PMLR, 2020: 590-602.
[2] Boosting multiagent reinforcement learning via permutation invariant and permutation equivariant networks[C]//The eleventh international conference on learning representations. 2022.
Questions for authors
-
Please clarify the key methodological differences between ALLOTS and directly applying existing Deep Sets or permutation-equivariant architectures to the actor and critic. (See Weakness 1)
-
Why are larger entity-level value scores interpreted as greater contributions to the global return? Is there any theoretical or empirical evidence supporting the relationship between the learned entity-level values and actual entity contributions? (See Weakness 2)
-
Since the global value term is intended to capture inter-entity interactions, why does the actor objective exclude this term? Would directly maximizing the centralized Q-value provide more effective policy gradients, particularly in tasks requiring strong coordination? (See Weakness 3)
-
Can ALLOTS generalize to previously unseen entity counts or configurations without further training? (See Weakness 4) For example, how would a policy trained with a limited range of entity counts perform when evaluated on substantially larger or smaller entity sets?
-
Can the authors include stronger set-based baselines with comparable network capacities (such as PIC, DPN, and HPN )?
2 comments
Reply to the authors' response
Thank you for the authors' response. However, I still find the novelty of this work limited compared to existing studies. Therefore, I will maintain my original score.
We thank the reviewer for the constructive feedback.
Resp. to W1 & Q1: ALLOTS addresses intra-agent dimensional variability: each agent's joint continuous action space changes with its entity set, not merely its observation size or the agent population. A permutation-equivariant set-to-set actor preserves entity–action alignment. Motivated by localized entity changes with potentially global consequences, ALLOTS constructs dual value pathways from paired state–action representations. Unlike conventional set-based actor–critic methods that maximize the full scalar Q, ALLOTS learns the full portfolio Q to accommodate non-additive interactions while adaptively aggregating locally structured entity-level gradients for policy optimization. Permutation symmetry alone does not ensure TD consistency across changing entity sets; ALLOTS bootstraps on the realized next active set without altering the Bellman equation. Despite retaining the set-to-set actor, all ablations yield lower mean returns across all 12 configurations (Tables 2, 7–8), supporting contributions beyond set-based representation.
Resp. to W2 & Q2:
We agree that softmax((q−b)/τw)=softmax(q/τw) for the entity-score vector q, as noted in Sec. 4.2. Mean-centering expresses relative scores without changing the weights.
Entity scores are learned under the full portfolio-level TD objective. Rather than uniformly averaging entity-level gradients, ALLOTS assigns higher-scoring entities' gradients larger relative weights in the shared policy update, enabling value-guided gradient aggregation over changing entity sets. The scores are not identifiable marginal contributions or measures of improvement potential; we will clarify their interpretation.
ALLOTS exceeds uniform weighting (No-EC) in mean return across all 12 configurations. Table 10 shows that the best intermediate temperature yields higher mean training returns than both tested extremes in each FOgym testbed, supporting controlled weighting concentration. Counterfactual inference offers a complementary approach to estimating reference-dependent contributions, whereas our mechanism focuses on policy optimization.
Resp. to W3 & Q3: Excluding the global term from Eq. (15) is deliberate. Adding its action gradients can introduce direct cross-entity gradient coupling through the pooled state–action representation. Given the localized nature of entity changes, ALLOTS restricts direct policy optimization to entity-level pathways to preserve locally structured action gradients (at fixed observations), while the global term participates in full-Q TD learning and jointly trains the shared critic encoder. The actor also remains portfolio-conditioned for coordination. ALLOTS achieves the highest mean returns in all six cooperative coverage configurations (Tables 5–6). Direct full-Q optimization could benefit tasks requiring stronger coordination, but its relative effectiveness with an otherwise identical critic remains untested. We will clarify that Eq. (15) is a deliberate surrogate rather than the full-Q policy gradient.
Resp. to W4 & Q4: ALLOTS adapts to entity-set changes through policy inference rather than parameter updates. Its shared encoder, permutation-invariant pooling, and shared action head (Eqs. 2–5) have cardinality-independent parameters, enabling aligned actions for larger or smaller entity sets given compatible entity interfaces and environment capacity. Our evaluation demonstrates adaptation to repeated stochastic churn events without retraining during execution. Since policies are trained and evaluated under their respective scale and churn settings, these results support within-distribution adaptation and applicability across scales and domains, not zero-shot performance generalization to unseen cardinality ranges, entity configurations, or churn distributions. We will clarify this distinction.
Resp. to W5 & Q5:
Beyond padding-based baselines, MAAC, and AGILE, we include SQDDPG-Set, MADDPG-Set, and MAPPO-Set, which incorporate active-set pooling while retaining their original actor–critic structures (Appendix C.1.3–C.1.4). ALLOTS outperforms all three in mean return across all six configurations (Tables 5–6). No-EC isolates the benefit of adaptive weighting under the same network architecture.
PIC employs a permutation-invariant graph critic, while DPN/HPN introduce entity-wise PI/PE network modules. In MPE, these methods focus on the joint critic of MADDPG. In contrast, ALLOTS addresses simultaneous continuous control over dynamically changing entity sets, with explicit entity–action alignment, active-set-consistent TD learning, and locally structured policy gradients. Unlike these permutation-aware architectures, ALLOTS explicitly exposes entity-level value signals for adaptive gradient aggregation. Direct capacity-matched comparisons with PIC, DPN, and HPN would further strengthen the evaluation.