acceptodds
Desk rejected at ICLR 2027

ALLOTS: Adaptive Multi-Agent Reinforcement Learning with Variable Observation and Action Spaces

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Many real-world applications involve dynamic entities, e.g., heat pumps and electric vehicles with flexible electricity consumption, which, in multi-agent reinforcement learning (MARL), lead to variable observation and action spaces. Existing MARL methods handle variability in observation or action spaces in isolation, but cannot directly generate entity-aligned action sets that adapt to a changing entity set. To address this issue, we propose the daptive Muti-Agent Reinforcement earning with Variable bservation and acion paces () method that simultaneously handles variable observation and action spaces. First, we present a permutation-equivariant set-to-set actor that maps variable observations to entity-aligned variable actions, preserving valid control semantics as the entity set evolves. Second, we present a set-structured critic that encodes entity-level state–action pairs as joint tokens and bootstraps value targets from the next-step active set, enabling stable learning and efficient adaptation without retraining. Experiments on energy system, multi-agent particle, and multi-drone environments show that ALLOTS outperforms state-of-the-art methods in dealing with variable observation and action spaces. Code is available at https://anonymous.4open.science/r/ALLOTS-EFB4.

Primary area:
reinforcement learning
Keywords:
Multi-agent Reinforcement Learning, Variable Observation and Action Spaces, Dynamic Entities

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FNWnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ALLOTS, an adaptive multi-agent reinforcement learning (MARL) framework for tasks with variable observation and action spaces induced by dynamically changing entity sets. Specifically, ALLOTS introduces a permutation-equivariant set-to-set actor that generates entity-aligned actions for varying numbers of entities. It further develops a set-structured critic that combines entity-level value contributions with a global value term and performs TD bootstrapping using the next-step active entity set. Additionally, an entity-level advantage weighting mechanism is introduced to guide actor updates. Experiments on FOgym, Modified MPE, and MultiDrone demonstrate that ALLOTS generally outperforms several MARL baselines under different entity churn intensities.

Critical strengths
  1. This paper addresses variable observation and action spaces caused by dynamic entity participation, which is an important challenge in practical multi-agent systems but is not directly supported by many conventional MARL algorithms.

  2. The permutation-equivariant set-to-set actor provides a straightforward way to accommodate varying numbers of entities while maintaining entity-action correspondence, without requiring architecture modifications when the entity set changes.

  3. The set-structured critic explicitly encodes entity-level state-action pairs and combines entity-level and global value representations, providing a flexible architecture for value estimation under changing entity configurations.

  4. The proposed method achieves promising results across three application domains and different entity churn intensities. The ablation experiments also provide evidence of the effectiveness of the proposed components.

Critical weaknesses
  1. The main architecture largely builds upon existing permutation-equivariant networks, Deep Sets, and conventional actor-critic learning. The proposed TD bootstrapping mechanism also follows the standard Bellman update using the next state. Consequently, the methodological contributions beyond integrating existing techniques for variable-sized entity sets are incremental.

  2. The proposed entity-level advantage is calculated by subtracting the mean entity value contribution from each entity's value score. However, this baseline cancels out under softmax normalization, making the resulting weights equivalent to directly applying softmax to the entity-level values. Moreover, a larger entity value does not necessarily indicate a greater contribution to the global return or a greater need for policy improvement. If you employ the counterfactual inference here, the reviewer may agree with the argument of contribution.

  3. The centralized critic combines entity-level contributions with a global value term, whereas the actor objective only maximizes a weighted sum of entity-level values. Consequently, the actor update does not directly incorporate the action gradients of the global value term, which may limit its ability to capture complex inter-entity coordination.

  4. Although the paper emphasizes adaptation without retraining, the experiments mainly evaluate performance under different entity churn intensities. It remains unclear whether ALLOTS generalizes to previously unseen entity counts, entity configurations, or churn distributions without additional policy updates.

  5. Most baselines are conventional MARL algorithms adapted through padding and masking. Comparisons with stronger set-based or permutation-equivariant actor-critic architectures (e.g., [1] and [2]) would help determine whether the observed improvements arise from the proposed learning mechanisms or primarily from the set-structured network architecture.

References:

[1] PIC: permutation invariant critic for multi-agent deep reinforcement learning[C]//Conference on robot learning. PMLR, 2020: 590-602.

[2] Boosting multiagent reinforcement learning via permutation invariant and permutation equivariant networks[C]//The eleventh international conference on learning representations. 2022.

Questions for authors
  1. Please clarify the key methodological differences between ALLOTS and directly applying existing Deep Sets or permutation-equivariant architectures to the actor and critic. (See Weakness 1)

  2. Why are larger entity-level value scores interpreted as greater contributions to the global return? Is there any theoretical or empirical evidence supporting the relationship between the learned entity-level values and actual entity contributions? (See Weakness 2)

  3. Since the global value term is intended to capture inter-entity interactions, why does the actor objective exclude this term? Would directly maximizing the centralized Q-value provide more effective policy gradients, particularly in tasks requiring strong coordination? (See Weakness 3)

  4. Can ALLOTS generalize to previously unseen entity counts or configurations without further training? (See Weakness 4) For example, how would a policy trained with a limited range of entity counts perform when evaluated on substantially larger or smaller entity sets?

  5. Can the authors include stronger set-based baselines with comparable network capacities (such as PIC, DPN, and HPN )?

2 comments

We thank the reviewer for the constructive feedback.

Resp. to W1 & Q1: ALLOTS addresses intra-agent dimensional variability: each agent's joint continuous action space changes with its entity set, not merely its observation size or the agent population. A permutation-equivariant set-to-set actor preserves entity–action alignment. Motivated by localized entity changes with potentially global consequences, ALLOTS constructs dual value pathways from paired state–action representations. Unlike conventional set-based actor–critic methods that maximize the full scalar QQ, ALLOTS learns the full portfolio QQ to accommodate non-additive interactions while adaptively aggregating locally structured entity-level gradients for policy optimization. Permutation symmetry alone does not ensure TD consistency across changing entity sets; ALLOTS bootstraps on the realized next active set without altering the Bellman equation. Despite retaining the set-to-set actor, all ablations yield lower mean returns across all 12 configurations (Tables 2, 7–8), supporting contributions beyond set-based representation.

Resp. to W2 & Q2:

  1. We agree that softmax((q−b)/τw)=softmax(q/τw)\mathrm{softmax}((q-b)/\tau_w)=\mathrm{softmax}(q/\tau_w) for the entity-score vector qq, as noted in Sec. 4.2. Mean-centering expresses relative scores without changing the weights.

  2. Entity scores are learned under the full portfolio-level TD objective. Rather than uniformly averaging entity-level gradients, ALLOTS assigns higher-scoring entities' gradients larger relative weights in the shared policy update, enabling value-guided gradient aggregation over changing entity sets. The scores are not identifiable marginal contributions or measures of improvement potential; we will clarify their interpretation.

  3. ALLOTS exceeds uniform weighting (No-EC) in mean return across all 12 configurations. Table 10 shows that the best intermediate temperature yields higher mean training returns than both tested extremes in each FOgym testbed, supporting controlled weighting concentration. Counterfactual inference offers a complementary approach to estimating reference-dependent contributions, whereas our mechanism focuses on policy optimization.

Resp. to W3 & Q3: Excluding the global term from Eq. (15) is deliberate. Adding its action gradients can introduce direct cross-entity gradient coupling through the pooled state–action representation. Given the localized nature of entity changes, ALLOTS restricts direct policy optimization to entity-level pathways to preserve locally structured action gradients (at fixed observations), while the global term participates in full-Q TD learning and jointly trains the shared critic encoder. The actor also remains portfolio-conditioned for coordination. ALLOTS achieves the highest mean returns in all six cooperative coverage configurations (Tables 5–6). Direct full-Q optimization could benefit tasks requiring stronger coordination, but its relative effectiveness with an otherwise identical critic remains untested. We will clarify that Eq. (15) is a deliberate surrogate rather than the full-Q policy gradient.

Resp. to W4 & Q4: ALLOTS adapts to entity-set changes through policy inference rather than parameter updates. Its shared encoder, permutation-invariant pooling, and shared action head (Eqs. 2–5) have cardinality-independent parameters, enabling aligned actions for larger or smaller entity sets given compatible entity interfaces and environment capacity. Our evaluation demonstrates adaptation to repeated stochastic churn events without retraining during execution. Since policies are trained and evaluated under their respective scale and churn settings, these results support within-distribution adaptation and applicability across scales and domains, not zero-shot performance generalization to unseen cardinality ranges, entity configurations, or churn distributions. We will clarify this distinction.

Resp. to W5 & Q5:

  1. Beyond padding-based baselines, MAAC, and AGILE, we include SQDDPG-Set, MADDPG-Set, and MAPPO-Set, which incorporate active-set pooling while retaining their original actor–critic structures (Appendix C.1.3–C.1.4). ALLOTS outperforms all three in mean return across all six configurations (Tables 5–6). No-EC isolates the benefit of adaptive weighting under the same network architecture.

  2. PIC employs a permutation-invariant graph critic, while DPN/HPN introduce entity-wise PI/PE network modules. In MPE, these methods focus on the joint critic of MADDPG. In contrast, ALLOTS addresses simultaneous continuous control over dynamically changing entity sets, with explicit entity–action alignment, active-set-consistent TD learning, and locally structured policy gradients. Unlike these permutation-aware architectures, ALLOTS explicitly exposes entity-level value signals for adaptive gradient aggregation. Direct capacity-matched comparisons with PIC, DPN, and HPN would further strengthen the evaluation.

Reviewer_FNWnOpenReview ↗

Reply to the authors' response

Thank you for the authors' response. However, I still find the novelty of this work limited compared to existing studies. Therefore, I will maintain my original score.

Reviewer_RHReOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents an adaptive MARL framework that addresses variable observation and action spaces through set-based architecture. Specifically, the authors propose ALLOTS which is an actor-critic framework composed of PE set-to-set actor mapping, a set-structured centralized critic with TD bootstrapping on the next-step active set enabling stable learning.

While the proposed approach is promising and relevant to many real-world applications, I think the paper needs a lot more work.

Critical strengths
  1. The problem is interesting and relevant.
  2. ALLOTS is evaluated across 3 domains FOgym, MPE in PettingZoo and MutliDrone 3D coverage environment again several baselines MARL solvers, and achieves clearly the highest mean reward compared to the baselines.
  3. The proposed approach is promising and relevant to many real-world applications.
Critical weaknesses
  1. In the introduction, the QMIX algorithm the authors are referring to is not from Zhang et al. in 2024 but it was published in 2018 and from Rashid and al. (https://arxiv.org/abs/1803.11485). Also, the RL^2 paper is from Duan et al. The one the authors cited is named AMAGO-2(https://arxiv.org/abs/2411.11188).
  2. In the problem formulation, the cited paper from Peralez et al. 2025 talked about simultaneous move Dec-POMDP, the more general Dec-POMDP formulation is from Bernstein et al. 2002. It’s even cited in Peralez paper. The authors should be careful about references.
  3. Another important point in Preliminary is that the authors should explicitly formalized the Dec-POMDP as tuple like {S, A, …} and explicitly define what is state space, action space, etc. I don’t understand what the output of the policy is, I don’t know if it’s entity-action pair A_t or the continuous action in Definition 3.2. If the action is continuous then, the policy should give as output a probability distribution over action and then we can sample from that. Please clarify.
  4. Figure 2 is informative, but it will be much more interesting and easy to understand if the authors can restructure it like maybe having some actor boxes and centralized critic boxes and specify what the inputs of the actors are and how they are linked to the critic.
  5. ALLOTS architecture seems to follow the CTDE paradigm, I don’t see anywhere that is mentioned in the paper. Can the authors clarify?
  6. In subsection 4.1, the authors discussed Assumption B.1 and Proposition B.3 that are not referenced anywhere in the paper.
  7. The main critical point to me is the centralized action-value in equation 10 is composed by mean entity value contributions and the global residual. However, in equation 15, the advantage concerns only the entity value contribution ignoring the residual. The authors should revise this or explain it better.
  8. ALLOTS is evaluated across 3 domains against several baselines MARL solvers, and achieves the highest mean reward. However, 3 random seeds are statistically not enough at all.
  9. I have seen only results on FOgym, even though the authors said they have evaluated ALLOTS across 3 domains. MPE and MultiDrone results are missing.
  10. It could be interesting to see domain specific metrics rather than relying only on average rewards.
Questions for authors

See weaknesses, especially W5, W7, W9, W10.

1 comment

We thank the reviewer for recognizing the relevance and empirical potential of ALLOTS.

Resp. to W1 & W2: We will cite the original works by Rashid et al. (2018) for QMIX, Duan et al. (2016) for RL², and Bernstein et al. (2002) for Dec-POMDPs.

Resp. to W3: We will formalize the problem as M=⟨I,S,O,A,P,O,R,μ0,γ,T⟩\mathcal M=\langle\mathcal I,\mathcal S,\mathcal O,\mathcal A,P,O,R,\mu_0,\gamma,T\rangle, where I\mathcal I is the fixed manager set, S\mathcal S the global state space, O,A\mathcal O,\mathcal A the families of variable-size local observation/action spaces, P,OP,O the joint transition and observation kernels, RR the family of manager-specific reward functions, μ0\mu_0 the initial state distribution, and γ,T\gamma,T the discount and horizon. This is a partially observable Markov game, with Dec-POMDPs as the shared-reward case. The global state StS_t includes all active sets; PP conditions on StS_t and the joint action. Manager ii observes oi,t=(gi,t,Xi,t)o_{i,t}=(g_{i,t},X_{i,t}) (context and entity states), with local feasible actions Ai(St)⊆∏d∈Di,tAd(St)\mathcal A_i(S_t)\subseteq\prod_{d\in D_{i,t}}\mathcal A_d(S_t) for active set Di,tD_{i,t}. The deterministic set-to-set actor (Eq. (5)) outputs one continuous vector ai,tda_{i,t}^d per active entity, forming Ai,tA_{i,t} as the set of pairs (d,ai,td)(d,a_{i,t}^d) for d∈Di,td\in D_{i,t}, rather than an action distribution. Training adds exploration noise.

Resp. to W4 & W5: ALLOTS uses the same core actor–critic architecture and learning rules across domains. Unlike conventional joint-action CTDE critics, our critic evaluates the dynamic entity state–action pairs within each agent, not joint actions across agents. Fig. 2 depicts one manager, with actor input (gi,t,Xi,t)(g_{i,t},X_{i,t}), output Ai,tA_{i,t}, and a portfolio-level critic conditioned on the available context. FOgym managers share actor/critic parameters and replay experience; Modified MPE and MultiDrone use separate actor–critic instances and replay buffers. Parameter sharing is optional; execution is decentralized in all cases. We will replace “centralized action-value” with “portfolio-level action-value” and revise Fig. 2 to clarify the agent and entity levels.

Resp. to W6: Both statements are provided in Appendix B (pp. 13–14). Assumption B.1 specifies order-invariant environment semantics, while Proposition B.3 proves actor equivariance through shared entity maps and permutation-invariant pooling. We will highlight these existing cross-references in Sec. 4.1.

Resp. to W7: Eq. (14) measures relative entity-level scores. Adding the same residual to each score yields (qtd+ρglob)−(bt+ρglob)=qtd−bt(q_t^d+\rho_{\mathrm{glob}})-(b_t+\rho_{\mathrm{glob}})=q_t^d-b_t, leaving the advantages and softmax weights unchanged.

The full TD objective includes the residual, jointly training the shared pair encoder and both heads. At fixed observations, each score depends only on its corresponding action, whereas the residual depends on pooled portfolio state–action representations. Eq. (15) intentionally restricts actor optimization to the explicitly weighted entity-level pathway with detached weights. Including the residual would introduce an additional portfolio-level gradient outside this weighting mechanism.

No-EC (Tables 2, 7, 8) supports adaptive weighting: uniform weights reduce mean returns in all 12 ablation settings, with a 16.4% reward reduction in FOgym-10M under high churn and average penalty increases of 8.6% and 10.2% in Modified MPE and MultiDrone. We will clarify that Eq. (15) is a surrogate, not a full-Q policy gradient, and does not guarantee full-Q improvement. The scores are learned signals, not uniquely identifiable marginal contributions. No-EC does not isolate the effect of omitting the residual gradient.

Resp. to W8: We acknowledge the limitation of three training seeds (ten evaluation episodes per seed). ALLOTS nevertheless achieves the highest mean return across all 12 configurations, and every ablation lowers its corresponding full-model mean. This consistency supports empirical effectiveness.

Resp. to W9: Results are in Appendix C.3 (Tables 5–6, p. 18), referenced in Sec. 5.2, with ablations in Tables 7–8. ALLOTS has the highest mean return in all six settings. Under high churn, it outperforms the strongest baseline, SQDDPG-Set, on Modified MPE (−14.42 vs. −16.09) and MultiDrone (−437.74 vs. −711.67). We will highlight them in the main text.

Resp. to W10: ALLOTS targets intra-agent dimensional variability across domains. Rewards include economic profit and user satisfaction in FOgym, coverage, local efficiency, and collision penalties in both coverage tasks, plus an energy penalty in MultiDrone (Appendix C.1). We report aggregate returns, not separate component metrics, and higher returns do not imply improvement in every component. We will clarify this scope.

Reviewer_wL4GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ALLOTS addresses multi-agent control scenarios in which each manager supervises a dynamic set of entities and, at each time step, generates a continuous action vector for active entities, preserving the correspondence between each entity and its assigned action. The approach integrates a permutation-equivariant actor that uses shared entity encoders and mean pooling with a critic that jointly encodes each entity's state and its corresponding action. Training leverages the realized next active set and uses softmax-normalized relative entity scores to weight actor updates. The experimental evaluation introduces within-episode arrivals and departures in FOgym energy management, a modified Multi-Agent Particle Environment (MPE), and MultiDrone. ALLOTS achieves the highest mean reward among evaluated methods across all reported churn settings. Additional set-enhanced baselines and component ablations are detailed in the supplement.

Critical strengths
  1. The problem is practically motivated, as a manager must maintain device-action correspondence while the controlled portfolio evolves. The actor provides a clear, structurally appropriate interface, and the distinction between arbitrary input order and action assignment is well articulated.

  2. The evaluation spans energy management and two coverage environments, including simulated quadrotor dynamics, providing evidence across diverse application settings. The supplement also includes set-enhanced baselines, making comparisons more informative than those against fixed-input implementations alone. ALLOTS reports the highest mean rewards across the tested churn settings, supporting the practical promise of the combined approach, although the limited number of training seeds warrants caution when interpreting small performance differences.

Critical weaknesses
  1. The novelty discussion does not sufficiently distinguish the proposed contribution from the established capabilities of prior methods. COPA, already cited, evaluates within-episode additions and, while MAAC jointly embeds each agent’s observation and action. UPDeT [Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers. ICLR, 2021], REFIL [Shariq Iqbal, Christian A. Schroeder de Witt, Bei Peng, Wendelin Boehmer, Shimon Whiteson, and Fei Sha. Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning. ICML, 2021], and CPE/GLPE [Zhuofan Xu, Benedikt Bollig, Matthias Fuegger, Thomas Nowak, and Vincent Le Dreau. Centralized Permutation Equivariant Policy for Cooperative Multi-Agent Reinforcement Learning. arXiv:2508.11706, 2025 (preprint).] also warrant explicit comparison. These methods do not necessarily address the same complete setting, but handling changing populations, preserving entity–action correspondence, and jointly encoding observations and actions are not individually sufficient novelty claims. The paper should explain what is new in their combination and in the proposed learning procedure. In addition, Section 2.1 misleadingly groups AGILE with fixed-size slate-selection methods, and its adaptation to simultaneous continuous device control remains insufficiently explained. The citations for RL^2 and QMIX should also be corrected: they currently point to AMAGO-2 and SQIX rather than the original sources.

  2. The ablations do not isolate the proposed mechanisms (Appendix C.4.1). No-TD includes nonexistent padded entities in the target rather than comparing with correctly masked bootstrapping. No-PS retains the same-device state-action pairing before pooling (Equations 22-24), testing early versus later fusion. Thus, the results do not support the stronger causal claims. Evaluation-time updates are unspecified, making "without retraining" ambiguous. The experimental section is insufficiently self-contained: Sec. 5.1 largely lists baseline names and citations without explaining their mechanisms or essential adaptations. Environments have brief domain descriptions, but entity counts, count-sensitive inputs, and quantitative churn definitions are deferred to the appendix. These belong in the main article to make the comparisons and the mean-pooling architecture assessable.

Questions for authors
  1. What specific capability or learning mechanism distinguishes ALLOTS from COPA's within-episode arrival/departure handling, MAAC's joint observation-action embeddings, and CPE/GLPE's centralized equivariant policy architecture? Is the principal contribution the combination of these elements for continuous device control, or a distinct mechanism beyond these established components?

  2. Appendix C.4.1 states that No-TD includes nonexistent padded devices in the target, while Equations 22-24 show that No-PS preserves the same-device state-action correspondence. Which reported results, if any, distinguish the benefit of the proposed bootstrapping procedure from correctly masked bootstrapping? Likewise, does No-PS support a conclusion beyond the benefit of early versus later state-action fusion?

  3. The appendix describes the SlotMapper, observation/action adapters, and set-enhanced baselines that pool across active entities. Do the baselines also exclude inactive entities when computing bootstrapped next-state values, and how is this handled in the set-enhanced variants? Separately, how was AGILE adapted to generate simultaneous continuous commands for all active devices: what do its graph nodes represent, and what is its action-output mechanism?

  4. The appendix describes evaluation after training. Were actor and critic parameters frozen throughout all evaluation episodes, including after entity arrivals and departures?

1 comment

Thank you for the constructive review.

Resp. to W1 & Q1:

(1) COPA. COPA handles changes in the agent population, where each player selects its own action conditioned on the coach's strategy. In ALLOTS, the agent population remains fixed, but each agent controls a changing set of entities, so its own observation and action dimensions vary simultaneously. Each manager must generate one continuous action per active entity.

(2) MAAC. MAAC embeds each agent's entire observation-action pair and attends across agents to estimate agent-level Q-values. Its original encoder and actor assume fixed per-agent dimensions. ALLOTS instead encodes device-level state-action pairs and uses device-level scores to weight policy gradients within each manager. MAAC weights inter-agent information; ALLOTS weights intra-agent policy gradients.

(3) UPDeT. UPDeT maps observation entities to discrete action groups, allowing an agent to select one action from a variable candidate set. ALLOTS simultaneously generates continuous commands for every active entity.

(4) REFIL. REFIL targets cross-task generalization and excludes within-episode entity additions (Sec. 2). It randomly partitions observed entities to construct an auxiliary Q-learning loss over agents' own actions. ALLOTS handles actual entity arrivals/departures and uses device scores directly in continuous joint-action optimization.

(5) CPE/GLPE. GLPE is agent-number-agnostic, but does not directly address dimensional changes within individual agents, as shown by parameter growth with individual observation size in SMAC (Sec. 5). CPE uses centralized execution and retains the underlying MARL learning objectives. ALLOTS additionally learns device-level scores and a global interaction term through TD regression (Eqs. 6–12), and directly optimizes the actor using a detached softmax-weighted sum of these scores (Eqs. 13–16). No-EC replaces these weights with uniform weights, increasing the average penalty by 10.2% in MultiDrone (Table 8).

We will clarify these distinctions, correct AGILE's characterization, and cite the original RL² and QMIX papers.

Resp. to W2 & Q2:

(1) No-TD. Our TD design ensures that next-step action generation and value estimation operate on the same realized active set. No-TD includes nonexistent padded entities in next-step action generation and value estimation, retaining the other components. Correctly masked bootstrapping under identical network operations is mathematically equivalent to our design. No-TD tests active-set consistency, not superiority over equivalent masking. No-TD reduces mean reward by up to 34.3% in FOgym and increases average penalties by 8.8% and 12.8% in Modified MPE and MultiDrone, respectively.

(2) No-PS. Our Pair-Set critic allows state-action interactions to be modeled from the first learned transformation. No-PS retains entity-action correspondence but replaces joint encoding with separate encoding followed by same-entity fusion (Eqs. 22–24). Proposition B.4 establishes assignment awareness at the input level, while No-PS assesses the contribution of direct joint encoding. Tables 7–8 report average penalty increases of 14.1% and 21.8% in Modified MPE and MultiDrone, respectively. We will clarify both ablations.

(3) Experimental details. Mean pooling is complemented by normalized device counts and controllable-device ratios in FOgym, and system-wide and controller-specific active ratios in Modified MPE and MultiDrone. FOgym has 4/10 managers and 118/328 devices; MPE/MultiDrone have four controllers with up to 7/10 entities each. Churn rates are 10–15%, 20–25%, and 30–35%. We will move these details, churn schedules, and baseline mechanisms/adaptations to Sec. 5.1.

Resp. to Q3: Fixed-slot MADDPG, MATD3, and SQDDPG mask executed actions but do not explicitly mask inactive target-action coordinates. MADDPG-Set uses the realized next-state mask for target actions but retains a flattened centralized critic. MAPPO-Set masks inactive action log-probabilities and uses a centralized state-value critic without target-action computation.

We adapt AGILE's graph learning by replacing candidate-action nodes with fully connected active-device nodes. Graph attention produces device embeddings and a masked mean-pooled summary, feeding a shared tanh head that outputs simultaneous continuous commands. Both target networks use the next-state mask. ALLOTS achieves higher mean rewards than AGILE and MADDPG-Set in all reported comparisons, supporting benefits beyond masking alone.

Resp. to Q4: Yes. Actor and critic parameters remain frozen throughout evaluation, including after entity arrivals/departures. Actions are generated by forward inference alone, without fine-tuning or retraining.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.