acceptodds
Under review as a conference paper at ICLR 2027

SKARL: Kernel Mean-Field Reinforcement Learning for Zero-Shot Population-Size Transfer

Abstract

Large-scale multi-agent reinforcement learning (MARL) requires policies that scale to many agents and tolerate population mismatch between training and deployment. Mean-field reinforcement learning (MFRL) provides a size-independent interface, but standard moment summaries discard distributional structure. We introduce **S**calable **K**ernel Me**A**n-Field Multi-Agent **R**einforcement **L**earning (SKARL), which embeds empirical populations by kernel mean embeddings and represents individual values by cylindrical kernel functionals. Under a characteristic kernel, we prove density of this function class. Under same-law sampling, we further bound the distance between the laws of empirical embeddings at sizes and by a term proportional to , yielding the same rate for a fixed Lipschitz cylindrical functional. This result isolates representation sensitivity rather than end-to-end return preservation. For smooth kernels, we derive an RKHS Riesz gradient and its projection onto a fixed-size Nystr\"om dictionary. Across variable-size MPE, MAgent2, and PettingZoo tasks, SKARL scales to large populations and transfers zero-shot. In MAgent2 pursuit, training with and testing with up to improves over first-moment MFRL by - return per agent; in combined arms, training with retains positive mean return from to , while all evaluated baselines are negative.

Primary area:
reinforcement learning
Keywords:
Multi Agent Reinforcement Learning, Mean Field Reinforcement Learning, Kernel Methods

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XZE9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SKARL, a kernelized mean field MARL framework for zero-shot transfer across different population sizes. SKARL represents local agent populations using kernel mean embeddings and models individual Q-functions with cylindrical kernel functionals. The authors provide theoretical results on function-class density and population-size sensitivity, and evaluate the method on variable-size tasks in MPE, MAgent2, and SISL.

Critical strengths
  1. SKARL provides a principled, size-agnostic population representation using kernel mean embeddings and cylindrical kernel functionals, going beyond first-moment mean-field summaries.

  2. The paper combines clear representation-level theory with large-scale zero-shot transfer experiments, including deployments far beyond the training population size.

Critical weaknesses

1.The paper does not clearly explain how the neighbor-action-dependent local Q-function is used during decentralized execution, or how it remains consistent with the claimed IGM/value-decomposition framework.

2.The current ablation studies do not sufficiently disentangle the contributions of the main components of SKARL. In particular, the key large-scale experiments use standard coefficient-space optimization rather than the proposed RKHS-preconditioned update, while the contribution of the latent first-order correction is not evaluated separately.

Questions for authors

1.How are neighbors’ actions obtained during decentralized execution, and how is this consistent with the IGM/value-decomposition framework?

2.What is the contribution of the first-order correction in Eq. (4), and how important is the RKHS-preconditioned update to the reported gains?

3.How does Theorem 4.1 differ from existing KME sampling or coupling bounds?

6 comments
AuthorsOpenReview ↗

Response to Reviewer XZE9 (1/3)

We thank the reviewer for the careful reading. We appreciate the recognition that the kernel mean embedding gives a size-agnostic population representation, and that the paper pairs representation-level theory with large-scale zero-shot transfer. The questions below concern decentralized execution, the first-order correction, the RKHS update, and Theorem 4.1.

W1 & Q1: Decentralized execution

The population embedding used at execution is constructed from observation encodings. Section 3.1 writes the general form xj=eψ(sj,aj)x^j = e_\psi(s^j,a^j), because a mean-field statistic can average actions, states, or joint state–action pairs [1–3]. In the reported experiments the encoder is instantiated without action dependence, xj=eψ(sj)x^j = e_\psi(s^j), on MPE, MAgent2, and SISL. No neighbor action is read at execution. The averaged object is an observation-latent embedding measure. The MFRL baseline aggregates neighbor observations in the same way, so that comparison is matched at the observation level.

This is also what keeps the local Q consistent with the value decomposition. Each agent evaluates its local Q from its own observation and the observation embedding of the other agents. The greedy local action does not require any other agent's action. Training uses a VDN sum of these local Q-functions, so the individual greedy choice is the choice that maximizes the decomposed value. We will state the executed inputs, and this relation to the decomposition, in the revision.

[1] Yang, Yaodong, et al. "Mean field multi-agent reinforcement learning." International Conference on Machine Learning. PMLR, 2018.

[2] Geist, Matthieu, et al. "Concave utility reinforcement learning: The mean-field game viewpoint." arXiv preprint arXiv:2106.03787 (2021).

[3] Hu, Anran. Learning in Mean-Field Games and Continuous-Time Stochastic Control Problems. Diss. University of California, Berkeley, 2022.

AuthorsOpenReview ↗

Response to Reviewer XZE9 (2/3)

Q2: First-order correction

The correction is the first-order term that remains when the Q-function is a nonlinear functional of an embedding measure. In MFRL, the expansion around the mean action makes the first-order deviation average to zero, so the mean-action Q-function is enough. SKARL averages an embedding measure ρ\rho, and Q(ρ)=h(c(ρ))Q(\rho)=h(c(\rho)) is nonlinear, so that average cannot be moved inside hh. The expansion of QQ on ρ\rho leaves a term that depends on the location xx in the embedding space. Aligning that term with the deviation from the mean embedding gives the correction in Eq. (4).

The comparison below changes only this term. The base model is the cylindrical kernel functional h(c(ρ))h(c(\rho)). The full model adds the correction. Both use the RKHS update from the reported results. Full Eq. (4) is the reported SKARL model. Differences are differences of the reported means.

On adversarial pursuit, the full model is above the base model at every population size. The correction adds 2.512.51 to 3.103.10. The kernel basis alone already reaches the low 4040s, and the correction raises that policy further. Standard deviations are large, so these are orderings of means.

MAgent2 adversarial pursuit, trained at N=101N=101.

MethodM=4M=4M=11M=11M=25M=25M=45M=45M=70M=70M=101M=101M=180M=180M=281M=281M=500M=500
Base h(c(ρ))h(c(\rho))44.23±4.8844.72±4.2343.24±3.4042.31±3.3742.03±2.8842.07±3.6241.49±3.3542.02±3.2141.64±3.06
Full Eq. (4)46.90±7.8347.44±3.6146.34±1.4444.82±2.2844.89±2.3145.11±2.7344.40±2.4844.71±2.4644.58±2.46
Δ\Delta(Full − Base)+2.67+2.72+3.10+2.51+2.86+3.04+2.91+2.69+2.94

On combined arms, the base model is positive at every size. The correction adds 0.150.15 at M=18M=18 and 0.650.65 to 1.141.14 at the larger populations. The kernel basis already learns the task, and the correction improves that policy. The standard deviations are large.

MAgent2 combined arms, trained at N=120N=120.

MethodM=18M=18M=45M=45M=72M=72M=120M=120M=242M=242M=450M=450
Base h(c(ρ))h(c(\rho))2.51±2.192.04±1.892.95±2.312.32±1.271.85±1.411.28±0.96
Full Eq. (4)2.66±2.262.91±2.493.60±3.013.35±2.872.99±2.732.16±2.11
Δ\Delta(Full − Base)+0.15+0.87+0.65+1.03+1.14+0.88

On SISL, the base model remains negative. The correction raises the mean by 3.463.46, 4.044.04, 3.903.90, and 1.621.62 at M=32M=32, 6464, 128128, and 256256, and moves the return above zero at the first three sizes. Standard deviations on the full model are large.

SISL pursuit, trained at N=64N=64.

MethodM=32M=32M=64M=64M=128M=128M=256M=256
Base h(c(ρ))h(c(\rho))−0.81±2.17−0.94±3.61−2.19±4.15−1.79±3.25
Full Eq. (4)2.65±7.103.10±8.751.71±7.79−0.17±6.54
Δ\Delta(Full − Base)+3.46+4.04+3.90+1.62

Across the three tasks, the correction raises the kernel basis at every reported size. On adversarial pursuit and combined arms, the kernel basis already carries a working policy, and the correction adds a further gain. On SISL, the base model stays negative, and the correction is what moves the return above zero at M=32M=32, 6464, and 128128. We will include this ablation in the revision.

AuthorsOpenReview ↗

Response to Reviewer XZE9 (3/3)

Q2: RKHS-preconditioned update

The results reported in the paper were trained with the RKHS-preconditioned update, including the MAgent2 and SISL benchmarks. The description in Appendix F.1 does not match the implementation used for the reported results. We will revise the paper, including Appendix F.1, so the text matches the optimizer used in the experiments.

To test whether that choice matters on the headline benchmarks, we retrain the reported model under a Euclidean coefficient-space update, with the same architecture. The RKHS column is the reported SKARL setting. Differences below are differences of the reported means.

On adversarial pursuit, the RKHS mean is above the Euclidean mean at every population size, by 2.032.03 to 3.063.06. The standard deviations are large, so the intervals overlap and the comparison is an ordering of means.

MAgent2 adversarial pursuit, trained at N=101N=101.

MethodM=4M=4M=11M=11M=25M=25M=45M=45M=70M=70M=101M=101M=180M=180M=281M=281M=500M=500
Euclidean44.43±3.3344.38±0.6643.53±2.3342.12±2.0542.07±2.4942.67±2.4842.37±2.5442.43±2.3142.24±2.13
RKHS46.90±7.8347.44±3.6146.34±1.4444.82±2.2844.89±2.3145.11±2.7344.40±2.4844.71±2.4644.58±2.46
Δ\Delta(RKHS − Euclidean)+2.47+3.06+2.81+2.70+2.82+2.44+2.03+2.28+2.34

On combined arms, the Euclidean update stays near −0.54-0.54 at every size. The RKHS update obtains the positive returns reported for SKARL. The gap is 2.702.70 to 4.144.14. The RKHS standard deviations are large.

MAgent2 combined arms, trained at N=120N=120.

MethodM=18M=18M=45M=45M=72M=72M=120M=120M=242M=242M=450M=450
Euclidean−0.54±0.00−0.54±0.00−0.54±0.00−0.55±0.01−0.54±0.00−0.54±0.01
RKHS2.66±2.262.91±2.493.60±3.013.35±2.872.99±2.732.16±2.11
Δ\Delta(RKHS − Euclidean)+3.20+3.45+4.14+3.90+3.53+2.70

On SISL, the Euclidean update stays negative at every size. The RKHS update is positive at M=32M=32, 6464, and 128128, and near zero at M=256M=256. Standard deviations on the RKHS runs are large.

SISL pursuit, trained at N=64N=64.

MethodM=32M=32M=64M=64M=128M=128M=256M=256
Euclidean−6.97±1.94−6.98±0.85−5.53±1.54−7.19±0.67
RKHS2.65±7.103.10±8.751.71±7.79−0.17±6.54
Δ\Delta(RKHS − Euclidean)+9.62+10.08+7.24+7.02

Across the three headline benchmarks, the RKHS update is above the Euclidean coefficient-space update at every reported size. We will include this comparison in the revision together with the corrected description of the update used in the paper.

Q3: Theorem 4.1

A generic KME sampling or coupling bound controls the distance between an empirical kernel mean and the kernel mean of the population law. That argument applies to a general Hilbert-valued empirical mean, and it stops at the embedding. Theorem 4.1 specializes the same coupling to the KME used by SKARL and to the gap of the cylindrical value functional. The result is a representation-level bound on how that value changes with population size under same-law sampling. The bound informs the fixed-dimensional population interface: the value reads a KME whose dimension does not grow with the number of neighbors, and the theorem describes the sensitivity of that representation. We will make this distinction explicit in the revision.

Reviewer_yPD7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SKARL, a kernel-based mean-field reinforcement learning framework for scalable multi-agent learning and zero-shot population-size transfer. It represents empirical populations using kernel mean embeddings and constructs local Q-functions through cylindrical functionals with a latent first-order correction, implemented using fixed-size kernel dictionaries. The theoretical contributions include a density result under characteristic kernels, a representation-level population-size sensitivity bound under same-law i.i.d. sampling, and an RKHS-gradient update with finite-dictionary projection. The empirical evaluation covers MPE, MAgent2, and SISL tasks, with comparisons against mean-field, value-decomposition, and policy-gradient baselines. The reported results show improved performance on several large-population cooperative tasks and zero-shot deployment to populations up to approximately 45 times the training size.

Critical strengths
  1. Promising empirical results on challenging tasks with large populations. The reported MAgent2 results suggest practical value for the proposed representation. In adversarial pursuit, training with N=101 agents yields a return of 44.58 per agent when deployed with M=500, compared with 26.78 for MFRL. In combined arms, SKARL maintains positive mean returns across all tested deployment sizes, while the evaluated baselines remain negative. These results provide encouraging evidence that the approach can support effective learning and deployment across population sizes.

  2. A coherent representation framework with clearly scoped theoretical analysis. The combination of kernel mean embeddings, cylindrical value functions, and finite dictionaries provides a tractable way to model population distributions with a fixed-dimensional interface. The density result and same-law population-size sensitivity bound characterize the representation’s expressiveness and sampling sensitivity. The paper explicitly distinguishes these guarantees from end-to-end policy-return preservation, making the theoretical contribution easier to interpret and providing a useful foundation for studying population-size transfer.

Critical weaknesses

1.The contribution of the first-order correction is not isolated experimentally. The base cylindrical value function already captures distributional information, but the paper does not directly compare it with the full model in Eq. (4). The existing ablations examine the coefficient-update rule, dictionary size, kernel type, and value-decomposition scheme; they do not establish the independent benefit of the correction term. A comparison of separately trained models with and without this term, under matched settings, is needed to assess whether this central design improves learning or population-size transfer. This omission limits confidence in the significance of the proposed methodological extension.

2.The evaluation provides insufficient comparison with methods specifically targeting population-size generalization. A particularly relevant omission is InforMARL[1], which uses graph-based neighborhood aggregation for variable-size systems and provides the basis for the particle tasks considered here. Furthermore, Fixed-Kernel is described as freezing SKARL’s own dictionary locations and expansion coefficients, leaving its correspondence to prior kernel-based mean-field methods unclear. Although the existing baselines provide useful reference points, these gaps make it difficult to establish SKARL’s advantage over directly relevant alternatives. A targeted comparison under matched information access and training budgets would strengthen the evidence for its contribution. [1] Nayak, Siddharth, et al. "Scalable multi-agent reinforcement learning through intelligent information aggregation." International conference on machine learning. PMLR, 2023.

Questions for authors

1.Kernel mean embeddings have previously been used in mean-field RL and control [1,2], and cylindrical function approximation has also been studied [3]. What is the principal methodological or theoretical advance of SKARL over these approaches? In particular, the coupling argument in Theorem 4.1 appears applicable to general Hilbert-valued empirical means. What additional insight does this result provide, and how does it inform the proposed algorithm? Please identify the main new component or combination and explain what evidence establishes its value beyond the existing building blocks.

2.What limitation of the base cylindrical value function �0��0� is the correction in Eq. (4) intended to address? Could you compare separately trained models with and without this term, using the same encoder, dictionary size, optimizer, and training budget on a representative main benchmark? How does the correction affect both learning performance and zero-shot population-size transfer?

3.Fixed-Kernel is described as freezing SKARL’s dictionary locations and expansion coefficients. Does this baseline faithfully implement the method of Cui et al. [2], or should it be interpreted as an internal ablation of SKARL? Please explain its correspondence to the cited method, including any architectural or training modifications. How are the encoder and downstream network capacities matched between SKARL and the representation baselines?

4.How does SKARL compare with InforMARL [4], which explicitly addresses variable-size multi-agent systems, on comparable particle tasks under matched observation access, communication assumptions, and training budgets? If such a comparison is not applicable, what specific differences in the problem formulation prevent a fair evaluation?

5.Section 3.1 defines the population embedding through �1��1�. Which neighbor actions, if any, are used during execution: current actions, previous-step actions, or predicted actions? If current actions are used, how are they obtained before each agent selects its own action? The public MPE implementation appears to construct embeddings from observations and recurrent states without explicit neighbor-action inputs. Please reconcile this implementation with the state–action formulation and specify the actual population inputs and information available during execution for each benchmark, including MAgent2 and SISL.

References

[1] Lingxiao Wang, Zhuoran Yang, and Zhaoran Wang. Breaking the Curse of Many Agents: Provable Mean Embedding Q-Iteration for Mean-Field Reinforcement Learning. ICML, 2020.

[2] Kai Cui, Sascha Hauck, Christian Fabian, and Heinz Koeppl. Learning Decentralized Partially Observable Mean Field Control for Artificial Collective Behavior. ICLR, 2024.

[3] Huyên Pham and Xavier Warin. Mean-field neural networks: Learning mappings on Wasserstein space. Neural Networks, 168:380–393, 2023.

[4] Siddharth Nayak, Kenneth Choi, Wenqi Ding, Sydney Dolan, Karthik Gopalakrishnan, and Hamsa Balakrishnan. Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregation. ICML, 2023.

1 comment
AuthorsOpenReview ↗

Gentle Reminder: Discussion Period Approaching Deadline

Dear Reviewer,

Hope you are doing well. As the discussion period is drawing to a close, we would really appreciate it if you could review our rebuttal. We have responded point-by-point to your feedback and added new results as requested.

If you think our responses adequately resolve your concerns, we kindly ask you to consider updating your evaluation. Feel free to raise any remaining questions, and we will reply accordingly.

We sincerely appreciate your valuable time.

Best regards,

Authors

Reviewer_WZfpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SKARL, a kernelized mean-field multi-agent reinforcement learning (MARL) framework designed for zero-shot population-size transfer. The method encodes neighbor states and actions into a latent space and constructs a population representation using kernel mean embeddings (KME) evaluated against a fixed-size dictionary of anchor points. Individual Q-functions are modeled as cylindrical kernel functionals combined with a latent first-order correction term motivated by the Lions derivative expansion. To train the model, the authors derive an RKHS-preconditioned Riesz gradient projected onto the finite Nyström dictionary, though standard coefficient-space autodiff is used in several benchmarks depending on the environment. The authors provide theoretical density and representation-sensitivity bounds and validate the approach across variable-size MPE, SISL, and MAgent2 environments.

Critical strengths
  • Presentation and Empirical Scalability: The paper is well-written, polished, and clearly structured. Empirically, the proposed framework demonstrates strong scalability and zero-shot population-size transfer across large swarm sizes (up to M=500M=500 in MAgent2), outperforming standard first-moment MFRL and value-decomposition baselines in complex coordination tasks. This is consequential because achieving reliable zero-shot transfer across drastic population scale changes is a central bottleneck in MARL.
Critical weaknesses
  • Lack of Key Baselines and Component Ablations (e.g., M3ARL\text{M}^3\text{ARL}, DeepSets, InforMARL): A central premise of the paper is moving beyond first-order moment summaries, and Eq. (1) and Eq. (4) directly build upon the Lions–Taylor derivative formulation from M3ARL\text{M}^3\text{ARL} (Tang et al., 2024). Despite M3ARL\text{M}^3\text{ARL} being a primary state-of-the-art baseline that already leverages higher-order moment statistics and Lions derivatives, the paper omits it from experimental comparisons. Furthermore, averaging pairwise latent encodings over a neighborhood closely resembles a DeepSets architecture (or graph/aggregation methods like InforMARL, which the paper cites for the MPE setup). While SKARL differs by using a kernel basis and the Lions-inspired correction term, the lack of comparisons to M3ARL\text{M}^3\text{ARL}, DeepSets, or InforMARL—alongside missing ablations isolating the kernel basis from the first-order correction term—makes it difficult to tell which component actually drives the empirical gains.
  • Inconsistent Use of a Core Contribution (RKHS-Preconditioned Gradients): The derivation of the RKHS Riesz gradient and its Nyström projection (Proposition 3.3, Eqs. 6–9) is highlighted in the abstract, introduction, and main text as a primary contribution of the paper. However, Appendix F.1 reveals that this update rule is only used in the MPE Move implementation, while the headline large-scale benchmarks (MAgent2 Adversarial-Pursuit, Combined-Arms, and SISL) drop Eqs. (6)–(8) entirely in favor of standard end-to-end autodiff in coefficient space. Relying on ordinary autodiff for the paper's most challenging benchmarks severely undermines the practical necessity and claimed significance of the RKHS-preconditioned optimization machinery.
Questions for authors

See weaknesses section

11 comments
AuthorsOpenReview ↗

Response to Reviewer WZfp (1/4)

We thank the reviewer for the careful reading. We appreciate the recognition that the paper is clearly structured and that the zero-shot population-size results cover large swarms. The concerns below are the missing baselines, the split between the kernel basis and the first-order correction, and the role of the RKHS-preconditioned update in the reported experiments.

W1: Baselines

We add the baselines requested for this comparison. DeepSets [1] is a permutation-invariant set model, so it is not a MARL algorithm. We implement it as a mean-pooling Deep Sets encoder inside the same VDN-style TD pipeline used for SKARL. M$^3$ARL is included as the higher-order moment and Lions-derivative baseline, and InforMARL as the graph-aggregation baseline. Both are reported on Move, MAgent2 and SISL.

On Move, with zero-shot transfer from N=16N=16, SKARL has the highest per-agent return at M=4M=4, 1616, and 6464. At M=256M=256, M$^3$ARL is slightly higher on return and mean speed. DeepSets and InforMARL stay below SKARL on return over the whole sweep, and both drop at the largest population, while SKARL and M$^3$ARL remain on the same high plateau. Dec-POMFPPO stays at a low return with very low speed, so its collision count is low because the agents barely move. Read together, return, speed, and collisions show that SKARL keeps a fast policy across the transfer range, and that the higher-order moment baseline is competitive specifically at the largest Move population.

Move: zero-shot population-size transfer from N=16N=16.

MetricMethodM=4M=4M=16M=16M=64M=64M=256M=256
Return per agent ↑M$^3$ARL201.74205.76207.69208.43
Return per agent ↑DeepSets141.6184.5196.6187.8
Return per agent ↑InforMARL203.29192.3201.9182.9
Return per agent ↑Dec-POMFPPO74.2674.4674.6174.49
Return per agent ↑SKARL236.9234.7225.4202.6
Collisions per agent ↓M$^3$ARL1.477±0.0550.656±0.0550.171±0.0120.038±0.016
Collisions per agent ↓DeepSets1.33±1.8780.49±0.02000.38±0.0580.19±0.025
Collisions per agent ↓InforMARL0.56±0.5750.65±0.4610.17±0.1050.068±0.052
Collisions per agent ↓Dec-POMFPPO0.055±0.0110±00±00±0
Collisions per agent ↓SKARL0±00.77±0.3650.17±0.1150.04±0.0221
Mean speed ↑M$^3$ARL0.1602±0.00030.1575±0.00020.1556±0.000040.1552±0.0002
Mean speed ↑DeepSets0.10±0.0260.12±0.0110.13±0.0010.14±0.009
Mean speed ↑InforMARL0.14±0.0240.13±0.0180.13±0.0020.13±0.020
Mean speed ↑Dec-POMFPPO0.0486±0.00020.0484±0.00010.0485±0.000040.0484±0.00003
Mean speed ↑SKARL0.16±0.004340.17±0.05000.15±0.03230.14±0.0542

On MAgent2 adversarial pursuit, trained at N=101N=101, the SKARL mean is above M$^3$ARL, DeepSets, InforMARL, and Dec-POMFPPO at every tested population. The margin over M$^3$ARL is under one return point at M=4M=4, 4545, 281281, and 500500, and about two points from M=101M=101 to M=180M=180. The margin over DeepSets is smallest at the training size (0.350.35 at M=101M=101) and larger at both ends of the sweep (1.991.99 at M=4M=4, 2.112.11 at M=500M=500). SKARL has wider run-to-run variation than these two baselines, so the intervals overlap and the comparison is an ordering of means. InforMARL stays near a return of 1111 across the sweep. Dec-POMFPPO remains near the floor.

MAgent2 adversarial pursuit, trained at N=101N=101.

MethodM=4M=4M=11M=11M=25M=25M=45M=45M=70M=70M=101M=101M=180M=180M=281M=281M=500M=500
M$^3$ARL46.28±6.1946.17±4.7245.20±1.6844.09±1.3343.38±0.8043.17±0.7642.22±0.4643.98±0.6743.74±0.46
DeepSets44.91±2.9546.50±2.3845.22±1.8044.27±0.8643.93±0.8744.76±1.0443.53±0.3942.76±0.5042.47±0.15
InforMARL10.92±1.3010.72±0.6810.70±0.4110.77±0.4910.86±0.2010.78±0.2310.73±0.1310.54±0.1110.65±0.13
Dec-POMFPPO1.75±0.121.47±0.101.50±0.051.62±0.051.67±0.021.57±0.021.67±0.021.55±0.011.58±0.01
SKARL46.90±7.8347.44±3.6146.34±1.4444.82±2.2844.89±2.3145.11±2.7344.40±2.4844.71±2.4644.58±2.46
AuthorsOpenReview ↗

Response to Reviewer WZfp (2/4)

On MAgent2 combined arms, trained at N=120N=120, M$^3$ARL and InforMARL stay near −0.54-0.54 at every size. Dec-POMFPPO remains near −5.5-5.5, with a tight spread. DeepSets and SKARL are the two methods that obtain a positive return. DeepSets peaks at the training population and falls to 1.111.11 at M=450M=450. The SKARL mean is above DeepSets at every size, and the drop at the largest population is smaller (2.162.16 at M=450M=450). The standard deviations on SKARL are large, so the intervals overlap and the result should be read as a consistent mean ordering.

MAgent2 combined arms, trained at N=120N=120.

MethodM=18M=18M=45M=45M=72M=72M=120M=120M=242M=242M=450M=450
M$^3$ARL−0.54±0.00−0.54±0.01−0.54±0.00−0.55±0.01−0.54±0.01−0.55±0.00
DeepSets2.08±0.912.60±0.552.79±0.543.15±0.622.19±0.411.11±0.34
InforMARL−0.54±0.01−0.55±0.00−0.54±0.01−0.54±0.01−0.54±0.00−0.54±0.00
Dec-POMFPPO−5.63±0.14−5.16±0.09−5.50±0.04−5.34±0.07−5.66±0.03−5.67±0.04
SKARL2.66±2.262.91±2.493.60±3.013.35±2.872.99±2.732.16±2.11

On SISL pursuit, trained at N=64N=64, the SKARL mean is highest at every population size. Near the training size, M$^3$ARL and DeepSets are close to SKARL, and InforMARL is positive but lower. All five methods lose return as the population grows. At M=128M=128, SKARL and DeepSets remain positive and the other three are negative. At M=256M=256, the SKARL mean is still near zero, while DeepSets, InforMARL, M$^3$ARL, and Dec-POMFPPO are negative, with M$^3$ARL and Dec-POMFPPO falling furthest. Standard deviations on this task are large for every method, so the comparison is an ordering of means.

SISL pursuit, trained at N=64N=64.

MethodM=32M=32M=64M=64M=128M=128M=256M=256
M$^3$ARL1.78±5.182.94±6.75−0.98±6.74−3.53±8.59
DeepSets1.81±8.312.91±9.130.63±8.79−1.33±8.62
InforMARL1.36±5.461.99±3.26−1.72±9.51−2.19±6.23
Dec-POMFPPO0.47±5.7−1.19±3.13−2.15±4.61−4.88±4.12
SKARL2.65±7.103.10±8.751.71±7.79−0.17±6.54

We will add these comparisons to the revised manuscript.

[1] Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R. R., & Smola, A. (2017). Deep sets. Advances in Neural Information Processing Systems, 30.

AuthorsOpenReview ↗

Response to Reviewer WZfp (3/4)

Ablation: kernel basis versus the first-order correction

The base model is the cylindrical kernel functional h(c(ρ))h(c(\rho)). The full model adds the first-order correction in Eq. (4). Both use the RKHS update from the reported results, so the comparison changes only the correction. Full Eq. (4) is the SKARL model in W1. Differences below are differences of the reported means.

On adversarial pursuit, the full model is above the base model at every population size. The correction adds 2.512.51 to 3.103.10. The kernel basis alone already reaches the low 4040s, and the correction raises that policy further. Standard deviations are large, so these are orderings of means.

MAgent2 adversarial pursuit, trained at N=101N=101.

MethodM=4M=4M=11M=11M=25M=25M=45M=45M=70M=70M=101M=101M=180M=180M=281M=281M=500M=500
Base h(c(ρ))h(c(\rho))44.23±4.8844.72±4.2343.24±3.4042.31±3.3742.03±2.8842.07±3.6241.49±3.3542.02±3.2141.64±3.06
Full Eq. (4)46.90±7.8347.44±3.6146.34±1.4444.82±2.2844.89±2.3145.11±2.7344.40±2.4844.71±2.4644.58±2.46
Δ\Delta(Full − Base)+2.67+2.72+3.10+2.51+2.86+3.04+2.91+2.69+2.94

On combined arms, the base model is positive at every size. The correction adds 0.150.15 at M=18M=18 and 0.650.65 to 1.141.14 at the larger populations. The kernel basis already learns the task, and the correction improves that policy. The standard deviations are large.

MAgent2 combined arms, trained at N=120N=120.

MethodM=18M=18M=45M=45M=72M=72M=120M=120M=242M=242M=450M=450
Base h(c(ρ))h(c(\rho))2.51±2.192.04±1.892.95±2.312.32±1.271.85±1.411.28±0.96
Full Eq. (4)2.66±2.262.91±2.493.60±3.013.35±2.872.99±2.732.16±2.11
Δ\Delta(Full − Base)+0.15+0.87+0.65+1.03+1.14+0.88

On SISL, the base model remains negative. The correction raises the mean by 3.463.46, 4.044.04, 3.903.90, and 1.621.62 at M=32M=32, 6464, 128128, and 256256, and moves the return above zero at the first three sizes. Standard deviations on the full model are large.

SISL pursuit, trained at N=64N=64.

MethodM=32M=32M=64M=64M=128M=128M=256M=256
Base h(c(ρ))h(c(\rho))−0.81±2.17−0.94±3.61−2.19±4.15−1.79±3.25
Full Eq. (4)2.65±7.103.10±8.751.71±7.79−0.17±6.54
Δ\Delta(Full − Base)+3.46+4.04+3.90+1.62

Across the three tasks, the correction raises the kernel basis at every reported size. On adversarial pursuit and combined arms, the kernel basis already carries a working policy, and the correction adds a further gain. On SISL, the base model stays negative, and the correction is what moves the return above zero at M=32M=32, 6464, and 128128. We will include this ablation in the revision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal phase

Dear Area Chair,

Thank you for considering SKARL. Thank you for taking the time to consider our submission. We’d like to summarize what the reviewers concerned about and how our rebuttal addressed it.

The three reviews show 2 (weak rejection). Reviewer WZfp and reviewer XZE9 explicitly maintained their scores. Reviewer XZE9 confirmed that our explanation resolved the decentralized-execution concern. Until we write this, reveiwer yPD7 has not been involved in the communication.

Concern 1. How does decentralized execution work?

Reviewers’ questions. Reviewer XZE9 and yPD7 asked how an agent could use its neighbors’ actions before choosing its own action. Reviewer XZE9 also asked whether this was consistent with value decomposition.

Our response. We clarified that the population embedding uses neighbors’ observations only. Each agent can therefore select its own greedy action independently. Together, these actions maximize the sum of local values used by VDN.

Reviewers’ reactions. Reviewer XZE9 explicitly confirmed that this resolved their concern.

Concern 2. What do the first-order correction and the RKHS update contribute?

Reviewers’ questions. All three reviewers asked us to isolate the first-order correction. Reviewer WZfp and XZE9 also identified a conflict between Appendix F.1 and the optimizer we reported using on MAgent2 and SISL.

Our response. We conduct two controlled ablations. First, we train the same model with and without the correction. Adding it raises mean return at every tested population size across three benchmarks. On SISL Pursuit with 64 agents, mean return rises from −0.94 to 3.10. Second, we keep the architecture fixed and change only the optimizer. On Combined Arms, the Euclidean update learns a stationary policy with return near −0.54, while the RKHS update yields positive mean returns at all six sizes. We acknowledged the error in Appendix F.1 and committed to documenting the optimizer used for each result.

Reviewers’ reactions. Reviewer WZfp acknowledged that we ran the requested ablations but remained concerned about variability across seeds. XZE9 recognized that the optimizer choice appears to matter and requested the exact configuration and code path for reproducibility.

Concern 3. How strong and fair are the baseline comparisons?

Reviewers’ questions. Reviewer WZfp requested M³ARL, DeepSets, and InforMARL comparisons. Reviewer yPD7 asked about matched training conditions and how the Fixed-Kernel baseline relates to prior work. Reviewer WZfp and XZE9 also questioned whether the weaker baselines had converged.

Our response. We conduct comparisons on Move, MAgent2, and SISL, including M³ARL, DeepSets, InforMARL and Dec-POMFPPO. SKARL exceeds DeepSets by 14.8–95.3 mean return points on Move and achieves the highest mean at all six Combined Arms sizes, where M³ARL stays near −0.54. For InforMARL, we test four communication radii and report a learning curve extending to two million transitions. Its return levels off near 10.8 after one million transitions.

Reviewers’ reactions. Reviewer WZfp agreed that the graph radius was unlikely to be InforMARL’s main bottleneck. Both reviewers still considered several margins small relative to seed variability. Reviewer WZfp’s last comment preceded our extended learning curve. We agreed to limit our claims to the observed mean returns.

Concern 4. What does the population-size bound establish?

Reviewers’ questions. Reviewer XZE9 and yPD7 asked what Theorem 4.1 adds to existing kernel mean embedding bounds and how it relates to policy transfer.

Our response. We explained how the coupling argument bounds the change in SKARL’s value estimate when different-sized populations are sampled from the same distribution. The zero-shot deployment experiments provide the evidence for policy transfer.

Reviewers’ reactions. Reviewer XZE9 recognized the theorem’s interpretation as a representation-sensitivity bound and emphasized that its connection to policy returns remains empirical. We agreed with that assessment.

In the revised paper, we have added experiments, explainations and updated related descriptions.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.