- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SKARL, a kernelized mean field MARL framework for zero-shot transfer across different population sizes. SKARL represents local agent populations using kernel mean embeddings and models individual Q-functions with cylindrical kernel functionals. The authors provide theoretical results on function-class density and population-size sensitivity, and evaluate the method on variable-size tasks in MPE, MAgent2, and SISL.
Critical strengths
-
SKARL provides a principled, size-agnostic population representation using kernel mean embeddings and cylindrical kernel functionals, going beyond first-moment mean-field summaries.
-
The paper combines clear representation-level theory with large-scale zero-shot transfer experiments, including deployments far beyond the training population size.
Critical weaknesses
1.The paper does not clearly explain how the neighbor-action-dependent local Q-function is used during decentralized execution, or how it remains consistent with the claimed IGM/value-decomposition framework.
2.The current ablation studies do not sufficiently disentangle the contributions of the main components of SKARL. In particular, the key large-scale experiments use standard coefficient-space optimization rather than the proposed RKHS-preconditioned update, while the contribution of the latent first-order correction is not evaluated separately.
Questions for authors
1.How are neighbors’ actions obtained during decentralized execution, and how is this consistent with the IGM/value-decomposition framework?
2.What is the contribution of the first-order correction in Eq. (4), and how important is the RKHS-preconditioned update to the reported gains?
3.How does Theorem 4.1 differ from existing KME sampling or coupling bounds?
6 comments
Response to Reviewer XZE9 (2/3)
Q2: First-order correction
The correction is the first-order term that remains when the Q-function is a nonlinear functional of an embedding measure. In MFRL, the expansion around the mean action makes the first-order deviation average to zero, so the mean-action Q-function is enough. SKARL averages an embedding measure , and is nonlinear, so that average cannot be moved inside . The expansion of on leaves a term that depends on the location in the embedding space. Aligning that term with the deviation from the mean embedding gives the correction in Eq. (4).
The comparison below changes only this term. The base model is the cylindrical kernel functional . The full model adds the correction. Both use the RKHS update from the reported results. Full Eq. (4) is the reported SKARL model. Differences are differences of the reported means.
On adversarial pursuit, the full model is above the base model at every population size. The correction adds to . The kernel basis alone already reaches the low s, and the correction raises that policy further. Standard deviations are large, so these are orderings of means.
MAgent2 adversarial pursuit, trained at .
| Method | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Base | 44.23±4.88 | 44.72±4.23 | 43.24±3.40 | 42.31±3.37 | 42.03±2.88 | 42.07±3.62 | 41.49±3.35 | 42.02±3.21 | 41.64±3.06 |
| Full Eq. (4) | 46.90±7.83 | 47.44±3.61 | 46.34±1.44 | 44.82±2.28 | 44.89±2.31 | 45.11±2.73 | 44.40±2.48 | 44.71±2.46 | 44.58±2.46 |
| (Full − Base) | +2.67 | +2.72 | +3.10 | +2.51 | +2.86 | +3.04 | +2.91 | +2.69 | +2.94 |
On combined arms, the base model is positive at every size. The correction adds at and to at the larger populations. The kernel basis already learns the task, and the correction improves that policy. The standard deviations are large.
MAgent2 combined arms, trained at .
| Method | ||||||
|---|---|---|---|---|---|---|
| Base | 2.51±2.19 | 2.04±1.89 | 2.95±2.31 | 2.32±1.27 | 1.85±1.41 | 1.28±0.96 |
| Full Eq. (4) | 2.66±2.26 | 2.91±2.49 | 3.60±3.01 | 3.35±2.87 | 2.99±2.73 | 2.16±2.11 |
| (Full − Base) | +0.15 | +0.87 | +0.65 | +1.03 | +1.14 | +0.88 |
On SISL, the base model remains negative. The correction raises the mean by , , , and at , , , and , and moves the return above zero at the first three sizes. Standard deviations on the full model are large.
SISL pursuit, trained at .
| Method | ||||
|---|---|---|---|---|
| Base | −0.81±2.17 | −0.94±3.61 | −2.19±4.15 | −1.79±3.25 |
| Full Eq. (4) | 2.65±7.10 | 3.10±8.75 | 1.71±7.79 | −0.17±6.54 |
| (Full − Base) | +3.46 | +4.04 | +3.90 | +1.62 |
Across the three tasks, the correction raises the kernel basis at every reported size. On adversarial pursuit and combined arms, the kernel basis already carries a working policy, and the correction adds a further gain. On SISL, the base model stays negative, and the correction is what moves the return above zero at , , and . We will include this ablation in the revision.
Response to Reviewer XZE9 (3/3)
Q2: RKHS-preconditioned update
The results reported in the paper were trained with the RKHS-preconditioned update, including the MAgent2 and SISL benchmarks. The description in Appendix F.1 does not match the implementation used for the reported results. We will revise the paper, including Appendix F.1, so the text matches the optimizer used in the experiments.
To test whether that choice matters on the headline benchmarks, we retrain the reported model under a Euclidean coefficient-space update, with the same architecture. The RKHS column is the reported SKARL setting. Differences below are differences of the reported means.
On adversarial pursuit, the RKHS mean is above the Euclidean mean at every population size, by to . The standard deviations are large, so the intervals overlap and the comparison is an ordering of means.
MAgent2 adversarial pursuit, trained at .
| Method | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Euclidean | 44.43±3.33 | 44.38±0.66 | 43.53±2.33 | 42.12±2.05 | 42.07±2.49 | 42.67±2.48 | 42.37±2.54 | 42.43±2.31 | 42.24±2.13 |
| RKHS | 46.90±7.83 | 47.44±3.61 | 46.34±1.44 | 44.82±2.28 | 44.89±2.31 | 45.11±2.73 | 44.40±2.48 | 44.71±2.46 | 44.58±2.46 |
| (RKHS − Euclidean) | +2.47 | +3.06 | +2.81 | +2.70 | +2.82 | +2.44 | +2.03 | +2.28 | +2.34 |
On combined arms, the Euclidean update stays near at every size. The RKHS update obtains the positive returns reported for SKARL. The gap is to . The RKHS standard deviations are large.
MAgent2 combined arms, trained at .
| Method | ||||||
|---|---|---|---|---|---|---|
| Euclidean | −0.54±0.00 | −0.54±0.00 | −0.54±0.00 | −0.55±0.01 | −0.54±0.00 | −0.54±0.01 |
| RKHS | 2.66±2.26 | 2.91±2.49 | 3.60±3.01 | 3.35±2.87 | 2.99±2.73 | 2.16±2.11 |
| (RKHS − Euclidean) | +3.20 | +3.45 | +4.14 | +3.90 | +3.53 | +2.70 |
On SISL, the Euclidean update stays negative at every size. The RKHS update is positive at , , and , and near zero at . Standard deviations on the RKHS runs are large.
SISL pursuit, trained at .
| Method | ||||
|---|---|---|---|---|
| Euclidean | −6.97±1.94 | −6.98±0.85 | −5.53±1.54 | −7.19±0.67 |
| RKHS | 2.65±7.10 | 3.10±8.75 | 1.71±7.79 | −0.17±6.54 |
| (RKHS − Euclidean) | +9.62 | +10.08 | +7.24 | +7.02 |
Across the three headline benchmarks, the RKHS update is above the Euclidean coefficient-space update at every reported size. We will include this comparison in the revision together with the corrected description of the update used in the paper.
Q3: Theorem 4.1
A generic KME sampling or coupling bound controls the distance between an empirical kernel mean and the kernel mean of the population law. That argument applies to a general Hilbert-valued empirical mean, and it stops at the embedding. Theorem 4.1 specializes the same coupling to the KME used by SKARL and to the gap of the cylindrical value functional. The result is a representation-level bound on how that value changes with population size under same-law sampling. The bound informs the fixed-dimensional population interface: the value reads a KME whose dimension does not grow with the number of neighbors, and the theorem describes the sensitivity of that representation. We will make this distinction explicit in the revision.
Response to Reviewer XZE9 (1/3)
We thank the reviewer for the careful reading. We appreciate the recognition that the kernel mean embedding gives a size-agnostic population representation, and that the paper pairs representation-level theory with large-scale zero-shot transfer. The questions below concern decentralized execution, the first-order correction, the RKHS update, and Theorem 4.1.
W1 & Q1: Decentralized execution
The population embedding used at execution is constructed from observation encodings. Section 3.1 writes the general form xj=eψ(sj,aj), because a mean-field statistic can average actions, states, or joint state–action pairs [1–3]. In the reported experiments the encoder is instantiated without action dependence, xj=eψ(sj), on MPE, MAgent2, and SISL. No neighbor action is read at execution. The averaged object is an observation-latent embedding measure. The MFRL baseline aggregates neighbor observations in the same way, so that comparison is matched at the observation level.
This is also what keeps the local Q consistent with the value decomposition. Each agent evaluates its local Q from its own observation and the observation embedding of the other agents. The greedy local action does not require any other agent's action. Training uses a VDN sum of these local Q-functions, so the individual greedy choice is the choice that maximizes the decomposed value. We will state the executed inputs, and this relation to the decomposition, in the revision.
[1] Yang, Yaodong, et al. "Mean field multi-agent reinforcement learning." International Conference on Machine Learning. PMLR, 2018.
[2] Geist, Matthieu, et al. "Concave utility reinforcement learning: The mean-field game viewpoint." arXiv preprint arXiv:2106.03787 (2021).
[3] Hu, Anran. Learning in Mean-Field Games and Continuous-Time Stochastic Control Problems. Diss. University of California, Berkeley, 2022.