- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies zero-shot generalization in offline multi-task cooperative MARL, where hierarchical skill-based methods are currently dominant. The authors identify a "hierarchical value gap": in existing methods (e.g., ODIS, HiSSD), value supervision stops at the skill-selection level, while the low-level decentralized executors are trained by uniform conditional behavior cloning — so successful and unsuccessful realizations of the same skill are imitated with equal weight, and selecting the right skill does not guarantee effective joint execution. To close this gap, the paper proposes STEP (Skill-conditioned Team-value Execution Projection), a training-time-only framework with three components: (1) an in-sample expectile team-value estimator (attention-mixed per-agent utilities) that produces one-step TD residuals from logged transitions; (2) a team-residual execution projection that converts standardized residuals into weights on the executors' imitation losses, favoring higher-value realizations of each skill; and (3) execution-consistent consequence conditioning, which injects the decoder-driving skill code into the backbone's consequence predictor. The method is formalized via an execution residual (Definition 1) with supporting propositions on the reweighted execution target. Validation is conducted on SMAC: the primary benchmark is Marine-Hard (3 source maps, 9 unseen maps, four data qualities), with Marine-Easy and Stalker–Zealot as secondary checks, using a reproduced HiSSD backbone as the direct paired control plus ODIS, UPDeT-m, and BC baselines. STEP improves unseen-map win rates over its backbone by +5.42 and +4.79 points on medium-expert and medium-replay data while matching it on expert data; ablations attribute the gains to residual-weighted execution, and a tested retrieval extension yields no additional benefit. The paper also includes holdout-map checks and final-checkpoint (selection-free) results.
Critical strengths
-
Well-diagnosed and clearly formalized problem. The "hierarchical value gap" is a genuine and under-appreciated limitation of the dominant skill-learning paradigm in offline multi-task MARL: skill-level objectives (value, contrastive, or dynamics) shape which skill is selected but say nothing about the quality of execution within a skill, and uniform conditional behavior cloning then averages over good and bad realizations. The execution-residual formulation (Definition 1) precisely characterizes the residual discrepancy that skill selection alone cannot remove, and Propositions 1–2 honestly delimit what reweighting does and does not guarantee. This conceptual clarity — identifying that the "last mile" of skill transfer lies in executor training rather than skill discovery — is the paper's main intellectual contribution and is consequential because it is general: any hierarchical skill method with a behavior-cloned executor could in principle adopt this supervision.
-
Rigorous, well-controlled empirical protocol with convincing causal evidence. The evaluation is unusually careful for this literature: paired seeds with the reproduced HiSSD backbone isolate STEP's effect; shuffled-weight controls (C-shuf) demonstrate that the association between residuals and logged actions matters rather than the weighting procedure per se; graded weighting is shown to beat hard return filtering; final-checkpoint (selection-free) re-evaluation confirms the gains are not an artifact of best-over-training reporting; and never-inspected holdout maps rule out development-loop selection pressure. The mechanism analysis (weight deciles tracking realized returns across expert/medium provenance, gains concentrated on maps where the backbone has headroom) aligns the empirical evidence with the theory. This evidentiary standard meaningfully raises confidence that the reported gains are real and attributable to the proposed mechanism.
Critical weaknesses
-
The skill signal itself is under-examined and largely inherited without justification. STEP is deliberately agnostic to how skills are discovered — it wraps the HiSSD backbone and adds execution-time value supervision. However, the paper provides no clarification of the skill discovery process on which everything depends: how many skills the backbone learns, whether the skill space is discrete/continuous, how skill semantics relate to the reweighting, or whether the quality of skill discovery limits the ceiling of execution projection. More importantly, the residual weighting operates on the team-level TD residual shared across all agents, which is a coarse and ambiguous signal: it carries no individual credit assignment, and as the paper itself notes (Appendix G), per-step residual-return correlations are weak and mixed in sign — the diagnostic supports only episode-level selection, not identification of which joint actions within an episode were good. There is also no guarantee that a high residual corresponds to skill-consistent execution rather than lucky stochastic outcomes; Proposition 2 is explicitly conditional on realized outcomes and does not imply improved environment return. Because the entire framework rests on this single scalar signal per transition, the absence of any analysis of the skill–residual interaction (e.g., do weights separate realizations within a skill, as claimed, or merely across tasks/returns?) leaves the central mechanism only partially validated.
-
Limited behavioral/visualization evidence and questionable value objective design. Despite the paper's interpretability-adjacent framing, there is no visualization or qualitative analysis of the learned skills or of inter-agent correspondence — e.g., which agents' actions drive high-residual episodes, whether reweighting sharpens coordination patterns (focus fire, kiting) on unseen maps, or how the consequence conditioning changes the skill representations. Given that the method's premise is that executors conflate good and bad coordinated behaviors, showing that STEP's policies actually exhibit cleaner coordinated behaviors would be direct evidence, and its absence weakens the claim. Regarding the value objective: the expectile team value (τ_e = 0.7, fixed without sensitivity analysis) is admittedly not an advantage estimate, and the design has observable costs — on expert data the weights carry no signal and reweighting becomes pure variance, producing a measurable cost at the final checkpoint (−2.64, CI excluding zero on HE), which the paper acknowledges but does not address with any mechanism to detect or suppress no-signal regimes. The consequence-conditioning component (D) also shows inconsistent, mostly non-significant effects (δ_forward: +2.46 on HE, −0.93 on HMR, CIs crossing zero), and its inclusion in the "full STEP" obscures that the method is effectively a single-idea contribution whose second and third components are not clearly pulling their weight.
-
Attribution of component contributions is unclear, the evaluation is confined to a single benchmark family (SMACv1), and the margins over the strongest existing baseline are modest and frequently not statistically significant. First, the ablation chain (Table 12) shows that residual weighting (C) accounts for essentially all of the observed gains, while the value estimator alone (B) is neutral-to-harmful and consequence conditioning (D) is inconsistent and mostly insignificant (δ_forward: +2.46 on HE, −0.93 on HMR, all CIs crossing zero). The method is therefore effectively a single-component contribution (residual-weighted BC) wrapped in a framework whose other two components neither help nor are cleanly attributed — the "three-component" framing overstates the substance. Second, all evidence comes from SMAC/SMACv1 micromanagement maps; no experiment on any other environment family (e.g., multi-robot control, Google Research Football, Melting Pot, or non-StarCraft domains) is provided, so the generality of the "hierarchical value gap" claim — which the paper presents as a fundamental limitation of the paradigm — rests entirely on one benchmark. Third, and most consequential for my assessment: the headline gains (+5.42/+4.79 average win-rate points on HME/HMR) are relative to a self-reproduced HiSSD backbone whose training protocol deliberately differs from both the original paper and its official repository (optimizer, learning rate, training steps — Table 4), and recent stronger baselines such as BiKT are not evaluated at all. Per-map paired differences are noisy: on HM the weighting gain is +1.06 ± 5.08 and on HE the method is worse than its own backbone at the final checkpoint (−2.64, CI excluding zero); the holdout-map evaluation (Table 13), which is the least selection-biased evidence in the paper, shows differences that mostly scatter around zero with CIs crossing zero in 8 of 10 cells. Taken together — one real component, one benchmark family, no comparison against the strongest published baselines under their official protocols, and frequently insignificant margins — the empirical support is not sufficient to justify accepting a method whose theoretical guarantees are explicitly conditional and limited.
========
After the rebuttal, I maintain my score and the reasons are shown below:
- Insufficient experimental evidence to support the paper's broad claims.
My primary concern is the substantial gap between the paper's broad claims and its limited experimental validation. STEP is presented as a general framework, yet it is evaluated exclusively on HiSSD, making it unclear whether the improvements stem from STEP itself or the strong underlying backbone.
Furthermore, all experiments are restricted to the standard SMAC benchmark used in ODIS[1], without considering SMACv2, MAMuJoCo, or Cooperative Navigation, despite their adoption in related works such as HiSSD, BiKT[2], and STAIRS-Former[3]. Notably, even the earlier ODIS paper evaluated Cooperative Navigation. The comparisons also omit stronger recent methods, particularly BiKT and STAIRS-Former.
As skill-discovery-based approaches, ODIS and BiKT would be natural alternative backbones for evaluating STEP's generality. Without cross-backbone and cross-benchmark validation, the claimed general applicability of STEP remains insufficiently supported.
- Underestimated computational cost relative to existing methods.
In their response to Reviewer GtAy, the authors report only 1.16×–1.18× training overhead and approximately 7% additional parameters relative to HiSSD. However, these seemingly modest figures overlook the substantial computational cost of HiSSD itself.
According to my measurements on Marine-Easy-Expert, ODIS has approximately 138k parameters, compared with 679k for HiSSD. Thus, STEP's additional parameters alone amount to roughly 34% of the entire ODIS model. Moreover, HiSSD already requires approximately 2.1× the training time of ODIS (including ODIS pretraining), suggesting that STEP could cost approximately 2.4–2.5× as much as ODIS. Similar findings regarding the overhead of HiSSD are also reported in Appendix H of STAIRS-Former[3].
These computational costs also raise concerns about the scalability of both HiSSD and STEP. In particular, it remains unclear whether these methods can efficiently scale to larger multi-agent systems, such as scenarios involving more than 25 agents, under a reasonable GPU memory budget.
Reporting efficiency solely relative to an already expensive backbone is therefore insufficient. A fair assessment should include absolute computational costs and comparisons with other competitive methods.
- Unfair evaluation protocols and incomplete baseline reporting.
Despite providing additional end-of-training results for HiSSD and STEP, the authors have not resolved a fundamental inconsistency in their comparisons. The BC-best, UPDeT-m, and ODIS results in Tables 1 and 7 appear to be directly taken from the original ODIS paper, which reports end-of-training performance, while HiSSD and STEP are evaluated using best-over-training checkpoints. This protocol mismatch unfairly favors the latter methods and undermines the validity of the comparisons.
The reporting is also incomplete: Table 6 in Appendix E omits all baselines except HiSSD and STEP on Marine-Easy, while Table 7 omits the unseen-task average performance of the other baselines.
These issues substantially weaken the credibility of the experimental conclusions. The authors should reproduce existing methods under a consistent evaluation protocol and provide complete baseline comparisons across all task sets.
References
[1] Zhang, F. et al., Discovering Generalizable Multi-agent Coordination Skills from Multi-task Offline Data. ICLR 2023.
[2] Zhang, J. et al., Bi-Level Knowledge Transfer for Multi-Task Multi-Agent Reinforcement Learning. NeurIPS 2025.
[3] Jeon, J. et al., STAIRS-Former: Spatio-Temporal Attention with Interleaved Recursive Structure TransFormer for Offline Mulit-task Multi-agent Reinforcement Learning. ICLR 2026.
Questions for authors
-
On the skill signal: Does the residual weighting actually separate realizations within a skill? Please provide an analysis conditioning on the skill code z (e.g., distribution of weights within each skill cluster, or within-skill covariance Cov(w, δ | z) per skill), rather than the current episode-level analysis. Without this, the "within-skill quality signal" claim (as opposed to a global return filter, which %BC already partially achieves) is not fully established.
-
On the value objective: Since the paper admits the weights add pure variance on expert data (final-checkpoint HE: −2.64, 95% CI [−5.10, −0.40]), is there a principled way to gate or regularize the weighting when residual dispersion is uninformative — e.g., scaling α by measured residual variance or disabling weighting below a dispersion threshold? Also, why is τ_e = 0.7 chosen, and how sensitive are the results to τ_e and α?
-
On component contribution: The ablation shows residual weighting (C) drives essentially all gains while consequence conditioning (D) has inconsistent, mostly insignificant effects. Would the simpler C configuration be the recommended method? If D is retained in "full STEP," what evidence justifies it beyond a small effect on HME?
-
On skill discovery clarification: Given that STEP inherits the entire skill-discovery pipeline from the reproduced HiSSD backbone, could you clarify what happens with alternative backbones (e.g., ODIS's discrete skills or different skill dimensionalities)? Does the magnitude of the execution gap, and hence STEP's benefit, depend on skill granularity? This would clarify the generality of the "hierarchical value gap" claim beyond one backbone implementation.
-
On visualization (suggestion for revision rather than a blocking issue): Would you consider adding qualitative analysis — e.g., replay visualizations or skill-code statistics on unseen maps — showing that STEP's executors produce visibly different (more coordinated) joint behaviors than the backbone for the same skill? This would directly substantiate the execution-quality claim that the current win-rate and weight-diagnostic analyses only support indirectly.
-
On significance and baselines: The holdout-map evaluation (Table 13) — the only evaluation free of checkpoint selection — shows paired differences whose CIs cross zero in 8 of 10 cells, and on expert data STEP is significantly worse than the backbone at the final checkpoint. Given this, what evidence beyond average win-rate deltas supports the claim of a robust improvement? Additionally, could you report results against BiKT (the strongest recent hierarchical skill-transfer baseline) using its official implementation and protocol, and provide per-map significance for the HME/HMR gains rather than only regime-level averages?
5 comments
Response to Reviewer scyo (Part 2/3)
Q1 / Weakness 1: What is the skill signal, and do the weights distinguish realizations within a skill?
The skill space. In the HiSSD backbone a skill is a continuous 64-dimensional code per agent and step. Vector quantization is not enabled, so there is no fixed list of skills. We measured the executed codes on HE, HME and HMR (five seeds, three source maps). Their effective dimension is moderate, with 11 to 24 principal components explaining 90% of the variance. They show no clear cluster structure, since the silhouette score is 0.12 to 0.42 and rises with K without a preferred K, whereas a synthetic set with eight clusters scores 0.60. They also drift smoothly over time, with a median cosine similarity of 0.76 to 0.95 between consecutive steps, compared with 0.20 to 0.47 for random pairs. The number of skills thus has no fixed answer, and our skill-group analyses use k-means with K = 16 as a working discretization (Appendix J.2).
Within a skill, or only across skills and returns? The weights vary almost entirely inside skill groups. Across the 45 cells, only 0.1% to 2.6% of the weight variance lies between the K = 16 groups, and at most 3.4% for K = 8 or 32. We then ran two interventions with five paired seeds.
| HME | HMR | |
|---|---|---|
| STEP − skill-group mean weights | +5.04 [+3.65, +6.42] | +2.82 [+1.54, +4.42] |
| STEP − return-to-go weights | +1.81 [+0.63, +2.99] | +3.46 [+2.25, +4.75] |
Skill-group mean weights keep the differences between skill groups but remove those within a group. This removes most of the gain on HME, where the result returns to the backbone level (63.15 vs. 62.78), and about 60% of it on HMR, so the gain comes mainly from within-skill variation rather than from reweighting across skills. Return-to-go weights are a soft form of the global return filter you mention. They are weaker than the TD residual in both regimes, as is the hard filter %BC (Table 11), so the weights do more than rank episodes by return. Return-to-go also favors early steps of each episode, since SMAC rewards are non-negative, whereas the residual compares each step with the learned baseline.
On , is monotone in , so the covariance is non-negative whenever varies within , and the informative question is what drives this variation. The variance decomposition shows that it is the dominant part of the weights, and the interventions show that it is needed for the gain. We also conditioned on a context cluster in addition to the skill cluster and tested whether the recorded action of each agent explains the weights. In this finer test the individual action alone did not reliably explain the weights. We therefore describe the mechanism at the level the interventions support, namely that STEP exploits within-skill differences in realized execution quality, which can reflect both the actions taken and the states they lead to.
Q2 / Weakness 2: The value target, its sensitivity, and expert data.
Sensitivity. We varied and one at a time on HMR, with seeds 1 to 5 paired with the backbone.
| Setting | Unseen-map win rate (%) | Gain vs. backbone |
|---|---|---|
| HiSSD backbone | 52.43 | (reference) |
| 57.13 | +4.70 | |
| , (default) | 57.22 | +4.79 |
| 55.88 | +3.45 | |
| 58.24 | +5.81 | |
| 55.22 | +2.79 |
Every setting has a higher mean than the backbone, and 24 of 25 seed-level differences are positive. Among the values the default 0.7 has the highest best-over-training mean, with −0.10 for 0.5 and −1.35 for 0.9. For , sharpening to 0.5 gives no reliable change (+1.01, 95% CI [−0.39, +2.42]), while flattening to 2 lowers performance (−2.00) but stays above the backbone. At the end of training also has the smallest seed-to-seed spread among the three settings, which makes the default a stable choice.
Expert data and gating. The diagnosis is accurate. On near-saturated expert data the residuals have little to rank, and because Eq. 9 standardizes residuals within each batch, the weights keep the same relative spread even when the raw residual spread is small. This is consistent with the end-of-training difference on HE (−2.64), while under the main protocol STEP is on par there (−0.56, CI includes zero). A new sweep over the expert share of the data (Appendix J.7) points the same way. Under the best-over-training rule, the gain is +1.96 at 0% (HM), +2.95 at 25%, +5.42 at 50% (HME), +1.11 at 75% and −0.56 at 100% (HE), largest at 50% and absent on pure expert data. Your suggestion of gating on the raw dispersion is a sound remedy, and we will develop and evaluate it. The revised abstract states this result under the best-over-training protocol.
Response to Reviewer scyo (Part 3/3)
Q3 / Weakness 3: Component attribution.
We agree that residual-weighted execution is the core of STEP, and the revision makes this explicit. To answer your question directly, residual-weighted execution (C) alone obtains the win-rate gains on mixed-quality data and can be used on its own. It improves on the backbone by +3.66 on HME and +5.72 on HMR (C − A), and on the value estimator alone by +5.73 and +4.14 (C − B). The value estimator (B) is not meant to improve performance by itself, since it supplies the residual that C uses, so B and C form one mechanism. Consequence conditioning (D) gives the executed skill code an explicit prediction signal, which lowers prediction error on HMR in all five seeds on the two unseen maps with offline data (5m and 10m), while its effect on win rate is within seed-to-seed variation. The abstract now states this attribution explicitly, as the contributions and conclusion already did. Residual-weighted execution changes only how the executor is trained, and the deployed architecture, inputs and inference cost are unchanged.
Q4: Other backbones and skill granularity.
Portability to other hierarchical backbones and a controlled study of skill granularity are directions we will pursue, and the Limitations section states that our evidence is on HiSSD.
Benchmark family. Our evidence spans three SMAC task sets with different unit types and team compositions, including the heterogeneous Stalker–Zealot set. SMAC is the standard benchmark for offline multi-task MARL transfer and is used by HiSSD. The backbone's MAMuJoCo data and code have not been released, and we plan to extend STEP to continuous control such as MAMuJoCo.
Q5: Qualitative analysis.
We have added a behavioral analysis on two unseen maps (Appendix J.3). We re-ran the medium-replay checkpoints of STEP and the backbone on 7m_vs_8m and 4m in the same evaluation mode as during training (five seeds, 32 episodes per seed and map, 640 episodes in total) and recorded every action. STEP concentrates fire modestly more (the share of attacking agents that target the most-attacked enemy is 0.694 vs. 0.666 on 7m_vs_8m and 0.863 vs. 0.836 on 4m) and moves more (the fraction of movement actions is 0.312 vs. 0.293 and 0.256 vs. 0.237), and it wins more of these episodes (20.0% vs. 12.5% and 85.0% vs. 72.5%). The appendix also shows the per-agent action traces of one episode per map for both methods, chosen without regard to outcome. The skill-code statistics in Part 2 are also added to the appendix.
We hope these additions, together with the general response, address the concerns that shaped your assessment.
Response to Reviewer scyo (Part 1/3)
We thank the reviewer for the detailed review. We take the score as a clear signal that the submission did not give enough evidence on several points, and we have worked to close these gaps. We read the main concerns as (a) whether the weights act within skills, (b) whether the gains are robust beyond regime averages and measured against a faithful backbone, (c) component attribution, and (d) generality. For (a) to (c) we ran new analyses, namely two weight controls with five seeds, a variance decomposition of the weights and skill-code statistics, a within-skill action analysis, a sensitivity study of τe and α with a sweep over the expert share of the data, a behavioral analysis on two unseen maps, and a per-map table. Generality (d) remains a limitation, which we address in Part 3. All results use the main-text protocol with paired seeds and are added to the revised appendix.
We begin with Q6 / Weakness 3, which you identified as most critical to your assessment. Part 2 addresses Q1 and Q2, and Part 3 addresses Q3, Q4 and Q5.
Q6 / Weakness 3: Is the improvement robust, and is the backbone comparison fair?
Beyond the regime average. The gains are consistent at the seed level. On HME and HMR all five paired seeds favor STEP (+2.36 to +7.71 on HME and +2.50 to +8.19 on HMR, Figure 2). They also hold without picking the best point of training. Besides the holdout maps, Appendix F reports every cell at the end of training (the win rate logged at 35k steps, with no maximum over training), where the gains are +4.40 [+2.22, +7.12] and +6.31 [+3.88, +8.70], again with five of five seeds positive. Per map, the mean difference is positive on 7 of 9 HME maps and 8 of 9 HMR maps. Five of the 18 map-level intervals exclude zero, and all five favor STEP (12m and 10m_vs_12m on HME, and 4m, 7m_vs_8m and 10m_vs_11m on HMR). This holds even though five seeds give wide per-map intervals and on HME the backbone is already at 99.5% to 99.9% on 4m and 5m. The full 18-cell table is added (Appendix J.8). On HM the gain is smaller (+1.96), consistent with the expert-share sweep in Part 2, where the gain is largest when the data mix expert and medium executions.
The reproduced backbone. Our backbone follows the official repository and inherits its optimizer and learning rate, on which the paper and the repository differ. The HiSSD paper trains for 30k steps and the official repository defaults to 251k, and we train all arms for 35k steps with the paper's best-over-training rule. Under this protocol the reproduced backbone is stronger than the published HiSSD results in all four regimes. Its unseen-map averages are 70.3, 51.9, 62.8 and 52.4 (HE, HM, HME and HMR), while the averages of the per-map values in Table 1 of the HiSSD paper are 61.9, 47.2, 55.8 and 48.1. STEP's gains are thus measured against a stronger backbone, not a weaker one. All arms share seeds, data, budget, optimizer and evaluation, so the paired differences isolate the effect of STEP. Section 5.1 now states this justification, with the numbers in Appendix D.
BiKT. We did not run BiKT. Reproducing it under our matched protocol was not feasible within the rebuttal period, and its reported numbers use a different protocol, so we make no claim relative to BiKT. The two methods act on different parts of the hierarchy. BiKT changes what the executor is conditioned on by reusing a learned tactic codebook, whereas STEP changes how the executor is trained, so the two could be combined. As a related check, attaching a codebook and segment library of our own design to paired STEP checkpoints adds no further gain (Table 9). Our Related Work places BiKT among other routes to skill transfer, and a matched comparison with BiKT is part of our future work.