acceptodds
Under review as a conference paper at ICLR 2027

SAW-v2: Supervised Dual-Stream State Adaptive Weight for High-Dimensional Multi-Agent Reinforcement Learning

10 views

Abstract

Multi-agent reinforcement learning (MARL) has advanced significantly in complex robotic systems, enabling decentralized agents to acquire coordinated behaviors for locomotion, dexterous manipulation, and cooperative decision-making under the centralized training and decentralized execution (CTDE). However, existing MARL algorithms often process heterogeneous and high-dimensional observations in a largely uniform manner, which can dilute task relevant state information, overemphasize redundant or noisy features, and make actor-critic optimization less efficient and less stable in strongly coupled multi-agent environments. To address this limitation, we propose SAW-v2, a state adaptive weighting framework that learns stable state-importance weights for actor side representations. By identifying informative observation components and suppressing less useful ones, SAW-v2 provides a lightweight plug-in module for improving policy learning. 108 groups experiment on MAMuJoCo, DexHands, SMACv2, and MPE demonstrate that SAW-v2 consistently improves representative MARL baselines across both continuous control and discrete cooperative benchmarks, achieving average 12.3% performance gains and showing stronger robustness in heterogeneous and high-dimensional coordination tasks.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, state adaptive weighting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2HNROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SAW-v2 is a sample-weighting module used when training multi-agent actor-critic methods. A small network looks at the observation and predicts a positive weight for each sample. The weights are normalized over the batch, detached, and used to scale the actor loss. The weight network is trained separately to match two targets: the size of the actor loss and, when twin critics exist, the disagreement between the two critics. The two targets are combined by taking the larger one. The experiments add the module to MAPPO, HAPPO and HASAC on 18 tasks from MAMuJoCo, DexHands, MPE and SMACv2, with three seeds each. The mean score improves in 48 of 54 comparisons, with an average relative gain of 12.3%.

Critical strengths

S1. Broad evidence that the full module helps: Table 1 covers three backbones and four benchmark families, with three seeds and full learning curves. I recomputed the printed scores and confirmed 48/54 positive differences and a mean relative gain of about 12.33%. Several gains in continuous control and manipulation are large, and the six negative cases are reported openly. This makes the method worth considering in practice.

S2. Easy to plug in, with a clean split of gradients: Eqs. (2)-(9) and Algorithms 1-2 keep the supervised weight learning separate from the actor update. Detaching stops the actor loss from sending gradients into the weight network, and the masked normalization keeps the average weight fixed. The network is only used during training (§3.5), so it adds no cost when choosing actions. These are useful properties, even without a convergence guarantee.

Critical weaknesses

W1. The experiments show that the full package helps, but not which part of it is new and useful: The paper motivates SAW-v2 as a fix for the collapse of jointly learned weights (introduction, §3.1), and highlights the dual-stream supervision (§3.4). But all experiments only compare each backbone with and without the full module. Three comparisons are missing:

  • the earlier SAW, or a matched jointly trained weighting;
  • policy-only versus dual-stream supervision on the same twin-critic backbone;
  • a simple control that uses the detached targets directly as weights, to show that the learned predictor is needed.

Algorithm 2 falls back to policy-only supervision when there are no twin critics, so gains across different backbones cannot show what the critic stream adds. A small study on a few tasks, focused on the first two comparisons, would address this. It is not necessary to show that every part is essential.

W2. Some details of the tested variants are missing:

  • Eq. (7) uses the per-sample actor loss as its target, but §3.5 does not say what this loss is for each backbone, for example whether entropy terms are included. This matters because the loss size sets the target weights.
  • Algorithm 2 mentions optional variance and temporal regularizers, but gives no formulas or coefficients, and does not say which experiments use them.
  • The appendix says the common settings in Table 4 apply to only 14 of the 18 HASAC variants (p. 13), but does not say what the other four use. It also promises task-specific choices from the ranges in Table 3, but the task sections only give task descriptions and results.

Because of this, I cannot rebuild exactly what was run, or tell which change produced the gains. This does not mean that the baselines were mismatched or that the experiments were not done.

Questions for authors

Q1. On a representative twin-critic task, how does SAW-v2 compare with a matched SAW or joint-weighting baseline? And what does critic disagreement add over policy-only supervision? Please use the same backbone and budget. Using the targets directly as weights, or a plot of the weight distribution during training, would also help explain the predictor and the collapse story. A small, focused study is enough.

Q2. What exact per-sample loss is used in Eq. (7) for each backbone? Which configurations use both streams? Were the optional regularizers turned on? Please list the four HASAC exceptions and the exact per-task settings, and confirm which backbone settings are kept the same in each baseline vs SAW-v2 pair.

Reviewer_HynpOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on the weight degradation problem caused by joint optimization in high-dimensional, heterogeneous MARL and proposes a lightweight state adaptive weighting framework SAW-v2. SAW-v2 learns observation-dependent importance weights through detached dual-stream supervision derived from policy difficulty and critic uncertainty. Experiments show that the proposed method achieves higher win rate compared with several baselines.

Critical strengths
  1. The dual-stream supervision mechanism leverages existing training signals without introducing additional environment interactions, offering a lightweight approach to state weighting.
  2. The experimental results demonstrate improvements across a broad range of tasks.
Critical weaknesses
  1. The experimental validation of key design choices is insufficient. The paper does not compare SAW-v2 directly with the original SAW, despite motivating the method as an improvement over it. Moreover, the paper lacks dedicated ablation studies of some key design choices, i.e., the BMN, the max-based fusion strategy, the update interval K.
  2. The method uses actor-loss magnitude as a proxy for policy difficulty, but the reliability of actor-loss magnitude as a supervision signal is unclear. A large actor loss does not necessarily indicate greater policy difficulty or higher learning value, as it may also result from noisy estimates or optimization fluctuations.
Questions for authors
  1. How does SAW-v2 perform compared with the original SAW?
  2. Is actor-loss magnitude a reliable proxy for policy difficulty?
  3. Which components contribute most to the effectiveness of SAW-v2? How sensitive are the results to the choice of fusion strategy and weight normalization?
Reviewer_xKZ2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SAW-v2, a state-dependent sample-weighting module for multi-agent reinforcement learning. A weight network predicts a scalar weight from each sample’s local observation. The weights are normalized by their batch mean and detached before being applied to the per-sample actor losses. The weight network is trained separately using targets derived from the absolute policy-loss magnitude and, when twin critics are available, their value disagreement. The two signals are scaled and combined through an element-wise maximum. The authors integrate the module into MAPPO, HAPPO, and HASAC and evaluate it on 18 tasks across MAMuJoCo, DexHands, MPE, and SMACv2, using three random seeds per configuration. The paper reports improvements in 48 of 54 paired comparisons and an average relative return improvement of 12.3%, with larger gains primarily observed in continuous-control and dexterous-manipulation tasks.

Critical strengths
  1. The method is relatively simple and has potential practical value. Separating weight learning from policy optimization prevents the weight network from directly manipulating the policy objective through its weights. Batch-mean normalization also helps reduce changes in overall loss scaling. Integration primarily requires access to the underlying algorithm's per-sample actor losses, and the weight network is not needed during execution. These properties make the approach a potentially useful training enhancement with limited integration requirements.
  2. The evaluation is broad and provides evidence for the complete method's empirical effectiveness. The experiments cover four benchmark families and three backbone algorithms, including continuous control, dexterous manipulation, cooperative communication, and discrete-action tasks. Most paired comparisons show positive gains, suggesting that the benefits are not confined to a single algorithm or environment. Several improvements on complex manipulation tasks are substantial.
Critical weaknesses
  1. The abstract and introduction emphasize identifying informative observation components and suppressing redundant features. However, the method in Section 3 assigns one scalar weight to each sample's actor loss rather than explicitly selecting or weighting observation dimensions. Consequently, improved performance on high-dimensional tasks does not by itself establish that redundant observation components are being suppressed. Avoiding weight degeneration under joint optimization is a more directly relevant motivation, but the paper does not demonstrate how this degeneration occurs in the studied settings or how independent supervision alleviates it.
  2. The potential novelty lies in combining independent supervision, policy-loss signals, critic disagreement, and normalization into a MARL training module. Such a combination can be valuable, but the experiments primarily compare the complete module against the unmodified backbones. They do not establish its advantages over the original SAW, directly weighting losses using the constructed supervision targets, or single-stream variants. It is therefore unclear whether the additional weight network improves upon simpler direct weighting, or whether dual-stream fusion and the maximum operator provide distinct benefits. In particular, configurations without twin critics use only the policy stream, so the overall gains cannot uniformly be attributed to the dual-stream design. This evidence gap limits the assessment of the method’s novelty and contribution.
Questions for authors
  1. Does the method always produce one scalar weight per training sample? If so, what evidence supports the claims about identifying observation components and suppressing redundant features? Please also specify which backbone configurations actually use twin-critic supervision and the exact terms included in the per-sample policy loss for each algorithm.
  2. Could you compare the original SAW, direct weighting using the constructed supervision targets, and weighting through the learned network on representative tasks? In particular, what benefit does the additional weighting network provide over direct weighting? This comparison would materially affect the assessment of the contribution.

After careful discussions, I think the author have largely addressed my concerns. Thanks for the efforts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.