acceptodds
Withdrawn from ICLR 2027

Steady, as she goes: Rethinking Observations and Rewards for Stability Regulation in Cooperative Multi-Agent Systems

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continuing multi-agent systems must maintain stable operation and recover safely and promptly from disturbances. Previous works mainly formulate it based on instantaneous observations and tracking-error-type rewards, which may not be appropriate. This mismatch arises from the information available to the learner and the recovery behaviors favored by the reward. Without explicit disturbance context, value learning can conflate external effects with those of agents' actions. Missing temporal information can hinder action selection when the same instantaneous deviation occurs during recovery or renewed departure. Tracking-error style deviation penalties can also favor slower recovery over a safe, faster return that involves temporary overshoot. We study observation and reward grounding for continuing multi-agent regulation to align learning with stable-operation and recovery requirements. We propose an observation design that combines disturbance context available before action selection with windowed deviation and its change, providing information about external conditions and recent evolution. We also propose a recovery-oriented reward that uses time spent outside the normal operating region as the base cost and a potential difference as auxiliary recovery feedback, with safety and continued task performance as requirements. We organize the evaluation around MAPPO in three environments: cooperative payload transport, power-grid voltage control, and traffic signal control, with comparisons that isolate the roles of observation and reward design. The evaluation examines recovery success and time alongside normal operation, failures, and task performance to assess whether improved recovery preserves safe and effective operation.

Primary area:
reinforcement learning
Keywords:
Stability, Reinfocement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mnn8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies stability regulation in cooperative MARL under external disturbances. They propose STAB, which combines three parts including (1) incorporating disturbance-related information into policy and value-function inputs, (2) using sliding-window statistics to characterize historical deviations and their trends, and (3) designing a recovery-oriented reward based on the duration of abnormal operation and potential-based reward shaping.

The proposed approach is evaluated with MAPPO across three environments, including MultiWalker, power-grid voltage control, and traffic signal control. The experimental results show improvements in the reported unstable rate compared with the considered baselines.

Critical strengths
  1. It is a meaningful problem setting. Improving the robustness and recovery capability of cooperative MARL systems under external disturbances is an important practical problem. The paper also raises a reasonable concern that conventional tracking-error objectives may not always align with the goal of rapid recovery.

  2. Simple and practical design (but I think it is just three engineering tricks ). The proposed components are relatively straightforward to implement and can be integrated into existing MARL frameworks without substantial architectural changes. This may be useful from an engineering perspective.

  3. The experiments cover three different cooperative control scenarios, and the proposed method consistently improves the reported unstable rate over the considered baselines. The observation ablations also provide some evidence for the usefulness of disturbance information and temporal features.

Critical weaknesses
  1. From a methodological perspective, STAB appears to be primarily a combination of several relatively standard design choices including incorporating additional disturbance information, using sliding-window statistics to summarize historical dynamics, and introducing a recovery-oriented reward with potential-based shaping. Each component is intuitive, but the paper does not sufficiently clarify what fundamentally new methodological insight emerges from their combination.

  2. The experimental design makes it difficult to identify which components are actually responsible for the reported improvements. Although the paper includes ablations of disturbance information and temporal features, it does not independently evaluate the effectiveness of the proposed recovery-oriented reward. In particular, a natural baseline would be MAPPO with the original observations but the proposed reward function. Another important comparison would retain the proposed observation design while using the original environment reward. Without these comparisons, it is unclear whether the performance improvements primarily come from the additional observations, the reward redesign, or their combination. This is particularly problematic because reward design is presented as one of the central contributions of the paper.

  3. The experimental evaluation is not sufficiently comprehensive given the relatively incremental nature of the proposed methodology. The comparisons are primarily based on MAPPO variants, and the paper lacks stronger alternatives involving recurrent policies, different temporal representations, or alternative reward and cost formulations.

Questions for authors

See in the Weaknesses

Reviewer_pyKhOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes STAB, an observation and reward design for stability regulation in cooperative multi-agent systems under disturbances. The method adds measured disturbance information and windowed deviation statistics to the observations, and replaces the original reward with a cost on time spent outside the normal operating region plus potential-based shaping. Experiments with MAPPO on MultiWalker, PowerGrid, and TrafficSignal report lower unstable rates than the baselines.

Critical strengths

None

Critical weaknesses
  1. No theoretical contribution. The method only modifies the observations and the reward of MAPPO. The components are used as they are: disturbance feedforward (µ_t), an RMS envelope of the deviation (D_t, ΔD_t), and potential-based shaping (Ng et al., 1999), whose policy invariance Appendix B only restates. The paper does not analyze why or when this design improves regulation, and "stability" is used without a control-theoretic definition; it refers only to the empirical fraction of time within the normal region.
  2. The objective equals the metric, and safety and task performance are missing. The UR thresholds equal the normal-region tolerances (Table 4), so STAB directly minimizes UR. Safety and task performance are stated as requirements, but they are neither in the objective (Appendix B) nor evaluated. No safety limit is defined; in PowerGrid, the normal region is the admissible voltage band itself, so excursions beyond it are voltage violations, not "safe overshoot". The reward also drops the forward-progress term (MultiWalker) and the reactive-power cost (PowerGrid). The abstract promises evaluation of recovery success and time, failures, and task performance, but only UR is reported. Addressing this requires redesigning the objective and the evaluation, not only additional experiments.
  3. The comparison is not fair. The baselines use the original rewards and observations (Sec. 6; Table 2, last row), so they neither optimize UR nor receive the measured disturbance µ_t. The ablation never changes the reward alone. The gains therefore cannot be attributed to the proposed observations.
  4. Reproducibility. Tables 5–6 list only MAPPO hyperparameters; the window length, the potential scale, the definition of µ_t, the disturbance protocol, and the number of training seeds are not reported. The ±1 std ranges in Table 7 overlap in most cells, and what the std is computed over is not stated, so the significance of several ablation differences (notably frame stack vs. STAB) cannot be judged.
Questions for authors
  1. Could the authors clarify what "stability" means in this work, and provide any analysis, even in a simplified setting such as the spring example in Fig. 1, of why the proposed observations and reward improve recovery?
  2. Is µ_t given to MAPPO Vanilla and MAPPO + AdvMARL?
  3. What values of the window length and the potential scale are used? Over what are the standard deviations computed, and how many training runs were performed?
Reviewer_gXcnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper considers training for multi-agent reinforcement learning with considerations for making the system more observable and safe. Unfortunately I found the motivating example, the problem setup, and the proposed solution, very limited in scope.

Critical strengths

With sincere apologies, I could not locate the contributions.

Critical weaknesses

The motivating example is very limited in scope. This is a spring mass damper system, while the paper studies multi-agent systems, so the connection is unclear. Again, regarding the motivating example, the problems that are being raised are quite well known in standard undergraduate level control systems. This is a second order system and requires known both states (x and xdot) for full state feedback control as well as for safe control. If one of the states is not available, then an observer needs to be built, which requires tracking a sequence of observations. SImilarly, if external forces are not known, they need to be estimated from observers or measured. The point about different recoveries is unclear.

Several aspects of the problem formulation are unclear. It is unclear what yt means and how it depends on the states.

'Observable through local observations' is unclear. Is this about observability in the control system sense?

The problem aim is unclear, the aim is to stay within the normal region N but also it is allowed to depart outside of the normal region for a bit and then repeat? So what exactly is an acceptable design or an optimal design?

Section 5, the development of the approach, contains limited novelty. Obviously, if the external disturbance is not present in the state, it can be added as an extra measurement.

The idea of having a window of time and observe the behavior over a window of time is standard, like the observer design I mentioned above. It is also standard practice in RL from images for example to include a window of past images as the observation.

The point about different ways of recovery, and providing two different sequences with different square tracking performance is quite unclear. I am not sure what the authors are aiming to motivate. That perhaps the square tracking performance is not always desirable and a different reward function can be constructed. I have no objection, but I am not sure what is being achieved here. Again, from a control system perspective, one could choose the objective that they care about in their application and optimize for it, no one would complain about this.

Questions for authors

I would recommend the authors take into my comments above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.