- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper investigates the safety inconsistency between centrally optimized joint occupancy measures and independently executed local policies in offline multi-agent reinforcement learning. Through sensitivity analysis, a four-action witness theorem, and a classification of Boolean cost functions, the authors characterize the conditions and structural causes underlying the failure of safety penalties. Building on these theoretical findings, the authors propose Deployed-Cost Projection (DCP), which updates the joint occupancy measure using the gradient of deployed cost while preserving Bellman flow constraints. Experiments on random MDPs and structured tasks involving up to 64 agents validate the theoretical results, demonstrating that DCP can overcome the cost floors of conventional penalty-based methods and retain more reward at the same deployed safety cost.
Critical strengths
-
Strong theoretical characterization of safety penalty failures. Theorem 4.1 distinguishes the responses of the training target cost and the deployed cost to changes in safety multipliers. Furthermore, the four-action witness theorem, the complete classification of Boolean cost functions, and the characterization of cost floors along the penalty path identify the specific structural reasons why increasing safety penalties may fail to improve deployed safety.
-
A principled connection between theoretical analysis and algorithmic correction. DCP differentiates the deployed cost through the policy extraction operator, incorporating counterfactual credit assignment and observation-frequency ratios. The resulting update direction is then projected to preserve Bellman flow constraints. The descent analysis under approximation errors and the covariance ablation experiments further establish a theoretically grounded connection between the identified safety discrepancy and a practical correction method.
Critical weaknesses
-
Insufficient comparisons with closely related methods and limited evidence of DCP's distinct advantages. Although the mechanism-oriented experiments support the effectiveness of DCP, its advantages over existing safe offline multi-agent reinforcement learning methods remain insufficiently established. In particular, the covariance ablation suggests that preserving cross-agent correlations plays an important role in its performance gains. It remains unclear whether similar improvements could be achieved by other optimization methods incorporating comparable correlation information. Direct comparisons with closely related approaches, particularly SamaDICE and ComaDICE, would provide a more comprehensive assessment of DCP's relative effectiveness and computational efficiency.
-
Limited empirical validation in general dynamic and fully offline multi-agent settings. Although the experiments validate the proposed mechanisms, the large-scale evaluations mainly rely on highly structured, block-separable binary safety-switch tasks. Experiments involving state transitions and finite-data estimation are limited to small random MDPs with two agents. Moreover, some evaluations use true observation-frequency information for step-size normalization. These limitations leave the effectiveness, robustness, and scalability of DCP in more general and fully offline multi-agent environments insufficiently demonstrated.
Questions for authors
Q1. Comparison with related methods and the distinct advantages of DCP.
Could the authors provide direct comparisons between DCP and existing safe offline multi-agent reinforcement learning methods, particularly SamaDICE and ComaDICE, under matched datasets and deployed-cost budgets? Reporting budget satisfaction, reward performance at matched deployed costs, and computational overhead would help clarify the practical advantages of DCP. Additionally, could the authors clarify whether DCP retains its advantages over alternative optimization methods that exploit comparable cross-agent correlation information?
Q2. Empirical applicability and robustness.
Could the authors further evaluate DCP in more general dynamic multi-agent environments with nonseparable safety constraints and more complex interactions? In particular, how does DCP perform when all required quantities, including observation-frequency ratios and step-size normalization, are estimated entirely from offline data rather than relying on true environment information?
Q3. Generalizability of the theoretical results.
Could the authors further clarify the scope and limitations of the structural theoretical results? In particular, to what extent can the four-action witness theorem and Boolean cost classification provide insights into settings with continuous action spaces, state-dependent safety costs, or more general multi-agent interactions?
Awaiting Your Feedback on Our Rebuttal
Dear Reviewer VvAX,
Thank you again for your review. As the discussion period is drawing to a close, we would appreciate it if you could take a look at our rebuttal. We would be happy to clarify any points or address any remaining concerns while the discussion is still open.
Best regards,
The Authors