acceptodds
Under review as a conference paper at ICLR 2027

Safety after Decentralization: A Theory of Policy Extraction

Abstract

Offline multi-agent reinforcement learning often learns a joint distribution of states and actions, then fits a separate policy for each agent. Acting independently changes which actions occur together and which states the team visits. As a result, a stronger safety penalty can lower the training target's cost while raising the deployed policy's cost, even when optimization and policy fitting are exact. In the one-state setting with Kullback-Leibler (KL) regularization, binary actions, and binary costs, every type of cost that allows such a reversal has an example using at most four joint actions; four are sometimes necessary. We classify the costs for which a stronger penalty never raises deployed cost. We also identify when no penalty can reach a budget that an independent policy can satisfy. These findings motivate Deployed-Cost Projection (DCP), which adjusts the joint target using the gradient of deployed cost, preserves the state-flow constraints, and refits the local policies. We derive this gradient and give conditions for cost reduction when the gradient, projection, and policy fitting are approximate. In a structured safety task with up to 64 agents, DCP reaches budgets that penalty increases cannot reach. Controlled comparisons show how cross-agent action covariance helps DCP retain more reward at the same cost.

Primary area:
reinforcement learning
Keywords:
offline multi-agent reinforcement learning, safe reinforcement learning, constrained reinforcement learning, decentralized policy extraction, occupancy measures, distribution correction, multi-agent coordination, policy factorization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ENh4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates the safety inconsistency between centrally optimized joint occupancy measures and independently executed local policies in offline multi-agent reinforcement learning. Through sensitivity analysis, a four-action witness theorem, and a classification of Boolean cost functions, the authors characterize the conditions and structural causes underlying the failure of safety penalties. Building on these theoretical findings, the authors propose Deployed-Cost Projection (DCP), which updates the joint occupancy measure using the gradient of deployed cost while preserving Bellman flow constraints. Experiments on random MDPs and structured tasks involving up to 64 agents validate the theoretical results, demonstrating that DCP can overcome the cost floors of conventional penalty-based methods and retain more reward at the same deployed safety cost.

Critical strengths
  1. Strong theoretical characterization of safety penalty failures. Theorem 4.1 distinguishes the responses of the training target cost and the deployed cost to changes in safety multipliers. Furthermore, the four-action witness theorem, the complete classification of Boolean cost functions, and the characterization of cost floors along the penalty path identify the specific structural reasons why increasing safety penalties may fail to improve deployed safety.

  2. A principled connection between theoretical analysis and algorithmic correction. DCP differentiates the deployed cost through the policy extraction operator, incorporating counterfactual credit assignment and observation-frequency ratios. The resulting update direction is then projected to preserve Bellman flow constraints. The descent analysis under approximation errors and the covariance ablation experiments further establish a theoretically grounded connection between the identified safety discrepancy and a practical correction method.

Critical weaknesses
  1. Insufficient comparisons with closely related methods and limited evidence of DCP's distinct advantages. Although the mechanism-oriented experiments support the effectiveness of DCP, its advantages over existing safe offline multi-agent reinforcement learning methods remain insufficiently established. In particular, the covariance ablation suggests that preserving cross-agent correlations plays an important role in its performance gains. It remains unclear whether similar improvements could be achieved by other optimization methods incorporating comparable correlation information. Direct comparisons with closely related approaches, particularly SamaDICE and ComaDICE, would provide a more comprehensive assessment of DCP's relative effectiveness and computational efficiency.

  2. Limited empirical validation in general dynamic and fully offline multi-agent settings. Although the experiments validate the proposed mechanisms, the large-scale evaluations mainly rely on highly structured, block-separable binary safety-switch tasks. Experiments involving state transitions and finite-data estimation are limited to small random MDPs with two agents. Moreover, some evaluations use true observation-frequency information for step-size normalization. These limitations leave the effectiveness, robustness, and scalability of DCP in more general and fully offline multi-agent environments insufficiently demonstrated.

Questions for authors

Q1. Comparison with related methods and the distinct advantages of DCP.

Could the authors provide direct comparisons between DCP and existing safe offline multi-agent reinforcement learning methods, particularly SamaDICE and ComaDICE, under matched datasets and deployed-cost budgets? Reporting budget satisfaction, reward performance at matched deployed costs, and computational overhead would help clarify the practical advantages of DCP. Additionally, could the authors clarify whether DCP retains its advantages over alternative optimization methods that exploit comparable cross-agent correlation information?

Q2. Empirical applicability and robustness.

Could the authors further evaluate DCP in more general dynamic multi-agent environments with nonseparable safety constraints and more complex interactions? In particular, how does DCP perform when all required quantities, including observation-frequency ratios and step-size normalization, are estimated entirely from offline data rather than relying on true environment information?

Q3. Generalizability of the theoretical results.

Could the authors further clarify the scope and limitations of the structural theoretical results? In particular, to what extent can the four-action witness theorem and Boolean cost classification provide insights into settings with continuous action spaces, state-dependent safety costs, or more general multi-agent interactions?

Reviewer_gJeJOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies a safety gap in offline MARL: a joint training target can satisfy a cost budget, while the extracted local policies violate it. Increasing the safety penalty can raise the deployed cost or fail to reach a feasible budget. The paper analyzes when these failures happen and proposes DCP, which updates the joint target using the deployed-cost gradient and then refits the local policies.

Critical strengths
  1. The paper identifies a clear problem in the training and deployment process. The distinction between a penalty reversal and a penalty floor is useful. It also shows that these problems can happen even with exact optimization and policy fitting.

  2. The experimental controls help explain the method. The local actor baseline uses the same cost information, and DCP-prod removes cross-agent covariance. Comparing reward at the same deployed cost provides useful evidence that the covariance contributes to DCP’s reward retention.

Critical weaknesses
  1. My main concern is the gap between local cost reduction and reaching a feasible budget. DCP provides conditions for a cost-decreasing step, but it is unclear when repeated updates will actually satisfy the budget. The reward loss during correction is also not bounded. The conclusion acknowledges these limitations, but they remain important for understanding when the proposed correction will work.

  2. The fully empirical version satisfies the true budget in 95.8% of runs. The main text does not report how large the violations are in the remaining runs.

Questions for authors
  1. Can DCP get stuck above the budget even when a feasible product policy exists? What additional conditions would ensure that the updates reach the budget?

  2. Does the covariance advantage remain when agents interact across blocks or when the local policies require neural function approximation?

  3. How large are the budget violations in the unsuccessful empirical runs, and how sensitive are these results to errors in the cost evaluator?

Reviewer_VvAXOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper consider an offline multi-agent reinforcement learning setting. It consider the problem that arises when policies that are learned jointly are then executed in a distributed way by agents. The challenge is that this can make actions taken by agents in a distributed way suboptimal and increase the cost. The paper characterizes theoretically this situation, provides an algorithm for improving the situation, and provides some numerical experimentation.

Critical strengths

Overall, the paper has a complete effort around the problem, the definition, the theoretical analysis, the proposed solution, and the numerical results.

The theoretical results give a characterization of when the cost increases in this setting, and this is new from my understanding

Critical weaknesses

Overall, the paper has a few limitations.

The problem studied is expected and unavoidable. If two agents need to coordinate in order to avoid a conflict (like enter the same corridor), then in some sense there is not much they can achieve without coordination. From that point of view, the paper studies a problem that is real, but not worthwhile if in such cases there is not much we can do. In some sense, this is a negative result.

A lot of the numerical results are set up in random MDP problems, with few agents, or finite states, or one-state problems (which to be honest is not clear to me what it means), or Boolean agents (I am not sure what this means) or boolean costs. Overall, this limits the scope of the paper. It is questionable if the approach is useful for more 'real' multi-agent problems.

Also, I am wondering, from the offline data and without knowing the exact MDP model, are there cases where it is impossible to predict how the decentralized execution may fail? Like cases where there is in some sense not enough information to tell what will happen?

The lines 170-175 present a very obvious example whose scope is quite limited

Questions for authors

Please consider my comments above.

Unfortunately I find the language unclear in many places especially for technical mathematical points: "Here is agent i’s action distribution given its observation under the joint target. Each local policy reproduces this distribution, but their product need not reproduce the joint target"

"The full finite model has full row rank; when restricting the support, we retain only independent flow rows."

"says that whenever a cost allows this failure, some witness needs only four joint actions."

"Two-coordinate XOR is one when the two actions differ."

"Following the deployed-cost gradient directly could violate the state-flow constraints"

2 comments
AuthorsOpenReview ↗

Awaiting Your Feedback on Our Rebuttal

Dear Reviewer VvAX,

Thank you again for your review. As the discussion period is drawing to a close, we would appreciate it if you could take a look at our rebuttal. We would be happy to clarify any points or address any remaining concerns while the discussion is still open.

Best regards,

The Authors

Thank you, I have read the response, but I find the wording in the response still unclear. The examples that are mentioned with binary actions, are still also quite limited in scope. I would like to keep my evaluation the same

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.