acceptodds
Under review as a conference paper at ICLR 2027

EpiPlan: Prospective Epigraph-Space Planning for Safe Multi-Agent Reinforcement Learning

Abstract

Epigraph-conditioned safe reinforcement learning trains a controller across task-budget levels, but deployment typically collapses this structure to the policy at a learned critic root. In distributed epigraph control, local critic roots can be aggregated into a public team boundary, but executing only that boundary member does not compare the joint feedback trajectories induced by nearby boundary-relative choices. We introduce Prospective Epigraph-Space Planning (EpiPlan), which treats the frozen controller as a one-dimensional family of complete team policies and selects a member at deployment without retraining or optimizing joint-action sequences. From the same augmented state, EpiPlan rolls out one-sided displacements, recomputes the public root along each branch, uses frozen task and safety critics as terminal estimates, and executes the lowest-cost predicted-feasible member while retaining its displacement. We formalize this actual-receding family and its deployment epigraph; under stated approximation and margin conditions, our analysis bounds finite-realization error, selected-policy feasibility residual, and task loss in terms of continuation, calibration, and coordinate resolution. Across 18 native-eligible frozen checkpoints in six settings drawn from five cooperative tasks, EpiPlan lowers episode unsafety relative to paired public-root execution, while every environment-mean reward loss remains within a prespecified 15% tolerance. An exact-marginal permutation control that preserves each checkpoint's offset histogram but breaks state–member pairing raises expected episode unsafety by 10.38 percentage points (p=.0001), showing that the gain depends on contextual assignment rather than offset frequencies alone.

Primary area:
applications to robotics, autonomy, planning
Keywords:
safe reinforcement learning, multi-agent reinforcement learning, multi-agent planning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_jkMHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EpiPlan, a test-time planning method for frozen epigraph-conditioned multi-agent policies. Rather than executing only the public-root policy used by the underlying DEF-MARL controller, EpiPlan considers a finite set of one-sided displacements from this root. For each candidate, it simulates a finite joint continuation, uses the frozen task and safety critics for terminal estimates, and selects the lowest-cost candidate predicted to be feasible. The method requires no retraining and searches over a one-dimensional policy family rather than joint action sequences.

The paper also develops conditional approximation and selection bounds for this deployment procedure. Experiments on 18 selected DEF-MARL checkpoints across six MPE-style settings show reductions in episode unsafety relative to public-root execution and a design-selected fixed displacement. An exact-marginal permutation experiment further shows that preserving the displacement distribution while breaking its correspondence with the initial state substantially degrades safety, suggesting that state-dependent member selection contributes to the observed improvement.

Critical strengths
  1. Simple post-training use of an existing policy structure. EpiPlan exploits the one-dimensional policy family already exposed by an epigraph-conditioned controller, without retraining the actor or optimizing over high-dimensional joint action sequences. This makes the proposed intervention conceptually simple and separates it from methods that modify the underlying policy.
  2. Careful mechanism-oriented controls and transparent reporting. The fixed-displacement comparison tests whether the gains can be reproduced by a single globally more conservative operating point, while the exact-marginal permutation preserves the selected displacement distribution but breaks state-displacement pairing. The latter increases episode unsafety by 10.38 percentage points on average (p=.0001), providing useful evidence that contextual assignment matters within the evaluated policy family. The paper also clearly reports checkpoint filtering, paired evaluation, calibration/test splits, and the limitations of its theoretical guarantees.
Critical weaknesses
  1. The main evaluation relies on exact simulator lookahead, leaving robustness to imperfect deployment dynamics insufficiently established. EpiPlan uses the exact simulator for candidate continuations, so there is no model error over the simulated prefix. With H=32, the planner directly simulates the first 25% of a 128-step episode, and planning is performed once at the initial state. Because the dynamics are deterministic and the planner uses the exact simulator, this simulated prefix coincides with actual execution. Consistent with this favorable setting, safety prediction reaches ROC-AUC .999/.963/.968 on Corridor/Target/Line-5 at H=32. The main robustness evidence is a relatively small actuation-gain audit on five policies, where safety-benefit retention drops to .58–.74 on Target. I would like to see evaluation with stochastic or perturbed dynamics whose realizations are not available to the planner, or with an imperfect planning model.
  2. The comparisons do not control for the additional simulator information and computation available to EpiPlan. The primary public-root and fixed-displacement comparators do not receive the same six H=32 candidate continuations. Thus, while the experiments show that state-dependent displacement selection outperforms fixed or randomly reassigned displacements, they do not establish whether the proposed selector is better than simpler methods given the same lookahead budget. A matched best-of-K/safest-of-K rule, a planning or shielding baseline, or another selector operating on the same candidate frontier would be informative. In addition, a separate H=4 audit finds that tuning DEF-MARL’s existing global margin ξ achieves lower unsafety than EpiPlan’s H=4 configuration on both Corridor and Target, albeit at somewhat higher task cost. Although the primary H=32 EpiPlan results are safer than these tuned-ξ values, the two were not evaluated on the same cohort. A same-cohort comparison against this simple, simulator-free alternative would therefore substantially strengthen the empirical case.
Questions for authors
  1. How does EpiPlan perform when execution includes stochastic or perturbed dynamics whose realizations are unavailable to the planner, or when candidate continuations use an imperfect (e.g., learned) dynamics model? The hidden actuation-gain experiment is useful, but covers only five policies and a narrow form of model mismatch.
  2. How does EpiPlan compare against simpler alternatives given comparable test-time information and computation? In particular, could you evaluate a basic safest/best-of-K rule or a planning/shielding baseline using the same candidate frontier? I would also find a same-cohort comparison with the tuned global-margin ξ baseline useful, using the same checkpoints and test episodes as the primary evaluation.
  3. What is the oracle best-candidate-in-hindsight performance on the stored six-candidate frontier? This would help quantify how much of the available safety–utility improvement EpiPlan’s selector actually captures.

Clarifications

  1. Several calibrated thresholds are +∞, including two of the three ConnectSpread checkpoints. What role does the safety filter play for these checkpoints, where all candidates are admitted, and how sensitive are the aggregate results to such cases?
  2. Have you evaluated true receding-horizon selection, where the displacement is reconsidered during execution rather than selected once at the initial state? If not, do you expect repeated replanning to materially change the safety–utility tradeoff or computational cost?

Presentation and protocol clarifications

Please clarify how the reported 1.29–1.39× runtime overhead was measured, including absolute latency and the role of vectorized candidate evaluation. It would also be useful to verify that gathering the selected trajectory from the vectorized candidate frontier produces the same result as an independent execute-only replay under the intended deployment protocol.

3 comments
AuthorsOpenReview ↗

Response to Reviewer jkMH

We thank the reviewer for the positive assessment of the policy-family approach and the suggestions for further evaluation. Below, we address the questions on model mismatch, matched comparisons, and implementation.

Q5 / Weakness 3: execution uncertainty unavailable to the planner.

The additional F16 experiments evaluate EpiPlan with a mismatch between planning and execution. The planner retains the nominal model, candidate policies, and calibration, while executed commands are scaled by .9 or .8 before the original clipping operation. With H=32, the comparison with the root baseline uses the same metrics as the main F16 experiment:

Command gainMethodPrecision (%)Coverage (%)FSR (%)Any50 (%)Task cost
1.0Root94.0771.255.9398.820.5094
1.0EpiPlan (K=6)94.5771.415.4398.780.4955
.9Root93.2071.256.8098.570.5125
.9EpiPlan (K=6)92.9971.417.0198.290.5004
.8Root91.3171.258.6997.980.5206
.8EpiPlan (K=6)90.5571.419.4597.460.5117

Coverage is the accepted fraction of the 65,280 test states. At gains .9 and .8, EpiPlan with K=6 reduces task cost by 2.4% and 1.7% relative to the root baseline, with slightly lower precision and stabilization rates (Supplement C.8).

We also evaluate hidden Gaussian action noise in MPETarget and MPECorridor, with the initial states and noise realizations paired across methods. EpiPlan uses K=6 and H=32, with nominal-model prediction and calibration retained at both noise levels. The comparison with the public-root baseline at ξ=.21 is:

TaskNoise σRoot failure (%)EpiPlan failure (%)Root costEpiPlan cost
MPETarget.0525.1316.80.22190.23331
MPETarget.1023.7016.15.22077.23174
MPECorridor.0522.664.69.27366.27667
MPECorridor.1024.027.81.27454.27778

Each noise level includes 768 MPETarget and 512 MPECorridor test instances, with results averaged equally across controllers. Relative to root execution, EpiPlan reduces failures by 31.9–33.2% in MPETarget and 67.5–79.3% in MPECorridor, with task-cost increases of approximately 5% and 1.1–1.2%, respectively (Supplement C.10).

Q6 / Weakness 4: matched selectors and a tuned global margin.

The new comparisons provide Safe-K and EpiPlan with identical candidate simulations. On MPETarget, EpiPlan with K=11 reduces the failure rate from 10.03% to 7.94% relative to Safe-K, with task cost changing from .251 to .256. It also improves both metrics over calibrated fixed selection and a fixed mixture:

MPETarget rule, K=11, H=32Failure (%)Task cost
Calibrated fixed candidate13.15.381
Fixed mixture12.94.374
Safe-K10.03.251
EpiPlan7.94.256

Results are averaged across trained controllers on matched test states. The K=11 candidate set adds 5 policies with different margins and δ=0 to the K=6 displacement candidates. EpiPlan calibrates using joint safety and task performance, while the comparators use a 5% empirical risk target (Supplement C.2). The comparisons therefore assess the complete calibrated selection rules with the same prediction information and simulation budget.

The results also extend to other systems. In Safe HalfCheetah, EpiPlan with K=12 halves failures from 22/256 to 11/256 while maintaining return (1585.12 versus 1584.66). In the nominal F16 comparison with K=6, EpiPlan increases precision from 92.37% to 94.57% and reduces the false-safe rate from 7.63% to 5.43% relative to Safe-K, at similar coverage (71.68% versus 71.41%). On the respective accepted sets, Any50 increases from 98.04% to 98.78% and task cost decreases from .4967 to .4955 (Supplement C.8).

We also compare against a globally tuned margin. Relative to this baseline, EpiPlan with K=11 and H=32 reduces the MPETarget failure rate from 12.11% to 7.94% and task cost from .344 to .256. In MPECorridor, the failure rate decreases from 1.17% to .39% and cost from .312 to .271.

To examine the role of instance-dependent selection, we also preserve candidate-use counts and uniformly permute policy assignments. In the K=6 evaluation across 6 navigation settings, this increases both expected failure rate and task cost in every setting. For example, the MPETarget failure rate increases from 15.36% to 26.62%. This comparison supports the benefit of matching policies to the current instance (Supplement C.9).

Q7: hindsight performance with K=6.

In a separate K=6 control experiment with 128 test instances per training seed, the safety oracle achieves failure rates of 2/256 in MPECorridor and 35/384 in MPETarget. The corresponding rates for EpiPlan are 3/256 and 53/384, respectively:

TaskInstances with no safe candidateEpiPlan failures despite a safe candidateTotal EpiPlan failures
MPECorridor213
MPETarget351853

This separates the safety limit of the candidate set from the remaining selection error. The safety oracle uses full executed outcomes to choose a safe candidate whenever one exists, giving the minimum attainable failure rate within the candidate set. Section 5 develops this decomposition, and Supplement C.5 reports it for the additional horizon and coverage study.

Q8: infinite thresholds and fallback.

When the calibrated threshold is cH=+∞c_H=+\infty, all candidates with finite scores are retained for task ranking. Excluding the 2 MPEConnectSpread controllers with infinite thresholds leaves 16 controllers, for which the mean failure rate is approximately 27.39% under root execution and 11.77% under EpiPlan. The aggregate improvement therefore persists without those cases.

When no candidate meets the calibrated safety criterion, EpiPlan selects the candidate with the lowest safety score. The failure decomposition in Section 5 includes this fallback. The task-cost bound applies to safe selections from the predicted-safe set.

Q9: periodic reselection.

Thank you for this helpful suggestion. We agree that periodically reselecting the displacement is a promising extension of EpiPlan, allowing policy selection to adapt to changes during execution. We have not evaluated this extension in the current study and leave it for future work.

Deployment-time computation.

EpiPlan evaluates candidate policies and selects the displacement once, before execution begins. During execution, the selected displacement remains fixed while roots and actions are updated from the current observations, without further candidate simulation.

We appreciate your suggestions and welcome any further questions on these results.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer jkMH, as the discussion period draws to a close, we would appreciate your feedback on our additional results. Our response includes comparisons against a tuned global-margin baseline and Safe-K with matched simulation access, together with tests under execution perturbations unavailable to the planner. Please let us know if these results leave any questions unresolved. Thank you again for your constructive suggestions.

Reviewer_entxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this work, the authors propose EpiPlan, a deployment procedure for frozen epigraph-conditioned multi-agent policies. The authors use pretrained policies from Def-MARL, modifying the policy deployment by adding an offset δ\delta to the original zz obtained from Def-MARL via root-finding. The authors choose δ\delta with a grid search, then choose the the value that has the best bootstrapped safety and cost. Simulation experiments MPE-style tasks used in Def-MARL show improvements higher safety rates than Def-MARL.

Critical strengths
  • The proposed method improves the performance of Def-MARL on the tested tasks without requiring any additional retraining
Critical weaknesses
  • While it builds off of Def-MARL (i.e., Epigraph RL), it does not use any of the terminology / concepts introduced in those works, making the paper difficult to understand. For example, the authors use terms like “local boundary inversion” to refer to the local root-finding in Def-MARL, making it very confusing to readers familiar with Def-MARL. This happens repeatedly throughout the paper.
  • The proposed method of computing the offset δ\delta requires multiple rollouts (of horizon H, 32 in the paper) at test-time. This requires the test-time controller to have access to the dynamics and cost functions, as well as requires additional computation time.
  • The authors only evaluate on five MPE-style tasks, and also only compare against Def-MARL. It is not clear whether these improvements carry to other environments / tasks, and how these improvements compare to other methods.
Questions for authors
  • How does the proposed method compare to other methods, on both the proposed MPE tasks and other tasks?
  • How does the performance vary as the budget and horizon for computing the δ\delta offset varies?
5 comments
AuthorsOpenReview ↗

Response to Reviewer entx

We thank the reviewer for the constructive comments on terminology, computation, and evaluation. Below, we clarify the connection to epigraph RL and present the additional comparisons.

Terminology and the connection to EFPPO/Def-MARL.

We have revised the presentation to improve clarity and ensure consistency with the terminology used in epigraph RL.

EpiPlan selects a policy parameterized by its displacement δ from the root computed by Def-MARL. It simulates the candidates once before execution and combines their predicted trajectories with learned terminal values. It then selects a policy based on predicted safety and task performance and retains δ throughout execution, while recomputing roots and actions. At a fixed margin, this provides a one-dimensional planning problem within the trained policy family, without retraining.

Our public-coordinate implementation uses the maximum-root aggregation described in Def-MARL (Eq. (15) and Algorithm 2). Supplement C.6 also evaluates local-coordinate execution using Safe-K, with prefix safety screening and simulated task cost for ranking.

Model access and computation.

Candidate evaluation uses a dynamics model, cost and constraint functions, and the joint initial state. Safe-K receives the same candidate simulations as EpiPlan, allowing us to compare the selection rules with access to the same information and the same simulation budget.

We also test model mismatch by retaining the nominal planning model and calibration while perturbing execution. Under hidden action noise at σ=.05 and .10, EpiPlan with K=6 reduces failures relative to the root baseline by 31.9–33.2% in MPETarget and 67.5–79.3% in MPECorridor, with task-cost increases of approximately 5% and 1.1–1.2%, respectively (Supplement C.10). In F16, EpiPlan with K=6 reduces task cost on the respective accepted sets by 2.4% and 1.7% relative to the root baseline at command gains .9 and .8, respectively, with similar coverage and slightly lower safety precision and stabilization rates (Supplement C.8).

A dynamics model learned from interaction data could replace the simulator used for candidate evaluation, allowing EpiPlan to operate without access to the true dynamics. We leave evaluation with learned models to future work.

Evaluating K candidate policies over a prediction horizon H requires KH simulated transitions.

Q1: additional methods and environments.

Section 6 now compares EpiPlan with a calibrated fixed candidate, an optimized fixed mixture, and Safe-K using the same candidate policies and prediction horizon. On MPETarget, EpiPlan with K=11 reduces the failure rate from 13.15% to 7.94% and task cost from .381 to .256 relative to fixed selection. Relative to Safe-K, the failure rate decreases from 10.03% to 7.94%, with cost changing from .251 to .256. Results are averaged across trained controllers on matched test states.

The additional Safe HalfCheetah 2×3 and F16 experiments test the same selection approach in locomotion and aircraft control. Both use H=32, with identical candidate simulations for each EpiPlan/Safe-K pair:

Safe HalfCheetahSafe-K failures / 256EpiPlan failures / 256Safe-K returnEpiPlan return
K=623151566.871579.75
K=1222111585.121584.66
F16 methodPrecision (%)Coverage (%)FSR (%)Any50 (%)Task cost
Root94.0771.255.9398.820.5094
Safe-K (K=6)92.3771.687.6398.040.4967
EpiPlan (K=6)94.5771.415.4398.780.4955

For F16, precision is the fraction of accepted initial states whose trajectories remain safe, the false-safe rate (FSR) is one minus precision, and coverage is the accepted fraction of the 65,280 test states. Any50 requires 50 consecutive steps in the target altitude band with no simulator failure. Any50 and task cost are evaluated on each method's accepted set, as in Section 6.2. The K=6 candidate set is selected from a pool of 11 candidates before calibration.

With K=12, EpiPlan halves HalfCheetah failures with a change in return of less than .03%. In F16, EpiPlan with K=6 reduces the false-safe rate from 7.63% to 5.43%, with slightly lower task cost at similar coverage. These results extend the evidence for predictive policy selection beyond the original navigation tasks.

HalfCheetah uses a Def-MARL controller adapted to the public simulator, and F16 uses an EFPPO controller. Training and calibration details are provided in Supplement C.2 and C.7–C.8. All EpiPlan results above use the combined score. These comparisons evaluate deployment rules applied to the same controller. Supplement C.9 reports the separate comparison across 6 navigation settings.

Q2: candidate budget and prediction horizon.

Section 6.3 evaluates the contribution of prediction using the EpiPlan variant with separate prefix and terminal safety checks (Split). We compare selection based on initial-state values (H=0) with 32-step prediction, using the same K=11 candidates, calibration data, and test states. In MPETarget, prediction reduces the failure rate from 13.80% to 8.07% and task cost from .359 to .256. In MPECorridor, the failure rate decreases from 2.93% to .59% and cost from .296 to .269 (Supplement C.4).

A separate MPETarget study evaluates denser displacement grids up to K=81 at a fixed margin of ξ=.21, using Safe-K throughout:

Displacement candidates KHFailure (%)Task cost
63212.80.266
61288.69.235
111286.90.228
211286.42.225
411285.81.222
811285.17.220

With K=6, increasing H from 32 to 128 reduces the failure rate from 12.80% to 8.69%. At H=128, refining the displacement grid from 6 to 81 candidates further reduces the failure rate to 5.17% and task cost from .235 to .220. Results are averaged with equal weight across trained controllers. Since execution lasts 128 steps, H=128 evaluates the complete trajectory without a terminal estimate and requires 4 times as many simulated transitions per candidate. Section 6.3 and Supplement C.5 relate these improvements to safe-candidate availability and selection errors.

We hope the revised presentation and additional results address your concerns and welcome further questions.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer entx, as the discussion period draws to a close, we would appreciate your feedback on our response. We have clarified the terminology and added comparisons with Safe-K in Safe HalfCheetah and F16, along with studies of candidate budget and prediction horizon. We would be happy to clarify any remaining questions. Thank you again for your time and consideration.

Reviewer_8c6jOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I did not use AI

Critical strengths

For the reasons below, I found the paper very hard to follow, and hard to identify the strengths. It studies an important problem, and it seems to offer a novelty compared to papers So &Fan '23 and Zhang et al '25, a different parametrization of the team policies, if I understand this correctly.

Critical weaknesses

The paper is hard to follow. The sentences are not well connected with each other.

Also, there is a lot of terminology that is unclear. It is likely that the terminology is needed for the specific approach that the paper follows, or that the paper follows a very specific terminology that was introduced in a previous paper in epigraph RL, but this should have been explained in the paper. I give some examples below.

The technical problem formulation is also hard to follow due to the notation. Again, it is likely that this notation is needed, but it has to be explained. At the present stage, it is unclear.

It seems that the numerical experiments do not compare with a range of other safe MARL works, please correct me if I understood this wrong.

Questions for authors

A range of terms are non-standard and have to be clarified. Continuation (could this mean the state rollouts?), Root or critic root or public root. Boundary. Member (could this mean a member of the family of policies or a member of some other set?). Displacement. Actual-receding. Coordinate (could this mean parametrization of the policies?)

Perhaps the authors could compare with some non-RL methods in multi-agent safe control, such as distributed MPC approaches: Farina, M. and Scattolini, R. (2012). Distributed predictive control: Anon-cooperative algorithmwith neighbor-to-neighborcommunicationforlinearsystems. Automatica,48(6),1088–1096. Plug and play distributed model predictive control for heavy duty vehicle platooning and interaction with passenger vehicles H Hu, Y Pu, M Chen, CJ Tomlin - 2018 IEEE Conference on

In the problem formulation. Is O_i(x) determinist function?

Clarify what are the augmented states.

'Experimental rates average the event ... over held out initial states' unclear

' When the robust problem is feasible and attened' which one is the robust problem? It has not been specified.

What does 'team boundary' mean?

What is V_ic^h(x,z) and how does it depend on z?

In (5) the policy depends explicitly not only on the local observation o_i but also on something that depends on the global (?) state \chi. How is this possible?

'Admissible coordinate interval' is not defined.

'The executed coordinate along its continuation' is unclear what it means.

What is \chi_t^\delta?

Theorem 1 says if \Delta is compact, but earlier this is defiend as a simple interval [0, \delta_max] so it is by default compact.

After this lack of clear setup, I could not read the rest of the theoretical development.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 8c6j

We thank the reviewer for identifying the presentation issues and apologize for the lack of clarity in the original submission. We have revised the exposition and notation to make the formulation and its connection to epigraph RL easier to follow.

Motivation and approach.

Epigraph RL methods such as EFPPO and Def-MARL learn policies conditioned on a task budget. In Def-MARL, root finding computes the task budget using a prescribed safety margin ξ. Our navigation experiments show that, for many initial states, other policies represented by the same controller offer a better safety–task trade-off than deployment with a fixed margin. This observation motivated us to investigate how the policies in the learned family differ in safety and task performance, and how these differences can guide policy selection for each deployment instance.

To this end, we propose EpiPlan, which plans over the policy family induced by the trained controller. At a fixed ξ, each candidate is parameterized by a displacement δ from the root, yielding a one-dimensional search without retraining. Before execution, EpiPlan uses short-horizon simulations and learned terminal values to predict each candidate's safety and task performance and select a policy. It retains the selected displacement throughout execution, while roots and actions adapt to the evolving state.

We evaluate EpiPlan on cooperative multi-agent navigation (Def-MARL, MPETarget and MPECorridor), constrained locomotion (Def-MARL, Safe HalfCheetah), and aircraft control (EFPPO, F16). These experiments demonstrate improvements in the safety–task trade-off over fixed deployment rules, with additional gains over Safe-K under the same candidate simulations. Future work could extend the discrete search to continuous displacement optimization.

The policy being selected and the connection to epigraph RL.

EpiPlan plans over the policy family represented by a trained epigraph-conditioned controller. Each candidate policy is parameterized by a displacement δ from the state-dependent root. For a fixed margin ξ, the candidate uses

b^∗ξ,tδ=max⁡ir^∗i,tδ,z∗tδ=clip⁡∗Z(b^∗ξ,tδ+δ),u∗i,tδ=π∗θ(o∗i,tδ,ztδ).\widehat b*{\xi,t}^\delta=\max_i\widehat r*{i,t}^\delta,\qquad z*t^\delta=\operatorname{clip}*{\mathcal Z}(\widehat b*{\xi,t}^\delta+\delta),\qquad u*{i,t}^\delta=\pi*\theta(o*{i,t}^\delta,z_t^\delta).

Here, r^∗i,tδ\widehat r*{i,t}^\delta is computed by bounded root finding for max⁡cV^h∗i,c(oi,mi,z)+ξ=0\max_c\widehat V^h*{i,c}(o_i,m_i,z)+\xi=0. The maximum across agents is the public root. Recurrent-state arguments are omitted in the displayed actor notation. During execution, the selected δ remains fixed while roots and actions are recomputed at every step. Thus, selecting δ specifies an entire policy whose actions adapt to the evolving state.

The margin ξ determines the safety-value threshold used for root finding, whereas δ shifts the resulting budget supplied to the actor. In navigation, the K=11 candidate set adds 5 policies with different margins and δ=0 to the K=6 displacement candidates. Each candidate is then identified by a pair (ξ, δ), and both values are retained during execution (Section 4.4).

Definitions.

Sections 3–4 now introduce the control problem and epigraph formulation before the policy family and selection rule. The remaining notation is defined as follows:

  • Z=[zmin⁡,zmax⁡]\mathcal Z=[z_{\min},z_{\max}] is the task-budget range supplied to the actor.
  • Oi(x)O_i(x) deterministically maps the physical state to agent i's local observation.
  • χ=(x,m)\chi=(x,m) combines the physical state with the recurrent states of the actor and critics. Along candidate δ's trajectory, this state is denoted by χtδ\chi_t^\delta. All candidates share the same initial state.
  • V^i,ch(oi,mi,z)\widehat V^h_{i,c}(o_i,m_i,z) estimates the safety value for agent i and constraint c under the policy conditioned on z. It is distinct from the physical constraint and is not a failure probability.

Information available to the planner and the agents.

In our experiments, EpiPlan uses the joint initial state and simulator dynamics to select a policy before execution, after which each agent acts on its local observation and the shared budget derived from Def-MARL's maximum-root aggregation.

Policy selection also benefits local-coordinate execution. In an additional MPECorridor evaluation, we use Safe-K to screen candidates for safety over the simulated trajectory and rank them by simulated task cost. With K=11, this reduces failures from 9/256 to 0/256 and task cost from .29641 to .25495 relative to the local-root baseline. This evaluation shows that planning over the epigraph policy family also benefits Def-MARL's local-root deployment, where agents compute their epigraph coordinates locally without exchanging them during execution (Supplement C.6).

We also evaluate policy selection when execution differs from the nominal planning model. Under hidden action noise at σ=.05 and .10, EpiPlan with K=6 reduces failures relative to the root baseline by 31.9–33.2% in MPETarget and 67.5–79.3% in MPECorridor, with task-cost increases of approximately 5% and 1.1–1.2%, respectively (Supplement C.10). In F16, EpiPlan with K=6 reduces task cost on the respective accepted sets by 2.4% and 1.7% relative to the root baseline at command gains .9 and .8, respectively, with similar coverage and slightly lower safety precision and stabilization rates (Supplement C.8). The planning model and calibration remain unchanged in both experiments.

A dynamics model learned from interaction data could replace the simulator used for candidate evaluation, allowing EpiPlan to operate without access to the true dynamics. We leave evaluation with learned models to future work.

Problem formulation and experimental rates.

Section 3 defines the underlying infinite-horizon safe optimal control problem. Section 4.2 formulates optimization within the available policy family, and Section 5 analyzes execution over T steps using a prediction horizon H≤T. Each result now refers explicitly to its optimization problem. The displacement domain is the compact interval [0,δmax⁡][0,\delta_{\max}], with no additional compactness assumption.

The navigation failure rate is the fraction of test instances in which at least one physical constraint is violated during execution. With multiple trained controllers, we compute this rate for each controller and report their equally weighted mean. Section 6.1 states this aggregation explicitly.

Comparisons with safe control methods.

We agree that comparisons with distributed MPC and other safe MARL methods would broaden the evaluation. Our baseline Def-MARL includes comparisons with centralized and decentralized MPC in hardware experiments (Zhang et al., 2025, Sec. VI). In this paper, we prioritized comparisons within the same trained controller to assess the contribution of policy selection.

The added experiments compare deployment rules using the same controller, including fixed margins, a calibrated fixed candidate, a fixed mixture, and Safe-K with the same candidate simulations. They also extend the evaluation to locomotion and aircraft control. For example, with K=12, EpiPlan reduces Safe HalfCheetah failures from 22/256 to 11/256 relative to Safe-K, with returns of 1584.66 and 1585.12, respectively. These comparisons directly assess the benefit of predictive selection within a trained policy family (Section 6; Supplement C).

We hope these clarifications make the formulation easier to follow and welcome any further questions.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer 8c6j, as the discussion period draws to a close, we would appreciate your feedback on our revised manuscript. We have reorganized the problem formulation and method sections and clarified the terminology and notation. Please let us know if any concerns remain. Thank you again for your time.

Reviewer_NfwtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EpiPlan is a training-free deployment procedure for frozen DEF-MARL controllers. DEF-MARL normally executes the policy at the learned public safety boundary. Instead, at the initial state EpiPlan simulates six displacements of the epigraph coordinate above that boundary for H = 32 steps in the exact simulator. It scores each with a calibrated safety surrogate and the task critic, then executes the cheapest predicted-safe member.

The paper provides approximation and selection bounds. Across 18 filtered checkpoints in six MPE-style settings, EpiPlan roughly halves episode unsafety relative to the native public-boundary execution, with reward loss within 15%. A permutation control is used to argue that the gain comes from per-state member selection.

Critical strengths

Strength 1

The method is simple, training-free, and cheap (1.29–1.39× runtime). It reduces unsafety on all 18 checkpoints, with small reward cost, so it is easy to adopt on top of DEF-MARL.

Strength 2

The evaluation is transparent. It uses frozen design/test splits, disclosed cohort filtering, hierarchical CIs, and per-checkpoint and fallback diagnostics, including cases where EpiPlan loses.

Critical weaknesses

Weakness 1

The gain is not attributed to the epigraph structure rather than to privileged simulator lookahead. Candidates are rolled out 32 steps in the exact, deterministic simulator, so near-term violations are observed directly (safety ROC-AUC .999 on Corridor, Supp. Table 10).

The fixed-offset and permutation controls only show that per-episode information helps, which holds for any outcome-informed selector. The missing baselines use the same simulator budget:

  • best-of-6 over fixed-z rollouts, other checkpoints, or other ξ values;
  • model-predictive shielding (e.g., MAMPS);
  • a same-cohort, design-tuned global ξ at H = 32.

The global-ξ baseline matters most. In the separate H = 4 cohort, it was safer than EpiPlan on Corridor (.057 vs .111).

Weakness 2

The theory is standard and disconnected from the deployed method. The results are an epigraph equivalence, max/min nonexpansiveness, a Lipschitz covering argument, and a constrained-selection bound. Their error envelopes are never estimated, and the experiments use the untightened branch. The fallback branch, used in up to 16% of episodes, is uncovered. The refinement-level analysis is never exercised, since only a fixed six-point grid is used. Calibrated thresholds also vary widely across seeds, including +∞, which suggests a poorly calibrated safety score.

Questions for authors

Question 1

  • How does EpiPlan compare with the baselines in Weakness 1 under the same simulator access, candidate budget, and cohort at H = 32?

Question 2

  • What fraction of the unsafety reduction comes from violations within the simulated 32-step prefix versus after it?

Additional questions

  • Planning occurs only once at the initial state. How do results change with periodic replanning or with an imperfect or learned model?
  • I would consider raising my score if Questions 1–2 are addressed convincingly.
4 comments
AuthorsOpenReview ↗

Response to Reviewer Nfwt

We thank the reviewer for the careful assessment and the suggestions on baseline comparisons and failure timing. Below, we address the two main questions with additional results.

Q1 / Weakness 1: comparisons with the same simulator access and budget.

The revised experiments compare EpiPlan with Safe-K on the same controllers, initial states, candidate policies, and 32-step simulations. In MPETarget, EpiPlan reduces both the failure rate and task cost with K=6. With K=11, it reduces the failure rate by 20.8% relative to Safe-K, with a 1.7% increase in task cost:

MPETarget candidatesRuleFailure (%)Task cost
K=6Safe-K12.63.263
K=6EpiPlan11.33.256
K=11Safe-K10.03.251
K=11EpiPlan7.94.256

Results are averaged across trained controllers on matched test states. The K=11 candidate set adds 5 policies with different margins and δ=0 to the K=6 displacement candidates. EpiPlan calibrates its score using joint safety and task performance. Safe-K calibrates its prefix safety threshold and terminal task-value setting using an empirical risk target of 5% (Supplement C.2).

The matched comparisons also show gains beyond navigation:

  • Safe HalfCheetah: with K=12 and H=32, EpiPlan halves failures from 22/256 to 11/256, with essentially unchanged return (1585.12 versus 1584.66). With K=6, failures decrease from 23 to 15 and return increases from 1566.87 to 1579.75.
  • F16: with K=6 and H=32, safety precision increases from 92.37% to 94.57% and the false-safe rate decreases from 7.63% to 5.43%, while coverage remains similar (71.68% versus 71.41%). On each method's accepted set, the 50-step stabilization rate (Any50) increases from 98.04% to 98.78% and task cost decreases from .4967 to .4955. Precision is the fraction of accepted states whose trajectories remain safe, and coverage is the fraction of the 65,280 test states that are accepted (Supplement C.8).

These results use the combined EpiPlan score with revised calibration and the stated candidate sets. Supplement C.9 reports the separate comparison.

Role of the policy family and the requested controls. EpiPlan uses the epigraph coordinate to parameterize a family of policies and compares their predicted trajectories before execution. EpiPlan selects a displacement from the Def-MARL root and retains the displacement while roots and actions update. The additional controls examine this choice directly:

  • Fixed initial budgets (K=6): in a separate MPETarget control experiment at H=32 with 128 test instances per training seed, each candidate's budget is computed from the initial root and held constant throughout execution. Selecting among these candidates gives a mean failure rate of 24.22%, compared with 13.80% for root-relative displacements and 11.20% for margin-based candidates. These results motivate including both displacements and margins in the expanded candidate set.
  • A globally tuned margin: this comparison uses the same controllers and test states as the main evaluation. Relative to the tuned margin, EpiPlan with K=11 and H=32 reduces the MPETarget failure rate from 12.11% to 7.94% and task cost from .344 to .256. In MPECorridor, the failure rate decreases from 1.17% to .39% and cost from .312 to .271. The margin is selected for each controller on separate calibration states by choosing the lowest-cost setting that meets the 5% empirical risk target, or the setting with the lowest failure rate if none meets it.
  • Preserving candidate frequencies while permuting instance assignments: in the separate K=6 evaluation across 6 navigation settings, we compute expected outcomes after uniformly permuting EpiPlan's policy assignments among test instances within each controller. Expected failure rate and task cost both increase in every setting. For example, failure rates increase from 2.08% to 16.44% in MPECorridor and from 15.36% to 26.62% in MPETarget. This control supports the benefit of matching policies to individual instances, with the candidate-use counts held fixed (Supplement C.9).

We agree that comparisons with MAMPS and selection across separately trained checkpoints would be valuable. In this revision, we prioritized comparisons with the same controller and simulation budget, together with experiments on Safe HalfCheetah and F16 using Def-MARL and EFPPO, respectively. These experiments assess both the contribution of the selection rule and its applicability across different controllers and system dynamics.

Q2: failures within and beyond the prediction horizon.

We separate first violations within H=32 from those occurring later in the revised evaluation. The table compares root execution, Safe-K, and EpiPlan on the same test states. Each entry gives the early / late failure counts.

TaskKTest instancesRoot: ≤32 / >32Safe-K: ≤32 / >32EpiPlan: ≤32 / >32
MPECorridor1151295 / 50 / 30 / 2
MPETarget11768162 / 6414 / 6314 / 47
Safe HalfCheetah122566 / 145 / 175 / 6
F16665,28019,741 / 1,23618,809 / 2,97318,809 / 1,838

In MPECorridor, 95 of the root rule's 100 failures occur within H=32, and eliminating these early failures accounts for 96.9% of EpiPlan's net failure reduction. The corresponding fraction is 89.7% in MPETarget. Compared with Safe-K, EpiPlan has identical early-failure counts and fewer late failures in all 4 settings. Thus, its advantage over a selector with the same simulations extends beyond the predicted prefix. This comparison assesses the complete selection rule, including calibration and ranking.

Root denotes public-root execution for the multi-agent tasks (ξ=.21 in navigation) and online root finding for F16. The F16 failure counts are computed over all test states. In F16, the early-failure sets are identical, and paired comparison shows 1,740 states where EpiPlan is safe and Safe-K fails, versus 605 with the converse outcome. Thus, the net reduction of 1,135 failures occurs after H=32 (Supplement C.8).

Weakness 2: connection between the analysis and the implemented selector.

Section 5 connects deployment performance to the availability of safe candidates and the selector's ability to identify them. The failure decomposition applies to every selection, including fallback. For safe selections from the predicted-safe set, the task-cost analysis separates losses due to safety screening from those due to task ranking. The ranking bound depends on differences in prediction errors, so a common additive error cancels.

The experiments in Section 6.3 and Supplement C.5 examine candidate availability and selection errors as the prediction horizon and candidate coverage vary. In MPETarget, refining the displacement grid from K=6 to K=81 at fixed ξ=.21 and H=128 reduces the failure rate from 8.69% to 5.17% and task cost from .235 to .220, using Safe-K throughout. The implemented selector chooses its safety threshold on held-out calibration data (Supplement C.2). Supplement A provides supporting bounds on grid approximation and prediction error under the stated assumptions. An infinite threshold admits every finite-score candidate, leaving task ranking to select the policy. Fallback minimizes the safety score.

Additional questions: model mismatch and replanning.

We evaluate hidden action noise in MPE and command-gain perturbations in F16 while retaining the nominal planning model and calibration. At σ=.05 and .10, EpiPlan with K=6 reduces failures relative to the root baseline by 31.9–33.2% in MPETarget and 67.5–79.3% in MPECorridor, with task-cost increases of approximately 5% and 1.1–1.2%, respectively (Supplement C.10). In F16, EpiPlan with K=6 reduces task cost on the respective accepted sets by 2.4% and 1.7% relative to the root baseline at command gains .9 and .8, respectively, with similar coverage and slightly lower safety precision and stabilization rates (Supplement C.8).

A dynamics model learned from interaction data could replace the simulator used for candidate evaluation, allowing EpiPlan to operate without access to the true dynamics. We leave evaluation with learned models to future work.

Thank you also for suggesting periodic displacement reselection. We agree that this is a promising extension and leave its evaluation for future work.

We hope these results and clarifications address your concerns and welcome any further questions.

Reviewer_NfwtOpenReview ↗

EpiPlan’s Matched Comparisons Address My Main Concerns

I thank the authors for the additional experiments. The tuned global-margin baseline and the matched Safe-K comparison address my main concerns. With identical early-failure counts, EpiPlan has fewer failures beyond the 32-step prediction horizon in all four settings. The specific contribution of the epigraph parameterization remains open, but the new evidence supports the selection rule, and I will raise my score accordingly.

AuthorsOpenReview ↗

Thank You for Your Feedback

Thank you for your careful reassessment and constructive suggestions. We are pleased that the additional evidence addresses your main concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.