- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EpiPlan, a test-time planning method for frozen epigraph-conditioned multi-agent policies. Rather than executing only the public-root policy used by the underlying DEF-MARL controller, EpiPlan considers a finite set of one-sided displacements from this root. For each candidate, it simulates a finite joint continuation, uses the frozen task and safety critics for terminal estimates, and selects the lowest-cost candidate predicted to be feasible. The method requires no retraining and searches over a one-dimensional policy family rather than joint action sequences.
The paper also develops conditional approximation and selection bounds for this deployment procedure. Experiments on 18 selected DEF-MARL checkpoints across six MPE-style settings show reductions in episode unsafety relative to public-root execution and a design-selected fixed displacement. An exact-marginal permutation experiment further shows that preserving the displacement distribution while breaking its correspondence with the initial state substantially degrades safety, suggesting that state-dependent member selection contributes to the observed improvement.
Critical strengths
- Simple post-training use of an existing policy structure. EpiPlan exploits the one-dimensional policy family already exposed by an epigraph-conditioned controller, without retraining the actor or optimizing over high-dimensional joint action sequences. This makes the proposed intervention conceptually simple and separates it from methods that modify the underlying policy.
- Careful mechanism-oriented controls and transparent reporting. The fixed-displacement comparison tests whether the gains can be reproduced by a single globally more conservative operating point, while the exact-marginal permutation preserves the selected displacement distribution but breaks state-displacement pairing. The latter increases episode unsafety by 10.38 percentage points on average (p=.0001), providing useful evidence that contextual assignment matters within the evaluated policy family. The paper also clearly reports checkpoint filtering, paired evaluation, calibration/test splits, and the limitations of its theoretical guarantees.
Critical weaknesses
- The main evaluation relies on exact simulator lookahead, leaving robustness to imperfect deployment dynamics insufficiently established. EpiPlan uses the exact simulator for candidate continuations, so there is no model error over the simulated prefix. With H=32, the planner directly simulates the first 25% of a 128-step episode, and planning is performed once at the initial state. Because the dynamics are deterministic and the planner uses the exact simulator, this simulated prefix coincides with actual execution. Consistent with this favorable setting, safety prediction reaches ROC-AUC .999/.963/.968 on Corridor/Target/Line-5 at H=32. The main robustness evidence is a relatively small actuation-gain audit on five policies, where safety-benefit retention drops to .58–.74 on Target. I would like to see evaluation with stochastic or perturbed dynamics whose realizations are not available to the planner, or with an imperfect planning model.
- The comparisons do not control for the additional simulator information and computation available to EpiPlan. The primary public-root and fixed-displacement comparators do not receive the same six H=32 candidate continuations. Thus, while the experiments show that state-dependent displacement selection outperforms fixed or randomly reassigned displacements, they do not establish whether the proposed selector is better than simpler methods given the same lookahead budget. A matched best-of-K/safest-of-K rule, a planning or shielding baseline, or another selector operating on the same candidate frontier would be informative. In addition, a separate H=4 audit finds that tuning DEF-MARL’s existing global margin ξ achieves lower unsafety than EpiPlan’s H=4 configuration on both Corridor and Target, albeit at somewhat higher task cost. Although the primary H=32 EpiPlan results are safer than these tuned-ξ values, the two were not evaluated on the same cohort. A same-cohort comparison against this simple, simulator-free alternative would therefore substantially strengthen the empirical case.
Questions for authors
- How does EpiPlan perform when execution includes stochastic or perturbed dynamics whose realizations are unavailable to the planner, or when candidate continuations use an imperfect (e.g., learned) dynamics model? The hidden actuation-gain experiment is useful, but covers only five policies and a narrow form of model mismatch.
- How does EpiPlan compare against simpler alternatives given comparable test-time information and computation? In particular, could you evaluate a basic safest/best-of-K rule or a planning/shielding baseline using the same candidate frontier? I would also find a same-cohort comparison with the tuned global-margin ξ baseline useful, using the same checkpoints and test episodes as the primary evaluation.
- What is the oracle best-candidate-in-hindsight performance on the stored six-candidate frontier? This would help quantify how much of the available safety–utility improvement EpiPlan’s selector actually captures.
Clarifications
- Several calibrated thresholds are +∞, including two of the three ConnectSpread checkpoints. What role does the safety filter play for these checkpoints, where all candidates are admitted, and how sensitive are the aggregate results to such cases?
- Have you evaluated true receding-horizon selection, where the displacement is reconsidered during execution rather than selected once at the initial state? If not, do you expect repeated replanning to materially change the safety–utility tradeoff or computational cost?
Presentation and protocol clarifications
Please clarify how the reported 1.29–1.39× runtime overhead was measured, including absolute latency and the role of vectorized candidate evaluation. It would also be useful to verify that gathering the selected trajectory from the vectorized candidate frontier produces the same result as an independent execute-only replay under the intended deployment protocol.
3 comments
Q7: hindsight performance with K=6.
In a separate K=6 control experiment with 128 test instances per training seed, the safety oracle achieves failure rates of 2/256 in MPECorridor and 35/384 in MPETarget. The corresponding rates for EpiPlan are 3/256 and 53/384, respectively:
| Task | Instances with no safe candidate | EpiPlan failures despite a safe candidate | Total EpiPlan failures |
|---|---|---|---|
| MPECorridor | 2 | 1 | 3 |
| MPETarget | 35 | 18 | 53 |
This separates the safety limit of the candidate set from the remaining selection error. The safety oracle uses full executed outcomes to choose a safe candidate whenever one exists, giving the minimum attainable failure rate within the candidate set. Section 5 develops this decomposition, and Supplement C.5 reports it for the additional horizon and coverage study.
Q8: infinite thresholds and fallback.
When the calibrated threshold is , all candidates with finite scores are retained for task ranking. Excluding the 2 MPEConnectSpread controllers with infinite thresholds leaves 16 controllers, for which the mean failure rate is approximately 27.39% under root execution and 11.77% under EpiPlan. The aggregate improvement therefore persists without those cases.
When no candidate meets the calibrated safety criterion, EpiPlan selects the candidate with the lowest safety score. The failure decomposition in Section 5 includes this fallback. The task-cost bound applies to safe selections from the predicted-safe set.
Q9: periodic reselection.
Thank you for this helpful suggestion. We agree that periodically reselecting the displacement is a promising extension of EpiPlan, allowing policy selection to adapt to changes during execution. We have not evaluated this extension in the current study and leave it for future work.
Deployment-time computation.
EpiPlan evaluates candidate policies and selects the displacement once, before execution begins. During execution, the selected displacement remains fixed while roots and actions are updated from the current observations, without further candidate simulation.
We appreciate your suggestions and welcome any further questions on these results.
Follow-up on Our Response
Dear Reviewer jkMH, as the discussion period draws to a close, we would appreciate your feedback on our additional results. Our response includes comparisons against a tuned global-margin baseline and Safe-K with matched simulation access, together with tests under execution perturbations unavailable to the planner. Please let us know if these results leave any questions unresolved. Thank you again for your constructive suggestions.
Response to Reviewer jkMH
We thank the reviewer for the positive assessment of the policy-family approach and the suggestions for further evaluation. Below, we address the questions on model mismatch, matched comparisons, and implementation.
Q5 / Weakness 3: execution uncertainty unavailable to the planner.
The additional F16 experiments evaluate EpiPlan with a mismatch between planning and execution. The planner retains the nominal model, candidate policies, and calibration, while executed commands are scaled by .9 or .8 before the original clipping operation. With H=32, the comparison with the root baseline uses the same metrics as the main F16 experiment:
Coverage is the accepted fraction of the 65,280 test states. At gains .9 and .8, EpiPlan with K=6 reduces task cost by 2.4% and 1.7% relative to the root baseline, with slightly lower precision and stabilization rates (Supplement C.8).
We also evaluate hidden Gaussian action noise in MPETarget and MPECorridor, with the initial states and noise realizations paired across methods. EpiPlan uses K=6 and H=32, with nominal-model prediction and calibration retained at both noise levels. The comparison with the public-root baseline at ξ=.21 is:
Each noise level includes 768 MPETarget and 512 MPECorridor test instances, with results averaged equally across controllers. Relative to root execution, EpiPlan reduces failures by 31.9–33.2% in MPETarget and 67.5–79.3% in MPECorridor, with task-cost increases of approximately 5% and 1.1–1.2%, respectively (Supplement C.10).
Q6 / Weakness 4: matched selectors and a tuned global margin.
The new comparisons provide Safe-K and EpiPlan with identical candidate simulations. On MPETarget, EpiPlan with K=11 reduces the failure rate from 10.03% to 7.94% relative to Safe-K, with task cost changing from .251 to .256. It also improves both metrics over calibrated fixed selection and a fixed mixture:
Results are averaged across trained controllers on matched test states. The K=11 candidate set adds 5 policies with different margins and δ=0 to the K=6 displacement candidates. EpiPlan calibrates using joint safety and task performance, while the comparators use a 5% empirical risk target (Supplement C.2). The comparisons therefore assess the complete calibrated selection rules with the same prediction information and simulation budget.
The results also extend to other systems. In Safe HalfCheetah, EpiPlan with K=12 halves failures from 22/256 to 11/256 while maintaining return (1585.12 versus 1584.66). In the nominal F16 comparison with K=6, EpiPlan increases precision from 92.37% to 94.57% and reduces the false-safe rate from 7.63% to 5.43% relative to Safe-K, at similar coverage (71.68% versus 71.41%). On the respective accepted sets, Any50 increases from 98.04% to 98.78% and task cost decreases from .4967 to .4955 (Supplement C.8).
We also compare against a globally tuned margin. Relative to this baseline, EpiPlan with K=11 and H=32 reduces the MPETarget failure rate from 12.11% to 7.94% and task cost from .344 to .256. In MPECorridor, the failure rate decreases from 1.17% to .39% and cost from .312 to .271.
To examine the role of instance-dependent selection, we also preserve candidate-use counts and uniformly permute policy assignments. In the K=6 evaluation across 6 navigation settings, this increases both expected failure rate and task cost in every setting. For example, the MPETarget failure rate increases from 15.36% to 26.62%. This comparison supports the benefit of matching policies to the current instance (Supplement C.9).