acceptodds
Under review as a conference paper at ICLR 2027

HOPE: Hands-Off Counterfactual Policy Explanations for Reinforcement Learning Failures

Abstract

Explaining a reinforcement learning (RL) failure requires identifying when and how the agent could have acted differently to avoid failure while still completing its task. Existing counterfactual approaches modify observations, generate open-loop action sequences, or learn alternative policies, leaving a gap between explaining a specific failure and preserving the deployed policy's feedback. We present HOPE (Hands-Off counterfactual Policy Explanation), a unified framework for generating trajectory-specific counterfactual policies across continuous, discrete, and hybrid action spaces. Each counterfactual policy applies sparse, time-indexed action interventions to change the failure outcome while preserving the deployed policy's feedback at all other times. HOPE separates validity search from intervention reduction, using a graded reach-avoid residual to guide search across continuous, discrete, and hybrid action spaces. Validity-preserving deletion, refitting, and magnitude contraction then reduce intervention burden, while Pareto filtering retains nondominated trade-offs between intervention number and average magnitude. Experiments on 600 failure incidents across three navigation and driving benchmarks yield validity rates of 81%, 74%, and 83% on the common solvable subsets. Across all benchmarks, HOPE ranks first or second in validity and first in single-intervention rate among the evaluated RL counterfactual methods.

Primary area:
interpretability and explainable AI
Keywords:
counterfactual explanations, reinforcement learning, explainable reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6KLFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HOPE explains an observed RL failure by generating a trajectory-specific counterfactual policy. The stated gap over prior work is that state counterfactuals may be dynamically unreachable, open loop action sequence methods don't define feedback off the realised path, and learned counterfactual policies produce task-level alternatives rather than sparse per-incident changes. HOPE unifies continuous, discrete, and hybrid action spaces under one validity condition and separates (a) validity search from (b) intervention reduction. For search it replaces the uninformative binary reach-avoid indicator with a graded reach-avoid residual (Eq 8–9) and uses space-specific operators: a smoothed joint local/SPSA-style model (continuous), loss-ranked beam search (discrete), and alternating type-selection + parameter refinement (hybrid). Reduction then applies validity-preserving deletion, support-restricted refitting, and magnitude contraction, followed by Pareto filtering over (sparsity N, proximity D).

Critical strengths
  1. Very well-posed, practically relevant formulation.
  2. Thorough empirical evaluation.
Critical weaknesses
  1. Validity is checked on exactly one reference randomisation, the method may report counterfactual policies that would not actually avoid failure under other plausible noise draws. For a counterfactual explanation whose whole point is "this change would have prevented the failure," robustness across realisations is central, and its absence is the most consequential soundness gap.
  2. Explanation quality is asserted via proxies, not demonstrated. No user study and no causal ground-truth evaluation that sparse interventions actually improve human understanding or identify the true failure cause.
Questions for authors
  1. How stable is a HOPE counterfactual policy's validity under resampled environment randomisation (not just u(0))?
  2. How would HOPE degrade when only logged trajectories (no re-simulation) are available, which is the realistic deployed-system case?
  3. Is there any evidence that the sparser/Pareto explanations improve a human's ability to diagnose the failure, beyond the sparsity proxy?
2 comments
AuthorsOpenReview ↗

Response (1/2): W1 and Q1, Robustness to Other Noise Draws

Thank you for the positive assessment and the constructive comments.

W1 and Q1 (validity under other noise draws).

Thank you for highlighting this concern. Validity under the factual exogenous sequence establishes that an intervention would have prevented the recorded failure, but does not establish robustness to other plausible realizations. We now make this distinction explicit and measure it. Section 5.5 and Table 5 replay the reported policies, without further search, under five changes (64 draws per incident for exogenous values, 32 for start states, and both one-step shifts; unsolved incidents count as failures):

  • Fresh draws a new exogenous sequence from PuP_{\mathbf u} after kk (the interventional query).
  • Post. keeps u(0)\mathbf u^{(0)} before the failure time tft_f and redraws the later values, which the observed trajectory does not determine.
  • Local mixes u(0)\mathbf u^{(0)} with a new draw at level ρ=0.1\rho=0.1.
  • Start perturbs the start state sk\mathbf s_k.
  • Shift moves every intervention one step earlier or later.

Table 5 (revised paper). Percentage of perturbed rollouts over all incidents that reach the goal without failure. Each cell: B1 / B2 / B3. π0\pi_0 is the deployed policy without interventions. Bold marks the best value among the counterfactual methods for each benchmark and condition.

MethodFresh ↑Post. ↑Local ↑Start ↑Shift ↑
Deployed π0\pi_086.3 / 25.5 / 20.60.0 / 0.0 / 0.014.3 / 5.2 / 2.44.1 / 4.2 / 3.10.0 / 0.0 / 0.0
Random Support38.7 / 20.0 / 19.142.4 / 43.5 / 50.328.3 / 41.1 / 38.331.3 / 24.0 / 27.78.5 / 13.5 / 17.0
HIGHLIGHTS9.2 / 9.2 / 8.810.4 / 35.2 / 28.86.6 / 26.2 / 20.55.0 / 20.2 / 16.12.0 / 7.0 / 6.5
RACCER40.6 / 10.5 / 11.446.0 / 24.1 / 34.830.8 / 18.5 / 25.233.3 / 17.4 / 17.19.5 / 3.0 / 11.5
ACTER65.5 / 18.2 / 23.375.5 / 43.6 / 48.852.1 / 36.7 / 42.557.6 / 30.8 / 43.116.5 / 10.5 / 5.5
CEM61.8 / 18.7 / 19.169.9 / 42.1 / 53.230.0 / 29.6 / 39.232.9 / 22.9 / 30.44.5 / 10.0 / 16.5
A∗A^\ast– / 12.8 / –– / 34.3 / –– / 26.8 / –– / 24.6 / –– / 8.5 / –
HOPE (ours)67.0 / 21.1 / 19.671.5 / 50.2 / 52.742.0 / 36.8 / 40.040.9 / 29.7 / 25.55.5 / 3.5 / 15.0

We interpret these results in two parts: sensitivity to randomness that was not observed after the failure, and sensitivity to changes in the incident conditions or intervention timing.

  • Values after the failure (Post.). HOPE succeeds in 71.5%, 50.2%, and 52.7% of the rollouts on B1–B3, close to its validity of 77.3%, 53.0%, and 58.0%, so its explanations mostly remain valid when these unobserved values change.
  • Fresh exogenous sequence. HOPE succeeds in 67.0%, 21.1%, and 19.6% of the rollouts, and all methods drop to 8.8–23.3% on B2 and B3. The deployed policy itself succeeds in 86.3%, 25.5%, and 20.6%, so many recorded failures do not recur under a new sequence. Fresh thus changes the failure scenario itself, whereas Post. keeps the observed pre-failure trajectory.
  • Small perturbations. Under Local and Start, HOPE succeeds more often than π0\pi_0 (e.g., 42.0% vs. 14.3% and 40.9% vs. 4.1% on B1). ACTER, whose sequences change many steps, is the most robust under Local and Start on B1 and B3.
  • Timing. A one-step shift leaves 5.5%, 3.5%, and 15.0% success, so the intervention times matter.

So the explanations prevent the failure under the factual realization and mostly stay valid when only post-failure values are redrawn, but their robustness to broader changes, especially in timing, is limited. Appendix D now states this limitation and discusses a robust variant that requires Pr⁡u[C(Δ;u)=1]≥1−δ\Pr_{\mathbf u}[C(\boldsymbol\Delta;\mathbf u)=1]\ge1-\delta over exogenous sequences close to u(0)\mathbf u^{(0)}, which the current method does not enforce.

AuthorsOpenReview ↗

Response (2/2): Q2, W2, and Q3, Logged Data and Human Understanding

Q2 (Logged trajectories only).

HOPE needs a simulator that restores the state at kk and replays u(0)\mathbf u^{(0)} (Appendix D). With logs only, ff and u(0)\mathbf u^{(0)} must be estimated. The counterfactual is then identified only up to the fitted model, and different models that fit the same trajectory can give different counterfactuals [1]. We expect validity to degrade with model error, especially when the interventions move the trajectory away from the logged data. The Post. test isolates one source of uncertainty, unknown values after tft_f, with the simulator fixed; it does not test errors of a learned model. Appendix D outlines a path: learned world models with uncertainty estimates and conservative acceptance, followed by independent high-fidelity checks.

W2 and Q3 (Human understanding).

We agree that sparsity and the Pareto set are proxy measures of explanation quality; Appendix D states this limitation. We use sparsity as a proxy because people tend to explain an outcome with a few selected causes [2] (Section 3.2), but proxy measures can disagree with real user performance [3]. Fig. 1 illustrates the intended benefit on one incident: HOPE changes a single action, whereas the other methods need 3 to 31 interventions, and the steps ranked highest by Q-value importance do not prevent the failure. A user study with safety engineers that measures diagnosis accuracy, decision time, and trust is the next step of this work.

On causal ground truth, validity itself is a causal test: each reported change is checked by counterfactual replay under the SCM to prevent this failure. However, our incidents have no labels of the true failure cause, so we do not claim that a sparse explanation identifies the true cause. Testing this on incidents with known causes is future work.

We hope the revision addresses your concerns, and we would be glad to discuss any remaining issues.

References

[1] Oberst, M., & Sontag, D. (2019). Counterfactual off-policy evaluation with gumbel-max structural causal models. ICML, 4881-4890.

[2] Miller, T. (2019). Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267, 1-38.

[3] Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020). Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems. IUI, 454-464.

Reviewer_A5jROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper explains RL failures by finding a small number of time-indexed action changes to the deployed policy that turn a failed rollout into a successful one. At steps without a change, the deployed policy keeps acting on the states actually reached. The search optimizes a continuous penalty built from hazard and goal distances, then greedily deletes, refits, and shrinks interventions while keeping the rollout successful. On Safety-Gymnasium PointGoal and highway-env intersection, HOPE reaches 0.81, 0.74, and 0.83 validity on the incidents some method could solve, and it more often finds single-intervention fixes than 3 other baselines and random time selection.

Critical strengths

The problem is useful. Letting the deployed policy keep acting while changing only a few of its actions is a reasonable middle ground between open-loop action sequences and learning a whole new policy. The Open-loop ablation is the most convincing result in the paper: it shows that letting the policy respond at counterfactual states substantially improves validity.

Critical weaknesses
  1. My main concern is that the statistics and reporting are weak; this is why I lean towards reject. There is a single run with no confidence intervals, and several rankings rest on gaps of one or two incidents. Medians are computed over each method’s own solved incidents, which differ across methods. The paper does not say which Pareto point enters Table 2. Validity is reported over the solvable subset only. The main ε and W are where HOPE looks best (on B3, Random Support leads at smaller ε and W), and there is no description of how the baselines were tuned.
  2. The baselines don’t get the reduction phase. Deletion, refitting, and contraction work on any valid intervention sequence; you should therefore ablate the baselines you compare against to include these processes. Without the same reduction applied, it is unclear what the residual-guided search adds.
  3. I also found the paper unclear at times. Many terms (reach-avoid residual, graded loss, temporal support) are used before they are defined, and at times it felt like a lot of phrases were technical jargon
Questions for authors

Does any baseline receive HOPE’s deletion, refit, and contraction steps? What happens to the comparison if all of them do?

How were W and ε chosen for the main results? Were the baselines tuned on the development incidents?

6 comments
AuthorsOpenReview ↗

Response (1/3): W1, Statistics and Reporting

Thank you for your valuable comments and constructive suggestions. We have revised the statistics, the reporting, and the baseline comparison as follows.

W1 (Statistics and reporting).

We revised the reporting to make the uncertainty, policy-selection rule, and comparison sets explicit. These changes distinguish performance over all incidents from explanation quality on incidents that both HOPE and the compared baseline solve.

  • Seeds. Tables 2 and 3 report the mean ± standard deviation over three seeds; HOPE on B2 and B3 and A∗A^\ast [1] are deterministic. Table R1 in the general response gives validity and P1P_1 for all methods.
  • Reported Pareto point. For each incident, we report the valid policy with the smallest NN and, among those, the smallest DD (Section 5.1).
  • Denominator. Validity and P1P_1 are percentages of all incidents, not of the solvable subset.
  • Different solved sets. Medians over each method's own solved incidents are replaced by paired comparisons on the incidents that both HOPE and the baselines solve (right part of Table 2, below). HOPE more often needs fewer interventions in all 16 baseline–benchmark comparisons and more often has the smaller total magnitude N⋅DN\cdot D in 12. Its individual interventions are often larger: ACTER and CEM [2] on B1 and every baseline on B3 more often have the smaller DD.
  • Close rankings. Table 2 bolds every validity and P1P_1 value within one standard deviation of the best. We draw no conclusion from small validity gaps: the validity of HOPE is 1.7% higher, 5.9% lower, and 1.2% higher, in relative terms, than that of the best other method on B1–B3. The main claim concerns sparsity: before the baselines receive the shared reduction, HOPE has the highest P1P_1 on every benchmark (21.7%, 29.0%, and 39.0% vs. at most 17.0%, 23.0%, and 29.3%).
  • Choice of WW and ϵ\epsilon; baseline tuning. See Q2.

Table 2, right part (revised paper). Paired comparison on the incidents that a method and HOPE both solve. nn: number of such incidents, pooled over seeds and counted once per incident; other columns: percentage of these nn incidents on which each has the lower value (the remaining incidents have equal values). Each cell: Method / HOPE. Bold: the better value of each pair.

Bench.MethodnnSparsity NN ↑Proximity DD ↑N⋅DN\cdot D ↑Recency RR ↑
B1Random Support802 / 510 / 959 / 9038 / 32
B1HIGHLIGHTS220 / 1000 / 1000 / 100100 / 0
B1RACCER942 / 770 / 962 / 980 / 74
B1ACTER1520 / 10054 / 450 / 10016 / 68
B1CEM1363 / 7184 / 1231 / 6924 / 56
B2Random Support968 / 4317 / 198 / 3447 / 21
B2HIGHLIGHTS740 / 9532 / 410 / 9562 / 19
B2RACCER460 / 3913 / 90 / 3513 / 22
B2ACTER900 / 10024 / 580 / 10062 / 24
B2CEM9211 / 1610 / 610 / 1050 / 10
B2A∗A^\ast680 / 9726 / 530 / 9774 / 15
B3Random Support1129 / 2184 / 1471 / 2730 / 50
B3HIGHLIGHTS640 / 8875 / 2516 / 8159 / 34
B3RACCER780 / 3872 / 2151 / 383 / 77
B3ACTER1000 / 10090 / 100 / 1006 / 80
B3CEM1169 / 2483 / 1674 / 2629 / 50
AuthorsOpenReview ↗

Response (2/3): W2 and Q1, Shared Reduction

W2 and Q1 (Shared reduction).

Thank you for this suggestion. We now apply the deletion, refitting, and contraction of HOPE to the reported policy of every baseline, including the new CEM [2] and A∗A^\ast [1] (+R; Section 5.4). Table 4 compares the results with HOPE in the format of Table 2.

Table 4 (revised paper). Shared reduction: each baseline after the reduction of HOPE (+R), in the format of Table 2. P1P_1 is given before → after reduction. Each paired cell: +R / HOPE. Bold: the highest P1P_1 after reduction and the better value of each pair.

Bench.MethodP1P_1 (%) ↑nnSparsity NN ↑Proximity DD ↑N⋅DN\cdot D ↑Recency RR ↑
B1Random Support+R17.0 → 20.3808 / 2944 / 5145 / 4935 / 32
B1HIGHLIGHTS+R0.0 → 1.0220 / 910 / 1000 / 100100 / 0
B1RACCER+R5.7 → 20.08811 / 2341 / 5743 / 552 / 68
B1ACTER+R0.0 → 25.315218 / 2157 / 3949 / 4939 / 30
B1CEM+R9.0 → 30.313613 / 2279 / 1576 / 1641 / 25
B1HOPE (ours)21.7–––––
B2Random Support+R18.7 → 23.7969 / 915 / 118 / 1147 / 13
B2HIGHLIGHTS+R1.0 → 8.7745 / 5130 / 2211 / 5159 / 22
B2RACCER+R12.0 → 17.0460 / 1713 / 00 / 1313 / 22
B2ACTER+R0.0 → 14.3907 / 4911 / 337 / 5171 / 16
B2CEM+R23.0 → 25.09210 / 46 / 610 / 652 / 6
B2A∗A^\ast+R1.0 → 13.0683 / 569 / 293 / 5674 / 15
B2HOPE (ours)29.0–––––
B3Random Support+R29.0 → 32.711214 / 786 / 1486 / 1434 / 50
B3HIGHLIGHTS+R3.3 → 11.3640 / 5059 / 3131 / 6259 / 28
B3RACCER+R18.3 → 21.0780 / 3669 / 2351 / 383 / 79
B3ACTER+R0.0 → 6.31008 / 6660 / 3026 / 6824 / 50
B3CEM+R29.3 → 29.711610 / 1784 / 1478 / 2129 / 47
B3HOPE (ours)39.0–––––

Reduction raises P1P_1 for every baseline (e.g., ACTER from 0.0% to 25.3% on B1) and narrows the paired gaps of Table 2: the advantage of HOPE in NN decreases against every baseline, and CEM+R on B2 and Random Support+R on B3 now more often need fewer interventions than HOPE. After reduction, HOPE keeps the highest P1P_1 on B2 and B3 (29.0% and 39.0% vs. at most 25.0% and 32.7%), but not on B1, where CEM+R reaches 30.3%.

Thus, part of the sparsity advantage comes from the reduction, while the residual-guided search still gives a higher P1P_1 on B2 and B3. The ablation (Table 3) also supports the search design: removing closed-loop feedback or validity-first search lowers validity by up to 37.9% and 26.6%, respectively.

AuthorsOpenReview ↗

Response (3/3): W3 and Q2, Clarity and Settings

W3 (Clarity).

We revised the paper for clarity. The introduction now explains validity search, intervention reduction, the reach-avoid residual and its graded loss, and temporal support at first mention. Section 3 gives the full problem before the method, Appendix A.1 (Table 6) lists the notation, and Fig. 1 shows the policies of all methods on one incident.

Q2 (Choice of WW and ϵ\epsilon; baseline tuning).

All settings are selected on development incidents and frozen before evaluation (Appendix B.4). The main setting uses W=min⁡(90,tf)W=\min(90,t_f) and ϵ=0.5\epsilon=0.5 on B1, W=36W=36 on B2, and W=36W=36 and ϵ=1\epsilon=1 on B3. The baselines use the same window, bound, and validity test as HOPE. Their per-incident transition caps equal those of HOPE on B2 and B3 (including reduction) and are larger on B1 (44.86M vs. 24.10M transitions in total; Appendices B.3–B.4).

To assess sensitivity beyond the main setting, Appendices C.2–C.3 report all methods across five windows and four magnitude bounds. On B1, the validity of HOPE stays at 74–78% for every ϵ\epsilon, and its P1P_1 is the highest at every bound. Larger bounds help CEM more: at ϵ=1\epsilon=1 and 22, its validity is 88% and 98% vs. 78% and 77% for HOPE, but its P1P_1 is 11% and 0%. As you note, Random Support has slightly higher validity on B3 for ϵ<0.75\epsilon< 0.75 and at W=18W=18; the appendix reports these results. The P1P_1 of HOPE is the highest or equal to the highest at every window except W=9W=9 on B3.

We hope the revision addresses your concerns, and we would be glad to discuss any remaining issues.

References

[1] Tsirtsis, S., & Rodriguez, M. (2023). Finding counterfactually optimal action sequences in continuous state spaces. NeurIPS, 36, 3220-3247.

[2] De Boer, P. T., Kroese, D. P., Mannor, S., & Rubinstein, R. Y. (2005). A tutorial on the cross-entropy method. Annals of Operations Research, 134(1), 19-67.

Reviewer_DE1KOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

HOPE starts from a simulator snapshot taken before an RL failure, plus the stored per-step randomization. It searches for sparse, time-indexed perturbations of the deployed policy's actions, executed in closed loop, that make the rollout reach the goal while avoiding failure. Over hundreds of incidents tested, the proposed method achieves a high validity rate.

Critical strengths

The paper is clearly written and well specified

Critical weaknesses
  • It's hard to talk about counterfactuals rigorously without mentioning any well established causal framework. In the current presentation, there is no Structural Causal Model (SCM) defined anywhere nor Potential Outcome (PO) framework used. The paper simply redefines "counterfactual" as "replay in a resetable simulator".
  • From a rigor SCM view, this reduces the problem to simply do interventions in a deterministic system without any identification issue which is the biggest challenges for real world ctf applications.
  • It's very possible that different SCMs can be compatible with the same observed trajecotry and the paper totally ignored this, which is plausible because no formal causal treatment is used in the discussion.
Questions for authors
  1. What per-step randomness exists in each benchmark? If there is none, why is this a "counterfactual" rather than an "interventional replay"?
  2. When the counterfactual trajectory consumes random draws differently, how is the stream of draws matched to the mechanisms that consume them?
  3. What fraction of HOPE's explanations remain valid under small perturbations of randomness, the initial state, or the intervention time?
  4. How does Random Support choose actions? What do CEM+deletion and Tsirtsis & Gomez-Rodriguez (2023, on B2) achieve at an equal budget?
  5. Could you rephrase your method in proper causal semantics and double check its validity under SCM framework?
13 comments
AuthorsOpenReview ↗

Response (1/4): W1–W3, Causal Formulation and Identification

Thank you for your valuable and constructive feedback, particularly on the causal formulation. We have restated our method in structural causal model (SCM) and added experiments to evaluate additional baselines and assess robustness.

W1 (Causal formulation).

Thank you for pointing out that the causal semantics were not explicit. Section 3.1 now formulates the problem as an SCM. The state st\mathbf s_t and action at\mathbf a_t are endogenous. The exogenous variable ut\mathbf u_t contains the random inputs at time tt, and u=(u0,…,uH−1)∼Pu\mathbf u=(\mathbf u_0,\ldots,\mathbf u_{H-1})\sim P_{\mathbf u}. The structural equations are

at:=π0(st),st+1:=f(st,at,ut).\mathbf a_t:=\pi_0(\mathbf s_t),\qquad \mathbf s_{t+1}:=f(\mathbf s_t,\mathbf a_t,\mathbf u_t).

Both π0\pi_0 and ff are deterministic, so all randomness of a rollout enters through u\mathbf u. A counterfactual policy intervenes on the action equation: a^t=πΔ(s^t,t):=φ(π0(s^t),Δt)\hat{\mathbf a}_t=\pi_{\boldsymbol\Delta}(\hat{\mathbf s}_t,t):=\varphi(\pi_0(\hat{\mathbf s}_t),\boldsymbol\Delta_t), where Δt\boldsymbol\Delta_t is the action intervention at tt and φ\varphi applies it to the action. An explanation answers a counterfactual query about one incident [1]: abduction recovers the factual exogenous sequence u(0)\mathbf u^{(0)}, action replaces the action equation with πΔ\pi_{\boldsymbol\Delta}, and prediction propagates s^t+1=f(s^t,πΔ(s^t,t),ut(0))\hat{\mathbf s}_{t+1}=f(\hat{\mathbf s}_t,\pi_{\boldsymbol\Delta}(\hat{\mathbf s}_t,t),\mathbf u^{(0)}_t) from s^k=sk\hat{\mathbf s}_k=\mathbf s_k. Simulator replay is therefore not a new meaning of "counterfactual": the simulator implements ff, the stored u(0)\mathbf u^{(0)} contains the result of abduction, and replay carries out the three steps exactly.

W2 (Scope and identification).

Thank you for raising the issue of causal identification. HOPE does not seek to identify causal structure from observational data. It assumes a known simulator transition function ff and the recorded exogenous sequence u(0)\mathbf u^{(0)}. Under this access, the counterfactual query is identified [2]. Prior work makes the similar assumption, with a known SCM [3,4] or a simulator that can be queried [5,6]. Section 3.1 and Appendix D state this access assumption and its limits. The system is still not deterministic: ut\mathbf u_t enters every step, and keeping u(0)\mathbf u^{(0)} is what makes the query counterfactual rather than interventional (Q1).

Identification lets us evaluate a given intervention, but does not tell us which intervention to choose. Our contribution is this search: finding sparse closed-loop interventions that prevent the failure. Unlike [3], which uses dynamic programming in finite discrete MDPs, and [4], which returns open-loop action sequences, HOPE keeps π0\pi_0 in control at every step without an intervention and handles continuous, discrete, and hybrid actions in a black-box simulator. The search is hard because the binary validity indicator cannot rank invalid candidates, and early interventions change later states and actions. The ablation (Table 3) shows that the search design matters: removing closed-loop feedback or validity-first search lowers validity by up to 37.9% and 26.6%, respectively.

W3 (SCM ambiguity from logs).

We agree that a logged trajectory alone does not identify a unique SCM, and different SCMs can give different counterfactuals [2]. HOPE does not infer ff and u(0)\mathbf u^{(0)} from the trajectory; it takes them from the simulator. Even with ff known, the trajectory would not fully determine u(0)\mathbf u^{(0)}: in B1, clipping the action to [−1,1][-1,1] hides part of the actuator noise, and in B2 and B3, an arrival proposal rejected by the spacing rule leaves no trace in the trajectory but can be accepted in a counterfactual rollout. We check the replay by restoring the stored snapshot and confirming that it reproduces the per-step state checksums and the observed failure. Our claim is therefore conditional on the specified simulator SCM; it is not a model-free conclusion from logs.

The exogenous values after the failure time tft_f are also not determined by the observed trajectory. When we redraw them (Post., Section 5.5), HOPE succeeds in 71.5%, 50.2%, and 52.7% of the rollouts on B1–B3, compared with its validity of 77.3%, 53.0%, and 58.0%. Appendix D discusses the limits of applying the method to logs or learned models.

AuthorsOpenReview ↗

Response (2/4): Q1–Q3, Randomness, Matching, and Robustness

Q1 (Per-step randomness).

Thank you for asking us to clarify the source and role of per-step randomness. Appendix B.1 specifies ut\mathbf u_t for each benchmark. B1: the MuJoCo dynamics are deterministic, the built-in action noise of Safety-Gymnasium is off, and ut\mathbf u_t is Gaussian actuator noise (σ=1.25\sigma=1.25 in normalized action units) on both action channels. B2 and B3: ut\mathbf u_t is one vehicle-arrival proposal per policy step, made with probability 0.167 (at 5 Hz), which draws the route origin, route offset, initial position, and initial speed of a new vehicle. The spacing rule of highway-env, which accepts or rejects each proposal, is part of ff.

Keeping u(0)\mathbf u^{(0)} fixed is what makes our query counterfactual rather than an interventional replay: it asks whether a change would have prevented this particular failure, whereas redrawing u\mathbf u asks whether the changed policy succeeds in a new realization. The deployed policy π0\pi_0 fails under u(0)\mathbf u^{(0)} by construction, but with a fresh exogenous sequence it succeeds in 86.3%, 25.5%, and 20.6% of the rollouts on B1–B3 (Table 5). An interventional replay would therefore often not even reproduce the observed failure, especially on B1, and could not explain it.

Q2 (Matching random draws).

Keeping u(0)\mathbf u^{(0)} fixed requires matching random inputs by their role, even when interventions change the trajectory. To ensure this, a counter-based random number generator (Philox) computes each value of ut\mathbf u_t from the incident seed, time step, entity, and channel. No value is taken from a shared random stream, so a changed action changes the states that later mechanisms receive but not their exogenous values, and all candidates are evaluated under the same u(0)\mathbf u^{(0)} (Section 3.1, Appendix B.1). For example, in B2 and B3, every candidate receives the same arrival proposal at each step; whether the proposal is accepted depends on the counterfactual state through ff. The values of u(0)\mathbf u^{(0)} before tft_f come from the observed episode. A counterfactual rollout can continue after tft_f until HH; the values of u(0)\mathbf u^{(0)} for these steps are generated from the same incident seed and are the same for all candidates. Post. in Q3 redraws them.

Q3 (Robustness to perturbations).

Thank you for this question. Section 5.5 replays the reported policies, without further search under five changes: Fresh draws a new exogenous sequence after kk (the interventional query); Post. keeps u(0)\mathbf u^{(0)} before tft_f and redraws the later values; Local mixes u(0)\mathbf u^{(0)} with a new draw at level ρ=0.1\rho=0.1; Start perturbs the start state sk\mathbf s_k; and Shift moves every intervention one step earlier or later. We use 64 draws per incident for the exogenous values, 32 for start states, and both shifts. The rates are over all incidents, and an unsolved incident counts as a failure, so they can be compared directly with validity.

Table R2. Successful rollouts (%) over all incidents on B1 / B2 / B3 (from Table 5 in the revised paper, which also reports Fresh and all methods).

Post. ↑ (values after tft_f)Local ↑ (ρ=0.1\rho=0.1)Start ↑Shift ↑ (±1 step)
HOPE71.5 / 50.2 / 52.742.0 / 36.8 / 40.040.9 / 29.7 / 25.55.5 / 3.5 / 15.0
Deployed π0\pi_00.0 / 0.0 / 0.014.3 / 5.2 / 2.44.1 / 4.2 / 3.10.0 / 0.0 / 0.0

Table R2 shows that robustness depends on which aspect of the incident is changed. Redrawing only the unobserved post-failure values keeps success at 50.2–71.5%. Local and Start give lower rates (36.8–42.0% and 25.5–40.9%), but still well above π0\pi_0, so the interventions keep their effect; ACTER, whose sequences change many steps, is more robust under Local and Start on B1 and B3 (Table 5). Timing is sensitive: a one-step shift leaves HOPE 3.5–15.0% success. Under a fresh exogenous sequence, HOPE succeeds in 67.0%, 21.1%, and 19.6% of the rollouts, so an explanation for one incident does not ensure success under a new sequence. Appendix D states this limitation and discusses a robust variant.

AuthorsOpenReview ↗

Response (3/4): Q4, Random Support and Equal-Budget Baselines

Q4 (Random Support; equal budget).

Random Support samples a support size from 1,2,3,4,6,8,12,16\\{1,2,3,4,6,8,12,16\\} and uniformly selects that many intervention times within the window. At each selected time, it samples an intervention: a random direction with norm ϵ\epsilon on B1, a random alternative action on B2, and a random intervention type and parameter on B3. It keeps every valid draw (Appendix B.3).

We added CEM [7] on all benchmarks and the A∗A^\ast search of [4] on B2, and applied the reduction of HOPE (deletion, refitting, and contraction) to every baseline (+R). Each baseline has a per-incident cap on simulator transitions. On B2 and B3, the cap equals the transitions that HOPE uses on the same incident, including reduction. On B1, the baselines get a larger budget: 44.86M transitions in total, versus 24.10M for HOPE (1.86 times; Appendix B.4). For +R, the reduction runs after the search without a cap, so +R baselines can use more transitions than HOPE.

Table R3. Validity, P1P_1, and P1P_1 after the reduction of HOPE (%; higher is better), from Tables 2 and 4 in the revised paper.

B1B2B3
CEM: validity / P1P_1 / P1P_1 (+R)70.7 / 9.0 / 30.350.3 / 23.0 / 25.057.3 / 29.3 / 29.7
A∗A^\ast: validity / P1P_1 / P1P_1 (+R)–38.0 / 1.0 / 13.0–
HOPE: validity / P1P_177.3 / 21.753.0 / 29.058.0 / 39.0

HOPE has higher validity than CEM on B1 and than A∗A^\ast on B2, and similar validity to CEM on B2 and B3. CEM+R has a higher P1P_1 than HOPE on B1, with the larger budget above. On B2 and B3, HOPE has the higher P1P_1, although CEM+R can use more transitions. Reduction raises the P1P_1 of A∗A^\ast from 1.0% to 13.0%.

A∗A^\ast has a low P1P_1 although 76.3% of its valid sequences change only one factual action. After that change, the trajectory leaves the observed one, so later replayed factual actions can differ from what π0\pi_0 would choose at the reached states. Counted against π0\pi_0 at those states, A∗A^\ast solves only 1.0% of the incidents with a single intervention (Section 5.2). HOPE avoids this mismatch because π0\pi_0 acts at every step without an intervention.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (Part 1/2)

We thank all reviewers for their time and constructive feedback. We have revised the manuscript based on the feedback. Revised text in the manuscript is highlighted using blue color.

The main revisions address the causal formulation, the fairness and reporting of the comparisons, and the robustness of the explanations:

  1. Causal formulation (DE1K). Section 3.1 now defines a structural causal model (SCM) with endogenous states and actions, exogenous variables ut\mathbf u_t, and the structural equations at:=π0(st)\mathbf a_t:=\pi_0(\mathbf s_t) and st+1:=f(st,at,ut)\mathbf s_{t+1}:=f(\mathbf s_t,\mathbf a_t,\mathbf u_t). An explanation is a counterfactual query computed by abduction, action, and prediction under the factual exogenous sequence u(0)\mathbf u^{(0)}. Section 5.1 and Appendix B.1 give ut\mathbf u_t for each benchmark and how it is matched across rollouts.
  2. Statistics and reporting (A5jR). Tables 2 and 3 report means ± standard deviations across three seeds. Validity and the single-intervention rate P1P_1 are evaluated over all 200 incidents in each benchmark. Sparsity, proximity, and recency are compared between HOPE and each baseline only on the incidents that both of them solve. For each incident, we report the valid policy with the fewest interventions NN and, among those, the smallest average magnitude DD (Section 5.1).
  3. Baselines (DE1K, A5jR). We added the cross-entropy method (CEM) [2] on all benchmarks and the A∗A^\ast search method of Tsirtsis & Gomez-Rodriguez [1] on B2. Each baseline has a per-incident cap on simulator transitions, equal to that of HOPE on B2 and B3 and larger on B1 (Appendices B.3–B.4). We also apply the reduction of HOPE to every baseline (+R; Section 5.4, Table 4).
  4. Robustness (DE1K, 6KLF). Section 5.5 and Table 5 evaluate the robustness of the reported policies by replaying them under five perturbations: a fresh exogenous sequence, redrawn post-failure values, local mixing with a new draw, a perturbed initial state, and a one-step shift in intervention timing. The deployed policy π0\pi_0 is included as a reference.
  5. Settings and sensitivity (A5jR). All settings are selected using development incidents and fixed before evaluation (Appendix B.4). Appendices C.2–C.3 report results for all methods across five intervention windows WW and four magnitude bounds ϵ\epsilon.
  6. Clarity (A5jR, 6KLF). Terms are defined at first use. Fig. 1 shows the policies produced by all methods for the same incident. Appendix D discusses simulator access, robustness, and user evaluation.

References

[1] Tsirtsis, S., & Rodriguez, M. (2023). Finding counterfactually optimal action sequences in continuous state spaces. NeurIPS, 36, 3220-3247.

[2] De Boer, P. T., Kroese, D. P., Mannor, S., & Rubinstein, R. Y. (2005). A tutorial on the cross-entropy method. Annals of Operations Research, 134(1), 19-67.

AuthorsOpenReview ↗

General Response (Part 2/2)

These revisions are supported by additional baseline, ablation, and robustness experiments. The main findings across 600 failure incidents (200 per benchmark) are as follows:

  • Validity. HOPE ranks first or second in validity (77.3%, 53.0%, and 58.0% on B1–B3). Its validity is 1.7% higher, 5.9% lower, and 1.2% higher, in relative terms, than that of the best other method (Table R1 below; Table 2 in the revised paper).
  • Sparsity. Before applying the shared reduction to the baselines, HOPE has the highest P1P_1 on every benchmark (21.7%, 29.0%, and 39.0%, compared with at most 17.0%, 23.0%, and 29.3% for the baselines). On the incidents that both HOPE and a baseline solve, HOPE more often needs fewer interventions in all 16 baseline–benchmark comparisons and more often has the smaller total intervention magnitude N⋅DN\cdot D in 12, although its individual interventions are often larger (right part of Table 2; full table in the response to Reviewer A5jR).
  • Ablation. Without closed-loop feedback or validity-first search, validity decreases by up to 37.9% and 26.6%, respectively, relative to the full method. Greedy instead of joint selection of intervention times lowers B1 validity by 39.2% (Table 3 in the revised paper).
  • Shared reduction. The intervention reduction of HOPE also applies to other methods and raises the P1P_1 of every baseline (e.g., ACTER from 0.0% to 25.3% on B1). After reduction, HOPE keeps the higher P1P_1 on B2 and B3 (29.0% and 39.0% vs. at most 25.0% and 32.7%), but not on B1, where CEM+R reaches 30.3% (Table 4; full table in the response to Reviewer A5jR).
  • Robustness. When the exogenous values after the failure are redrawn, HOPE succeeds in 71.5%, 50.2%, and 52.7% of the rollouts, close to its validity. Under local exogenous changes and perturbed start states, it succeeds more often than π0\pi_0. A one-step shift of the interventions leaves 5.5%, 3.5%, and 15.0% success, and fresh exogenous sequences do not ensure success, which Appendix D states as a limitation (Table 5; full table in the response to Reviewer 6KLF).

Table R1. Validity and P1P_1: percentages of all incidents solved by a valid policy and by one with a single intervention, mean ± standard deviation over three seeds (left part of Table 2 in the revised paper). HOPE on B2 and B3 and A∗A^\ast are deterministic. Bold: the highest value in each column.

MethodB1 Validity (%) ↑B1 P1P_1 (%) ↑B2 Validity (%) ↑B2 P1P_1 (%) ↑B3 Validity (%) ↑B3 P1P_1 (%) ↑
Random Support45.3 ± 1.217.0 ± 4.456.3 ± 2.918.7 ± 3.556.3 ± 1.229.0 ± 2.6
HIGHLIGHTS11.0 ± 0.00.0 ± 0.039.7 ± 1.51.0 ± 0.033.0 ± 0.03.3 ± 0.6
RACCER47.3 ± 1.55.7 ± 0.626.0 ± 0.012.0 ± 0.039.0 ± 0.018.3 ± 0.6
ACTER76.0 ± 0.00.0 ± 0.051.0 ± 1.70.0 ± 0.054.3 ± 3.10.0 ± 0.0
CEM (new)70.7 ± 0.69.0 ± 2.050.3 ± 2.323.0 ± 1.057.3 ± 1.529.3 ± 2.5
A∗A^\ast (new)––38.01.0––
HOPE77.3 ± 1.521.7 ± 1.553.029.058.039.0

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.