- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HOPE explains an observed RL failure by generating a trajectory-specific counterfactual policy. The stated gap over prior work is that state counterfactuals may be dynamically unreachable, open loop action sequence methods don't define feedback off the realised path, and learned counterfactual policies produce task-level alternatives rather than sparse per-incident changes. HOPE unifies continuous, discrete, and hybrid action spaces under one validity condition and separates (a) validity search from (b) intervention reduction. For search it replaces the uninformative binary reach-avoid indicator with a graded reach-avoid residual (Eq 8–9) and uses space-specific operators: a smoothed joint local/SPSA-style model (continuous), loss-ranked beam search (discrete), and alternating type-selection + parameter refinement (hybrid). Reduction then applies validity-preserving deletion, support-restricted refitting, and magnitude contraction, followed by Pareto filtering over (sparsity N, proximity D).
Critical strengths
- Very well-posed, practically relevant formulation.
- Thorough empirical evaluation.
Critical weaknesses
- Validity is checked on exactly one reference randomisation, the method may report counterfactual policies that would not actually avoid failure under other plausible noise draws. For a counterfactual explanation whose whole point is "this change would have prevented the failure," robustness across realisations is central, and its absence is the most consequential soundness gap.
- Explanation quality is asserted via proxies, not demonstrated. No user study and no causal ground-truth evaluation that sparse interventions actually improve human understanding or identify the true failure cause.
Questions for authors
- How stable is a HOPE counterfactual policy's validity under resampled environment randomisation (not just u(0))?
- How would HOPE degrade when only logged trajectories (no re-simulation) are available, which is the realistic deployed-system case?
- Is there any evidence that the sparser/Pareto explanations improve a human's ability to diagnose the failure, beyond the sparsity proxy?
2 comments
Response (2/2): Q2, W2, and Q3, Logged Data and Human Understanding
Q2 (Logged trajectories only).
HOPE needs a simulator that restores the state at and replays (Appendix D). With logs only, and must be estimated. The counterfactual is then identified only up to the fitted model, and different models that fit the same trajectory can give different counterfactuals [1]. We expect validity to degrade with model error, especially when the interventions move the trajectory away from the logged data. The Post. test isolates one source of uncertainty, unknown values after , with the simulator fixed; it does not test errors of a learned model. Appendix D outlines a path: learned world models with uncertainty estimates and conservative acceptance, followed by independent high-fidelity checks.
W2 and Q3 (Human understanding).
We agree that sparsity and the Pareto set are proxy measures of explanation quality; Appendix D states this limitation. We use sparsity as a proxy because people tend to explain an outcome with a few selected causes [2] (Section 3.2), but proxy measures can disagree with real user performance [3]. Fig. 1 illustrates the intended benefit on one incident: HOPE changes a single action, whereas the other methods need 3 to 31 interventions, and the steps ranked highest by Q-value importance do not prevent the failure. A user study with safety engineers that measures diagnosis accuracy, decision time, and trust is the next step of this work.
On causal ground truth, validity itself is a causal test: each reported change is checked by counterfactual replay under the SCM to prevent this failure. However, our incidents have no labels of the true failure cause, so we do not claim that a sparse explanation identifies the true cause. Testing this on incidents with known causes is future work.
We hope the revision addresses your concerns, and we would be glad to discuss any remaining issues.
References
[1] Oberst, M., & Sontag, D. (2019). Counterfactual off-policy evaluation with gumbel-max structural causal models. ICML, 4881-4890.
[2] Miller, T. (2019). Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267, 1-38.
[3] Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020). Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems. IUI, 454-464.
Response (1/2): W1 and Q1, Robustness to Other Noise Draws
Thank you for the positive assessment and the constructive comments.
W1 and Q1 (validity under other noise draws).
Thank you for highlighting this concern. Validity under the factual exogenous sequence establishes that an intervention would have prevented the recorded failure, but does not establish robustness to other plausible realizations. We now make this distinction explicit and measure it. Section 5.5 and Table 5 replay the reported policies, without further search, under five changes (64 draws per incident for exogenous values, 32 for start states, and both one-step shifts; unsolved incidents count as failures):
Table 5 (revised paper). Percentage of perturbed rollouts over all incidents that reach the goal without failure. Each cell: B1 / B2 / B3. π0 is the deployed policy without interventions. Bold marks the best value among the counterfactual methods for each benchmark and condition.
We interpret these results in two parts: sensitivity to randomness that was not observed after the failure, and sensitivity to changes in the incident conditions or intervention timing.
So the explanations prevent the failure under the factual realization and mostly stay valid when only post-failure values are redrawn, but their robustness to broader changes, especially in timing, is limited. Appendix D now states this limitation and discusses a robust variant that requires Pru[C(Δ;u)=1]≥1−δ over exogenous sequences close to u(0), which the current method does not enforce.