- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This work proposes Counterfactual KKT (CF-KKT), a constraint learning framework that leverages learned dynamics and locally optimal demonstrations without requiring risky exploration. The proposed approach uses a locally learned differentiable dynamics model to impose KKT-inspired optimality conditions directly on the demonstrations. The learnt dynamics are used to generate reward-improving counterfactual behaviors near the demonstrations, revealing behaviors that would be preferable in the absence of the unknown constraint, providing synthetic infeasible data. Authors prove that positive counterfactual advantage in an H-step horizon implies infeasibility. Using a set of counterfactual state-action pairs, the proposed approach trains the average learned constraint value along the rollout to exceed a positive margin m. As authors demonstrate, the proposed approach learns neural constraint representations with improved safety and data efficiency relative to state-of-the-art offline ICRL baselines.
Critical strengths
- Very concisely documented research effort, with a clear formulation, analysis and stated assumptions.
- The use of perturbations in known safe trajectories to create adversarial examples that are considered unsafe (without requiring interaction in any setting) is an interesting, novel approach
Critical weaknesses
- The main assumption that perturbed safe (demonstrated, expert) trajectories are unsafe does not hold in general. For instance, this happens in cases where expert trajectories cross states that are “far” from unsafe states (e.g. spatially far in the maze settings)
- The assumption that demonstrated trajectories are the optimal ones, in practice breaks down, and this causes Lemma1 to break down. This is also implied by the example trajectories shown in the Appendix (Figure 3, purple routes are the oracles, and in reporting the violations related to oracles).
- Hit maps showing states’ occupancy by demonstrated / expert trajectories must be provided, along with perturbed ones: This is very relevant to uncovering the constraints.
- A study on the effects of the “degree” of perturbation is very relevant.
- Authors must discuss cases where baselines score higher violation but also, higher return: Methods spending on cost may result into more effective tsk execution. The balance achieved in optimizing must be commented.
- Authors do miss a very relevant work (https://doi.org/10.65109/RJIB1203): This should be mentioned and be used as a baseline.
Minor: V* is mentioned as cost-to-go, I guess this is erroneous and should be reward-to-go as in other places in the paper. Otherwise, please explain.
After the rebuttal: I am satisfied by the answers and revision provided by authors.
Questions for authors
Can you please address the weaknesses provided above?
After the rebuttal: I am satisfied by the answers and revision provided by authors.
7 comments
Response to W2
W2: We agree that demonstrations are not exactly optimal in practice, and our experiments reflect this. Grid-based Dijkstra paths with PD tracking are not optimal in the continuous maze, and the MuJoCo experts are trained PPO-Lagrangian policies with no optimality guarantee. Lemma 1 is the idealized statement, and the robustness extension of Lemma 1 we establish in App. E.2 covers an -suboptimal expert and shows that suboptimality only produces conservative errors. Empirically, App. C validates the generated counterfactuals against ground truth using the learned, expert-only dynamics. In App. F.1, we report the constraint learning results from expert demos that are intentionally made suboptimal by inflating the obstacle size in the Maze experiments, showing that CF-KKT can learn a more conservative yet still safe constraint map with such experts that maintain additional safety margins.
Response to W3
W3: We have added such hit maps in App. F.3, Fig. 6. For each maze we show the occupancy density of the expert demonstrations, of all perturbed rollouts, and of the rollouts accepted by our rule, overlaid on the arena. It can be observed that not all perturbations result in an infeasible trajectory, while our optimality-based counterfactual selection rule correctly identifies trajectories taking shortcuts through the obstacle, violating constraints to improve their reward.
Response to W1
W1: We agree that "perturbed safe (demonstrated, expert) trajectories are unsafe" does not hold in general, and we wish to emphasize that our method does not, in fact, make such an assumption. Perturbations are not labelled unsafe by default. Each candidate rollout is scored by its value-augmented advantage over the expert from the same state, and only candidates that improve on the expert are labelled (wk=1[A^V(k)>0], line 294), while all others are discarded. The selection of counterfactual states is therefore targeted, retaining only perturbations whose reward improvement can only be explained by a constraint violation. Under the assumptions of Lemma 1, feasible perturbations cannot induce positive advantage. When the expert trajectory lies strictly inside the feasible region, sufficiently small perturbations remain feasible. Under local optimality, these perturbations cannot improve the expert’s value-augmented return, so they have non-positive advantage and are rejected by the selection rule. Lemma 1 formalizes this for an optimal expert, and the new robust lemma (App. E.2) extends it to approximate dynamics and to an εE-suboptimal expert, where the guarantee holds for perturbations that outperform the expert by more than a margin τ. Empirically, App. C validates the generated counterfactuals against ground truth using the learned, expert-only dynamics, reporting that the generated counterfactual trajectory has >90 chance of containing a violating transition.