acceptodds
Under review as a conference paper at ICLR 2027

Learning Unknown Constraints without Unsafe Data via Optimality and Counterfactual Regularization

Abstract

Learning from demonstrations (LfD) provides a framework for inferring unknown constraints from locally optimal, constraint-satisfying expert behavior. Existing approaches largely fall into two paradigms, constrained inverse optimal control (CIOC) and inverse constrained reinforcement learning (ICRL). CIOC exploits optimality conditions such as the Karush–Kuhn–Tucker (KKT) conditions but typically assumes known dynamics and structured constraint representations. Meanwhile, ICRL accommodates complex unknown constraints and unknown transition dynamics but often requires extensive online exploration, during which unsafe constraint violations may occur. In this work, we introduce Counterfactual KKT (CF-KKT), a constraint learning framework that leverages learned dynamics and locally optimal demonstrations to recover unknown constraints without requiring known dynamics or additional risky exploration, thereby combining the data efficiency and safety advantages of CIOC with the flexibility of ICRL. First, we use a locally learned differentiable dynamics model to impose KKT-inspired optimality conditions directly on the demonstrations. Second, we use the learned dynamics to generate reward-improving counterfactual behaviors near the demonstrations, revealing behaviors that would be preferable in the absence of the unknown constraint and thus providing synthetic infeasible data. When the constraint parameterization is known, the same learned-dynamics framework enables direct CIOC-based parameter recovery, and we characterize its sensitivity to dynamics misspecification. Across high-dimensional robotic control tasks, our approach learns neural constraint representations with improved safety and data efficiency relative to state-of-the-art offline ICRL baselines.

Primary area:
reinforcement learning
Keywords:
learning from demonstrations, inverse constrained reinforcement learning, inverse optimal control, safe reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rJWCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work proposes Counterfactual KKT (CF-KKT), a constraint learning framework that leverages learned dynamics and locally optimal demonstrations without requiring risky exploration. The proposed approach uses a locally learned differentiable dynamics model to impose KKT-inspired optimality conditions directly on the demonstrations. The learnt dynamics are used to generate reward-improving counterfactual behaviors near the demonstrations, revealing behaviors that would be preferable in the absence of the unknown constraint, providing synthetic infeasible data. Authors prove that positive counterfactual advantage in an H-step horizon implies infeasibility. Using a set of counterfactual state-action pairs, the proposed approach trains the average learned constraint value along the rollout to exceed a positive margin m. As authors demonstrate, the proposed approach learns neural constraint representations with improved safety and data efficiency relative to state-of-the-art offline ICRL baselines.

Critical strengths
  • Very concisely documented research effort, with a clear formulation, analysis and stated assumptions.
  • The use of perturbations in known safe trajectories to create adversarial examples that are considered unsafe (without requiring interaction in any setting) is an interesting, novel approach
Critical weaknesses
  • The main assumption that perturbed safe (demonstrated, expert) trajectories are unsafe does not hold in general. For instance, this happens in cases where expert trajectories cross states that are “far” from unsafe states (e.g. spatially far in the maze settings)
  • The assumption that demonstrated trajectories are the optimal ones, in practice breaks down, and this causes Lemma1 to break down. This is also implied by the example trajectories shown in the Appendix (Figure 3, purple routes are the oracles, and in reporting the violations related to oracles).
  • Hit maps showing states’ occupancy by demonstrated / expert trajectories must be provided, along with perturbed ones: This is very relevant to uncovering the constraints.
  • A study on the effects of the “degree” of perturbation is very relevant.
  • Authors must discuss cases where baselines score higher violation but also, higher return: Methods spending on cost may result into more effective tsk execution. The balance achieved in optimizing must be commented.
  • Authors do miss a very relevant work (https://doi.org/10.65109/RJIB1203): This should be mentioned and be used as a baseline.

Minor: V* is mentioned as cost-to-go, I guess this is erroneous and should be reward-to-go as in other places in the paper. Otherwise, please explain.

After the rebuttal: I am satisfied by the answers and revision provided by authors.

Questions for authors

Can you please address the weaknesses provided above?

After the rebuttal: I am satisfied by the answers and revision provided by authors.

7 comments
AuthorsOpenReview ↗

Response to W1

W1: We agree that "perturbed safe (demonstrated, expert) trajectories are unsafe" does not hold in general, and we wish to emphasize that our method does not, in fact, make such an assumption. Perturbations are not labelled unsafe by default. Each candidate rollout is scored by its value-augmented advantage over the expert from the same state, and only candidates that improve on the expert are labelled (wk=1[A^V(k)>0]w_k = \mathbf 1[\hat A_V^{(k)} > 0], line 294), while all others are discarded. The selection of counterfactual states is therefore targeted, retaining only perturbations whose reward improvement can only be explained by a constraint violation. Under the assumptions of Lemma 1, feasible perturbations cannot induce positive advantage. When the expert trajectory lies strictly inside the feasible region, sufficiently small perturbations remain feasible. Under local optimality, these perturbations cannot improve the expert’s value-augmented return, so they have non-positive advantage and are rejected by the selection rule. Lemma 1 formalizes this for an optimal expert, and the new robust lemma (App. E.2) extends it to approximate dynamics and to an εE\varepsilon_E-suboptimal expert, where the guarantee holds for perturbations that outperform the expert by more than a margin τ\tau. Empirically, App. C validates the generated counterfactuals against ground truth using the learned, expert-only dynamics, reporting that the generated counterfactual trajectory has >90> 90\\% chance of containing a violating transition.

AuthorsOpenReview ↗

Response to W2

W2: We agree that demonstrations are not exactly optimal in practice, and our experiments reflect this. Grid-based Dijkstra paths with PD tracking are not optimal in the continuous maze, and the MuJoCo experts are trained PPO-Lagrangian policies with no optimality guarantee. Lemma 1 is the idealized statement, and the robustness extension of Lemma 1 we establish in App. E.2 covers an εE\varepsilon_E-suboptimal expert and shows that suboptimality only produces conservative errors. Empirically, App. C validates the generated counterfactuals against ground truth using the learned, expert-only dynamics. In App. F.1, we report the constraint learning results from expert demos that are intentionally made suboptimal by inflating the obstacle size in the Maze experiments, showing that CF-KKT can learn a more conservative yet still safe constraint map with such experts that maintain additional safety margins.

AuthorsOpenReview ↗

Response to W3

W3: We have added such hit maps in App. F.3, Fig. 6. For each maze we show the occupancy density of the expert demonstrations, of all perturbed rollouts, and of the rollouts accepted by our rule, overlaid on the arena. It can be observed that not all perturbations result in an infeasible trajectory, while our optimality-based counterfactual selection rule correctly identifies trajectories taking shortcuts through the obstacle, violating constraints to improve their reward.

Reviewer_F5P1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a method for inferring unknown constraints from expert demonstrations. In contrast to much prior work, only safe demonstrations are required. The approach assumes (local) optimality of the demonstrations for a known reward function, and derives a loss function based on KKT optimality conditions. A dynamics model is learnt from the safe training data, and used to counterfactually generate perturbed trajectories locally around the demonstrations, yielding a value-augmented counterfactual advantage, which is used as an infeasibility signal (Lemma 1). Experiments are provided on tabular mazes and MuJoCo tasks, showing favorable performance compared even with methods that rely on unsafe demonstrations.

Critical strengths
  • The problem of learning safety constraints without requiring unsafe demonstrations / risky exploration is very important especially in high-stakes applications.
  • The experiments, while not entirely avoiding constraint violations, shows that the approach outperforms existing methods that require additional unsafe data.
Critical weaknesses
  • The results are primarily empirical; Lemma 1 only provides a sufficient but not necessary condition to detect infeasibility. Moreover, Lemma 1 assumes access to the true dynamics, which I expect would typically not be available in practice for many applications.
  • The approach (via exploiting optimality conditions) relies on the assumption that the expert demonstrations are (locally) optimal. Perhaps more critically, since the approach relies on complementary slackness, it requires that active constraints are satisfied with equality, i.e., leaving no slack. I could imagine that in many practical safety-critical applications, experts might trade optimality with additional slack to ensure robustness.
Questions for authors
  • How robust is the approach against sub-optimal (yet nevertheless safe) demonstrations?
  • How robust is the result from Lemma 1 w.r.t. approximate dynamics models?
  • How does the approach relate to prior work [1] which also considers learning unknown constraints from safe offline data?

[1] D. Linder et al. “Learning Safety Constraints from Demonstrations with Unknown Rewards”, AISTATS 2024

8 comments
AuthorsOpenReview ↗

Response to W1

W1: Lemma 1 is used as a certificate for labeling if a rollout is infeasible, not as a classifier of all infeasible trajectories, so sufficiency is what the method needs, enabling us to select the infeasible states among random rollouts. The counterfactual loss requires some correctly labeled infeasible examples, not all of them, and a missed infeasible rollout only reduces coverage without introducing a wrong label. A necessary condition is also not available from expert data alone, since a constraint that never binds for the expert leaves no trace in its behaviour and cannot be identified by any method. To address the reliance on true dynamics, the revision adds a robust version of Lemma 1 (App. E.2). If the learned model's one-step error is at most ε\varepsilon along the rollouts, the terminal value error at most εV\varepsilon_V, and the expert within εE\varepsilon_E of optimal, then a model-predicted advantage above τ=2δJ+εE\tau = 2\delta_J + \varepsilon_E certifies that the true rollout is infeasible, where δJ\delta_J is the induced return error and grows with ε\varepsilon through the propagation factor Γh=∑k<hLfk\Gamma_h = \sum_{k<h} L_f^k. The exact-model, optimal-expert case τ=0\tau=0 recovers the original Lemma 1. Empirically, App. C validates the generated counterfactuals against ground truth using the learned, expert-only dynamics.

AuthorsOpenReview ↗

Response to W2

W2: We agree that this is a modelling assumption. However, when constructing our loss functions, we do not model demonstrations as being exactly optimal. We minimize a KKT inspired cost rather than strictly ensure the KKT conditions. Moreover, the experiments we conduct in the paper are with suboptimal expert data. In the mazes, expert paths come from Dijkstra on a 61×6161\times61 grid tracked by a PD controller, which is not optimal for the continuous double integrator system, and in MuJoCo the experts are the trained policies shipped with the ICSDICE benchmark, which are trained with PPO-Lagrangian without any optimality guarantee. CF-KKT works well with the aforementioned non-ideal expert.

In the robust lemma (W1) suboptimality enters only through εE\varepsilon_E, and its errors are one-sided, since a suboptimal expert can cause feasible states to be labeled unsafe, never the reverse, so the learned feasible set shrinks rather than grows, becoming more conservative. We test this with deliberately conservative experts that plan around inflated obstacles for the Maze experiments in the revised manuscript (App. F.1). Constraint learning from the conservative expert recovers a conservative estimate of the feasible region.

AuthorsOpenReview ↗

Response to Q1

Q1: In App. F.1, we report the constraint learning results from expert demos that are intentionally made suboptimal by inflating the obstacle size in the Maze experiments, showing that CF-KKT can learn a more conservative yet still safe constraint map with such experts that maintain additional safety margins.

Reviewer_sU84OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method for offline inverse constraint learning under unknown dynamics and a known analytical task reward by using a KKT-informed loss and counterfactual rollouts. Precisely, the proposed method first learns a dynamics model from expert trajectories and then uses this learned dynamics model in place of a known dynamics model in the tandard KKT conditions for constrained inverse optimal control. They propose two variations (1) directly solving the KKT conditions (with relaxed stationarity condition) under the learned dynamics model using a MIP solver when a tractable parameterization of the cost function is known or (2) further relaxing all KKT conditions into penalities to construct a loss that can be optimized by gradient-based methods. Further, they propose an additional counterfactual rollout loss which encourags the cost function to assign higher costs on trajectories generated by following the learned dynamics model on perturbed expert actions. They demonstrate their method in three increasingly challenging domains (1) a tractable parametric loss function (2) a generic loss function parameterized by a neural network in continuous maze environments and (3) same as (2) but in high-dimensional locomotion environments. In the first case, they demonstrate more accurate parameter recovery than prior methods in (2) they demonstrate more accurate constraint recovery as measured by IoU and performance (violation rate/returns) of a downstream planner and in (3) improved performance of a policy obtained by offline policy extraction.

Critical strengths

(1) Experimental results are strong compared to baselines: Maze results IoU demonstrate that the learned constraints are much closer to ground truth than prior methods. Policy returns/violation rates in maze and Mujoco demonstrate generally lower violation rate than prior methods with similar returns. I particularly appreciate the constraint inference + planning/offline-policy set up for the evaluations which seperates the constraint learning from policy learning for evaluatoins. Appropriate benchmarks are included. Ablation of supervision signals in the Appendix demonstrates that both the counterfactual loss and the KKT-inspired loss are important for achieving calibrated constraints in the Mujoco environments. Consider including these in the main paper.

(2) Clarity: The writing is clear and well-structured. Relevant related work is included, the background section provides appropriate level of detail. The proposed method is explained clearly with appropriate detail. The experiments are well-motivated and easy to follow.

Critical weaknesses

(1) Robustness of learned dynamics model: As I understand the learned dynamics model is trained entirely on non-violating expert trajectories. It is therefore questionable how well such a model can be used for counterfactual rollouts, since the model only sees non-violating states and expert actions. Further the text states that in the parametric case the dynamics model is trained on a single trajectory - this warrants further explanation as to how the model is not overfitting. It would be useful if the authors could provide some conditions on the expert data/environments under which learning a dynamics model exclusively on expert data is able to generate useful counterfactual trajectories and ablations that demonstrate these conditions experimentally.

(2) Novelty: The KKT conditions of the CIOC problem under known dynamics and the relaxation of the stationarity condition to form a optimization problem are available from prior work. The core contribution is thus (a) using a learned dynamics model in place of known dynamics (b) further softening the satisfaction of the KKT condtions to penalize primal infeasability and complementary slackness to form a loss function that can be minimized by standard gradient based learning and (c) the counterfactual loss that encourages higher costs on non-expert trajectories. The first two contributions are relatively small, so the bulk of the novelty is in the counterfactual loss however it is unclear to me how robust these counterfactual rollouts are given the dynamics model is trained exclusively on expert data (see point 1)

I would be glad to increase my score if the authors could provide a better explanation of how the counterfactual rollouts are able to extrapolate outside of the expert data.

Questions for authors

(1) What is the difference between a "known parametric function" and a neural network? This is repeatedly referred throughou the paper, however a neural network has a known parametric form. The important contrast seems to be tractability, not known parameterization. Please elaborate.

(2) Can you elaborate on how the dynamics model can be trained with a single expert trajectory in the parametric case and how it extrapolates outside of the safe states seen by the expert by training only on expert rollouts? How does the quality of the dataset and/or environment dynamics influence the robustness of the learned dynamics model and the counterfactual rollouts?

(3) How is Lemma 1 related to the proposed counterfactual loss? The loss does not seem to account for advantages but rather increases cost on any off-expert trajectory? Please clairfy.

(4) Table 13 does not include KKT-only performance - can this be included?

(5) The bolding in Table 3 is confusing - what are the rules for bolding? It appears the IQ-Learn achieves lower violation in several environments. If a reward threshold is being used, please clairfy.

12 comments
AuthorsOpenReview ↗

Response to W1

Robustness of learned dynamics model: As I understand the learned dynamics model is trained entirely on non-violating expert trajectories. It is therefore questionable how well such a model can be used for counterfactual rollouts, since the model only sees non-violating states and expert actions. Further the text states that in the parametric case the dynamics model is trained on a single trajectory - this warrants further explanation as to how the model is not overfitting.

W1: The two settings (parametric and neural) use the learned dynamics differently. In the setting with a known parametric form, the CIOC formulation in Section 4.1 (lines 219–220) does not involve counterfactual rollouts: the dynamics model is fitted to a single expert trajectory and evaluated only along that trajectory. Since CIOC evaluates the dynamics only at the demonstrated state–action pairs, no extrapolation is involved. Empirically, we observe a moderate relative Jacobian error of 0.11 when evaluating the model along expert trajectories. Proposition 1 (Appendix E.1) bounds how the recovered parameters change with the dynamics parameters.

In the neural setting, counterfactual rollouts start from expert states and perturb the demonstrated action sequence over H=24H=24 steps, targeting deviations near the expert trajectories. When the unknown constraint is nearly active at some states along an optimal expert trajectory (c(x)≈0c(x) \approx 0), even small deviations can reach states with c(x)>0c(x)>0, potentially improving the reward by violating the constraint. Thus, counterfactuals can reveal constraint violations near the demonstrations. In Appendix C of the original manuscript, we empirically verify that, with learned dynamics and a suboptimal expert, the generated counterfactual trajectories contain a violating transition more than 90% of the time.

It would be useful if the authors could provide some conditions on the expert data/environments under which learning a dynamics model exclusively on expert data is able to generate useful counterfactual trajectories and ablations that demonstrate these conditions experimentally.

Following the reviewer's suggestion to specify the data and environment conditions, the revised manuscript formally states these conditions in a robust version of Lemma 1 (Appendix E.2), accounting for dynamics-model error, value-estimation error, and expert suboptimality. A counterfactual is guaranteed to violate the constraint if its predicted return improvement exceeds what these errors could jointly explain. Larger error bounds therefore require a larger improvement margin to certify infeasibility.

Specifically, suppose the model's one-step error is at most ε\varepsilon along the rollouts, the terminal return-to-go estimation error is at most εV\varepsilon_V, and the expert's return is within εE\varepsilon_E of the optimal safe return. Accepting a counterfactual only when its model-predicted advantage exceeds

τ=2δJ+εE,\tau = 2\delta_J + \varepsilon_E,

where

\qquad \Gamma_h = \sum_{k=0}^{h-1}L_f^k.$$ certifies that the corresponding true rollout violates the constraint and therefore contains at least one infeasible state. The conditions are thus that the model is accurate within the local rollout tube and that the constraint boundary is reachable within the perturbation budget $(\sigma,H)$.
AuthorsOpenReview ↗

Response to W2

W2: We thank the reviewer for summarizing our contributions. We would like to clarify why contribution (b) is substantial.

Classical CIOC enforces trajectory-level stationarity, with costates coupling all time steps. To evaluate CIOC with a neural constraint, we implemented it using learned dynamics and adjoint costates over both 16-step windows and full episodes. We tested several stabilization and calibration techniques: costate clipping, residual normalization, discounting, a 100×100\times primal-feasibility weight, and their combination. In every variant, the learned constraint remained positive (indicating infeasibility) on the expert's own states, and the mask admitted approximately 7% of the offline data Dπ\mathcal{D}_{\pi}, compared with the ground-truth safe fractions of 26–77% across tasks. Costates are computed through backward recursion using the learned model's Jacobians, so model errors compound along the trajectory, making the coupled system ill-conditioned. Replacing the KKT threshold with a manually selected quantile of the cost also failed to yield consistent results across tasks. Appendix F.6 reports the full results, and Remark 1 states this limitation of applying CIOC directly without contribution (b).

The transition-wise residuals with bounded multipliers (Equations 8–10) decouple time steps, so each residual depends only on a one-step model prediction. This formulation makes the expert-side supervision numerically stable. The ablation in Appendix D further shows that these terms calibrate the mask's operating threshold. Counterfactual supervision alone ranks states well but rejects every transition across all five tasks (Table 11), whereas adding the KKT terms moves the admission rate toward the ground-truth safe fraction.

The robustness of contribution (c) is addressed in our response to W1.

AuthorsOpenReview ↗

Response to Q1

Q1: We agree and thank the reviewer for pointing this out. A neural network also has a known parametric form. The distinction we intend to make is computational tractability. By “known parametric function,” we mean a constraint with a known geometric functional form (for example, x2+y2≥θx^2 + y^2 \geq \theta, where θ\theta is unknown) and sufficiently few unknown parameters for the KKT recovery problem to be solved exactly as a mixed-integer program. In our experiments, the neural network has too many parameters for this approach to be computationally tractable, which motivates the use of a gradient-based loss in the neural setting.

Comments on the paper

1 comment

We thank the reviewers for their thoughtful comments and helpful suggestions. We have uploaded a revised manuscript incorporating additional experiments, analyses, and clarifications in response to your feedback. The corresponding additions and revisions are highlighted in red in the main paper and appendix for ease of reference. We appreciate your time and consideration and welcome any further questions.

We will respond to each reviewer's individual raised weakness and questions on the paper under their own official review sections. We use "W1", "W2", etc. (respectively, "Q1", "Q2", etc.) to refer to the weaknesses identified (respectively, questions raised) by the reviewer, in the order in which they appear in the reviewer's official reviews.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.