acceptodds
Under review as a conference paper at ICLR 2027

Multi-Objective Constraint Inference using Inverse reinforcement learning

Abstract

Constraint inference from expert demonstrations is widely considered essential to align reinforcement learning agents with safety boundaries and operational guidelines. However, existing approaches typically assume homogeneous demonstrations, generated by a single expert or multiple experts with identical objectives. They also have limited ability in capturing individual preferences and often suffer from computational inefficiencies. In this paper, we introduce Multi-Objective Constraint Inference (MOCI), a novel framework to jointly extract shared constraints and individual preferences from heterogeneous expert trajectories, where multiple experts pursue different objectives while adhering to the same safety constraints. MOCI effectively models and learns from diverse, and potentially conflicting, behaviors. Empirical evaluations demonstrate that MOCI improved predictive performance, showed competitive computational efficiency, and recovered clinically credible preferences and constraints in a longitudinal lung cancer treatment setting. These results establish MOCI as an accurate, flexible, and practical approach to the real-world constraint inference and preference learning tasks in inverse reinforcement learning with heterogeneous experts.

Primary area:
reinforcement learning
Keywords:
Inverse Reinforcement Learning, Safe Reinforcement Learning, Constraint Inferencing

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_BivPOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes multi objective constraint inference, a framework for jointly learning shared constraints and heterogeneous expert preferences from unlabeled demonstrations. It combines maximum-entropy inverse reinforcement learning with EM to estimate latent expert preferences, followed by likelihood based constraint inference. The method also introduces softened constraint penalties and local policy likelihoods. Experiments on gridworlds and a synthetic lung cancer treatment dataset demonstrate promising constraint recovery, with improvements over the selected baselines.

Critical strengths

Distinguishing shared constraints from individual preferences is a meaningful problem in inverse reinforcement learning particularly when expert demonstrations are heterogeneous.

MOCI achieves strong precision and F1 scores on the reported gridworld benchmark relative to the selected baselines, and the authors evaluate reward and cluster recovery separately. The experiments across multiple seeds and cluster settings provide useful initial evidence that the integrated framework can operate in a controlled heterogeneous-expert setting. These results justify further investigation, although they are not sufficient to establish the broader claims.

Critical weaknesses

The paper does not establish when its central objective is identifiable. An action may remain unvisited because it is prohibited or simply because all experts strongly dislike it. Both explanations can produce identical observed behavior. Although the paper introduces heterogeneous experts and joint optimization, it does not provide sufficient assumptions or evidence showing how this ambiguity is resolved. There are also apparent inconsistencies in the likelihood normalization and insufficient justification for the claimed advantage of the local likelihood formulation. These concerns directly affect the validity of the proposed inference method.

The inconsistency between the softened constraint likelihood in Equation (6) and the allowed-action normalization in Equations (7) and (9) is a substantive mathematical concern. Since likelihood differences determine both latent cluster responsibilities and greedy constraint selection, this is not merely a notational detail.

the local policy likelihood appears algebraically equivalent to the global path likelihood under the deterministic finite-horizon assumptions used in the gridworld experiments. The paper does not provide a concrete example or derivation demonstrating how the local formulation identifies constraints that the global formulation cannot detect. These issues directly concern the validity and novelty of the proposed inference procedure.

A particularly relevant missing work is Lindner et al., Learning Safety Constraints from Demonstrations with Unknown Rewards (AISTATS 2024), which explicitly addresses recovering shared constraints from demonstrations generated under different unknown rewards. This weakens the stated premise that existing constraint-inference approaches necessarily assume homogeneous demonstrations or known rewards.

Questions for authors

How are the soft violation penalties incorporated into the partition function and local policy normalization in the actual implementation? Can the authors provide a mathematically consistent derivation establishing normalization and demonstrating a concrete example where the proposed local likelihood detects a constraint that the global path likelihood cannot?

How does MOCI differ substantively from CoCoRL (Lindner et al., 2024), which already considers shared constraints under heterogeneous unknown rewards? Can the authors provide a direct constraint-recovery comparison, where applicable, and compare against a baseline combining existing multi-intention IRL with conventional constraint inference? These comparisons would materially affect the assessment of novelty and effectiveness.

Under what assumptions can MOCI distinguish an action that is genuinely infeasible from one that all observed experts strongly dislike? Figure 3 assigns substantial negative water rewards to both expert types even though water is also designated as a hard constraint. How should this be interpreted in light of the claimed separation between constraints and preferences?

Reviewer_4KfYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers the problem of constraint inference in the setting where expert demonstrations are produced by multiple experts with different reward profiles, but who share a common set of constraints. Both rewards and constraints are unknown. The authors focus on a the case of a deterministic, finite MDP with linear rewards, where a simple closed-form likelihood for the MaxEnt IRL can be evaluated. The authors propose to use an EM-style algorithm to jointly optimize cluster assignments of the experts to one of a set of KK (prefixed) clusters, as well as the reward coefficient vectors for each cluster. To incorporate constraints, the authors propose to greedily add constraints to the constraint set based on the log-likelihood gain achieved by the addition of the proposed state-action pairs to the constraint set. They test their method on a simple gridworld environment and demonstrate that the method successful recovers reasonable weights for two different reward profiles and a shared constraint. Finally, they test their method on a realworld dataset for lung cancer treatment.

Critical strengths

(1) The greatest strength of the paper is the lung cancer treatment experiment, which applies the proposed method to a real-world scenario, demonstrating the utility of such a method beyond toy environments. The results in the lung cancer treatment domain are quite promising, with the method recovering constraints on treatments at a high level of precision (though recall is less satisfactory).

Critical weaknesses

(1) Limited novelty and uncited prior work: My primary concern is the scope of the contribution, particularly in light of substantial uncited relevant work. While the authors do cite some relevant work in ICRL, the background section does not describe the ICRL method from [1,2] in detail, and rather these details are combined with their own contributions section which doesn't fully credit the origins of the method. In particular, the constrained path likelhood is presented as a new result, however, this doesn't appear different from that used in [1] and [2]. Further, the greedy likelihood maximization method for building the constraint set is similar to in [1]. Additionally, though the title of the paper is multi-objective IRL (multiple reward functions), it appears that the actual problem formulation is much closer to multi-intention IRL (multiple expert intentions in the dataset) on which there is a prior body of work that is completely uncited here. In particular the EM algorithm for clustering experts and assigning weights to a linear reward function for each cluster appears to have been derived in [3]. At a minimum [3] and subsequent work should be cited. However, that given the prior work in this area, the incremental contribution of the paper is minimal as it simply combines the greedy constraint building process from [1] with the EM algorithm from [3]. Both components are known in the context of IRL and their combination holds minimal novelty, particularly so since these methods rely on fairly restrictive assumptions i.e. deterministic, finite MDPs with linear rewards which later work [2] in ICRL discard.

(2) Experimental results and baselines are missing/unclear: The implementation of the ICRL baselines for the experiments are unclear. The methods from [1] and [2] are designed to be used with a known reward function so it is unclear how they are applied here and I do not see further explanation of this provided in either the text or the Appendix. Further, a simple and important baseline is missing - treating the constratint as a shared weight in the reward vector. Without this, it is not clear to me that this method improves on prior work in multi-intention IRL. Further while the author's do ablate the impact of kk in the appendix, the impact of k>1k>1 actually appears to be very small, with no affect on precision and a boost of only 0.02 on recall. Finally, baseline results are only provided for the GridWorld experiments and not the lung cancer dataset which is the much more interesting case.

(3, Minor) Some notation is extremely confusing, for example, it appears that πk\pi_k is being used to both represent the policy and the prior probabilities for the cluster. Another example is the PpathP_\text{path} and PlocalP_\text{local} are defined but in the EM steps a bare PP is used so its not clear which likelihood is being referred to.

[1] Scobee, Dexter RR, and S. Shankar Sastry. "Maximum likelihood constraint inference for inverse reinforcement learning." arXiv preprint arXiv:1909.05477 (2019). [2]Malik, Shehryar, et al. "Inverse constrained reinforcement learning." International conference on machine learning. PMLR, 2021. [3] Babes, Monica, et al. "Apprenticeship learning about multiple intentions." Proceedings of the 28th international conference on machine learning (ICML-11). 2011.

Questions for authors

(1) Where is the path likelihood vs global likelhood used in the EM method? (2) Since the constraints are soft, i,e, they allow a small non-zero probability of violation, would it not be possible to simply model the constraint as a shared weight in the reward vector so that rewards become ϕ(s,a)Twk+wCϕ(s,a)\phi(s,a)^Tw_k + w_{C}\phi(s,a)? This seems like a simple baseline for comparison? (3) How are the ICRL methods in the baselines adapted to the setting without known reward functions? (4) Why is a negative weight assigned to water in the GridWorld experiments? One motivation for learning constraints indepdently from rewards is so that unmodeled constraint impacts do not leak into rewards, but if the rewards are learning a negative weight on the constrained states, is the seperate constraint inference process necessary?

Reviewer_9W4FOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to relax the two common assumptions about the (hard) constraint inference problem (identifying a set of forbidden states or actions): (1) the dataset is homogeneous and (2) the expert’s reward function is assumed known. To address this, this work proposes Multi-Objective Constraint Inference (MOCI), a new framework that jointly extracts shared constraints and individual preferences from heterogeneous expert trajectories. Specifically, MOCI combines maximum entropy inverse RL with multiple latent experts, and then it optimizes the objective function (essentially the joint log-likelihood, which depends on the constraint, preference weight, and policy) via Expectation-Maximization (EM). Then, the candidate constraints are evaluated based on state-level model-expert policy discrepancy with safety pre-screening. MOCI is then evaluated on Gridworld and lung cancer treatment, and the experimental results demonstrate that MOCI can achieve constraint inference under heterogeneous experts.

Critical strengths
  • This paper studies a new practical problem setting in constraint inference, and the proposed solution is conceptually simple and sound.

-The experiments on small-scale and synthetic problems are quite helpful in describing the main technical idea.

Critical weaknesses
  • While MOCI offers a practical extension to the conventional single-expert constraint inference (Scobee and Sastry, 2020; Malik et al., 2021), the methodology appears incremental as MOCI appears to directly combine maximum entropy formulation with multiple experts and solve it by EM. The paper currently provides limited evidence of a substantial advance in constraint inference.

  • The constraint candidate screening approach in Equations (17)-(18) appears somewhat heuristic as it is not well justified why prioritizing by state-level model-expert policy discrepancy is a theoretically grounded design. Moreover, in Equations (16)-(18), the proposed method presumes that the state and action spaces are discrete. It remains unclear how this can extend to continuous control problems.

  • Moreover, this work assumes the knowledge of some features ϕ(s,a)\phi(s,a) such that the rewards are linear in ϕ\phi. While this is usually acceptable for goal-oriented problems (e.g., gridworld), features are not directly available in general (continuous) control problems. It is not immediately clear how to jointly learn the features and other required components. This can be a fundamental limitation to the proposed approach.

  • The paper introduces two candidate likelihood formulations, namely global path likelihood and local policy likelihood. However, it is not clear how these two candidates are combined or integrated.

  • The experiments are quite limited. MOCI is evaluated only on small tabular gridworlds and a synthetic clinical dataset. Moreover, while the constraint precision can suggest that the predicted restrictions are usually correct, it does not necessarily mean the resulting policy is safe. It would be helpful to evaluate policies trained under the inferred constraints.

  • The benefit of heterogeneous modeling appears not that significant. Specifically, in Table 4, increasing the mixture size from K=1K=1 to K=3K=3 only marginally improves the constraint F1. More evidence is needed to showcase the benefits of heterogeneous modeling.

References:

  1. Scobee and Sastry, “Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning,” ICLR 2020.

  2. Malik et al.,, “Inverse Constrained Reinforcement Learning,” ICML 2021

Questions for authors
  • What are the fundamental differences between MOCI and the conventional constraint inference methods (aside from adding multiple latent experts)?

  • Can the authors justify or better motivate the constraint candidate screening approach?

  • Can the MOCI approach apply to more general control problems without assuming the knowledge of features.

  • Can the authors explain how the two candidate likelihood functions are jointly used or integrated?

  • Does MOCI remain effective for more complex problems of larger scales, e.g., the benchmark tasks in (Liu et al., 2023)?

Reference:

Liu et al., “Benchmarking Constraint Inference in Inverse Reinforcement Learning,” ICLR 2023.

3 comments

We thank reviewer 9W4F for their thoughtful review and please find our response to each point below.

1.1: Fundamental differences from conventional constraint inference

Beyond modeling multiple experts, MOCI (i) extends constrained MaxEnt (Scobee & Sastry, 2020) from a single known reward to learned cluster-specific rewards wkw_k with a step-wise local likelihood for unreachable states, and (ii) extends multi-intention EM (Babes et al., 2011) by conditioning likelihoods on a shared constraint set CC inferred inside the M-step. MOCI differs in three ways:

  1. Learned rewards: Prior methods assume known rewards (Sec. 1); MOCI jointly learns wkw_k and CC (Sec. 3, Eqs. 11–15).
  2. Separate constraints: To disambiguate constraints from low rewards, CC enters the likelihood solely as a multiplicative factor over forbidden pairs rather than modifying rewards (Eqs. 6–7; Appdx. B).
  3. Off-path evaluation: Path likelihoods normalize only at s0s_0, assigning zero gain to unreachable states. Our local likelihood normalizes at every visited state (Eqs. 9–10), enabling inference on observational clinical data.

1.2: Justification of candidate screening

In the revised version, we reorganised Constraint Candidate Screening & Safety Pre-Screening into four parts (candidate screening, discrepancy ranking, signature grouping, acceptance criterion) with explicit justifications. For candidate screening:

Screening is an exact necessary condition for acceptance rather than a heuristic filter. Under the hard indicator (η=0\eta = 0), any trajectory containing a forbidden pair has zero likelihood (Eq. 6); adding a demonstrated pair (s,a)(s,a) to CC thus yields ΔLtotal=−∞\Delta\mathcal{L}_{\text{total}} = -\infty, violating Eq. 20. Under the soft indicator (η>0\eta > 0, Eq. 7), each observed occurrence of (s,a)(s,a) incurs a finite penalty log⁡η\log \eta, and thresholds nmax⁡n_{\max} and ϵ\epsilon filter out pairs whose accumulated penalty n(s,a)log⁡ηn(s,a)\log \eta cannot be outweighed by the likelihood gain.

1.3: Generalizing beyond known features

While MOCI currently assumes linear rewards over predefined features ϕ(s,a)\phi(s,a) (noted in Sec. 6), its constraint inference (Eqs. 11–12, 17–20) depends on rewards solely via trajectory likelihoods P(ξ∣C,wk)P(\xi \mid C, w_k); only the M-step gradient (Eq. 14) uses linearity. MOCI can extend to unknown features via a shared feature network with cluster-specific heads:

Rk(s,a)=wk⊤ϕψ(s,a)R_k(s,a) = w_k^\top \phi_\psi(s,a)

For continuous control, CC can be replaced by a feasibility function cθ(s,a)c_\theta(s,a) (Malik et al., 2021) with partition functions estimated via MaxEnt RL. While the separate multiplicative constraint factor (Eqs. 6–7) helps prevent learned features from absorbing constraints into low rewards, formal identifiability and empirical validation remain future work (now in Sec. 6).

1.4: Role of the two likelihood functions

They are alternatives, not a combined objective: PpathP_{\text{path}} applies when demonstrations start from a common s0s_0 under known dynamics (e.g., GridWorld, where both coincide), whereas PlocalP_{\text{local}} handles observational data with varying entry and off-path states (e.g., lung cancer). We clarified this in Sec. 3 before their definitions and directly after Eq. 11:

Lavg(C,{wk},{ρk})=1∣D∣∑i=1∣D∣log⁡(∑k=1KρkP(ξi∣C,wk))\mathcal{L}_{\mathrm{avg}}(C, \lbrace w_k \rbrace, \lbrace \rho_k \rbrace) = \frac{1}{|\mathcal{D}|} \sum_{i=1}^{|\mathcal{D}|} \log \left( \sum_{k=1}^K \rho_k P(\xi_i \mid C, w_k) \right)

Here P(ξ∣C,wk)P(\xi \mid C, w_k) denotes the selected likelihood formulation, either PpathP_{\text{path}} (Eq. 6) or PlocalP_{\text{local}} (Eq. 9), which is used consistently in the mixture likelihood (Eq. 11), responsibilities (Eq. 12), and constraint score (Eq. 20).

1.5: Larger-scale benchmarks (Liu et al., 2023)

Those benchmarks are single-expert continuous-control tasks with known rewards; applying MOCI requires the continuous extension outlined above in 1.3 and in sec-6 in more details.

1.6: Significance of heterogeneous modeling

We agree: in Table 4, moving from K=1K = 1 to K≥3K \ge 3 barely changes constraint detection because the constraint is shared and off all experts' preferred routes. Rather, the mixture's primary role is preference disentanglement, K=1K = 1 fits one reward vector and cannot separate experts, whereas K=2K = 2 recovers expert-specific reward weights and assignments (Fig. 7). As added in Appx. D.2: "The main benefit of the mixture is the recovery of expert-specific preferences, which a single model cannot provide."

AuthorsOpenReview ↗

Following up on rebuttal

Dear 9W4F,

Thank you once more for your thoughtful review of our work. We hope that our rebuttal and proposed updates address your key concerns, and we would greatly appreciate any feedback you might have on them. If anything remains unclear, please let us know. We remain available to clarify any remaining points during the rest of the discussion period.

Best regards, The Authors

Thank you for the detailed response and for answering my questions. I appreciate the clarification regarding the path and local likelihoods used as alternatives.

However, I still have the following concerns:

  1. Novelty beyond adding latent experts: The rebuttal clarifies that MOCI jointly learns cluster-specific rewards and a shared constraint set, and that the local likelihood makes it applicable to observational data. These are useful extensions. My concern is whether their combination provides a substantial methodological advance in constraint inference, as also pointed out by Reviewer 4KfY.

  2. Justification for constraint candidate ranking: The rebuttal explains why observed state-action pairs can be excluded from the candidate set under hard constraints and feasible demonstrations. My concern is about the subsequent ranking step. It remains unclear why state-level model-expert policy discrepancy is a reliable criterion for prioritizing the remaining candidates.

  3. Generalization beyond predefined features and discrete spaces: The proposed feature network and feasibility function offer a plausible direction for continuous control. However, it is still unclear to me how candidate screening and ranking would work in continuous spaces, or how the method would distinguish shared constraints from low rewards when the features are learned jointly. Moreover, the scope of the experiments is still somewhat limited.

  4. Safety of policies trained under the inferred constraints: Constraint precision and F1 can measure recovery of the constraint set. My concern is whether a policy trained under the inferred constraints avoids violating the true constraints.

Given the above remaining concerns, I am inclined to maintain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.