- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes multi objective constraint inference, a framework for jointly learning shared constraints and heterogeneous expert preferences from unlabeled demonstrations. It combines maximum-entropy inverse reinforcement learning with EM to estimate latent expert preferences, followed by likelihood based constraint inference. The method also introduces softened constraint penalties and local policy likelihoods. Experiments on gridworlds and a synthetic lung cancer treatment dataset demonstrate promising constraint recovery, with improvements over the selected baselines.
Critical strengths
Distinguishing shared constraints from individual preferences is a meaningful problem in inverse reinforcement learning particularly when expert demonstrations are heterogeneous.
MOCI achieves strong precision and F1 scores on the reported gridworld benchmark relative to the selected baselines, and the authors evaluate reward and cluster recovery separately. The experiments across multiple seeds and cluster settings provide useful initial evidence that the integrated framework can operate in a controlled heterogeneous-expert setting. These results justify further investigation, although they are not sufficient to establish the broader claims.
Critical weaknesses
The paper does not establish when its central objective is identifiable. An action may remain unvisited because it is prohibited or simply because all experts strongly dislike it. Both explanations can produce identical observed behavior. Although the paper introduces heterogeneous experts and joint optimization, it does not provide sufficient assumptions or evidence showing how this ambiguity is resolved. There are also apparent inconsistencies in the likelihood normalization and insufficient justification for the claimed advantage of the local likelihood formulation. These concerns directly affect the validity of the proposed inference method.
The inconsistency between the softened constraint likelihood in Equation (6) and the allowed-action normalization in Equations (7) and (9) is a substantive mathematical concern. Since likelihood differences determine both latent cluster responsibilities and greedy constraint selection, this is not merely a notational detail.
the local policy likelihood appears algebraically equivalent to the global path likelihood under the deterministic finite-horizon assumptions used in the gridworld experiments. The paper does not provide a concrete example or derivation demonstrating how the local formulation identifies constraints that the global formulation cannot detect. These issues directly concern the validity and novelty of the proposed inference procedure.
A particularly relevant missing work is Lindner et al., Learning Safety Constraints from Demonstrations with Unknown Rewards (AISTATS 2024), which explicitly addresses recovering shared constraints from demonstrations generated under different unknown rewards. This weakens the stated premise that existing constraint-inference approaches necessarily assume homogeneous demonstrations or known rewards.
Questions for authors
How are the soft violation penalties incorporated into the partition function and local policy normalization in the actual implementation? Can the authors provide a mathematically consistent derivation establishing normalization and demonstrating a concrete example where the proposed local likelihood detects a constraint that the global path likelihood cannot?
How does MOCI differ substantively from CoCoRL (Lindner et al., 2024), which already considers shared constraints under heterogeneous unknown rewards? Can the authors provide a direct constraint-recovery comparison, where applicable, and compare against a baseline combining existing multi-intention IRL with conventional constraint inference? These comparisons would materially affect the assessment of novelty and effectiveness.
Under what assumptions can MOCI distinguish an action that is genuinely infeasible from one that all observed experts strongly dislike? Figure 3 assigns substantial negative water rewards to both expert types even though water is also designated as a hard constraint. How should this be interpreted in light of the claimed separation between constraints and preferences?
We thank reviewer 9W4F for their thoughtful review and please find our response to each point below.
1.1: Fundamental differences from conventional constraint inference
Beyond modeling multiple experts, MOCI (i) extends constrained MaxEnt (Scobee & Sastry, 2020) from a single known reward to learned cluster-specific rewards wk with a step-wise local likelihood for unreachable states, and (ii) extends multi-intention EM (Babes et al., 2011) by conditioning likelihoods on a shared constraint set C inferred inside the M-step. MOCI differs in three ways:
1.2: Justification of candidate screening
In the revised version, we reorganised Constraint Candidate Screening & Safety Pre-Screening into four parts (candidate screening, discrepancy ranking, signature grouping, acceptance criterion) with explicit justifications. For candidate screening:
1.3: Generalizing beyond known features
While MOCI currently assumes linear rewards over predefined features ϕ(s,a) (noted in Sec. 6), its constraint inference (Eqs. 11–12, 17–20) depends on rewards solely via trajectory likelihoods P(ξ∣C,wk); only the M-step gradient (Eq. 14) uses linearity. MOCI can extend to unknown features via a shared feature network with cluster-specific heads:
Rk(s,a)=wk⊤ϕψ(s,a)
For continuous control, C can be replaced by a feasibility function cθ(s,a) (Malik et al., 2021) with partition functions estimated via MaxEnt RL. While the separate multiplicative constraint factor (Eqs. 6–7) helps prevent learned features from absorbing constraints into low rewards, formal identifiability and empirical validation remain future work (now in Sec. 6).
1.4: Role of the two likelihood functions
They are alternatives, not a combined objective: Ppath applies when demonstrations start from a common s0 under known dynamics (e.g., GridWorld, where both coincide), whereas Plocal handles observational data with varying entry and off-path states (e.g., lung cancer). We clarified this in Sec. 3 before their definitions and directly after Eq. 11:
Lavg(C,{wk},{ρk})=∣D∣1∑i=1∣D∣log(∑k=1KρkP(ξi∣C,wk))
1.5: Larger-scale benchmarks (Liu et al., 2023)
Those benchmarks are single-expert continuous-control tasks with known rewards; applying MOCI requires the continuous extension outlined above in 1.3 and in sec-6 in more details.
1.6: Significance of heterogeneous modeling
We agree: in Table 4, moving from K=1 to K≥3 barely changes constraint detection because the constraint is shared and off all experts' preferred routes. Rather, the mixture's primary role is preference disentanglement, K=1 fits one reward vector and cannot separate experts, whereas K=2 recovers expert-specific reward weights and assignments (Fig. 7). As added in Appx. D.2: "The main benefit of the mixture is the recovery of expert-specific preferences, which a single model cannot provide."