- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes an inverse reinforcement learning framework for learning from multiple suboptimal demonstrators without access to an optimal demonstrator. Instead of estimating a single reward, the authors construct a feasible reward set by intersecting demonstrator-specific suboptimality constraints (Definition 3.1). Proposition 3.2 provides an LP characterization of this set.
The main theoretical contributions include a sufficient condition for strict feasible-set shrinkage (Theorem 3.4), an optimality-gap bound based on proximity to the unobserved optimal occupancy (Theorem 3.5, Eq. 16), and complementary coverage-based guarantees (Theorem H.1). The authors further propose reward-selection and offline function-approximation methods (Eq. 17; Eqs. 20–21).
Experiments on grid-world tasks (Figure 2) and LLM arithmetic tool-use tasks (Figure 3) show improvements over a MaxEnt baseline.
Critical strengths
Definition 3.1 and Proposition 3.2 (Eqs. 12–13) formulate reward learning through demonstrator-specific suboptimality constraints without requiring an optimal demonstrator. This is a meaningful extension of the feasible-reward-set perspective and provides an interpretable alternative to likelihood-based reward fitting.
Theorem 3.5 (Eq. 16) and Theorem H.1 (Eqs. 85–87) identify conditions under which suboptimal demonstrations collectively constrain rewards associated with optimal behavior. The Case-2 grid-world experiment (Figure 2) provides encouraging empirical evidence for this complementary-coverage phenomenon.
Critical weaknesses
1). The framework assumes that every demonstrator's true suboptimality is bounded by its declared εᵏ (Section 3.1, Definition 3.1, Eqs. 11–12). These bounds determine the feasible set in Eq. 13 and are central to interpreting the theoretical guarantees.
However, Section 4.1 (lines 414–420) fixes εᵏ = 0.1 for all grid-world demonstrators. In contrast, Appendix O.2 (Eq. 169 and the accompanying discussion) reports that the mean oracle suboptimality in the Case-2 environment is 0.233 ± 0.069.
Unless a different reward normalization explains this discrepancy, at least some demonstrators violate their declared bounds. Consequently, the true reward may not belong to the feasible set in Eq. 12, undermining the connection between the theoretical assumptions and the principal experimental results.
This is particularly consequential because the sensitivity experiment in Table 1 (p. 30) demonstrates that underestimated bounds can produce degenerate rewards with artificially low measured suboptimality.
Additionally, the Case-1 experiment explicitly includes an optimal demonstrator at K = 20 (Section 4.1, p. 8, lines 405–408), so its endpoint does not test the claimed setting without an optimal demonstrator.
2). The theoretical reward set imposes simplex normalization (Eq. 8; Eq. 12). However, the neural implementation in Section 3.3 (Eqs. 20–21) only bounds reward values individually, while the advantage-parameterized objective in Appendix N (Eq. 163) does not impose an explicit magnitude or normalization constraint.
As written, Eq. 163 can be unbounded below. For example, consider a one-state, two-action system with demonstrator occupancy (1, 0), baseline occupancy (0, 1), and advantage values (0, −M). Both hinge penalties vanish for any M > 0, while the reward-selection objective equals −M. Thus the loss tends to negative infinity as M increases.
The suggested nonpositive parameterization Aψ = −softplus(fψ) ( lines 1499–1501) does not address this magnitude issue. This raises concerns about whether the implementation preserves the normalized-reward assumptions and optimization properties of the theoretical LP.
Furthermore, the grid-world evaluation uses SubOpt(r̂, d*ₑ) (Section 4.1), which can be zero for degenerate constant rewards. The authors explicitly demonstrate this failure mode in Table 1 ( Δ = −0.3). Yet Figure 2 does not report reward non-degeneracy or the performance of policies optimized under the recovered rewards.
Similarly, the LLM experiment reports pairwise response-ranking accuracy (Eq. 171), rather than the success rate of an actual policy trained or optimized using the recovered reward. Hence Figure 3 provides evidence of discrimination between two supplied responses, but not necessarily successful recovery of better decision-making behavior.
These issues substantially weaken the practical conclusions.
Questions for authors
How are the soft violation penalties incorporated into the partition function and local policy normalization in the actual implementation? Can the authors provide a mathematically consistent derivation establishing normalization and demonstrating a concrete example where the proposed local likelihood detects a constraint that the global path likelihood cannot?
How does MOCI differ substantively from CoCoRL (Lindner et al., 2024), which already considers shared constraints under heterogeneous unknown rewards? Can the authors provide a direct constraint-recovery comparison, where applicable, and compare against a baseline combining existing multi-intention IRL with conventional constraint inference? These comparisons would materially affect the assessment of novelty and effectiveness.
Under what assumptions can MOCI distinguish an action that is genuinely infeasible from one that all observed experts strongly dislike? Figure 3 assigns substantial negative water rewards to both expert types even though water is also designated as a hard constraint. How should this be interpreted in light of the claimed separation between constraints and preferences?
7 comments
Thank you for your detailed response and clarifications.
First, I apologize for an inadvertent error in my original review. The three questions under "Questions for Authors" were mistakenly carried over from another manuscript. Please disregard those questions. The paper summary, strengths, weaknesses, and other technical concerns in my review refer to the present submission.
I appreciate the authors' responses, particularly the additional reward-variance results and the acknowledgment that Eq. 163 requires an explicit magnitude constraint. I have the following revised questions.
Validity and calibration of demonstrator suboptimality bounds
Section 4.1 specifies εᵏ = 0.1, whereas Appendix O.2 reports mean oracle suboptimality of 0.233 ± 0.069. The response acknowledges that the declared bounds can exclude the ground-truth reward from the feasible set.
Could the authors clarify whether these quantities use exactly the same reward normalization? If so, what fraction of demonstrators in Figure 2 satisfy their declared suboptimality bounds?
I appreciate the proposed additional experiments with εᵏ = 0.2 and 0.3. It would be particularly informative to repeat the Case-2 experiment using individual oracle bounds εᵏ = εᵏ*, which are available in this synthetic environment.
This would help distinguish the empirical consequences of valid suboptimality constraints from those of deliberately misspecified bounds.
Well-posedness and normalization of the advantage-parameterized objective
I appreciate the acknowledgment that Eq. 163 requires an explicit magnitude constraint.
However, a distinction remains between the theoretically bounded optimal advantage A*r induced by a simplex-normalized reward and a freely parameterized neural advantage Aψ.
Although the former is necessarily bounded, the parameterization in Appendix N does not appear to guarantee that Aψ corresponds to any admissible normalized reward. In particular, restricting Aψ to nonpositive values does not impose a lower magnitude bound.
Could the authors specify:
- The exact magnitude constraint imposed on Aψ in their implementation, if any.
- Whether this constraint was used to obtain the results in Figure 3 or is being introduced only in the revision.
- How the resulting parameterization preserves the normalization underlying Eqs. 8, 12–13, given that Eqs. 20–21 impose only elementwise reward bounds and use a support-limited Bellman penalty.
If no explicit magnitude constraint was used in the reported experiments, additional results with the corrected objective would strengthen the practical contribution.
Evaluation of meaningful policy recovery
I appreciate the additional reward-variance results. These establish that the learned rewards do not collapse to constants in the reported experiments.
However, nonconstant rewards do not necessarily induce policies that perform well under the ground-truth reward.
Figure 2 measures SubOpt(r̂, d*ₑ), which evaluates whether the ground-truth optimal occupancy remains nearly optimal under the recovered reward. This is different from evaluating whether a policy optimized under r̂ performs well under the true reward.
Since the grid-world experiments have access to the synthetic ground-truth reward, could the authors additionally report the true-reward performance or regret of policies optimized under the recovered rewards, together with the corresponding MaxEnt results?
Similarly, Figure 3 measures the ability to rank an optimal response above a supplied near-optimal response. This is a useful discriminative evaluation, but it does not establish that the recovered reward enables the model to produce better tool-use sequences.
Could the authors provide an evaluation of actual tool-use success or response selection from a more diverse candidate set?
These measurements would help establish whether the proposed framework recovers rewards that improve decision-making rather than only separating particular demonstrated trajectories.
** MaxEnt baseline calibration**
Thank you for clarifying that Case 1 uses demonstrator-specific inverse temperatures βᵏ = k and that the common value βᵏ = 1.0 applies to Case 2. This resolves part of my original concern, and I appreciate the planned manuscript correction.
However, the relative performance of MaxEnt can depend substantially on temperature calibration.
Could the authors report results with a validation-tuned inverse temperature in Case 2 and the LLM experiment, where Table 2 reports β = 1.0?
A comparison against a reasonably tuned baseline would help determine how much of the improvement comes from the feasible-set formulation rather than the baseline's behavioral-model calibration.
I appreciate the authors' willingness to clarify these points. The proposed feasible-set perspective is interesting, and resolving the questions above would substantially strengthen the relationship between the theoretical formulation and the empirical claims.
We thank you for taking the time to carefully read our rebuttals and for clearly articulating the remaining concerns. We believe there are four remaining concerns regarding the evaluation of the proposed algorithms. We have conducted additional experiments to address these concerns as follows:
1. Could the authors clarify whether these quantities use exactly the same reward normalization? If so, what fraction of demonstrators satisfy their declared bounds, and what happens with or individual oracle bounds?
Yes, the reported Case-2 oracle suboptimality and are on the same reward scale. On the 20 maps underlying Figure 2, only 21/400 demonstrators (5.25%) satisfy ; the corresponding fractions for and are 126/400 (31.5%) and 350/400 (87.5%). Thus, represents the substantially misspecified case.
Following the suggestion, we repeated Case 2 on the same 20 maps, using constant bounds , and also individual ground-truth bounds . The following table reports the mean ± sample standard deviation of across maps.
| Case-2 bound | ||||
|---|---|---|---|---|
The consistent improvement as increases can be clearly observed for both and , and also for the ground-truth suboptimality bound without any misspecification. Thus, our evaluation holds for the ground-truth suboptimality-bound case, which the theory addresses. We will add this result to the revised manuscript.
2. How well do policies optimized under the recovered rewards perform on goal reaching?
We agree that it would be better to report the performance of an optimal policy under the obtained rewards. In the grid-world environment, for each recovered reward , we compute an optimal policy . We then evaluate the probability of reaching the goal within 400 steps under the uniform initial-state distribution.
The table reports goal-reaching rates (%) as the mean sample standard deviation over 20 maps.
Case 1:
| Method | ||||
|---|---|---|---|---|
| Proposed | ||||
| MaxEnt |
Case 2:
| Method | ||||
|---|---|---|---|---|
| Proposed | ||||
| MaxEnt |
In Case 1, our method shows consistent improvement as increases. For MaxEnt, the performance increases abruptly as approaches 20, showing that MaxEnt requires near-optimal demonstrators. In Case 2, our method shows consistent improvement as increases, while MaxEnt does not show a performance improvement. These results coincide with those of our original suboptimality metric in Figure 2, wihch shows that our suboptimality metric is highly correlated with the performance of the optimized policy. We will add this result to the revised manuscript.
In the LLM experiment, instead of , we report the win rate, which measures whether the obtained reward can distinguish the optimal trajectory from the erroneous trajectory. Therefore, the quality of the reward is reported directly, without suffering from the scale issue (degeneracy). Nevertheless, we will also add the LLM experimental results to the final manuscript, although we cannot present them now because of time and computational limits.
We thank the reviewer for the concrete checks on the experiments and the practical objective. We clarify each concern below and will incorporate these points into the revised manuscript.
Section 4.1 fixes ϵk=0.1, while Appendix O.2 reports a Case-2 mean oracle suboptimality of 0.233±0.069.
We would like to clarify that the true ϵk cannot be known in practice, so we set ϵk=0.1 as an arbitrary constant matched to the reward scale. As discussed in the manuscript, the proposed framework works under misspecified suboptimality bounds, even when the ground-truth reward does not lie in the feasible set. The effect of such a misspecified suboptimality bound is already explained in detail in Appendix O.2 (Table 1).
As we use a looser bound, that is, as ϵk increases, each demonstration becomes less informative, and the slope in Figure 2 becomes flatter. This behavior is expected and unavoidable in our framework, and it does not indicate a weakness of the evaluation. We will add results and figures for ϵk=0.2 and ϵk=0.3 in Appendix of the revised draft to clarify this point.
The evaluation metric SubOpt(r^,de⋆) can be zero for a constant reward.
We agree that fixing ϵk=0.1 below the true mean suboptimality overly tightens the feasible-set constraints and can produce a more constant (degenerate) reward. However, because we used the techniques specified in Section 3.2 to avoid degenerate rewards, the resulting reward does not easily become constant, as the variance reported in Table 1 shows.
To show this, the table below reports the mean population variance of the learned reward in Case 2 for K∈{1,5,10,20}. The variance stays near 0.10 from K=1 to K=20 and does not collapse to zero, so the recovered rewards do not converge to a constant reward, and SubOpt(r^,de⋆) is not vacuous metric. We will include the full data in the appendix.
Case 1 includes an optimal demonstrator at K=20.
We do not see what is problematic about Case 1 including an optimal demonstrator at K=20. Case 1 represents a setting in which optimality improves incrementally as K increases from 1 to 20, and for K≤19 the dataset contains no optimal demonstrator.
The advantage objective in Eq. 163 can be unbounded below.
We thank the reviewer for identifying this. The simplex normalization fixes the scale of reward r, and the corresponding advantage Ar⋆ should be bounded as well. Thus, there exists an implicit magnitude constraint in Eq. 163. With that constraint, the objective is not unbounded below. We will add the explicit magnitude constraint.
We would like to clarify that this constraint is already implicit by our definition, and it does not imply that our loss function is wrong. Therefore, we do not see this as a critical weakness.
The LLM experiment reports only pairwise ranking accuracy. Why not compare the policy trained on the recovered reward?
In this environment, correct and wrong trajectories are clearly distinguished, so the quality of the learned reward can be evaluated directly without training a policy. Training a policy would mix in an additional policy-optimization error and make the conclusion less clear.
MaxEnt uses βk=1.0 for every demonstrator despite heterogeneous expertise.
We would like to clarify that the common inverse temperature βk=1.0 is used only in Case 2, where every demonstrator knows exactly one portal and the demonstrators therefore have the same quality. In Case 1, where the demonstrators have heterogeneous expertise, the MaxEnt baseline uses a demonstrator-specific inverse temperature: a demonstrator that knows k portals is given βk=k. Even with these demonstrator-specific temperatures, MaxEnt remains substantially suboptimal. We thank the reviewer for pointing this out, and we will revise lines 419–420.
Additional questions in the Questions For Authors section.
We believe the reviewer submitted the wrong questions from a different ICLR submission.
Thank you again for the comments. If you have any further questions or would like additional clarification, we would be glad to respond. Otherwise, we would be grateful if you might reconsider your score in light of the points above.