acceptodds
Under review as a conference paper at ICLR 2027

Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach

Abstract

Inverse reinforcement learning (IRL) typically assumes demonstrations from a single optimal demonstrator, but in many applications data come from multiple imperfect demonstrators with heterogeneous suboptimality levels. We study reward learning in this setting through a feasible-reward-set framework: for each demonstrator, we encode its declared suboptimality level as a linear constraint and intersect the resulting feasible sets across demonstrators. Our theoretical analysis shows that the joint feasible set shrinks monotonically as data are added and provides a sufficient condition for strict shrinkage. We further establish optimality guarantees for the unobserved optimal demonstrator under every reward in the resulting feasible set. On the practical side, we introduce strategies to address the inherent reward ambiguity in the obtained reward set and provide an offline algorithm with function approximation for high-dimensional environments. Experiments in tabular grid-world and large language model (LLM) fine-tuning settings support the theoretical predictions and demonstrate improved performance over baselines.

Primary area:
reinforcement learning
Keywords:
Inverse Reinforcement Learning, Reward Learning, Imperfect Demonstrations, Heterogeneous Demonstrators, Feasible Reward Sets

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_a2P1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an inverse reinforcement learning framework for learning from multiple suboptimal demonstrators without access to an optimal demonstrator. Instead of estimating a single reward, the authors construct a feasible reward set by intersecting demonstrator-specific suboptimality constraints (Definition 3.1). Proposition 3.2 provides an LP characterization of this set.

The main theoretical contributions include a sufficient condition for strict feasible-set shrinkage (Theorem 3.4), an optimality-gap bound based on proximity to the unobserved optimal occupancy (Theorem 3.5, Eq. 16), and complementary coverage-based guarantees (Theorem H.1). The authors further propose reward-selection and offline function-approximation methods (Eq. 17; Eqs. 20–21).

Experiments on grid-world tasks (Figure 2) and LLM arithmetic tool-use tasks (Figure 3) show improvements over a MaxEnt baseline.

Critical strengths

Definition 3.1 and Proposition 3.2 (Eqs. 12–13) formulate reward learning through demonstrator-specific suboptimality constraints without requiring an optimal demonstrator. This is a meaningful extension of the feasible-reward-set perspective and provides an interpretable alternative to likelihood-based reward fitting.

Theorem 3.5 (Eq. 16) and Theorem H.1 (Eqs. 85–87) identify conditions under which suboptimal demonstrations collectively constrain rewards associated with optimal behavior. The Case-2 grid-world experiment (Figure 2) provides encouraging empirical evidence for this complementary-coverage phenomenon.

Critical weaknesses

1). The framework assumes that every demonstrator's true suboptimality is bounded by its declared εᵏ (Section 3.1, Definition 3.1, Eqs. 11–12). These bounds determine the feasible set in Eq. 13 and are central to interpreting the theoretical guarantees.

However, Section 4.1 (lines 414–420) fixes εᵏ = 0.1 for all grid-world demonstrators. In contrast, Appendix O.2 (Eq. 169 and the accompanying discussion) reports that the mean oracle suboptimality in the Case-2 environment is 0.233 ± 0.069.

Unless a different reward normalization explains this discrepancy, at least some demonstrators violate their declared bounds. Consequently, the true reward may not belong to the feasible set in Eq. 12, undermining the connection between the theoretical assumptions and the principal experimental results.

This is particularly consequential because the sensitivity experiment in Table 1 (p. 30) demonstrates that underestimated bounds can produce degenerate rewards with artificially low measured suboptimality.

Additionally, the Case-1 experiment explicitly includes an optimal demonstrator at K = 20 (Section 4.1, p. 8, lines 405–408), so its endpoint does not test the claimed setting without an optimal demonstrator.

2). The theoretical reward set imposes simplex normalization (Eq. 8; Eq. 12). However, the neural implementation in Section 3.3 (Eqs. 20–21) only bounds reward values individually, while the advantage-parameterized objective in Appendix N (Eq. 163) does not impose an explicit magnitude or normalization constraint.

As written, Eq. 163 can be unbounded below. For example, consider a one-state, two-action system with demonstrator occupancy (1, 0), baseline occupancy (0, 1), and advantage values (0, −M). Both hinge penalties vanish for any M > 0, while the reward-selection objective equals −M. Thus the loss tends to negative infinity as M increases.

The suggested nonpositive parameterization Aψ = −softplus(fψ) ( lines 1499–1501) does not address this magnitude issue. This raises concerns about whether the implementation preserves the normalized-reward assumptions and optimization properties of the theoretical LP.

Furthermore, the grid-world evaluation uses SubOpt(r̂, d*ₑ) (Section 4.1), which can be zero for degenerate constant rewards. The authors explicitly demonstrate this failure mode in Table 1 ( Δ = −0.3). Yet Figure 2 does not report reward non-degeneracy or the performance of policies optimized under the recovered rewards.

Similarly, the LLM experiment reports pairwise response-ranking accuracy (Eq. 171), rather than the success rate of an actual policy trained or optimized using the recovered reward. Hence Figure 3 provides evidence of discrimination between two supplied responses, but not necessarily successful recovery of better decision-making behavior.

These issues substantially weaken the practical conclusions.

Questions for authors

How are the soft violation penalties incorporated into the partition function and local policy normalization in the actual implementation? Can the authors provide a mathematically consistent derivation establishing normalization and demonstrating a concrete example where the proposed local likelihood detects a constraint that the global path likelihood cannot?

How does MOCI differ substantively from CoCoRL (Lindner et al., 2024), which already considers shared constraints under heterogeneous unknown rewards? Can the authors provide a direct constraint-recovery comparison, where applicable, and compare against a baseline combining existing multi-intention IRL with conventional constraint inference? These comparisons would materially affect the assessment of novelty and effectiveness.

Under what assumptions can MOCI distinguish an action that is genuinely infeasible from one that all observed experts strongly dislike? Figure 3 assigns substantial negative water rewards to both expert types even though water is also designated as a hard constraint. How should this be interpreted in light of the claimed separation between constraints and preferences?

7 comments

We thank the reviewer for the concrete checks on the experiments and the practical objective. We clarify each concern below and will incorporate these points into the revised manuscript.

 

  • Section 4.1 fixes ϵk=0.1\epsilon^k=0.1, while Appendix O.2 reports a Case-2 mean oracle suboptimality of 0.233±0.0690.233\pm 0.069.

    We would like to clarify that the true ϵk\epsilon^k cannot be known in practice, so we set ϵk=0.1\epsilon^k=0.1 as an arbitrary constant matched to the reward scale. As discussed in the manuscript, the proposed framework works under misspecified suboptimality bounds, even when the ground-truth reward does not lie in the feasible set. The effect of such a misspecified suboptimality bound is already explained in detail in Appendix O.2 (Table 1).

    As we use a looser bound, that is, as ϵk\epsilon^k increases, each demonstration becomes less informative, and the slope in Figure 2 becomes flatter. This behavior is expected and unavoidable in our framework, and it does not indicate a weakness of the evaluation. We will add results and figures for ϵk=0.2\epsilon^k=0.2 and ϵk=0.3\epsilon^k=0.3 in Appendix of the revised draft to clarify this point.

 

  • The evaluation metric SubOpt(r^,de⋆)\mathrm{SubOpt}(\hat r, d_e^\star) can be zero for a constant reward.

    We agree that fixing ϵk=0.1\epsilon^k=0.1 below the true mean suboptimality overly tightens the feasible-set constraints and can produce a more constant (degenerate) reward. However, because we used the techniques specified in Section 3.2 to avoid degenerate rewards, the resulting reward does not easily become constant, as the variance reported in Table 1 shows.

    To show this, the table below reports the mean population variance of the learned reward in Case 2 for K∈{1,5,10,20}K\in\{1,5,10,20\}. The variance stays near 0.100.10 from K=1K=1 to K=20K=20 and does not collapse to zero, so the recovered rewards do not converge to a constant reward, and SubOpt(r^,de⋆)\mathrm{SubOpt}(\hat r,d_e^\star) is not vacuous metric. We will include the full data in the appendix.

    KK115510102020
    Variance of rewards0.095±0.0080.095\pm 0.0080.105±0.0100.105\pm 0.0100.108±0.0100.108\pm 0.0100.109±0.0110.109\pm 0.011

 

  • Case 1 includes an optimal demonstrator at K=20K=20.

    We do not see what is problematic about Case 1 including an optimal demonstrator at K=20K=20. Case 1 represents a setting in which optimality improves incrementally as KK increases from 11 to 2020, and for K≤19K\le 19 the dataset contains no optimal demonstrator.

 

  • The advantage objective in Eq. 163 can be unbounded below.

    We thank the reviewer for identifying this. The simplex normalization fixes the scale of reward rr, and the corresponding advantage Ar⋆A_r^\star should be bounded as well. Thus, there exists an implicit magnitude constraint in Eq. 163. With that constraint, the objective is not unbounded below. We will add the explicit magnitude constraint.

    We would like to clarify that this constraint is already implicit by our definition, and it does not imply that our loss function is wrong. Therefore, we do not see this as a critical weakness.

 

  • The LLM experiment reports only pairwise ranking accuracy. Why not compare the policy trained on the recovered reward?

    In this environment, correct and wrong trajectories are clearly distinguished, so the quality of the learned reward can be evaluated directly without training a policy. Training a policy would mix in an additional policy-optimization error and make the conclusion less clear.

 

  • MaxEnt uses βk=1.0\beta^k=1.0 for every demonstrator despite heterogeneous expertise.

    We would like to clarify that the common inverse temperature βk=1.0\beta^k = 1.0 is used only in Case 2, where every demonstrator knows exactly one portal and the demonstrators therefore have the same quality. In Case 1, where the demonstrators have heterogeneous expertise, the MaxEnt baseline uses a demonstrator-specific inverse temperature: a demonstrator that knows kk portals is given βk=k\beta^k = k. Even with these demonstrator-specific temperatures, MaxEnt remains substantially suboptimal. We thank the reviewer for pointing this out, and we will revise lines 419–420.

 

  • Additional questions in the Questions For Authors section.

    We believe the reviewer submitted the wrong questions from a different ICLR submission.

 

Thank you again for the comments. If you have any further questions or would like additional clarification, we would be glad to respond. Otherwise, we would be grateful if you might reconsider your score in light of the points above.

Thank you for your detailed response and clarifications.

First, I apologize for an inadvertent error in my original review. The three questions under "Questions for Authors" were mistakenly carried over from another manuscript. Please disregard those questions. The paper summary, strengths, weaknesses, and other technical concerns in my review refer to the present submission.

I appreciate the authors' responses, particularly the additional reward-variance results and the acknowledgment that Eq. 163 requires an explicit magnitude constraint. I have the following revised questions.

Validity and calibration of demonstrator suboptimality bounds

Section 4.1 specifies εᵏ = 0.1, whereas Appendix O.2 reports mean oracle suboptimality of 0.233 ± 0.069. The response acknowledges that the declared bounds can exclude the ground-truth reward from the feasible set.

Could the authors clarify whether these quantities use exactly the same reward normalization? If so, what fraction of demonstrators in Figure 2 satisfy their declared suboptimality bounds?

I appreciate the proposed additional experiments with εᵏ = 0.2 and 0.3. It would be particularly informative to repeat the Case-2 experiment using individual oracle bounds εᵏ = εᵏ*, which are available in this synthetic environment.

This would help distinguish the empirical consequences of valid suboptimality constraints from those of deliberately misspecified bounds.

Well-posedness and normalization of the advantage-parameterized objective

I appreciate the acknowledgment that Eq. 163 requires an explicit magnitude constraint.

However, a distinction remains between the theoretically bounded optimal advantage A*r induced by a simplex-normalized reward and a freely parameterized neural advantage Aψ.

Although the former is necessarily bounded, the parameterization in Appendix N does not appear to guarantee that Aψ corresponds to any admissible normalized reward. In particular, restricting Aψ to nonpositive values does not impose a lower magnitude bound.

Could the authors specify:

  • The exact magnitude constraint imposed on Aψ in their implementation, if any.
  • Whether this constraint was used to obtain the results in Figure 3 or is being introduced only in the revision.
  • How the resulting parameterization preserves the normalization underlying Eqs. 8, 12–13, given that Eqs. 20–21 impose only elementwise reward bounds and use a support-limited Bellman penalty.

If no explicit magnitude constraint was used in the reported experiments, additional results with the corrected objective would strengthen the practical contribution.

Evaluation of meaningful policy recovery

I appreciate the additional reward-variance results. These establish that the learned rewards do not collapse to constants in the reported experiments.

However, nonconstant rewards do not necessarily induce policies that perform well under the ground-truth reward.

Figure 2 measures SubOpt(r̂, d*ₑ), which evaluates whether the ground-truth optimal occupancy remains nearly optimal under the recovered reward. This is different from evaluating whether a policy optimized under r̂ performs well under the true reward.

Since the grid-world experiments have access to the synthetic ground-truth reward, could the authors additionally report the true-reward performance or regret of policies optimized under the recovered rewards, together with the corresponding MaxEnt results?

Similarly, Figure 3 measures the ability to rank an optimal response above a supplied near-optimal response. This is a useful discriminative evaluation, but it does not establish that the recovered reward enables the model to produce better tool-use sequences.

Could the authors provide an evaluation of actual tool-use success or response selection from a more diverse candidate set?

These measurements would help establish whether the proposed framework recovers rewards that improve decision-making rather than only separating particular demonstrated trajectories.

** MaxEnt baseline calibration**

Thank you for clarifying that Case 1 uses demonstrator-specific inverse temperatures βᵏ = k and that the common value βᵏ = 1.0 applies to Case 2. This resolves part of my original concern, and I appreciate the planned manuscript correction.

However, the relative performance of MaxEnt can depend substantially on temperature calibration.

Could the authors report results with a validation-tuned inverse temperature in Case 2 and the LLM experiment, where Table 2 reports β = 1.0?

A comparison against a reasonably tuned baseline would help determine how much of the improvement comes from the feasible-set formulation rather than the baseline's behavioral-model calibration.

I appreciate the authors' willingness to clarify these points. The proposed feasible-set perspective is interesting, and resolving the questions above would substantially strengthen the relationship between the theoretical formulation and the empirical claims.

We thank you for taking the time to carefully read our rebuttals and for clearly articulating the remaining concerns. We believe there are four remaining concerns regarding the evaluation of the proposed algorithms. We have conducted additional experiments to address these concerns as follows:

 

1. Could the authors clarify whether these quantities use exactly the same reward normalization? If so, what fraction of demonstrators satisfy their declared bounds, and what happens with ϵk=0.2,0.3\epsilon^k = 0.2, 0.3 or individual oracle bounds?

Yes, the reported Case-2 oracle suboptimality and ϵk=0.1\epsilon^k = 0.1 are on the same reward scale. On the 20 maps underlying Figure 2, only 21/400 demonstrators (5.25%) satisfy ϵk=0.1\epsilon^k = 0.1; the corresponding fractions for ϵk=0.2\epsilon^k = 0.2 and ϵk=0.3\epsilon^k = 0.3 are 126/400 (31.5%) and 350/400 (87.5%). Thus, ϵk=0.1\epsilon^k = 0.1 represents the substantially misspecified case.

Following the suggestion, we repeated Case 2 on the same 20 maps, using constant bounds ϵk=0.2,0.3\epsilon^k = 0.2, 0.3, and also individual ground-truth bounds ϵk=ϵk,⋆\epsilon^k = \epsilon^{k,\star}. The following table reports the mean ± sample standard deviation of SubOpt⁡(r^,de⋆)\operatorname{SubOpt}(\widehat r,d_e^\star) across maps.

Case-2 boundK=1K = 1K=5K = 5K=10K = 10K=20K = 20
ϵk=0.1\epsilon^k = 0.10.389±0.1660.389 \pm 0.1660.207±0.1190.207 \pm 0.1190.149±0.1030.149 \pm 0.1030.043±0.0570.043 \pm 0.057
ϵk=0.2\epsilon^k = 0.20.441±0.1630.441 \pm 0.1630.344±0.1330.344 \pm 0.1330.293±0.1010.293 \pm 0.1010.175±0.0880.175 \pm 0.088
ϵk=0.3\epsilon^k = 0.30.442±0.1640.442 \pm 0.1640.357±0.1400.357 \pm 0.1400.321±0.1120.321 \pm 0.1120.208±0.1240.208 \pm 0.124
ϵk=ϵk,⋆\epsilon^k = \epsilon^{k,\star}0.440±0.1670.440 \pm 0.1670.339±0.1560.339 \pm 0.1560.299±0.1330.299 \pm 0.1330.177±0.1330.177 \pm 0.133

The consistent improvement as KK increases can be clearly observed for both ϵk=0.2\epsilon^k = 0.2 and 0.30.3, and also for the ground-truth suboptimality bound without any misspecification. Thus, our evaluation holds for the ground-truth suboptimality-bound case, which the theory addresses. We will add this result to the revised manuscript.

 

2. How well do policies optimized under the recovered rewards perform on goal reaching?

We agree that it would be better to report the performance of an optimal policy under the obtained rewards. In the grid-world environment, for each recovered reward r^\widehat r, we compute an optimal policy π^∈arg⁡max⁡π⟨r^,dπ⟩\widehat\pi \in \arg\max_\pi \langle\widehat r,d^\pi\rangle. We then evaluate the probability of reaching the goal within 400 steps under the uniform initial-state distribution.

The table reports goal-reaching rates (%) as the mean ±\pm sample standard deviation over 20 maps.

Case 1:

MethodK=1K = 1K=5K = 5K=10K = 10K=20K = 20
Proposed25.71±23.0425.71 \pm 23.0451.48±27.0451.48 \pm 27.0461.45±20.2261.45 \pm 20.2282.16±12.6882.16 \pm 12.68
MaxEnt27.59±38.8727.59 \pm 38.8748.27±41.8448.27 \pm 41.8449.60±43.6549.60 \pm 43.6591.88±25.0891.88 \pm 25.08

Case 2:

MethodK=1K = 1K=5K = 5K=10K = 10K=20K = 20
Proposed23.52±22.9623.52 \pm 22.9638.50±29.6938.50 \pm 29.6941.97±23.5041.97 \pm 23.5045.24±26.8545.24 \pm 26.85
MaxEnt22.54±35.8222.54 \pm 35.8216.90±26.3816.90 \pm 26.3819.01±30.7019.01 \pm 30.7016.84±27.1016.84 \pm 27.10

In Case 1, our method shows consistent improvement as KK increases. For MaxEnt, the performance increases abruptly as KK approaches 20, showing that MaxEnt requires near-optimal demonstrators. In Case 2, our method shows consistent improvement as KK increases, while MaxEnt does not show a performance improvement. These results coincide with those of our original suboptimality metric in Figure 2, wihch shows that our suboptimality metric is highly correlated with the performance of the optimized policy. We will add this result to the revised manuscript.

In the LLM experiment, instead of SubOpt⁡(r^,de⋆)\operatorname{SubOpt}(\widehat r,d_e^\star), we report the win rate, which measures whether the obtained reward can distinguish the optimal trajectory from the erroneous trajectory. Therefore, the quality of the reward is reported directly, without suffering from the scale issue (degeneracy). Nevertheless, we will also add the LLM experimental results to the final manuscript, although we cannot present them now because of time and computational limits.

Reviewer_MQUYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits concepts of feasible reward sets in the context of inverse reinforcement learning, where their goal is to recover a reward function given a collection of suboptimal expert demonstrations (and some estimate of their suboptimality). The paper defines a notion of reward feasibility sets given expert demonstrations, which are sets of reward functions inducing a set of occupancy measures containing some margin around the expert demonstrations, with the margin given by the per-expert optimality estimate. The paper first shows conditions under which adding new suboptimal demonstrations will necessarily shrink the feasible reward set, and ultimately provides bounds on the utility of certain rewards extracted from the resulting reward sets, suggesting that a sufficient condition for IRL is that a mixture of suboptimal expert policies is close to optimal. Their analysis naturally suggests an IRL approach based on linear programming in the tabular case, and the authors additionally introduce an algorithm based on function approximation, enabling scaling to LLM fine-tuning.

Critical strengths

Despite my presentation score, I thought the paper was quite well written for the most part, mainly up until section 3.2. The paper was otherwise a joy to read, and discusses an interesting, relevant, and important problem. I was particularly interested in how ambitious the authors were with respect to the suboptimality of the experts – these weren't just suboptimal experts, but rather bad policies, and it is interesting how such bad policies can be leveraged in conjunction to learn something nontrivial. I also appreciated the extension to LLM fine-tuning and found the empirical result there to be pretty neat, though I do have some questions about the methodology, etc, as mentioned later in the review.

Critical weaknesses

I believe there are a few main weaknesses in this work, namely with respect to the feasible reward framework, the considerations in Section 3.2 and 3.3, and the experiments.

First, in terms of theory, I am not totally clear on what is new. Clearly the concept of feasible reward set is not new (the authors cite Kim et al., 2024, for example). It looks like the main addition here is the concept of a feasible reward set induced by a set of demonstrator-suboptimality pairs. As I mention in the Questions section, though, the theoretical results regarding this set are not necessarily very strong, in the sense that they seem to require very strong assumptions on the supports of the occupancy measures in question. Proposition 3.6 seems to give some motivation for why the resulting LP approach is better than existing tabular methods (because for those tabular methods, you can lower bound the suboptimality), but I believe the same is true for this proposed method (e.g., just construct a dataset where all experts are CC-suboptimal?).

Next, regarding section 3.2, I found this section largely rushed and a little handwavy. This is particularly true for equation 17, which I ask about in the Questions section; it seems certain correctness properties are lost.

Furthermore, I am a little confused by Section 3.3 (again, see Questions). Here, the authors propose a function approximation IRL method, but the connection to the previous analysis about feasible reward sets seems to have disappeared. What we're left with is a loss that looks very familiar in the context of IRL.

Finally, the experiments are missing details and baselines. Again, see Questions for conrete questions I had about those. But one thing is for sure – the experiments seem to be missing many relevant baselines, at the very least, some of the ones that are discussed at length in the paper (e.g. T-REX). This is especially true in the context of the LLM experiment. Especially in this function approximation regime, where we have seemingly deviated from the feasible set maintenance framework, there is a whole slew of popular IRL methods that are worth comparing to. I suppose the point of this paper is not to provide a SOTA LLM finetuning algorithm, so this isn't a critical issue on its own. That said, there is a lot of ambiguity about how the experiment was conducted, to the point where I can't draw any conclusions from these results yet myself.

Questions for authors

Theorem 3.4 seems potentially almost vacuous, though this depends on the setting. I suppose it would be helpful to better frame when it is actually "triggered". This is addressed slightly in the text, but I think more intuition is warranted. For example, any time any of your suboptimal experts' occupancy measure is supported on the whole state-action space (say, your expert has some softmax policy, etc), then Theorem 3.4 is a no-op. Another concern is that the denominator of equation 14 scales with the support of the occupancy measures, while the numerator only scales with the mass (which could potentially grow much less quickly).

I really don't understand equation 17. Where does this come from? Why is it appropriate to subtract off the base policy's occupancy measure here, especially if you expect the base policy to be reasonably good? For example, suppose there is a high reward at state 1, and the base occupancy measure agrees with d‾e\overline{d}_e at that state. Then the reward you extract from equation 17 may assign 00 reward to that state, even though all experts and the base policy visit it. Is there something that prevents this? Or is there a reason why the feasibility of the rewards in the search space guarantees that this is actually not a problem? I'm not seeing it.

What is the connection between equation 20/21 with the method proposed before in the tabular case? I don't see any notion of reward feasibility left. In fact, it looks like this is a very similar loss as what is used in the SFM paper [1], but consuming multiple expert demonstrations. In fact, how do you represent the estimated occupancy measures (given that they're distributions over very large spaces)?

I'm also not sure logistically how the LLM experiment works. Looking past the issue of how to represent the occupancy measures, it looks like you'll still need one phase where you estimate the reward and value function, and then you still need to train a policy – do you just do RL with the reward function you recover from some finite amount of optimization steps on equation 20/21?

References

  1. Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching. Jain et al., ICLR 2025. https://proceedings.iclr.cc/paper_files/paper/2025/file/60a3b2b3210acc010ecfbbd6d5afc8b2-Paper-Conference.pdf.
6 comments

We appreciate that you enjoyed reading our paper, and we thank you for the careful comments on both the theory and the empirical evaluation. We clarify each concern below and will incorporate these points into the revised manuscript.

 

  • In terms of theory, it is unclear what is genuinely new, as the concept of feasible reward set is not new.

    As described in the manuscript, the presented intersection-based feasible reward set under multiple suboptimal demonstrators (Eqs. 11–13) is not itself new, and we do not claim it as our novelty. Our contribution is the set of theoretical guarantees for this intersection without an optimal demonstrator: the feasible set shrinks monotonically as demonstrations are added (Theorem 3.4), every retained reward makes the unobserved optimal demonstrator nearly optimal when a mixture of the demonstrators is close to optimal (Theorem 3.5), and finitely many suboptimal demonstrators suffice to drive this gap below any prescribed level (Theorem G.1). These properties are generally unavailable to MLE-based methods with demonstrator-specific temperatures (Proposition 3.6). To our knowledge, these guarantees have not been clearly investigated in the literature.

 

  • Theorem 3.4 looks nearly vacuous whenever any suboptimal expert has full state–action support, as with a softmax policy.

    We would like to clarify two points regarding this comment. First, Theorem 3.4 is designed for the complementary-coverage regime. The exact criterion is Lemma C.2: the feasible set shrinks if and only if ϵ′<max⁡r∈R(D)SubOpt(r,de′)\epsilon'<\max_{r\in\mathcal{R}(\mathcal{D})}\mathrm{SubOpt}(r,d_e'). This condition can hold even when the supports overlap, and it provides the tightest bound.

    Second, although we allow stochastic policies, the analysis is for an MDP without entropy regularization, so a suboptimal demonstrator need not visit every state–action pair. The full-support softmax policy arises instead under entropy or KL regularization. There the suboptimality is the gap in the KL-regularized return, SubOpt(r,π)=Jrβ(πr)−Jrβ(π)\mathrm{SubOpt}(r,\pi)=J_r^\beta(\pi_r)-J_r^\beta(\pi), where Jrβ(π)J_r^\beta(\pi) is the return of π\pi minus a KL penalty against a reference policy. Under this setting, the constraint is no longer linear, but it remains convex. The same arguments therefore still apply once the declared bounds are adjusted by this additional KL term. We will add this in the revised draft.

 

  • In Eq. 14, the denominator scales with the support of the occupancy measures, while the numerator scales only with the mass on the relevant set, which can grow much more slowly.

    We do not see this as critical. As described in lines 224–230, the reward lies in the simplex Δ(S×A)\Delta(S\times A) rather than in [0,1]∣S∣∣A∣[0,1]^{|S||A|}. Its average value is therefore 1/(∣S∣∣A∣)1/(|S||A|), and ϵ\epsilon is on the same small scale. The denominator in Eq. 14 is a consequence of this normalization. Moreover, the main implication of Theorem 3.4 is that shrinkage is guaranteed under this condition. The theorem is not intended to be a strong quantitative bound. The exact bound is provided in Lemma C.2.

 

  • Proposition 3.6 seems to motivate the LP by showing that existing tabular methods can suffer a suboptimality lower bound. The same construction appears to apply here: take a dataset in which every expert is CC-suboptimal.

    This is not the case in our framework. A near-zero bound is still possible when every demonstrator is CC-suboptimal, provided the coverage is sufficient. Appendix G gives a sufficient condition that does not require a near-optimal demonstrator and can be satisfied by finitely many demonstrators. Appendix H gives a tighter bound that depends on the coverage weights. We will move a more detailed discussion of these results into the main text, since this is an important advantage of our framework. We understand that reviewers are not expected to read the appendix, but we would be grateful if you could check Appendices G and H.

    The empirical evaluation shows the same point: demonstrators whose suboptimality stays fixed can still produce a meaningful reward when their coverage is complementary, as in Case 2 of Figure 2 and the LLM experiment in Figure 3.

  • Section 3.2 is rushed and handwavy, and Eq. 17 seems to drop correctness properties of the feasible set. Where does this objective come from, and why subtract the base occupancy, especially if the base policy is reasonably good?

    Equation 17 is a selection rule inside the feasible set, which remains the constraint set in Eq. 13. The guarantees in the previous section therefore continue to apply to the selected reward. When the demonstrators provide rich information, the constraints rule out arbitrarily bad rewards.

    In addition, the max-margin idea in Eq. 17 is a standard approach in IRL. Apprenticeship Learning via Inverse Reinforcement Learning (Abbeel and Ng, 2004), one of the earliest works in IRL, introduced this idea, and it has since been widely used. As stated in the manuscript, the baseline occupancy is assumed to be substantially worse than the demonstrators. Examples are the uniform policy and, in the LLM experiment, a trace that replaces every tool call by a placeholder.

    We appreciate the comment regarding the readability of Section 3.2. We reduced the surrounding context and moved the details to the appendix because of the space limit, since this section only adds a rule for selecting one reward from the feasible set. Following the comment, we will expand this discussion in the revised manuscript.

 

  • The function-approximation method no longer maintains a feasible reward set, and the loss looks like a standard IRL objective, such as SFM (Jain et al., 2025).

    The feasible-set structure is maintained in the function-approximation loss (Eqs. 20–21). These equations are the empirical form of the same program: the first term is the return gap in Eq. 17, and the other two terms are soft penalties for the suboptimality and Bellman constraints in Eq. 13. The hard polytope is replaced by penalties, while the per-demonstrator budgets ϵk\epsilon^k remain in the loss. Therefore, if the number of samples goes to infinity and the two penalty terms are driven to zero, the learned reward satisfies rθ∈R(D)r_\theta\in\mathcal{R}(\mathcal{D}). In large spaces, the occupancy does not need to be stored as an explicit distribution. For example, as in Eq. 163, each inner product can be computed as a discounted sum of advantages along the observed trajectories.

    We thank the reviewer for introducing this related paper. While SFM (Jain et al., 2025) addresses non-adversarial imitation from a single expert by matching successor features, our loss is for multiple suboptimal demonstrators and keeps a suboptimality bound ϵk\epsilon^k for each demonstrator. SFM has no such bound. We will clarify this distinction in the revised manuscript.

 

  • The LLM experiments omit baselines such as T-REX. For the LLM experiment, once the method leaves explicit feasible-set maintenance, many standard IRL methods are relevant comparisons.

    We would like to separate our response into two parts. First, the experiments are designed to illustrate the theoretical claim that the intersection-based method outperforms MLE-based methods when the demonstrators are mostly suboptimal (Proposition 3.6). We do not claim a practical, deployable IRL algorithm with SOTA efficiency. We will make this scope clear in the revised manuscript.

    Second, as discussed above, the method does not leave feasible-set maintenance. Eqs. 20–21 keep the per-demonstrator suboptimality ϵk\epsilon^k, and the learned reward lies in R(D)\mathcal{R}(\mathcal{D}) when the penalty terms vanish.

    We agree that adding a ranking baseline such as T-REX would be a helpful addition. Proposition 3.6 already separates this estimator in theory, but the LLM experiment would benefit from an additional case in which the demonstrators have different but comparable suboptimality, together with a ranking-based baseline. We will run this experiment and include it in the revised manuscript.

 

  • The LLM training procedure still seems to need one phase that estimates the reward and value, and another that trains a policy. Is the policy trained by RL on the reward recovered from finitely many steps of Eqs. 20–21?

    There is no second RL stage in our LLM experiment. We report the quality of the trained reward model by how often the optimal trace outscores a trace with one wrong tool name (Eq. 171), precisely to avoid confounding the comparison with a further policy-optimization stage. We will explicitly discuss this in Section 4.2 to avoid confusion.

 

Thank you again for the careful comments. If you have any further questions or would like additional clarification, we would be glad to respond. Otherwise, we would be grateful if you might reconsider your score in light of the points above.

Thanks to the authors for the detailed comments and clarifications. I'll try to address them one by one.

1: Novelty of theory. Fair enough, I think my concern largely lies then in Theorem 3.4 (next). That said, while Theorem G.1 that you point out in the rebuttal seems interesting, this theorem lived entirely in the appendix with no reference from the main text; I did not see this during my review, and I do not have time to sufficiently appreciate it now. If it's a crucial result, it should be in the main text.

2: Theorem 3.4. Again, I appreciate the detailed rebuttal, I'll break my response up in two as you did, which was helpful.

  1. I do understand that the paper is focused on showing how complementary coverage can be leveraged, which is interesting. Ignoring Lemma C.2 for now, though, it seems very hard to guarantee that your individual experts occupy basically a partition of the state space, which, as I understand it, is roughly what Theorem 3.4 requires if your individual experts are suboptimal. That said, it seems you're suggesting that the assumption in Theorem 3.4 can be sharpened with Lemma C.2, but I don't see the connection immediately. Would you mind spelling it out for me?

  2. Regarding entropy-regularized RL -- this is not necessarily what I was referring to. There are just certain methods that employ stochastic policies because the method works well, e.g. SAC. While SAC was built upon the entropy-regularized RL framework, it is still a strong policy optimizer in the classical RL sense where you don't care about the entropy-regularized objective. Moreover, the paper has an LLM experiment, which is another example of a softmax policy. My point was that, even though these types of policies are largely concentrated on a "good" set of states, they technically can have full state support, which makes the assumption in Theorem 3.4 much much weaker due to the dependence on the cardinality of the support.

3: Denominator scaling. Addressed above.

4: Proposition 3.6. Thanks for the appendix pointers, but again I do not have the bandwidth to piece this all together myself now. The way appendices G and H are framed in the main text is that they provide supplementary results, not results that are critical for demonstrating the power of the proposed method. Again, if you can spell it out for me more precisely, that would help. I also haven't noticed any amendments to the draft yet; perhaps it would be useful to integrate some of these changes during the discussion period where it might be most clear.

5: Section 3.2 / Equation 17. Thanks for the clarification, I understand. This is no longer a concern.

6: Function approximation method. Thanks for the clarification. I think when I originally wrote my review, I was mainly hung up on the distinction between the hard constraint vs soft constraint, but I think this is fair. Regarding the comparison to SFM -- to be clear, I totally agree that your paper contributes something new (e.g. multiple AND suboptimal demonstrators), just that some parts of it look mainly superficially similar, and was curious if there was a deeper connection.

7: LLM Baselines. See above -- agreed that the function approximation methods do account for the feasible reward set. Would still be curious to see a comparison with T-REX, not so much for the purpose of establishing SOTA, but just to understand how they differ in practice.

8: LLM training procedure. I see -- I think I misunderstood the LLM experiment as you suggested. So to be totally clear, you are not doing any form of policy improvement, you are just evaluating the quality of the inferred reward model via win rate, which is defined roughly as the probability that the learned reward model scores a correct completion higher than an incorrect one? This makes sense and addresses my original concern. Is it surprising that for 1 suboptimal expert, your proposed method already is quite a bit better than the baseline? I'd expect your method to mainly shine as the number of experts increases / the coverage increases.

Reviewer_JBGCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies learning rewards from multiple imperfect demonstrators without assuming access to an optimal demonstrator. It uses assumed bounds on each demonstrator’s suboptimality to narrow the set of rewards that could explain their behavior. The theory gives conditions under which combining demonstrations reduces reward ambiguity. The authors also propose a neural implementation and test the approach in a grid world and a synthetic arithmetic tool-use task using an LLM.

Critical strengths
  • The problem is interesting, particularly the possibility of learning from demonstrators with different strengths when none is optimal. The paper examines when their complementary information can help reduce reward ambiguity.

  • The main finite-MDP results are supported by detailed proofs that appear sound. The analysis gives explicit conditions under which additional demonstrations provide useful information, and the linear-programming formulation makes the tabular problem concrete.

Critical weaknesses
  • I am not sure about the significance of the contribution relative to existing feasible-reward-set approaches. The paper should explain more clearly what new insight its guarantees provide. The relationship to Lazzati et al. 2025, Reward Compatibility: A Framework for Inverse RL, should also be discussed, particularly its treatment of bounded demonstrator suboptimality. The claim of opening a new avenue beyond likelihood-based IRL seems too broad.

  • The suboptimality bounds are a substantial practical assumption. Obtaining valid, informative bounds without knowing the true reward or optimal behavior may be difficult. The proposed calibration heuristic is discussed, but its reliability, tuning requirements, and cost are not really demonstrated. This is important because the guarantees depend on these bounds, and overly tight choices can produce uninformative rewards.

  • I am not convinced that the experiments demonstrate useful reward learning. The sensitivity experiment produces a constant reward with a perfect value of the main tabular metric. The main plots therefore need additional checks to distinguish meaningful improvement from reward degeneracy. The reward-selection objectives are specified, but it is unclear whether the additional coverage penalty or pairwise constraints are used, and the grid-world baseline occupancy is not specified. There is also no ablation showing how much these choices matter. A metric approaching zero is not enough on its own. Whether an agent trained using the learned reward actually reaches the goal efficiently is not demonstrated.

  • The LLM experiment is too narrow to provide strong evidence of usefulness for LLM applications. The task consists of short synthetic traces using only five arithmetic tools, with demonstrators constructed to make prescribed errors. Evaluation checks whether a supplied correct trace scores above a supplied trace with one incorrect tool name. This is a limited scoring test. Using an LLM does not, by itself, show that the method works for less structured language tasks. There is no evaluation of improved generation, successful tool execution, or more complex tasks. The reward-selection baseline is also artificially constructed using unknown tool calls, rather than testing the pretrained-model baseline motivated earlier. I would view this as a small proof of concept.

  • The demonstration counts are reported, but it is still unclear how much data the method needs. The grid world uses 100 trajectories per demonstrator. The LLM setup uses 2,000 training tasks with one constructed completion from each demonstrator per task. The paper does not show what happens when fewer demonstrations are available from each demonstrator. Adding demonstrators in the grid world also increases the total data, making it difficult to separate complementary information from additional samples. The jump to neural function approximation feels abrupt, and the data requirements and reliability of the implemented method remain unclear. The cited occupancy-estimation bounds do not resolve these questions.

  • The baseline comparison is insufficiently justified. MaxEnt barely improves as demonstrators are added in Figure 2, right, and Figure 3. The authors attribute this to fitting aggregated suboptimal demonstrations, but do not test whether this persists under appropriate tuning. Its inverse temperature is fixed across demonstrators, and relevant newer methods are cited but not evaluated. I am especially concerned about omitting the ranking baseline in Case 1 because it reportedly performed worse. Those results should be shown. The lack of a strict ordering can explain its exclusion from Case 2, but does not justify withholding the Case 1 results. The claim of comparable training time also needs numerical runtimes.

  • The main text relies too heavily on the appendix. Essential information, including the LLM training objective, baseline construction, and reward-collapse results, should be explained alongside the main results. Deferring proofs is reasonable, but readers should be able to understand what is trained, which constraints are applied, and what the metrics establish from the main text.

  • The conclusion invokes “realistic scenarios,” but the experiments do not support this claim. A small grid world and a constructed arithmetic task can illustrate the framework, but neither demonstrates practical effectiveness or reliable calibration of the required bounds. These limitations extend beyond implementation efficiency and should be reflected in the conclusion.

Questions for authors
  1. How were the suboptimality bounds chosen in each experiment? Was the proposed calibration procedure actually used, and if so, what external quality signal and stopping criterion were used?

  2. What baseline occupancy was used in the grid-world experiment? Were the additional coverage penalty or pairwise constraints applied in any of the reported experiments?

  3. Given that a constant reward can achieve a perfect tabular metric, what additional evidence shows that the rewards learned in the main experiments are useful? Can the authors report whether policies trained using these rewards perform the intended task well?

  4. How were MaxEnt’s hyperparameters selected, and were tuning or convergence checks performed to investigate its nearly flat performance? Please also report the ranking-baseline results for Case 1.

4 comments

We thank the reviewer for the deatiled comments on our work. We clarify each concern below and will incorporate these points into the revised manuscript.

 

  • [W1] It is unclear what the guarantees add beyond existing feasible-reward-set methods.

    As described in the manuscript, the presented intersection-based feasible reward set under multiple suboptimal demonstrators (Eqs. 11–13) is not itself new, and we do not claim it as our novelty. Our contribution is the set of theoretical guarantees for this intersection without an optimal demonstrator: the feasible set shrinks monotonically as demonstrations are added (Theorem 3.4), every retained reward makes the unobserved optimal demonstrator nearly optimal when a mixture of the demonstrators is close to optimal (Theorem 3.5), and finitely many suboptimal demonstrators suffice to drive this gap below any prescribed level (Theorem G.1). These properties are generally unavailable to MLE-based methods with demonstrator-specific temperatures (Proposition 3.6). To our knowledge, these guarantees have not been clearly investigated in the literature.

 

  • [W1] The relation to Lazzati et al. (2025) should be discussed.

    We thank the reviewer for introducing this related paper (Lazzati et al., 2025). They also use a suboptimality budget, for a single expert whose suboptimality lies in a known interval [L,U][L,U], and their results replace that binary constraint by a compatibility score and its sample complexity. We instead intersect upper bounds ϵk\epsilon^k from several suboptimal demonstrators, with no optimal demonstrator, and Theorems 3.4, 3.5, and G.1 control the gap to the feasible set of an unobserved optimal demonstrator. We will discuss this distinction explicitly in the revised manuscript.

 

  • [W1] The claim of a new avenue beyond likelihood-based IRL is too broad.

    We will revise and narrow this sentence in conclusion: Proposition 3.6 shows an advantage over likelihood-based IRL in the specific setting with no optimal demonstrator and many suboptimal demonstrators of similar quality.

 

  • [W2] Valid suboptimality bounds are difficult to obtain, and the calibration heuristic is not shown to be reliable. How were ϵk\epsilon^k chosen?

    We would like to clarify that the true ϵk\epsilon^k cannot be known in practice, so we set ϵk=0.1\epsilon^k=0.1 as an arbitrary constant matched to the reward scale. As discussed in the manuscript, the proposed framework works under misspecified suboptimality bounds, even when the ground-truth reward does not lie in the feasible set. The effect of such a misspecified suboptimality bound is already explained in detail in Appendix O.2 (Table 1).

 

  • [W3] A constant reward can score perfectly on the tabular metric. The main plots therefore need additional checks to distinguish meaningful improvement from reward degeneracy.

    Fixing ϵk=0.1\epsilon^k=0.1 below the true mean suboptimality overly tightens the feasible-set constraints and can produce a more constant (degenerate) reward. The constant reward reported in Table 1 (Δ=−0.3\Delta=-0.3) is that misspecification case.

    In our experiments, the learned reward is not constant, because we used the techniques specified in Section 3.2 to avoid degenerate rewards. To show this, the table below reports the mean population variance of the learned reward in Case 2 for K∈{1,5,10,20}K\in\{1,5,10,20\}. The variance stays near 0.100.10 from K=1K=1 to K=20K=20 and does not collapse to zero, so the recovered rewards do not converge to a constant reward, and SubOpt(r^,de⋆)\mathrm{SubOpt}(\hat r,d_e^\star) is not a vacuous metric. We will include the full data in the appendix.

    KK115510102020
    Variance of rewards0.095±0.0080.095\pm 0.0080.105±0.0100.105\pm 0.0100.108±0.0100.108\pm 0.0100.109±0.0110.109\pm 0.011

 

  • [W3] No trained policy is evaluated.

    This is because our goal is to learn a meaningful reward, and our experiment validates the main theoretical analysis, which is stated for the metric SubOpt(r,de)\mathrm{SubOpt}(r, d_e). In the LLM environment, correct and wrong trajectories are clearly distinguished, so the quality of the learned reward can be evaluated directly by how often the optimal trace outscores a trace with one wrong tool name (Eq. 171). Training a policy would mix in an additional policy-optimization error and make the conclusion less clear. We will explicitly discuss this in Section 4.2.

 

  • [W4] The LLM experiment is too narrow to provide strong evidence of usefulness for LLM applications.

    The experiments are designed to illustrate the theoretical claim that the intersection-based method outperforms MLE-based methods when the demonstrators are mostly suboptimal (Proposition 3.6). We do not claim a practical, deployable IRL algorithm for LLM with SOTA efficiency. We will make this scope clear in the revised manuscript.

  • [W5] The paper does not show what happens when fewer demonstrations are available from each demonstrator. The cited occupancy-estimation bounds do not resolve these questions.

    Could the reviewer elaborate on this weakness? The effect of using fewer trajectories from each demonstrator is already stated in the manuscript. Lines 349-359 cite the finite sample-complexity bounds for the occupancy estimate d^ek\hat d_e^k (Kim et al., 2024). Lines 360–365 then give the exact consequence: for example, if ∥d^ek−dek∥1≤ηk\|\hat d_e^k-d_e^k\|_1\le\eta_k holds with high probability, replacing ϵk\epsilon^k with ϵk+ηk\epsilon^k+\eta_k preserves compatibility with the population constraint.

 

  • [W6] MaxEnt stays nearly flat, and the authors do not test whether this persists under appropriate tuning. Its inverse temperature is fixed across demonstrators.

    We would like to clarify that the common inverse temperature βk=1.0\beta^k=1.0 is used only where the demonstrators have the same quality. This is Case 2, where every demonstrator knows exactly one portal, and it is the LLM experiment, where every demonstrator errs on all but one tool. In Case 1, where the demonstrators have heterogeneous expertise, the MaxEnt baseline uses a demonstrator-specific inverse temperature: a demonstrator that knows kk portals is given βk=k\beta^k=k. Even with these demonstrator-specific temperatures, MaxEnt remains substantially suboptimal. We thank the reviewer for pointing this out, and we will revise lines 419–420.

 

  • [W6] Relevant newer methods are cited but not evaluated. The Case 1 ranking baseline is omitted.

    Following the reviewer's suggestion, we will include the ranking baseline (T-REX) curve in the Case 1. We observed that the ranking baseline is almost flat and performs even worse than MaxEnt.

 

  • [W7] Too much essential material is only in the appendix.

    We thank the reviewer for pointing this out. We reduced the surrounding context and moved the details to the appendix because of space limitations. However, we agree that a reader should still be able to see the training objective, the baseline, and what the metrics establish without consulting the appendix. We will move these to the main text.

 

  • [W8] The conclusion says “realistic scenarios,” which the experiments do not support.

    The phrase refers to the problem setting that the theory addresses: imperfect and heterogeneous demonstrators, with no optimal demonstrator. The grid world and the arithmetic task are constructed instances of that setting, and the following sentence already states that the paper focuses on the theoretical advantages of the framework. We will revise the wording from “realistic scenarios” to “settings with no optimal demonstrator,” so it is not read as a claim of practical deployment.

  • [Q1] How were the suboptimality bounds chosen in each experiment? Was the proposed calibration procedure actually used, and if so, what external quality signal and stopping criterion were used?

    In the grid-world experiment, the bound is chosen arbitrarily and matched to the reward scale to reflect the practical scenario, as we responded in [W2]. For the LLM experiment, we used the proposed calibration: we set the bounds that the penalty term goes to zero and verified the feasibility of the trained reward.

 

  • [Q2] What baseline occupancy was used in the grid-world experiment? Were the additional coverage penalty or pairwise constraints applied in any of the reported experiments?

    The reported runs solve Eq. 17. In the grid world, the baseline occupancy is the uniform policy, which is the default in Section 3.2. The optional coverage penalty and pairwise constraints in Appendices L and M were not used.

 

  • [Q3] Given that a constant reward can achieve a perfect tabular metric, what additional evidence shows that the rewards learned in the main experiments are useful? Can the authors report whether policies trained using these rewards perform the intended task well?

    The trained policy performs the intended task well because the obtained reward is non-degenerate (as shown in our response) and the true optimal policy is nearly optimal. In the grid-world environment, for each recovered reward r^\widehat r, we compute an optimal policy π^∈arg⁡max⁡π⟨r^,dπ⟩\widehat\pi \in \arg\max_\pi \langle\widehat r,d^\pi\rangle. We then evaluate the probability of reaching the goal within 400 steps under the uniform initial-state distribution.

    The table reports goal-reaching rates (%) as the mean ±\pm sample standard deviation over 20 maps.

    Case 1:

    MethodK=1K = 1K=5K = 5K=10K = 10K=20K = 20
    Proposed25.71±23.0425.71 \pm 23.0451.48±27.0451.48 \pm 27.0461.45±20.2261.45 \pm 20.2282.16±12.6882.16 \pm 12.68
    MaxEnt27.59±38.8727.59 \pm 38.8748.27±41.8448.27 \pm 41.8449.60±43.6549.60 \pm 43.6591.88±25.0891.88 \pm 25.08

    Case 2:

    MethodK=1K = 1K=5K = 5K=10K = 10K=20K = 20
    Proposed23.52±22.9623.52 \pm 22.9638.50±29.6938.50 \pm 29.6941.97±23.5041.97 \pm 23.5045.24±26.8545.24 \pm 26.85
    MaxEnt22.54±35.8222.54 \pm 35.8216.90±26.3816.90 \pm 26.3819.01±30.7019.01 \pm 30.7016.84±27.1016.84 \pm 27.10

    In Case 1, our method shows consistent improvement as KK increases. For MaxEnt, the performance increases abruptly as KK approaches 20, showing that MaxEnt requires near-optimal demonstrators. In Case 2, our method shows consistent improvement as KK increases, while MaxEnt does not show a performance improvement. These results coincide with those of our original suboptimality metric in Figure 2, wihch shows that our suboptimality metric is highly correlated with the performance of the optimized policy. We will add this result to the revised manuscript.

    In the LLM experiment, instead of SubOpt⁡(r^,de⋆)\operatorname{SubOpt}(\widehat r,d_e^\star), we report the win rate, which measures whether the obtained reward can distinguish the optimal trajectory from the erroneous trajectory. Therefore, the quality of the reward is reported directly, without suffering from the scale issue (degeneracy). Nevertheless, we will also add the LLM experimental results to the final manuscript, although we cannot present them now because of time and computational limits.

 

  • [Q4] How were MaxEnt's hyperparameters selected, and were tuning or convergence checks performed to investigate its nearly flat performance? Please also report the ranking-baseline results for Case 1.

    The convergence of the reward is checked during the optimization. Tuning is not required in Case 2. In Case 1, we used a demonstrator-specific inverse temperature, as we responded in [W6]. We will include the ranking baseline (T-REX) curve for Case 1, but we observed that the ranking baseline is almost flat and performs even worse than MaxEnt.

 

Thank you again for the detailed comments. If you have any further questions or would like additional clarification, we would be glad to respond. Otherwise, we would be grateful if you might reconsider your score in light of the points above.

Thank you again for your thoughtful review. With the discussion period closing soon, we would appreciate any follow-up thoughts on our rebuttal, particularly whether our responses address your concerns and which points, if any, remain unresolved. We would be happy to provide further clarification during the remaining discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Results

We thank all reviewers for their constructive feedback, which has strengthened both the theoretical scope and the empirical evidence of our work. We summarize the main revisions and additional results below.

 

1. Extended theoretical guarantees (Reviewer MQUY): We added a concentration-based condition guaranteeing strict shrinkage even when all demonstrators have full state–action support, with the original theorem recovered as a special case (Appendix E, Theorem E.1). We also moved the necessary-and-sufficient characterization of strict shrinkage into the main text (Lemma 3.4) and clarified our optimality guarantees beyond convex-mixture approximation (lines 264–279).

2. Additional validation of reward usefulness (Reviewers a2P1 and JBGC): We verified that the learned rewards do not collapse to constants and evaluated policies optimized under these rewards (Appendix R). Goal-reaching performance improves as more demonstrators are incorporated in both grid-world settings, supporting meaningful policy improvement and corroborating the trends in our original reward-level evaluation.

3. Forthcoming experiments: As promised in our responses, we will report downstream LLM performance after policy optimization using the recovered rewards and include comparisons with T-REX in settings with demonstrators of differing expertise.

 

We hope that these revisions and additional results, together with our detailed responses to the individual reviews, address the reviewers' concerns. We would be grateful if the reviewers could take these updates into account in their final assessments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.