- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3(+1)
Read review
Paper summary
This paper introduces an efficient multi-task inverse RL (IRL) method based on low-rank policy representations. The approach factorises the policy logits with a set of basis functions that are shared across tasks, enabling shared representation learning across different states. Both the basis functions and corresponding coefficients are learned by maximum likelihood estimation on the trajectory data. Following existing theory from single-task IRL, the method then recovers reward functions from the trained policies with fitted Q-iteration (FQI). Crucially, the factorisation enables to approximately recover rewards with only one FQI solve per basis function, rather than one FQI per task, increasing scalability. A theoretical analysis establishes finite-sample guarantees and an experimental comparison demonstrates effective reward recovery and favourable scaling of the proposed approach.
Critical strengths
- The idea of factorising policy logits with shared basis functions is well-motivated and natural. This allows sharing of representations across tasks in the multi-task IRL setting, which is demonstrated experimentally.
- The approximation of reward recovery by only performing steps of FQI is interesting and enables scaling independent of the number of tasks.
- An extensive theoretical analysis of the approach is provided, which establishes finite-sample guarantees (although I have unfortunately been unable to verify the details of the proofs).
- The experimental results demonstrate improved reward recovery with lower computational cost than the baseline methods.
Critical weaknesses
- The paper makes several strong assumptions in the development of its method. Assumption (1) requires all true reward functions to satisfy the same known normalisation under a reference policy . While any reward can be normalised accordingly in the source environment, this generally depends on the environment dynamics and may not preserve optimal behaviour in the target environment. A discussion of this assumption and cases where it holds in practice would strengthen the paper. The experiments also do not investigate how the method performs when this assumption is violated, as the ground-truth rewards are deliberately normalised. A robustness study for a violation of this assumption would be appreciated.
- The theoretical analysis relies on known dynamics, so has limited applicability in practice. Moreover, the primary motivation of the proposed approach is to improve scalability by only computing the FQI iterations for the basis functions. However, the entire theoretical analysis only applies for the exact plug-in reward, which scales linearly in the number of tasks. This seems to be a significant mismatch between the theory and the motivation and practical implementation of the method. The empirical comparison in Appendix B.10 is appreciated, but does not address the theoretical concerns.
- Some important experimental results are delegated to the appendix. In particular, I think showing the impact of the rank and number of tasks on reward recovery is particularly important, since this directly measures the usefulness of the method (e.g. if maintaining acceptable recovery performance required increasing rank similarly to the number of tasks, this would significantly limit the scaling claims). I encourage the authors to include the promising results of Table 2 and Figure 8 in the main text. Furthermore, while these results are encouraging, they do not establish how well the method scales with heterogeneous task populations. An experiment investigating how the required rank (and hence computational cost) scale with increasing task diversity (e.g. by increasing the number of underlying reward features) would strengthen the paper.
- Finally, I think that the exposition could be significantly improved. As a reader not overly familiar with some of the prior works, I struggled to follow the paper in several sections. I would encourage the authors to separate the currently entangled method section into separate background/problem statement and method sections. The transition from policy learning to reward recovery required substantial effort to follow: Assumption 1 is never motivated (nor is explained where and come from), and Lemma 1 and Equation (3) are simply stated without much explanation. The formulation also leaves the cost of inversion implicit: as stated in Equation (3), it seems like we could simply compute the matrix inverse once, whereas this in practice requires one FQI iterations per basis or task. It could potentially be helpful to explain these ideas in the context of single-task IRL (referring to van der Laan for details) and then move on to the setting considered in the paper.
Questions for authors
- When is Assumption (1) justified in practice, especially when transferring to target environments with different dynamics?
- Can you provide a robustness study of the performance of the method if Assumption (1) is not met?
- How does reward recovery and the required number of basis functions scale with an increasing number of heterogeneous tasks?
4 comments
Part 1
Thank you for the detailed review. The revised paper now includes target-regret and policy-KL bounds for omitting log-sum-exp and a finite-sample theorem for fitted Algorithm 1 with sampled source transitions (Section 4.4, Theorem 2; proofs in Appendices A.9–A.10). The direct transfer comparison and normalization-misspecification study are in General Response Parts 3 and 5, Tables A and B; we will restructure the manuscript exposition as suggested.
The theoretical contribution connects two steps. First, the coverage constant characterizes when pooled observations identify the shared logit structure despite unobserved task–state blocks; Example 6 in Appendix A.4.1 illustrates recovery through information supplied by other tasks. Second, Theorem 1 propagates estimation error through reward recovery to target-value, regret, and policy-KL guarantees under changed dynamics.
Weakness 1: Assumption 1 and transfer
Assumption 1 is a condition on the true reward. Under MaxEnt behavior, source demonstrations identify rewards only up to potential shaping. As you note, normalizing an arbitrary reward using source dynamics can change optimal behavior under target dynamics. When the invariant true reward itself satisfies
however, the condition remains valid across environments because it involves no dynamics. We will make this distinction explicit.
This is a standard form of identification condition, encompassing a known reference-action reward and the normalization used by GenPQR (General Response Part 5). Its applicability requires justification from task semantics or external information.
The policy-estimation stage is separate from reward identification, making combinations with other identification strategies, such as identification from multiple environments (Rolland et al., 2022), a promising extension. Transfer guarantees for those combinations would require additional analysis.
Our transfer evaluation already uses RecSim's generating reward, which violates the uniform- normalization used by the method. Part 5, Table B extends this to larger violations and an alternative normalization. The generating reward or alternative normalization changes LowRank's score by at most 2.6 points relative to the correctly specified case; LowRank leads the listed baselines through twice the generating reward's deviation.
At four times the deviation, exact recovery of the assumed-normalized reward itself scores below random, and LowRank also deteriorates substantially. We will discuss this identification limitation alongside the observed tolerance to moderate violations.
Weakness 2: known dynamics and the approximate estimator
We now establish a finite-sample target-regret guarantee for Algorithm 1 using sampled source transitions without knowing the source kernel. In the tabular setting with , under the stated global-optimization, regression-realizability, and pooled-coverage conditions, the guarantee retains Theorem 1’s statistical radii and adds explicit terms for normalizer omission, finite fitted iteration, and source-transition estimation. The latter two contributions vanish as the number of updates and samples increase, respectively (General Response Part 2(iii)).
We also bound the normalizer-omission reward discrepancy and its linear contribution to target regret separately (General Response Part 1(i)–(ii)). The matched calibration in Part 2 measures this penalty using exact source recovery in both arms.
Part 3, Table A separately evaluates the neural fitted-iteration implementation. At the reported endpoints, fitted iteration and exact inversion differ in return by at most 2.6 percentage points for each estimator. The exact plug-in leads the approximation by approximately 3 percentage points with fitted iteration and 4–7 with exact inversion, quantifying the accuracy–computation trade-off from using rather than reward-recovery solves.
As the rebuttal discussion period draws to a close, we wanted to follow up on the revisions prompted by your review. Your suggestions led to substantial additional work that we believe has strengthened the paper, and we thank you again for them. Substantive additions in the updated manuscript are highlighted in blue.
For Assumption 1, we now motivate the known-reference-action case and clarify that the assumption supplies information about the invariant true reward. Appendix B.6 (Table 3) tests normalization misspecification, including evaluation under the original generating reward and larger violations.
To address the theory–implementation mismatch, Section 4.4 now provides normalizer-omission bounds and a finite-sample guarantee for Algorithm 1's fitted basis recovery from sampled source transitions without knowing the source kernel, under explicit tabular fitting conditions. Table 1 in the main text directly compares exact and approximate recovery using both exact inversion and fitted iteration, reporting target return and soft regret.
We also evaluated reward recovery and runtime as the population grows to 128 tasks at fixed rank (Figure 5; Table 4). Fitted-rank sensitivity and expert spectra are now reported in the main text (Figure 3; Section 5.3), complemented by the task-diversity sweep in the General Response examining when additional bases help.
For exposition, Section 3.2 now introduces single-task reward identification and explains the recovery map before the multi-task method. Section 3.4 clarifies the distinction between reusing a known tabular operator and running task-specific or shared-basis fitted solves.
We would greatly appreciate hearing whether these changes address your concerns and change your overall assessment of the paper, or whether a specific issue remains unresolved.
Many thanks for the rebuttal. The finite-sample guarantee in particular is useful to reduce the gap between theory and practical implementation, and the revised paper makes Assumption 1 clearer. This has resolved most of my concerns, and I have hence increased my score to 3. I retain some reservations about scalability for populations with substantially higher effective policy rank.
Part 2
Weakness 3 / Question 3: task diversity
Empirically, the tested populations retain strong approximate low-rank policy structure as reward diversity increases. Expanding the reward-weight span from four to eight dimensions increases the expert rank needed to capture 99% of anchored-logit energy only from four to four–five. This concentration is not guaranteed by the reward construction: the nonlinear MaxEnt optimality map need not preserve reward rank.
More broadly, low-dimensional shared structure has a substantial empirical track record in real recommender systems, including latent-factor models developed for the Netflix Prize (Koren, Bell and Volinsky, 2009). Related low-rank preference models are also established in discrete-choice modeling (Kallus and Udell, 2020). These precedents motivate exploiting shared structure in heterogeneous populations. Our assumption concerns sequential policy logits and is not implied by low-dimensional preferences; we therefore assess it directly through the expert spectra and diversity sweep.
We added the requested diversity sweep to measure what exploiting this structure buys in recovery and computation (General Response Part 6, Table C). Four bases consistently outperform two in the richer populations; additional bases yield smaller, regularization-dependent gains. Thus the results support exploiting approximate low-rank structure in these populations, while identifying the associated accuracy–cost trade-off. Populations with nearly N dominant policy directions remain an untested regime.
We keep the same four-topic RecSim MDP and vary the reward-weight span:
Each task's reward is normalized to a common RMS, and expert temperature is fixed. All ranks in this experiment use SVD initialization and identical data splits within each population/replicate. We select c∈0,0.05,0.2,0.8 by held-out policy NLL separately for each (d,k,replicate) and evaluate fixed-c comparisons, including the paper's c=0.2.
This complements Table 2's population-size sweep and Figure 8's rank ablation on a fixed population. The coverage differences and effective-rank limitation are stated alongside Table C.
Weakness 4: exposition and operator cost
We will split Section 3 into single-task background and the multi-task method, addressing the three points you identify:
Identification. Explain the reward ambiguity and motivate Assumption 1 through a known reference-action reward or other information justifying μ,g.
Reward recovery. Derive Lemma 1 from the soft Bellman equation logπ=r+γPV−V. Averaging over μ and applying Assumption 1 gives a linear equation for V, yielding Equation (3).
Computational cost. With an explicit finite transition matrix, its factorization can be computed once and reused across right-hand sides. Our implementation instead estimates the operator's action from sampled transitions using fitted fixed-point solves. The saving is reducing these from N task-specific inputs to k shared basis inputs. We will state this distinction next to Equation (3).
Questions 1–2 are addressed under Weakness 1; Question 3 is addressed under Weakness 3.
We hope the new approximation bound, transfer comparison, normalization-misspecification study, and planned exposition changes sufficiently address the concerns underlying your recommendation. If so, would you consider raising your overall score?
Reference: Rolland et al. (2022), Identifiability and generalizability from multiple experts in inverse reinforcement learning.