acceptodds
Under review as a conference paper at ICLR 2027

Scalable Multi-Task Inverse Reinforcement Learning

Abstract

By learning transferable rewards, inverse reinforcement learning (IRL) enables counterfactual evaluation of agents under modified environments. Such transfer places strict requirements on coverage since target environments affect agents' state occupancy. We propose a multi-task IRL method that pools data across multiple agents with different rewards in the same environment under a low-rank assumption. In addition to alleviating coverage requirements, so each task need not visit every state as long as others do, the method offers scalable evaluation of multiple tasks under new environments as computationally intensive planning scales with rank rather than the number of tasks. We provide finite sample guarantees on reward recovery and on policy learning in new environments. Experiments show our method is robust to limited coverage, recovers rewards on and off of each task's support, transfers to target environments at lower regret than baselines, with its computational advantage over per-task methods widening as tasks grow.

Primary area:
reinforcement learning
Keywords:
inverse reinforcement learning, multi-task inverse reinforcement learning, reward learning, low-rank structure, transfer learning, maximum entropy reinforcement learning, offline reinforcement learning, matrix completion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Sg6KOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3(+1)
Read review
Paper summary

This paper introduces an efficient multi-task inverse RL (IRL) method based on low-rank policy representations. The approach factorises the policy logits with a set of kk basis functions that are shared across tasks, enabling shared representation learning across different states. Both the basis functions and corresponding coefficients are learned by maximum likelihood estimation on the trajectory data. Following existing theory from single-task IRL, the method then recovers reward functions from the trained policies with fitted Q-iteration (FQI). Crucially, the factorisation enables to approximately recover rewards with only one FQI solve per basis function, rather than one FQI per task, increasing scalability. A theoretical analysis establishes finite-sample guarantees and an experimental comparison demonstrates effective reward recovery and favourable scaling of the proposed approach.

Critical strengths
  • The idea of factorising policy logits with shared basis functions is well-motivated and natural. This allows sharing of representations across tasks in the multi-task IRL setting, which is demonstrated experimentally.
  • The approximation of reward recovery by only performing kk steps of FQI is interesting and enables scaling independent of the number of tasks.
  • An extensive theoretical analysis of the approach is provided, which establishes finite-sample guarantees (although I have unfortunately been unable to verify the details of the proofs).
  • The experimental results demonstrate improved reward recovery with lower computational cost than the baseline methods.
Critical weaknesses
  • The paper makes several strong assumptions in the development of its method. Assumption (1) requires all true reward functions to satisfy the same known normalisation under a reference policy μ\mu. While any reward can be normalised accordingly in the source environment, this generally depends on the environment dynamics and may not preserve optimal behaviour in the target environment. A discussion of this assumption and cases where it holds in practice would strengthen the paper. The experiments also do not investigate how the method performs when this assumption is violated, as the ground-truth rewards are deliberately normalised. A robustness study for a violation of this assumption would be appreciated.
  • The theoretical analysis relies on known dynamics, so has limited applicability in practice. Moreover, the primary motivation of the proposed approach is to improve scalability by only computing the FQI iterations for the basis functions. However, the entire theoretical analysis only applies for the exact plug-in reward, which scales linearly in the number of tasks. This seems to be a significant mismatch between the theory and the motivation and practical implementation of the method. The empirical comparison in Appendix B.10 is appreciated, but does not address the theoretical concerns.
  • Some important experimental results are delegated to the appendix. In particular, I think showing the impact of the rank and number of tasks on reward recovery is particularly important, since this directly measures the usefulness of the method (e.g. if maintaining acceptable recovery performance required increasing rank similarly to the number of tasks, this would significantly limit the scaling claims). I encourage the authors to include the promising results of Table 2 and Figure 8 in the main text. Furthermore, while these results are encouraging, they do not establish how well the method scales with heterogeneous task populations. An experiment investigating how the required rank (and hence computational cost) scale with increasing task diversity (e.g. by increasing the number of underlying reward features) would strengthen the paper.
  • Finally, I think that the exposition could be significantly improved. As a reader not overly familiar with some of the prior works, I struggled to follow the paper in several sections. I would encourage the authors to separate the currently entangled method section into separate background/problem statement and method sections. The transition from policy learning to reward recovery required substantial effort to follow: Assumption 1 is never motivated (nor is explained where μ\mu and gg come from), and Lemma 1 and Equation (3) are simply stated without much explanation. The formulation also leaves the cost of inversion implicit: as stated in Equation (3), it seems like we could simply compute the matrix inverse once, whereas this in practice requires one FQI iterations per basis or task. It could potentially be helpful to explain these ideas in the context of single-task IRL (referring to van der Laan for details) and then move on to the setting considered in the paper.
Questions for authors
  • When is Assumption (1) justified in practice, especially when transferring to target environments with different dynamics?
  • Can you provide a robustness study of the performance of the method if Assumption (1) is not met?
  • How does reward recovery and the required number of basis functions scale with an increasing number of heterogeneous tasks?
4 comments

Weakness 3 / Question 3: task diversity

Empirically, the tested populations retain strong approximate low-rank policy structure as reward diversity increases. Expanding the reward-weight span from four to eight dimensions increases the expert rank needed to capture 99% of anchored-logit energy only from four to four–five. This concentration is not guaranteed by the reward construction: the nonlinear MaxEnt optimality map need not preserve reward rank.

More broadly, low-dimensional shared structure has a substantial empirical track record in real recommender systems, including latent-factor models developed for the Netflix Prize (Koren, Bell and Volinsky, 2009). Related low-rank preference models are also established in discrete-choice modeling (Kallus and Udell, 2020). These precedents motivate exploiting shared structure in heterogeneous populations. Our assumption concerns sequential policy logits and is not implied by low-dimensional preferences; we therefore assess it directly through the expert spectra and diversity sweep.

We added the requested diversity sweep to measure what exploiting this structure buys in recovery and computation (General Response Part 6, Table C). Four bases consistently outperform two in the richer populations; additional bases yield smaller, regularization-dependent gains. Thus the results support exploiting approximate low-rank structure in these populations, while identifying the associated accuracy–cost trade-off. Populations with nearly NN dominant policy directions remain an untested regime.

We keep the same four-topic RecSim MDP and vary the reward-weight span:

  • d=2d=2: two topic-pair directions, e.g. wi=ai[1,1,0,0]+bi[0,0,1,1]w_i=a_i[1,1,0,0]+b_i[0,0,1,1], retaining positive reward on all topics.
  • d=4d=4: four independently weighted topic features.
  • d=8d=8: the same underlying four-topic weights plus four consumption-modulated features, ϕt(s,a)ct(s)/bt\phi_t(s,a)c_t(s)/b_t. Their coefficients are wcnt,it=wbase,itρitw_{\mathrm{cnt},it}=w_{\mathrm{base},it}\rho_{it}, with ρit∼U(−0.9,0.9)\rho_{it}\sim U(-0.9,0.9) independently across tasks and topics.

Each task's reward is normalized to a common RMS, and expert temperature is fixed. All ranks in this experiment use SVD initialization and identical data splits within each population/replicate. We select c∈0,0.05,0.2,0.8c\in\\{0,0.05,0.2,0.8\\} by held-out policy NLL separately for each (d,k,replicate)(d,k,\text{replicate}) and evaluate fixed-cc comparisons, including the paper's c=0.2c=0.2.

This complements Table 2's population-size sweep and Figure 8's rank ablation on a fixed population. The coverage differences and effective-rank limitation are stated alongside Table C.

Weakness 4: exposition and operator cost

We will split Section 3 into single-task background and the multi-task method, addressing the three points you identify:

  1. Identification. Explain the reward ambiguity and motivate Assumption 1 through a known reference-action reward or other information justifying μ,g\mu,g.

  2. Reward recovery. Derive Lemma 1 from the soft Bellman equation log⁡π=r+γPV−V\log\pi=r+\gamma PV-V. Averaging over μ\mu and applying Assumption 1 gives a linear equation for VV, yielding Equation (3).

  3. Computational cost. With an explicit finite transition matrix, its factorization can be computed once and reused across right-hand sides. Our implementation instead estimates the operator's action from sampled transitions using fitted fixed-point solves. The saving is reducing these from NN task-specific inputs to kk shared basis inputs. We will state this distinction next to Equation (3).

Questions 1–2 are addressed under Weakness 1; Question 3 is addressed under Weakness 3.

We hope the new approximation bound, transfer comparison, normalization-misspecification study, and planned exposition changes sufficiently address the concerns underlying your recommendation. If so, would you consider raising your overall score?

Reference: Rolland et al. (2022), Identifiability and generalizability from multiple experts in inverse reinforcement learning.

Thank you for the detailed review. The revised paper now includes target-regret and policy-KL bounds for omitting log-sum-exp and a finite-sample theorem for fitted Algorithm 1 with sampled source transitions (Section 4.4, Theorem 2; proofs in Appendices A.9–A.10). The direct transfer comparison and normalization-misspecification study are in General Response Parts 3 and 5, Tables A and B; we will restructure the manuscript exposition as suggested.

The theoretical contribution connects two steps. First, the coverage constant c0c_0 characterizes when pooled observations identify the shared logit structure despite unobserved task–state blocks; Example 6 in Appendix A.4.1 illustrates recovery through information supplied by other tasks. Second, Theorem 1 propagates estimation error through reward recovery to target-value, regret, and policy-KL guarantees under changed dynamics.

Weakness 1: Assumption 1 and transfer

Assumption 1 is a condition on the true reward. Under MaxEnt behavior, source demonstrations identify rewards only up to potential shaping. As you note, normalizing an arbitrary reward using source dynamics can change optimal behavior under target dynamics. When the invariant true reward itself satisfies

∑aμ(a∣s)ri(s,a)=g(s),\sum_a\mu(a\mid s)r_i(s,a)=g(s),

however, the condition remains valid across environments because it involves no dynamics. We will make this distinction explicit.

This is a standard form of identification condition, encompassing a known reference-action reward and the normalization used by GenPQR (General Response Part 5). Its applicability requires justification from task semantics or external information.

The policy-estimation stage is separate from reward identification, making combinations with other identification strategies, such as identification from multiple environments (Rolland et al., 2022), a promising extension. Transfer guarantees for those combinations would require additional analysis.

Our transfer evaluation already uses RecSim's generating reward, which violates the uniform-μ\mu normalization used by the method. Part 5, Table B extends this to larger violations and an alternative normalization. The generating reward or alternative normalization changes LowRank's score by at most 2.6 points relative to the correctly specified case; LowRank leads the listed baselines through twice the generating reward's deviation.

At four times the deviation, exact recovery of the assumed-normalized reward itself scores below random, and LowRank also deteriorates substantially. We will discuss this identification limitation alongside the observed tolerance to moderate violations.

Weakness 2: known dynamics and the approximate estimator

We now establish a finite-sample target-regret guarantee for Algorithm 1 using sampled source transitions without knowing the source kernel. In the tabular setting with g=0g=0, under the stated global-optimization, regression-realizability, and pooled-coverage conditions, the guarantee retains Theorem 1’s statistical radii and adds explicit terms for normalizer omission, finite fitted iteration, and source-transition estimation. The latter two contributions vanish as the number of updates and samples increase, respectively (General Response Part 2(iii)).

We also bound the normalizer-omission reward discrepancy and its linear contribution to target regret separately (General Response Part 1(i)–(ii)). The matched calibration in Part 2 measures this penalty using exact source recovery in both arms.

Part 3, Table A separately evaluates the neural fitted-iteration implementation. At the reported endpoints, fitted iteration and exact inversion differ in return by at most 2.6 percentage points for each estimator. The exact plug-in leads the approximation by approximately 3 percentage points with fitted iteration and 4–7 with exact inversion, quantifying the accuracy–computation trade-off from using kk rather than NN reward-recovery solves.

As the rebuttal discussion period draws to a close, we wanted to follow up on the revisions prompted by your review. Your suggestions led to substantial additional work that we believe has strengthened the paper, and we thank you again for them. Substantive additions in the updated manuscript are highlighted in blue.

For Assumption 1, we now motivate the known-reference-action case and clarify that the assumption supplies information about the invariant true reward. Appendix B.6 (Table 3) tests normalization misspecification, including evaluation under the original generating reward and larger violations.

To address the theory–implementation mismatch, Section 4.4 now provides normalizer-omission bounds and a finite-sample guarantee for Algorithm 1's fitted basis recovery from sampled source transitions without knowing the source kernel, under explicit tabular fitting conditions. Table 1 in the main text directly compares exact and approximate recovery using both exact inversion and fitted iteration, reporting target return and soft regret.

We also evaluated reward recovery and runtime as the population grows to 128 tasks at fixed rank (Figure 5; Table 4). Fitted-rank sensitivity and expert spectra are now reported in the main text (Figure 3; Section 5.3), complemented by the task-diversity sweep in the General Response examining when additional bases help.

For exposition, Section 3.2 now introduces single-task reward identification and explains the recovery map before the multi-task method. Section 3.4 clarifies the distinction between reusing a known tabular operator and running task-specific or shared-basis fitted solves.

We would greatly appreciate hearing whether these changes address your concerns and change your overall assessment of the paper, or whether a specific issue remains unresolved.

Many thanks for the rebuttal. The finite-sample guarantee in particular is useful to reduce the gap between theory and practical implementation, and the revised paper makes Assumption 1 clearer. This has resolved most of my concerns, and I have hence increased my score to 3. I retain some reservations about scalability for populations with substantially higher effective policy rank.

Reviewer_QJNdOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

This paper proposes a multi-task IRL method using a low-rank approximation of task policies, thus avoiding scaling inverse reward solutions with the number of tasks, but rather keeping it limited to the assumed low rank. The paper provides theoretical guarantees for the proposed method, and with mild approximations, evaluates it empirically on a few different environments, demonstrating its efficiency and efficacy.

Critical strengths

The methodological contributions made in the paper seem sound and meaningful. Particularly, rank-constraining the policy and not the reward directly is a compelling choice. The text explicitly explains the construction of the method, clearly states assumptions and approximations and provides theoretical guarantees.

In addition, the experimentation explicitly stressing the assumptions of the method (removing samples from particular latent directions rather than randomly; comparing on-support to off-support performance; etc.) strengthens the claims made in the paper, providing evidence to support the assumptions and approximations made in the design of the method.

Critical weaknesses

The central weakness I find in this paper is that the theoretical guarantees support the exact plug-in reward and not the approximate reward, which is central to the computational savings the algorithm provides. I acknowledge that the empirical work in the experiments section shows that this is not a particularly meaningful issue in practice (on the evaluated environments), it would strengthen the paper if it could provide some guarantees or bounds for the error caused by the approximate reward.

Questions for authors
  1. How realistic is the assumption on \mu and g? As they are used to form an assumption on the hidden true rewards, wouldn’t a misspecification of them hurt transfer to different dynamics? Could this be a limitation to implementing the proposed method in more complex environments?

  2. Is the low-rank assumption a limitation on the usability of the proposed algorithm? In the three tested environments, the underlying reward structure is low-rank by design, but as the authors mention, the space of true expert policies could be higher-rank (especially as they are learned with PPO and NN policies (Appendix B.1). Could this be limiting in cases where the learned policies vary greatly in their resulting strategies?

  3. Given that only RecSim displays meaningful state occupancy shifts when dynamics are perturbed, it seems like the claims on transfer between MDPs may be slightly overstated. Can the authors provide additional evidence that LowRank is effective in this regime? At the very least, a characterization of environments where this would be effective could strengthen the case on the matter.

4 comments

Thank you for recognizing the methodological contribution and the experiments testing its mechanism. Your questions led to bounds for the approximation, a finite-sample guarantee for Algorithm 1, and the transfer, normalization-misspecification, and task-diversity experiments in the General Response.

Critical Weakness: guarantees for the approximation

Please see General Response Parts 1–4 for the bounds on the discrepancy in reward estimates and target regret from omitting log-sum-exp, and the finite-sample guarantee for Algorithm 1 using sampled source transitions. The assumptions and proof sketches are collected there; the revised full paper gives the complete proofs of these three bounds.

General Response Part 3, Table A separately measures its effect on transfer. The exact plug-in leads by approximately 3 percentage points with fitted iteration and 4–7 with exact inversion, quantifying the trade-off from using kk rather than NN reward-recovery solves. These empirical results complement the finite-sample target-regret guarantee, which holds under the tabular fitting and pooled-coverage conditions stated in General Response Part 2.

Q1: realism and misspecification of μ,g\mu,g

Assumption 1 supplies information needed to identify rewards from source demonstrations. A known reference-action reward is one example; an arbitrary imposed normalization need not identify the invariant true reward. We will make this distinction explicit and discuss the related identification schemes summarized in General Response Part 5.

The policy-estimation stage is separate from reward identification, making combinations with other identification strategies, such as identification from multiple environments (Rolland et al., 2022), a promising extension. Transfer guarantees for those combinations would require additional analysis.

General Response Part 5, Table B directly tests misspecification. RecSim's generating reward already violates the uniform-μ\mu normalization used by our method. Evaluating under that reward or the alternative normalization changes LowRank's score by at most 2.6 points relative to the correctly specified case, and LowRank leads the listed baselines through twice the generating reward's deviation.

At four times that deviation, transfer deteriorates substantially: even exact recovery of the assumed-normalized reward scores below random. This documents both tolerance to the tested moderate violations and a failure regime. General robustness in more complex environments remains an empirical question.

Q2: limitations of low rank

We agree that sufficiently heterogeneous policies can limit the gains. Low-rank rewards do not imply low-rank log-policies, because the MaxEnt optimality map is nonlinear; the PPO-trained experts also introduce approximation beyond the theoretical setting.

Although the rewards use four underlying features, this does not impose rank four on the expert log-policies: the MaxEnt optimality map is nonlinear. We measured the trained experts' log-policy spectra on 10,000 pooled states. The top four singular directions capture 96.2%, 98.9%, and 99.4% of the energy in FourRooms, Highway, and RecSim, respectively, indicating strong approximate low-rank structure. Figure 8 separately tests sensitivity to the fitted rank. For populations with substantially higher effective rank, the pooling and computational gains can diminish.

Figure 8 further shows that even with k=2k=2, LowRank exceeds the best baseline in every domain.

Q3: evidence and scope of transfer

We agree that the distinction should be explicit: reward recovery is evaluated in three domains, while the counterfactual transfer evidence is from RecSim. The dynamics perturbations we tried in FourRooms and Highway did not produce substantial occupancy shifts.

Within RecSim, Figure 3 shows the advantage growing as occupancy moves off each task's source support. At the strongest shift in General Response Part 3, Table A, LowRank achieves 65.0% of optimum, compared with 39.9% for behavior cloning and 38.1% for IQ-Learn.

The advantage over per-task IRL is most relevant when:

  • Each task leaves important states poorly covered.
  • Other tasks cover those states and span the relevant latent directions.
  • Target dynamics make those states consequential.
  • Rewards remain invariant and the identification assumptions hold.

We will state this characterization and the scope of the empirical transfer evidence explicitly. The theoretical connection is the target-occupancy factor ∥Lμ⊤ρ2π∥2\lVert L_\mu^\top\rho_2^\pi\rVert_2 in Rn(π)R_n(\pi), which weights reward-recovery error by its consequences under target visitation.

You identified the absence of guarantees for the approximation as the central weakness. We hope the bounds for omitting log-sum-exp, the finite-sample guarantee for Algorithm 1, and the direct transfer comparison address that concern. If they do, would you consider raising your overall score?

Reference: Rolland et al. (2022), Identifiability and generalizability from multiple experts in inverse reinforcement learning.

Thank you for the detailed response and additional analysis, both here and in the general responses.

Although my concerns have mostly been resolved, I will keep my score as it is (weak acceptance) pending further discussion with the other reviewers.

We appreciate the quick response and thank you again for the thorough review. We’re happy to answer any further questions while the rebuttal discussion remains open to us.

Reviewer_sD7AOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an offline multi-task IRL method that needs no reward features: it fits rank-k anchored policy logits jointly across tasks and recovers per-task rewards by inverting k shared bases rather than N tasks, with finite-sample guarantees for an exact normalized variant under a pooled-coverage condition and synthetic experiments on reward recovery, RecSim transfer, and runtime.

Critical strengths
  1. The idea makes sense. Sharing rank-k logit bases can make a task's reward be recovered at states only other tasks visited and give large off-support gains over per-task GenPQR.
  2. It contributes a coverage-to-transfer theory. The pooled-coverage condition tolerates unobserved task–state blocks, carries logit error through to target value, regret, and policy KL, and is supported by a targeted-deletion experiment with volume-matched controls.
Critical weaknesses
  1. The guarantees and the efficiency gains seems to come from different estimators. Theorem 1 covers exact normalized recovery with tabular states and known dynamics, whereas the reported experiments use the k-basis shortcut, whose reward bias might not vanish even with perfect logits. Besides Appendix B.10 compares the two only by reward correlation, not by the target regret or policy KL that the transfer claims rely on.
  2. The benefit of low rank itself is not isolated in experiments. There is no baseline shares a representation across tasks without the rank-k limit. Besides, shared reward bases with continuous per-task weights already appear in uncited IRL work like ELIRL & BASIS.
Questions for authors
  1. Is Figure 3's oracle computed from the reward that generated the demonstrations or from its µ-normalized version, and if the latter, can you report results against the original reward?
  2. Can you add a baseline that trains one shared network with task-specific heads but no rank-k limit?
4 comments
AuthorsOpenReview ↗

Response Part 1

Thank you for recognizing the pooled-coverage gains, transfer theory, and targeted-deletion experiments. We added bounds for omitting log-sum-exp and a finite-sample guarantee for Algorithm 1, together with the requested transfer comparison, a shared-network baseline without an explicit rank constraint, and evaluation against the original generating reward.

Critical Weakness 1: exact versus approximate recovery

General Response Parts 1–2 separate three guarantees. For the anchored logits returned by Algorithm 1’s classification step, r^i\widehat r_i and r^iapprox\widehat r_i^{\mathrm{approx}} apply the same population recovery map with and without log-sum-exp normalization, using the true source dynamics. Part 1(i) bounds ∥r^iapprox−r^i∥∞\lVert\widehat r_i^{\mathrm{approx}}-\widehat r_i\rVert_\infty by the log-sum-exp’s variation across states. Part 1(ii) retains the form of Theorem 1’s statistical terms and adds 2∥r^iapprox−r^i∥∞/(1−γ2)2\lVert\widehat r_i^{\mathrm{approx}}-\widehat r_i\rVert_\infty/(1-\gamma_2) to the target-regret bound. Multiplying the regret bound by 1−γ21-\gamma_2 gives the corresponding occupancy-weighted policy-KL guarantee. Appendix A.9 of the revised paper gives the full omission bounds and proofs.

Part 2(iii), stated as Theorem 2 in Section 4.4 of the revised paper with proof in Appendix A.10, covers Algorithm 1’s actual output r~i\widetilde r_i, obtained by running K≥1K\ge1 fitted-Q updates on each of the kk shared logit bases using sampled source transitions and combining the recovered reward bases with the fitted task weights. Under the statistical and fitting conditions stated there—including g=0g=0, known sampling design, global tabular classification with bounded anchored logits, and zero-initialized exact least-squares fitting of representable empirical Bellman updates—the bound adds iteration and transition-estimation contributions that decay as O(γ1K+1)O(\gamma_1^{K+1}) and O(n−1/2)O(n^{-1/2}), respectively, with other parameters fixed. Recovery therefore does not require knowledge of P1\mathsf{P}_1; target dynamics remain known for replanning. The discrepancy from omitting log-sum-exp can persist.

The matched calibration in General Response Part 2 compares the policies under r^i\widehat r_i, while Part 3, Table A evaluates target return and soft regret under the true reward. In RecSim, the kk-basis approximation nearly matches fitted exact plug-in recovery, with a return loss of 2.9–3.4 percentage points of the true-reward optimum; with exact inversion, the paired loss is 4.3–6.8 points. Thus reducing recovery from NN task-specific solves to kk shared basis solves comes with a small measured performance loss, and the theory gives an explicit target-regret guarantee with an additional normalizer-omission penalty. The fitted rows provide empirical evidence for the survival-weighted implementation variant. The full conditions and proof sketches are in General Response Parts 1–4.

AuthorsOpenReview ↗

Response Part 2

Critical Weakness 2: isolating low rank and related work

We added TaskCond, which shares a network across tasks and conditions its logits nonlinearly on a learned task embedding:

logitsi(s)=f(enc(s),ei).\mathrm{logits}_i(s)=f(\mathrm{enc}(s),e_i).

It has no explicit rank-kk constraint. TaskCond matches LowRank's parameter count within 0.5% and uses the same encoder, per-task dimension (d=k=4d=k=4), training protocol, and reward operator, with regularization selected using held-out likelihood.

Across the tested data budgets, LowRank achieves higher off-support reward correlation in all three domains. On RecSim transfer, TaskCond reaches 28% → 19% of optimum with fitted iteration and 34% → 22% with exact inversion, compared with 32% → 19% for random. This comparison supports the benefit of LowRank's bilinear constraint beyond the representation sharing provided by TaskCond.

Thank you for pointing us to ELIRL and BASIS; we will cite both and clarify the overlap. ELIRL uses supplied reward features and, in its MaxEnt formulation, known dynamics. BASIS relies on reward-observed forward-RL pretraining with environment interaction. Our setting uses offline demonstrations and pooled transitions without supplied reward features. Our contribution places structure on anchored policy logits and connects it to pooled coverage and transfer guarantees; low-rank rewards alone do not imply low-rank policy logits.

Q1: Figure 3's oracle and the original reward

All method returns in Figure 3 are evaluated under the generating reward ψ⊤w\psi^\top w. The ambiguity concerns the reference line, which uses a policy planned under the μ\mu-normalized reward. We now compare directly against the oracle obtained by exact value iteration on the generating reward.

On the published runs (5 seeds × 8 tasks), LowRank leads at every shift, with its margin over the best baseline growing from 8 to 21 percentage points of the original-reward optimum.

We also identified a correction to the reference line: its normalization was computed using target rather than source dynamics. Using source dynamics places that reference at 99.7% of the original-reward optimum or above. Correcting the reference leaves every method's return unchanged. We will correct Figure 3 and add the original-reward comparison.

Q2: shared network without a rank limit

TaskCond provides this comparison through a shared encoder and nonlinear task conditioning, without requiring task logits to lie in a shared kk-dimensional linear subspace. Its matched parameter budget, training protocol, and results are described above.

We will include these additions and distinguish statistical estimation error, the discrepancy from omitting log-sum-exp, and the finite-iteration and transition-estimation errors in Algorithm 1.

We hope the new guarantees for Algorithm 1, target-performance comparison, TaskCond baseline, and original-reward evaluation address your main concerns. If so, would you consider raising your overall score?

Thank you for the response! The rebuttal solved most my concerns but it seems like there are major updates needed. I'll consider raising my score and keep a low confidence for now.

Thank you for considering our responses and additional analysis. We’re glad that these have resolved most of your concerns.

We would like to clarify that the revisions listed in our general “Updated manuscript” comment have already been incorporated into the uploaded manuscript. Substantive changes are highlighted in blue, and the main text remains within the ten-page limit. In particular, regarding your principal concerns:

  • Guarantees for the implemented method: Section 4.4 bounds the normalizer-omission error and its contribution to target regret. The new Theorem 2 covers Algorithm 1’s fitted recovery from sampled transitions under explicit tabular classification and regression conditions, with proofs in Appendices A.9–A.10.
  • Direct transfer comparison: Main-text Table 1 compares exact plug-in and approximate recovery under both exact source-operator inversion and fitted iteration, reporting target return and soft regret.
  • Original-reward evaluation: The transfer evaluation now uses the original generating reward’s optimum under target dynamics. Appendix B.6 and Table 3 additionally examine identification misspecification.
  • Representation-sharing baseline and related work: We added TaskCond, a shared-network baseline without an explicit rank constraint, and expanded the discussion of ELIRL and BASIS.

These updates are complete and available in the current revised manuscript for your assessment.

Comments on the paper

7 comments
AuthorsOpenReview ↗

General response part 1

We thank all reviewers. Below are the new analyses and experiments referenced in our replies; all will be added to the revision.

Section 1 spans Parts 1–4; Section 2 is in Part 5, and Section 3 is in Part 6.

1. Approximation: theory and transfer performance (sD7A, QJNd, Sg6K)

We address the gap between exact-recovery theory and Algorithm 1 raised by sD7A (Weakness 1), QJNd (Critical Weakness), and Sg6K (Weakness 2) through three theoretical bounds: (i) the discrepancy in reward estimates caused by omitting the log-sum-exp in r^iapprox\widehat r_i^{\mathrm{approx}}, (ii) a target-regret guarantee with an additional normalizer-omission penalty for r^iapprox\widehat r_i^{\mathrm{approx}}, and (iii) a finite-sample guarantee for Algorithm 1, which approximates r^iapprox\widehat r_i^{\mathrm{approx}} using KK fitted-Q updates on sampled source transitions for each of the kk learned logit bases, then combines the recovered reward bases using the fitted task weights.

Matched calibration measures the loss in target value from using r^iapprox\widehat r_i^{\mathrm{approx}} instead of r^i\widehat r_i, with both policies evaluated under r^i\widehat r_i; transfer experiments quantify the return and regret differences between exact recovery and the kk-basis approximation under both exact inversion and fitted iteration. Algorithm 1 thus substantially reduces recovery computation from NN task-specific solves to kk shared basis solves while nearly matching fitted exact recovery in experiments, inducing a loss of 2.9–3.4 percentage points of the true-reward optimum in RecSim. Our updated theory gives an explicit target-regret guarantee with an additional normalizer-omission penalty.

(i) Discrepancy in reward estimates ∣r^iapprox−r^i∣∞\\|\widehat r_i^{\mathrm{approx}}-\widehat r_i\\|_\infty

Given the anchored logits Θ^=W^Φ^\widehat\Theta=\widehat W\widehat\Phi returned by Algorithm 1’s classification step, let r^i\widehat r_i be the reward recovered from the corresponding fitted log-policy using the paper’s exact normalized recovery map. Let r^iapprox\widehat r_i^{\mathrm{approx}} be the reward obtained by applying the same population recovery map directly to Θ^i\widehat\Theta_i, omitting the log-sum-exp. Both rewards are defined using the true source dynamics P1\mathsf{P}_1:

\widehat r_i &=C_\mu(I-\gamma_1\mathsf{P}_1\mu)^{-1} \left[ \widehat\Theta_i-\log\sum_b e^{\widehat\Theta_{i,(\cdot,b)}}-g \right]+g,\\\\ \widehat r_i^{\mathrm{approx}} &=C_\mu(I-\gamma_1\mathsf{P}_1\mu)^{-1} (\widehat\Theta_i-g)+g. \end{aligned}$$ Here $C_\mu f=f-\mu f$, $(\mu f)(s)=\sum_a\mu(a\mid s)f(s,a)$, and $g$ is the known reward normalization $\mu r_i=g$; the discrepancy between these reward estimates satisfies $$\lVert\widehat r_i^{\mathrm{approx}}-\widehat r_i\rVert_\infty\le\frac{\gamma_1}{1-\gamma_1}\left[\max_s\log\sum_b e^{\widehat\Theta_{i,(s,b)}}-\min_s\log\sum_b e^{\widehat\Theta_{i,(s,b)}}\right].$$ In particular, $\widehat r_i^{\mathrm{approx}}=\widehat r_i$ when the log-sum-exp is constant across all states or when source transitions are action-independent. #### (ii) Target regret penalty resulting from the discrepancy in reward estimates Omitting the normalizer introduces an additional term in the target-regret guarantee that is linear in the discrepancy between the reward estimates. Let $\widehat\pi_i$ and $\widehat\pi_i^{\mathrm{approx}}$ be soft-optimal target policies for $\widehat r_i$ and $\widehat r_i^{\mathrm{approx}}$, and let $\pi_{2,i}^\star$ be optimal for the true reward $r_i$. With entropy coefficient 1 and Theorem 1’s value-error radii $R_n(\pi)$, its simultaneous estimation event of probability at least $1-\delta$ gives $$\begin{aligned} &J_{2,i}(\pi_{2,i}^\star;r_i) -J_{2,i}(\widehat\pi_i^{\mathrm{approx}};r_i)\\\\ &\quad\le R_n(\pi_{2,i}^\star) +R_n(\widehat\pi_i^{\mathrm{approx}}) +\frac{2\\|\widehat r_i^{\mathrm{approx}}-\widehat r_i\\|_\infty} {1-\gamma_2}. \end{aligned}$$ The two $R_n$ terms account for statistical estimation error; the last term bounds the additional penalty from omitting the normalizer.
AuthorsOpenReview ↗

General response part 2

1. Approximation: theory and transfer performance (continued)

(iii) Finite sample guarantee for Algorithm 1

We give a finite-sample guarantee for Algorithm 1 under the fitting conditions below, using sampled source transitions without knowledge of P1\mathsf{P}_1. Suppose we obtain r~i\widetilde r_i by running K≥1K\ge1 fitted-Q updates on each of the kk shared logit bases and combining the recovered reward bases with task ii’s fitted weights, as in Algorithm 1. Let π~i\widetilde\pi_i be the soft-optimal target policy under r~i\widetilde r_i.

We assume Theorem 1’s statistical conditions with g=0g=0. The classification step returns a global minimizer of the tabular factorized objective at λn\lambda_n, with anchored fitted logits satisfying ∣Θ^∣∞≤B\\|\widehat\Theta\\|_\infty\le B. Fitted recovery starts from zero, and each regression exactly minimizes squared loss over a class that represents the empirical Bellman update.

For δ∈(0,1)\delta\in(0,1), α∈(0,1−δ)\alpha\in(0,1-\delta), and

\ge8\log(2SA/\alpha),$$ with probability at least $1-\delta-\alpha$, simultaneously for every task, $$\begin{aligned} &J_{2,i}(\pi_{2,i}^{\star};r_i) -J_{2,i}(\widetilde\pi_i;r_i)\\\\ &\quad\le \underbrace{ R_n(\pi_{2,i}^{\star})+R_n(\widetilde\pi_i) }_{\text{statistical estimation}}\\\\ &\qquad+ \underbrace{ \frac{2\\|\widehat r_i^{\mathrm{approx}}-\widehat r_i\\|_\infty} {1-\gamma_2} }_{\text{omitting log-sum-exp}}\\\\ &\qquad+ \underbrace{ \frac{4\gamma_1^{K+1}\\|\widehat\Theta_i\\|_\infty} {(1-\gamma_1)(1-\gamma_2)} }_{\text{finite fitted iteration}}\\\\ &\qquad+ \underbrace{ \frac{8\gamma_1\\|\widehat\Theta_i\\|_\infty} {(1-\gamma_2)(1-\gamma_1)^2} \sqrt{ \frac{S\log 2+\log(2SA/\alpha)} {n\min_{s,a}\sum_{i=1}^N d_{i,s}\pi_{1,i}^\star(a\mid s)} } }_{\text{source-transition estimation}}. \end{aligned}$$ This bound controls target regret for Algorithm 1’s fitted recovery using $k$ shared basis fits (rather than $N$ task-specific solves) and sampled source transitions, without knowing $\mathsf{P}_1$. It retains the form of Theorem 1’s statistical terms and adds contributions from omitting log-sum-exp, finite fitted iteration, and source-transition estimation. With other parameters fixed, the contribution to the regret bound from finite fitted-Q iteration decays as $O(\gamma_1^{K+1})$, while the contribution from source-transition estimation decays as $O(n^{-1/2})$, so both vanish as $K,n\to\infty$. The discrepancy in reward estimates from omitting log-sum-exp need not vanish, but its magnitude is bounded by (i). Note that Assumption 3’s coverage condition, $c_0(d,\Theta^\star)>0$, guarantees $\min_{s,a}\sum_{i=1}^N d_{i,s}\pi_{1,i}^\star(a\mid s)>0$. This guarantee directly addresses Sg6K’s Weakness 2 and sD7A’s Weakness 1. We can also construct instances where the target-regret bound’s linear dependence on $\\|\widehat r_i^{\mathrm{approx}}-\widehat r_i\\|_\infty$ is tight. We next evaluate these effects empirically; proof sketches for (i)–(iii) follow the experimental comparisons. **Matched calibration.** We directly evaluate the shortcut policy's soft-value loss under the exact plug-in reward in a separate five-seed diagnostic covering eight tasks and five target settings. We use two target initial-state distributions: cold starts at zero clicks, as in training, and warm starts uniformly over states with 1–8 total clicks, representing partly fatigued users. Both evaluations use the same converged policies. We assess the normalizer-omission penalty separately from transition-estimation and finite-iteration error, using exact source recovery in both arms. The measured loss satisfies this penalty in every tested case. Average per-task loss-to-bound ratios are approximately 1% under the warm initial-state distribution and 4–6% under the cold distribution; the largest individual ratio is 0.16. Losses and bounds are computed from the same retrained fits. The worst-case penalty remains nearly constant across target settings, while the measured loss varies with target occupancies.
AuthorsOpenReview ↗

General response part 3

1. Approximation: transfer experiment and proof sketches (continued)

Transfer experiment. Table A separately measures performance under the true reward over 10 seeds × 8 tasks. We compare the exact plug-in (NN solves) with the approximation (kk solves), each using exact inversion or fitted iteration. Soft regret is occupancy-weighted policy KL divided by 1−γ21-\gamma_2.

For RecSim's action-dependent exits, fitted Bellman targets use an estimated survival probability instead of the realized survival indicator. We apply this change to every fitted reward-recovery method.

Table A. RecSim transfer, no shift → strongest shift. Fitted rows use survival-weighted targets.

Method% of optimumSoft regret
Exact plug-in, exact inversion78.2 → 68.41.95 → 0.71
Exact plug-in, fitted iteration75.6 → 67.92.04 → 0.73
Approximation, exact inversion71.3 → 64.12.18 → 0.92
Approximation, fitted iteration (LowRank)72.2 → 65.02.17 → 0.90
Behavior cloning62.3 → 39.96.03 → 5.95
IQ-Learn per task59.8 → 38.14.69 → 3.70
Random32.0 → 19.03.67 → 1.15

The paired return cost of the approximation is 3.4±0.53.4\pm0.5 → 2.9±1.82.9\pm1.8 percentage points with fitted iteration and 6.8±0.66.8\pm0.6 → 4.3±1.64.3\pm1.6 with exact inversion (mean ± sd across seeds after averaging tasks). For each estimator, fitted iteration is within 2.6 percentage points of exact inversion at these endpoints. These comparisons quantify the observed accuracy–computation trade-off.

Proof sketches for (i)–(iii)

For (i), subtracting the recovery formulas and cancelling the immediate state-only normalizer gives

\widehat r_i^{\mathrm{approx}}-\widehat r_i &=\gamma_1(\mathsf{P}_1-\mu\mathsf{P}_1) (I-\gamma_1\mu\mathsf{P}_1)^{-1}\\\\ &\quad\times\left[\log\sum_b e^{\widehat\Theta_{i,(\cdot,b)}}\right]. \end{aligned}$$ The discounted sum under the source dynamics and $\mu$ has range at most $1/(1-\gamma_1)$ times the normalizer’s range. The difference between its next-state expectations under $\mathsf{P}_1$ and $\mu\mathsf{P}_1$ is bounded by that range, proving (i). The identity also shows that the discrepancy vanishes when the normalizer is constant across states or source transitions are action-independent. For (ii), changing rewards changes a fixed policy’s value by at most $\\|\widehat r_i^{\mathrm{approx}}-\widehat r_i\\|_\infty/(1-\gamma_2)$. Applying this twice and using optimality of $\widehat\pi_i^{\mathrm{approx}}$ gives the factor 2. Then $J_{2,i}(\pi_{2,i}^\star;\widehat r_i)\le J_{2,i}(\widehat\pi_i;\widehat r_i)$ and Theorem 1’s simultaneous value-error bound add the two statistical terms.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.