acceptodds
Under review as a conference paper at ICLR 2027

Towards Recoverable Reward Transfer Geometry from IRL Gradients

Abstract

Inverse reinforcement learning (IRL) aims to recover reward functions from expert demonstrations, but reward ambiguity fundamentally limits their transferability across environments. Recent theory characterizes transfer sensitivity through principal-angle geometry between Potential Shaping Spaces, yet computing this latent geometry requires known or estimated transition laws. We show that this model-level route can be bypassed: we establish an observable route through IRL gradients, linking the transfer-relevant reward structure to gradient-space geometry and enabling direct estimation of this geometry. In practice, however, noisy or approximately evaluated gradients can lead to rank inflation and degeneracy of empirical principal angles, while complementary representations introduce structural angles unrelated to transfer. To enable estimation, we develop Dimension-Calibrated Truncated SVD (DC-TSVD), which uses intrinsic dimensions determined by IRL identifiability to suppress perturbation-induced directions and remove structural angles. Theoretical analysis establishes principal-angle estimation error bounds, and experiments on synthetic and finite-demonstration IRL environments demonstrate reliable estimation of the latent transfer geometry.

Primary area:
reinforcement learning
Keywords:
Inverse Reinforcement Learning, Reward Transferability, Reward Transfer Geometry, Principal Angle Estimation, Observable IRL Gradients

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_a4TxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the reward transfer properties (and errors) of rewards recovered by inverse reinforcement learning. Prior work showed that transfer error depends on the principal angles between the Potential Shaping Spaces of the source environments. However, this requires access to transition laws. The authors propose to estimate these angles from IRL gradients instead, using a dimension-calibrated truncated SVD (DC-TSVD). The method is evaluated on synthetic subspaces and small WindyGridWorld environments.

Critical strengths
  • The question of whether transfer geometry can be estimated without the transition model is interesting.
  • The observation that IRL gradients lie in the Jacobian image of the identifiable reward space is simple and elegant.
Critical weaknesses
  • The paper is very hard to follow. The introduction mainly lists related work instead of explaining the problem, and the background is unintuitive. For example, it is never explained why principal angles between shaping spaces matter for transfer, or what the constants in Eq. 1 represent. The only motivating example is deferred to App. A, and its figure is cluttered and hard to understand, with a placeholder caption. The writing must be significantly improved, e.g. with an intuitive explanation of the problem and a dedicated related-work section, so that the reader can follow the proposed method.
  • The method section introduces many named objects in quick succession (Identifiable Reward Space, Theoretical Gradient Space, Ideal Gradient Matrix, Ideal Model) with little intuition. The notation is overloaded: rr denotes both the reward and dim⁡(UP)\dim(\mathcal{U}_P), cc denotes different constants in Assumptions 2 and 3, and hh denotes both the regularizer and an ambient dimension. Also, the main real-environment results are only in the appendix.
  • The claim of bypassing the transition model is not substantiated. Computing an IRL gradient requires the optimal policy for each sampled reward, which itself requires the transition law or an RL solve, and the paper does not say how this policy is obtained.
Questions for authors
  • Computing the IRL gradient requires the soft-optimal policy for each sampled reward. How is this policy obtained in your experiments without the transition model?
  • How does DC-TSVD compare to simply estimating the transition laws from the same trajectories and computing the principal angles directly?
Reviewer_qhvHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces the DC-TSVD method for estimating reward transferability without needing to model the environment dynamics. It does so by measuring the angles between spaces of reward changes that preserve expert behavior and using truncated SVD to remove noisy gradient dimensions. The paper provides bounds on the estimation errors and uses simulations in simple environments to demonstrate that DC-TSVD can recover angles more accurately than using the full gradient space, potentially leading to better reward transfer.

Critical strengths
  1. The paper demonstrates a connection between IRL gradients and the geometry of reward transfer without requiring estimation of the transition dynamics.
  2. The paper shows why noisy gradients can lead to misleading notions of transferability and proposes the DC-TSVD method to address this issue.
  3. The simulated experiments test different noise levels and demonstrate that DC-TSVD performs better than full SVD.
Critical weaknesses

I am writing this review without being familiar with the prior work on the geometry of potential shaping spaces upon which this paper builds. Thus, I may be missing the significance of some of the results.

  1. DC-TSVD appears to have limited practical significance. From my understanding, the paper focuses on recovering the "transfer geometry," but it does not empirically evaluate how the proposed technique improves reward function recovery. The paper also does not test whether angle recovery predicts transfer performance. Furthermore, the paper provides no comparisons to prior IRL approaches and their ability to transfer rewards. This lack of empirical evaluation makes it difficult to assess the contributions of the method and its relationship to prior IRL work.
  2. I found it difficult to distinguish the paper's novel contributions from its background material. Much of the writing appears to describe general properties of subspaces and SVD, and it is unclear how this exposition establishes a novel method. The paper should more clearly connect these results to the reward transfer problem and explain what limitations of prior approaches they address.
  3. The pipeline is only evaluated in a single WindyGridWorld environment. It is unclear whether the proposed approach can generalize beyond this environment or to more complex settings.
  4. A central motivation of the paper is avoiding the need to estimate the transition dynamics. However, the proposed approach requires estimating gradients using additional trajectories. The paper should therefore compare this process against directly estimating the transition function to demonstrate whether the proposed approach offers a meaningful advantage.
Questions for authors

See weaknesses section.

Reviewer_EMWnOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper relies on a result from a previous paper, Schlaginhaufen & Kamgarpour, 2024, which roughly claims that the "transferability" of rewards recovered from regularized IRL from an expert in two envs P0,P1, depends on the "principal angle" between P0,P1. That said, the goal of the paper is to provide an alternative way for the computation of this principal angle that does not require explicitly computing it through P0,P1, but instead passing through what they call "IRL gradients". The authors show that an SVD-based approach can provably recover the principal angle from IRL gradients at the price of some additional assumptions.

Critical strengths
  • Strong theoretical formalism
  • The identifiability and transferability problems in IRL are important and of interest for the community
Critical weaknesses
  • No clear motivation for why the proposed approach is better or even needed -> The authors do not provide any sample complexity or computational complexity comparisons between the naive estimation approach (Schlaginhaufen & Kamgarpour, 2024) and the proposed approach, which I think is crucial to support the proposed method. Is the naive approach that bad?
  • More generally, there is no theoretical result of sample complexity; how many expert demonstrations do we need? How many interactions with the env? These are quite important questions to be addressed
  • I think that the presentation should be improved. For instance, in Section 2 the PSS is not explained very well, and the authors basically assume that the reader perfectly knows the paper (Schlaginhaufen & Kamgarpour, 2024), since they don't explain what kind of transferability guarantee they are studying (that is, lines 133-145, they just say "a transferability guarantee"). In other words, it might be hard for the reader to be interested and understand the contributions of this specific paper if even understanding the motivations and problem setting requires reading other papers.
Questions for authors

Please, clarify my points above.

Moreover, explain why you believe that the 3 assumptions you introduce are reasonable.

Reviewer_rRJcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper estimates the principal angles between Potential Shaping Spaces, which control how well IRL rewards transfer, directly from IRL gradients rather than from estimated transition laws. It proposes DC-TSVD, a truncated SVD at the rank implied by identifiability theory, proves error bounds for it, and validates it on synthetic data and on 3×3 and 5×5 WindyGridWorld.

Critical strengths
  • A simple link between gradients and transfer geometry. The observation that IRL gradients lie in the identifiable reward space, and so carry the geometry that controls transfer, is elegant. It opens a new and observable route to a quantity that previously needed the transition laws.
  • A clear diagnosis with a simple fix. Theorem 3 neatly explains why naive SVD collapses every cosine to one. DC-TSVD fixes this without any tuning parameters and works well in the reported experiments.
Critical weaknesses

The evaluation scope is narrow. The experiments use gridworlds with at most 25 states and known, invertible Jacobians. The gradients are sampled at random reward parameters to ensure coverage, rather than taken from an actual IRL run. So it’s unclear whether the method holds up in the settings that motivate the paper.

Questions for authors

Could you show that estimated θ₂ predicts actual transfer error?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.