- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the reward transfer properties (and errors) of rewards recovered by inverse reinforcement learning. Prior work showed that transfer error depends on the principal angles between the Potential Shaping Spaces of the source environments. However, this requires access to transition laws. The authors propose to estimate these angles from IRL gradients instead, using a dimension-calibrated truncated SVD (DC-TSVD). The method is evaluated on synthetic subspaces and small WindyGridWorld environments.
Critical strengths
- The question of whether transfer geometry can be estimated without the transition model is interesting.
- The observation that IRL gradients lie in the Jacobian image of the identifiable reward space is simple and elegant.
Critical weaknesses
- The paper is very hard to follow. The introduction mainly lists related work instead of explaining the problem, and the background is unintuitive. For example, it is never explained why principal angles between shaping spaces matter for transfer, or what the constants in Eq. 1 represent. The only motivating example is deferred to App. A, and its figure is cluttered and hard to understand, with a placeholder caption. The writing must be significantly improved, e.g. with an intuitive explanation of the problem and a dedicated related-work section, so that the reader can follow the proposed method.
- The method section introduces many named objects in quick succession (Identifiable Reward Space, Theoretical Gradient Space, Ideal Gradient Matrix, Ideal Model) with little intuition. The notation is overloaded: denotes both the reward and , denotes different constants in Assumptions 2 and 3, and denotes both the regularizer and an ambient dimension. Also, the main real-environment results are only in the appendix.
- The claim of bypassing the transition model is not substantiated. Computing an IRL gradient requires the optimal policy for each sampled reward, which itself requires the transition law or an RL solve, and the paper does not say how this policy is obtained.
Questions for authors
- Computing the IRL gradient requires the soft-optimal policy for each sampled reward. How is this policy obtained in your experiments without the transition model?
- How does DC-TSVD compare to simply estimating the transition laws from the same trajectories and computing the principal angles directly?