acceptodds
Under review as a conference paper at ICLR 2027

Hyperbolic Inverse Reinforcement Learning

Abstract

Inverse reinforcement learning (IRL) seeks to infer the reward function underlying observed behavior, but its predictive accuracy and interpretability depend on how that reward is computed from the underlying features. While prior IRL reward functions have been implemented through a variety of techniques including linear weights, neural networks, or other function approximators, none make assumptions about the geometry of the underlying feature space. Inspired by geometric organization observed in neural circuits, we introduce Hyperbolic inverse reinforcement learning (Hyperbolic IRL), which parameterizes reward as a function of Hyperbolic distance between the feature vector and a learned parameter vector, as well as its Euclidean distance variant. We tested our methods against a range of linear and nonlinear IRL baselines under both deterministic and stochastic policies. Our Hyperbolic approach achieved comparable performance to state-of-the-art models on established benchmarks and excelled in the low data limit and across dense environments. Applied to human navigation among multiple exhibits in a museum room, Hyperbolic IRL reveals reward structure that depends not only on individual exhibit-distance features, but also on non-separable conjunctions between them. Together, these results demonstrate that a Hyperbolic reward parameterization can improve both the predictive accuracy and interpretability of IRL, providing a framework for studying how non-separable, conjunctive structure can begin to explain behavior.

Primary area:
reinforcement learning
Keywords:
inverse reinforcement learning, hyperbolic geometry, reward learning, maximum entropy, neuroscience

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_pfCfOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a hyperbolic reward function parameterization for inverse reinforcement learning. The hyperbolic parameterization has a attractive properties of hierarchical representation and packing dense objects into tight space. The learning algorithm is unchanged and uses the maximum entropy IRL algorithm with Riemannian gradient descent. The authors tested the proposed algorithm in object world and a museum environment against linear reward parameterization and deep IRL. The results show hyperbolic IRL is more sample efficient and better captures feature interaction.

Critical strengths
  • Using hyperbolic distance to parameterize reward functions for IRL is an interesting idea. This is a useful prior if applicable to the problem at hand.
  • The paper is well written and relatively straightforward to understand for reader not familiar with hyperbolic space.
  • Results show better sample complexity of the hyperbolic model compared to euclidean or classic linear model and better captures human behavior in the co_attendance analysis.
Critical weaknesses
  • This paper angles towards neuroscience applications. I think the paper would benefit much more from evaluating on a wider range of neuroscience tasks and environments and demonstrating its impact. Furthermore, the paper would benefit from a more in-depth discussion of the type of environment and tasks the hyperbolic model is suitable for and unsuitable for. The current paper only describes itself as suitable for "densely distributed reward-relevant objects". Also are there non-neuroscience environments which this model may be good for?
Questions for authors
  • For the object world environment, is expected value difference of 10 good or not good?
  • The museum environment is interesting but I wonder whether it breaks the Markovian assumption of normal IRL? I expect human behavior to be history dependent.
Reviewer_JUsNOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The manuscript introduces Hyperbolic Inverse Reinforcement Learning (Hyperbolic IRL), a framework for distance-based IRL in which the reward is derived from the distance between a state's feature vector and a learned parameter vector, computed in either hyperbolic or Euclidean space. Its key advantage is that the hyperbolic reward can capture non-additive interactions between task features. The approach is partly motivated by neuroscience literature where neural circuits show hyperbolic organization (Zhang et al., 2023; Zhou et al., 2018). The authors argue that 'a reward geometry grounded in the brain’s own representational structure is a natural fit' for such behavioral analysis.

The authors evaluate their approach in two settings. First, to test whether the model can recover a nonlinear rewards, a simulated gridworld benchmark (Objectworld) whose true reward is a known nonlinear function was used. Second, to test whether modeling real human behavior benefits from accounting for interactions between features, the model was fit to recorded human trajectories of 26 museum visitors obtained from the HUMEX dataset. The authors report that Hyperbolic IRL outperforms the other models in both settings.

Critical strengths

The feature coupling analysis in Section 3.3 shows that the hyperbolic reward can capture interaction between features and provides an interpretable measure of feature coupling (conjunctive effect) of any feature pair.

Treating curvature as a hyperparameter is a clever and cheap feature that introduces more flexibility in the model: a single number can be used to control the shape of the reward surface, so the geometry can be tuned to the setting.

Critical weaknesses

Several approaches are compared to demonstrate the efficacy of Hyperbolic IRL. However, the authors did not include Gaussian Process IRL, which was introduced in the same paper as the Objectworld benchmark, to test nonlinear reward recovery. This is a serious limitation of this study.

The limitations of the approach are not clearly stated. First, validation studies are limited to spatial navigation tasks. The approach is clearly suited to such tasks, but the authors should state that the method was evaluated in such tasks for its natural compatibility with the structure of these tasks, and its efficacy in other domains remains to be tested. Second, the authors do state that this approach is especially useful when data are scarce, which is consistent with the trends in Figure 2, where the plots are cut off at a 128 demonstrations, when the DeepIRL catches up to the hyperbolic approaches. Based on the trends of the DeepIRL approach, given enough data, it may be able to outperform the hyperbolic approaches. The authors should mention this in a limitations section, and make a stronger case for their approach in light of these results.

Line 280 (pg 6) states that "Hyperbolic κ\kappa = −4 had the best ADE and FDE performance...", but in Figure 4B–C the error bars overlap, which does not suggest a significant difference between models with κ\kappa = −4 and -1. Please report statistical significance test for the results.

The statements "curvature of κ\kappa = −4 had the best performance with increased sample count" in the Figure 2 caption and "with Hyperbolic (κ\kappa = −4) showing the best performance ..." in line 238 (pg 5) do not match my assessment of the plots in Figure 2. If I am interpreting the plots correctly, the EVD for κ\kappa = −1 in the deterministic cases is lower than that of κ κ\kappa= −4, and therefore is the better model.

I have several concerns about the plots in Figure 3: How many examples were used for generating it? Did the authors test if these results hold under varying number of demonstrations? The authors do not specify the number of repeats per condition in Figure 3.

Also, please state what the shaded bands represent Figures 2 and 3. Are they standard error or standard deviation?

Questions for authors

Please see critical weaknesses

Reviewer_kLiVOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The paper parameterizes rewards as the negative squared distance between state features and a learned reference point, using either Euclidean or hyperbolic geometry within maximum-entropy IRL. Experiments on Objectworld and museum visitor trajectories compare policy performance, while feature interaction probes and exhibit co-attendance statistics are used to interpret the learned rewards.

Critical strengths

The reward parameterization is compact, requiring 15 parameters in the museum experiment.

Critical weaknesses
  • The interaction probes do not establish behavioral preferences. Figures 5–6 vary exhibit distances while holding the visitor’s coordinates fixed (L268–270, L318–326). These combinations generally cannot correspond to actual positions. The resulting interactions therefore describe the model’s extrapolation outside feasible states, rather than demonstrating conjunctive preferences supported by the observations.

  • The behavioral check reuses the training data. Co-attendance is computed across all visitors, including those used to fit the reward (L270–271, L364–371). The label-permutation test provides no out-of-sample validation. The reported correlation therefore does not establish whether the inferred interactions generalize beyond the trajectories used for fitting.

  • The comparisons do not support the broader claims. Comparing only with MaxEnt IRL and DeepIRL (L210–212) is insufficient to support the abstract’s state-of-the-art claim. GPIRL and smaller, properly tuned neural rewards are relevant controls for the low-data claim. Moreover, weak interactions in one fitted DeepIRL model do not establish a representational advantage: DeepIRL can already model nonlinear feature interactions, contrary to the distinction drawn at L411–413.

  • The mathematical specification is inconsistent. The appendix gives incompatible reward definitions, R=−12d2R=-\frac12d^2 and R=ϕ⊤θR=\phi^\top\theta (L586–588). The sign error at L715 puts the argument of arcosh⁡\operatorname{arcosh} outside its real domain, while Eqs. (52)–(53) replace trajectory-level exponential weights with single-state weights without justification.

Questions for authors
  • Do the reported gains persist with the exact gradient, using the same curvature, training budget, and stability controls?

  • Can the interaction claims be tested using feasible feature combinations and visitors excluded from reward fitting?

  • What exactly is “scale-matched” in Figures 2–3? How were feature scale, reward scale, and policy temperature controlled, and what validation criterion was used to select curvature?

  • Please provide the learning rates, training iterations, trajectory horizons, discount factors, and DeepIRL architecture. How were the baselines tuned relative to the proposed method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.