acceptodds
Under review as a conference paper at ICLR 2027

Inverse Reinforcement Learning with State-Dependent Foresight for Human Decision-Making

Abstract

Computational modeling of behavior represents individuals with a model fitted to their observed behavior and characterizes them through its parameters. Naturalistic tasks incorporate the complexity of real environments and yield high-dimensional behavioral trajectories. Inverse reinforcement learning (IRL) infers the latent reward function underlying these trajectories. Standard IRL, however, assumes effectively infinite-horizon planning, whereas human foresight is both bounded and state-dependent. Here we introduce **SC-AIRL** (**S**tate-**C**onditional Adversarial IRL), which builds these two cognitive properties into the planner. Per-depth policies share one reward but differ in how far ahead they sum it explicitly. A state-conditional router mixes them according to the current state. Across held-out episodes in pedestrian crossing and highway overtaking tasks, SC-AIRL improves imitation over Standard AIRL, raising top-1 accuracy from 0.549 to 0.652 and from 0.629 to 0.718, respectively, while more closely reproducing participant behavior in rollouts. Inferred planning depth correlates with self-reported anxiety and motor impulsivity. SC-AIRL thus reproduces human behavior more closely than Standard AIRL while providing an interpretable, model-derived measure of state-dependent foresight.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Inverse reinforcement learning, Planning depth, State-dependent planning, Computational cognitive science, Naturalistic reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AFgkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SC-AIRL, an extension of AIRL that incorporates state-dependent planning depth through a mixture-of-experts architecture. Evaluated on held-out episodes from pedestrian-crossing and highway-overtaking tasks, SC-AIRL improves over baselines.

Critical strengths
  • MoE + AIRL seems to be interesting
Critical weaknesses
  • limited methodological novelty
  • missing motivation and necessity of MoE
  • missing enough justification of assumptions
  • limited evaluations
Questions for authors

Limited methodological novelty.

SC-AIRL appears to be a relatively straightforward extension of AIRL by incorporating a state-conditioned routing architecture.

A state-conditional router mixes them according to the current state.

  1. Could the authors clarify the methodological novelty compared with existing approaches, particularly InfoGAIL, Option-GAIL, and other multi-policy imitation learning methods?

1.1 What are the major theoretical contributions of the proposed framework? How do they differ from those of existing GAIL or AIRL approaches?

4.3 MULTI-POLICY ARCHITECTURE Under Assumption (A2) ....

1.2 GAE truncation primarily controls temporal credit assignment during policy optimization and does not necessarily correspond to human planning depth.

Could the authors clarify the theoretical justification for interpreting different GAE truncation horizons as different levels of human foresight? What empirical evidence supports this interpretation?

  1. Routing / MoE architectures have been extensively explored in LLMs. What specifically motivates the use of MoE in the proposed framework, and why is state-conditioned routing necessary for the two considered tasks? Could similar improvements be achieved using a single policy with state-dependent planning parameters or a simpler multi-policy architecture?

missing enough justification of assumptions

3.1 ASSUMPTIONS (A1) Shared reward. A set of depth-indexed policies ... shares a single reward ... (A2) Bounded foresight. (A3) State-conditional depth.

  1. Insufficient justification of assumptions. The assumptions in Section 3.1 (A1–A3) require further theoretical or empirical justification.

In particular, 3.1 Why should different planning depths share an identical reward function?

3.2 How can the inferred routing weights be interpreted as human planning depth rather than general policy specialization? What is the relationship between routing weights and human planning?

Evaluations are limited

  1. The experiments are restricted to two simulator-based tasks involving pedestrian crossing and highway overtaking. Could the authors provide evaluations on additional environments, such as established robotic control (e.g., Pendulum, Ant, Swimmer, Half-Cheetah, the ones evaluated in GAIL [1] or AIRL[2] ) or sequential decision-making benchmarks, to demonstrate the generalizability of SC-AIRL?

Option-GAIL adds ... Single-h AIRL uses ... Episode-AIRL uses the same per-depth policies ...

SC-AIRL has 14,101 parameters versus 3,472 for Standard AIRL on the highway task (Appendix G, Table 14)

  1. Missing important ablation studies. Although the paper includes comparisons with Single-h AIRL, Episode-AIRL, further controlled experiments would help distinguish the effects of state-dependent foresight from increased model capacity and policy diversity.

In particular, how does SC-AIRL compare with a parameter-matched MoE-AIRL model in which the expert policies use the same planning horizon? How can the authors demonstrate that the improvements specifically result from modeling different planning depths rather than simply combining multiple independently trained policies?

Furthermore, SC-AIRL has a substantially larger total parameter count than Standard AIRL. Could the authors clarify how this difference affects the performance comparison?

Table 2: Held-out imitation fidelity on both tasks.

  1. Table 2 reports results averaged over a small number of training runs, while some baselines are evaluated using only a single run. Why? Could the authors provide more extensive evaluations across random seeds and report consistent uncertainty estimates, such as standard deviations or confidence intervals for the main performance improvements, to better establish their statistical robustness?

[1] "Generative adversarial imitation learning." Advances in neural information processing systems 29 (2016).

[2] "Learning robust rewards with adversarial inverse reinforcement learning." arXiv preprint arXiv:1710.11248 (2017).

4 comments
AuthorsOpenReview ↗

Response to Reviewer AFgk (1/2)

We thank the reviewer for the insightful and comprehensive feedback. The comments have been very helpful in thoroughly revisiting the theoretical foundations of our work and refining our methodology.

Assumptions

Q1.2. GAE truncation does not necessarily correspond to human planning depth.

A. We thank the reviewer for this important comment. In response, we revised the model so that hh better represents a bounded planning horizon. Now, each policy explicitly optimizes an hh-step return (Eq. 1) rather than GAE truncated at hh steps (Eq. 2).

G_t^{(h)}=\sum_{k=0}^{h-1}\gamma^k r_\theta(s_{t+k},a_{t+k}),\qquad A_t^{(h)} = G_t^{(h)} - V^h(s_t) \tag{1}

\delta_t^{(h)} = r_\theta(s_t,a_t) + \gamma V^h(s_{t+1}) - V^h(s_t),\qquad A_t^{(h)} = \sum_{k=0}^{\min(h,T-t)-1}(\gamma\lambda)^k\delta_{t+k}^{(h)} \tag{2}

Q3. Justify assumptions in Section 3.1 (A1-A3)

Q2. Why are MoE and state-conditioned routing necessary?

A. The motivation and assumptions underlying SC-AIRL are grounded in prior work on human decision-making and bounded rationality. We have cited relevant references to support each assumption.

  • Assumption 1: shared reward (Q3.1). Human choices can be modeled as a weighted combination of strategies that accumulate the same reward to different planning depths [1]. Accordingly, SC-AIRL represents each strategy as a separate head, all sharing the same reward function rθr_\theta.
  • Assumption 2: bounded foresight (Q1.2). Tree-search models with truncated planning horizons explained human choices better than models with full horizons [2]. Accordingly, each πh\pi^h now maximizes Gt(h)=∑k=0h−1γkrθ(st+k,at+k)G_t^{(h)}=\sum_{k=0}^{h-1}\gamma^k r_\theta(s_{t+k},a_{t+k}), with VhV^h only as a baseline.
  • Assumption 3: state-conditional depth (Q2). The router's weights over the heads serve as a per-participant proxy for planning depth. Planning depth varies with immediate needs within a person [3], so the router is conditioned on the state. State-conditional weights reproduce held-out actions more closely than state-independent weights, in Top-1, KL and rollouts.

Q3.2. Justify interpretation of the inferred routing weights as human planning depth.

A. The heads share the same reward and architecture and differ only in their planning horizons (hh). Thus, we believe the routing weights can be interpreted as proxies for planning depth.

Novelty

Q1. Clarify the methodological novelty compared with existing approaches, particularly InfoGAIL, Option-GAIL, and other multi-policy imitation learning methods.

Q1.1. What are the major theoretical contributions of the proposed framework? How do they differ from those of existing GAIL or AIRL approaches?

A. SC-AIRL’s key contribution is to explicitly represent individual differences in planning depth within an inverse reinforcement learning framework, guided by principles of human bounded rationality from the assumptions above.

  • Multi-policy imitation: InfoGAIL and Option-GAIL offer limited interpretation of their sub-policy selection as a predefined quantity, such as planning depth, because the role of each sub-policy is identified only after training. In contrast, SC-AIRL allows this interpretation by assigning each sub-policy a planning horizon hh before training.
  • IRL: SC-AIRL integrates two lines of IRL research by using a state-conditional mixture of horizon-specific policies under a shared reward: Hierarchical AIRL selects sub-policies based on the state [4], while other IRL methods model planning horizon or depth using agent-specific discount factors [5] or participant-level depth-cost parameters [6]. To our knowledge, these ideas have not been integrated: the sub-policies in [4] are not horizon-specific, whereas [5, 6] use a fixed horizon per agent or participant rather than allowing it to vary by state.

[1] "Adaptive integration of habits into depth-limited planning defines a habitual-goal-directed spectrum." Proceedings of the National Academy of Sciences 113.45 (2016): 12868-12873.

[2] "Bonsai trees in your head: How the Pavlovian system sculpts goal-directed choices by pruning decision trees." PLoS Computational Biology 8.3 (2012): e1002410.

[3] "Planning in the brain." Neuron 110.6 (2022): 914-934.

[4] "Multi-task hierarchical adversarial inverse reinforcement learning." Proceedings of the 40th International Conference on Machine Learning, PMLR 202 (2023): 4895-4920.

[5] "Inverse reinforcement learning with multiple planning horizons." Reinforcement Learning Journal 3 (2024).

[6] "Measuring individual differences in the depth of planning." PsyArXiv preprint (2022).

AuthorsOpenReview ↗

Response to Reviewer AFgk (2/2)

Evaluation

Q2. Could a simpler multi-policy architecture achieve similar improvements?

A. As seen in the table below, state-conditional weights outperform simpler multi-policy alternatives on most metrics: achieving the highest Top-1 and the lowest KL, mean |ΔAction|, and outcome gap under the same shared reward and per-depth policies. State-independent weights perform better only on NLL. This comes from the 10.5% of held-out steps in which either model assigns the expert's action a probability below 1% (SC-AIRL 8.32 vs. 7.79), and on the remaining 89.5% of steps the two are on par (SC-AIRL 0.704 vs. 0.711).

Pedestrian, held-out, N = 52wt(h)w_t(h) in πmix=∑hwt(h) πh\pi_{\text{mix}}=\sum_h w_t(h)\,\pi^hNLL ↓Top-1 ↑KL ↓Mean |ΔAction| ↓Outcome gap ↓SC-AIRL better (of 52)
Uniform1/∣H∣1/\lvert\mathcal{H}\rvert1.5040.6330.1110.0550.19732 / 38 / 48 / 46 / 29
State-independentw(h)w(h)1.4290.6340.1090.0540.1979 / 42 / 47 / 47 / 32
One depth per participant1[h=h∗]\mathbb{1}[h=h^*]1.8950.6080.1130.0410.19852 / 50 / 40 / 38 / 31
Fixed per episode (Episode-AIRL)wep(h)w_{\text{ep}}(h)1.5740.6140.1140.0460.21541 / 50 / 43 / 44 / 33
State-conditional (SC-AIRL)pϕ(h∣st)p_\phi(h\mid s_t)1.4860.6500.0840.0330.183–

Mean |ΔAction| and outcome gap come from rollouts (Table 2 protocol); outcome gap is the total variation distance between the model's and the participant's success, collision and timeout rates.

Q5. How does SC-AIRL compare with a parameter-matched MoE-AIRL model in which the expert policies use the same planning horizon?

A. SC-AIRL fits better than MoE-AIRL for 52/52 (NLL) and 50/52 (Top-1) participants. We trained a parameter-matched MoE-AIRL in which all four expert policies use the same, full planning horizon under the Standard AIRL objective (MoE-AIRL 52,166 vs. SC-AIRL 51,173 parameters).

Pedestrian, held-out, N = 52Difference from SC-AIRLNLL ↓Top-1 ↑SC-AIRL better (NLL / Top-1)
SC-AIRL (revised)–1.4860.650–
MoE-AIRLSame planning horizon for all experts6.4610.59452 / 50

Q6. Could the authors provide more extensive evaluations across random seeds and report consistent uncertainty estimates?

A. The performance improvements of SC-AIRL exceed the variability observed across random seeds in the adversarial baselines, with participant-level 95% confidence intervals (CIs) for the performance differences excluding zero for all baselines. We retrained GAIL and Option-GAIL with three seeds and report the SD across training runs below.

Held-out, mean ± SD over training runsPedestrian NLLPedestrian Top-1Highway NLLHighway Top-1
SC-AIRL (revised)*1.4860.650––
SC-AIRL (submitted)1.579 ± 0.0260.647 ± 0.0041.505 ± 0.0680.718 ± 0.002
Episode-AIRL (submitted heads)1.656 ± 0.0270.617 ± 0.0031.742 ± 0.0590.675 ± 0.001
Single-hh2.109 ± 0.0520.610 ± 0.0022.681 ± 0.0160.654 ± 0.005
Standard AIRL15.67 ± 1.000.549 ± 0.0054.422 ± 0.0520.629 ± 0.006
GAIL14.22 ± 1.410.460 ± 0.0045.761 ± 0.1070.611 ± 0.003
Option-GAIL15.54 ± 1.670.363 ± 0.00915.35 ± 0.420.172 ± 0.007

*One training run. Highway Standard AIRL has two runs and every other row has three. Both SC-AIRL rows use the 500-step router.

On the highway task, the Option-GAIL policy selects IDLE, the action participants choose most often, in 14% of steps (GAIL 63%), which places its Top-1 below 1/5.

Pedestrian, revised SC-AIRL − baseline (95% CI over participants)NLLTop-1
Episode-AIRL−0.170 [−0.213, −0.127]0.033 [0.025, 0.041]
Single-hh−0.623 [−0.694, −0.556]0.040 [0.032, 0.048]
Standard AIRL−14.19 [−16.73, −11.76]0.100 [0.089, 0.111]
GAIL−12.73 [−15.68, −9.99]0.190 [0.179, 0.201]
Option-GAIL−14.06 [−15.89, −12.38]0.287 [0.272, 0.303]

Q4. Could the authors provide evaluations on additional environments, such as established robotic control or sequential decision-making benchmarks, to demonstrate the generalizability of SC-AIRL?

A. To our knowledge, we identified only two publicly available datasets containing both human demonstrations and psychometric measures (e.g., STAI and BIS), which are required for validating the planning depths that SC-AIRL infers. In future work, we plan to collect human demonstrations along with participants' psychometric measures in additional environments, including sequential decision-making benchmarks, to further evaluate the generalizability of SC-AIRL.

Thank you for the detailed response and clarifications. However, I am still not convinced that the paper’s contribution is sufficiently novel. It is still unclear why "routing weights can be interpreted as planning depth." Additionally, the evaluations are still limited.

I’ll keep my current score for now and consider the other reviewers’ feedback and the discussion before finalizing my rating.

AuthorsOpenReview ↗

Follow-up Response to Reviewer AFgk

We sincerely thank the reviewer for the thoughtful response and continued consideration of our work. We would be grateful if the reviewer could consider the following two points during the final discussion and evaluation.

Applicational novelty. SC-AIRL enables us to model how individuals dynamically adjust their planning horizons according to the current state. This allows us to characterize how these state-dependent adaptations differ across individuals in naturalistic decision-making tasks and how they relate to individual psychological traits.

Interpretation of planning depth. As shown in the table in our Response to Reviewer GEFp (1/2), the horizon-specific policies exhibit intuitive differences in action selection: longer-horizon policies act more cautiously, moving forward less frequently and moving backward or sideways more frequently, with lower collision rates and higher success rates in rollouts. The fitted router identifies the state-dependent combination of these policies that best explains each participant’s behavior. We therefore interpret the routing weights as a model-based proxy for dynamic planning depth.

We are very grateful for the reviewer’s constructive feedback, which has helped us strengthen our work.

Reviewer_xAsNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work introduces SC-AIRL (State-Conditional Adversarial Inverse Reinforcement Learning), an approach for explaining boundedly rational behavior in computationally constrained agents. The key motivation behind the study rests on the premise that humans can not plan over long horizons, especially for complex real-world tasks, and inverse RL approaches that do not account for such cognitive limitations do not accurately explain human behavior in such tasks. SC-AIRL uses a state-dependent mixed policy that weights the action probabilities of several policies that are trained to plan over distinct planning horizons, to model human behavior in complex tasks.

The authors claim novelty in their operationalization of limited horizon planning. First, SC-AIRL limits planning during training by summing rewards explicitly up to the specified horizon, as opposed to prior approaches that achieve this by scaling the rewards with agent-specific discount factors, which down-weight future rewards but never truncate it. Second, this approach uses a mixed policy that weights the action probabilities of policies trained on different planning horizons, with weights that depend on the current state, allowing a state-dependent model of foresight.

Critical strengths

In my view, the greatest strength of the model lies in its ability to model agents that dynamically adapt their behavioral parameters to their active task context, which the model achieves through state-dependent planning horizons.

Critical weaknesses

For the pedestrian task, Table 2's held-out results use a router stopped early on a validation split with NLL= 1.499, while in Section 6.3, the authors use a router fitted for 10,000 steps with NLL= 2.197, which suggests that the 10,000-step router overfits the training data. Additionally, Table 34 results show no significant correlations for the early-stopped router. The authors claim that the fit length was chosen by seed reliability of the measures. However, I am not convinced that seed reliability is a valid criterion for choosing an overfitted model to evaluate trait correlations. The authors need to better justify their methodological approach here.

Questions for authors

See weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer xAsN

Thank you for your positive evaluation of our paper and your insightful feedback.

W1. Is seed reliability a valid criterion for choosing an overfitted (10,000-step) router to evaluate trait correlations?

A. We now select the router fit length using a fixed criterion based on imitation performance and depth-measure reliability. At the selected fit length, the revised SC-AIRL’s depth measures remain significantly correlated with psychometric scores (I(s;h)I(s;h) × STAI rs=−.40r_s = -.40, p=.003p = .003; score-controlled Δh\Delta h × BIS-motor rs=−.33r_s = -.33, p=.018p = .018; see our response to Reviewer GEFp for the revised SC-AIRL). The criterion has two steps:

  1. Prediction performance on the validation set. Retain lengths whose NLL on the validation split (a random 20% of training-split transitions withheld from the router fit) is below that of a uniform mixture of the same per-depth policies. This step excludes every pedestrian fit from 1,000 steps on, including the 10,000-step fit, and the 2,000-step highway fit.
  2. Reliability for individual differences. Among these, select the length with the highest cross-seed reliability of the two depth measures (ICC(A,k) across three training seeds), so that the depth measures capture individual differences that are stable across training seeds.

Neither step uses psychometric scores or held-out data. We applied this rule to the three training runs of the submitted model, which select 500 steps for the pedestrian task and 600 steps for the highway task. We use these lengths for the revised model and will re-apply the rule when its remaining training runs finish.

Model (task, fit length)Held-out NLL ↓Top-1 ↑I(s;h)I(s;h) × STAIΔh\Delta h × BIS-motor
Revised SC-AIRL, seed 42 (pedestrian, 500)1.4860.650−.40 (.003) / −.37 (.007)−.23 (.095) / −.33 (.018)
Submitted SC-AIRL, 3-seed mean (pedestrian, 500)1.5790.647−.30 (.033) / −.23 (.108)−.18 (.214) / −.25 (.080)
Submitted SC-AIRL, 3-seed mean (highway, 600)1.5050.718n/a+.29 (.048) / +.42 (.003)

Correlations are Spearman rsr_s (pp), raw / score-controlled; STAI was not collected on the highway task.

Fit-length sweep and held-out states. For the revised model, both correlations keep their sign at every fit length from 250 steps on, including when restricted to held-out states.

Revised SC-AIRL, router fitValidation NLLHeld-out NLLI(s;h) × STAI, raw / controlledΔh × BIS-motor, raw / controlledHeld-out only: I(s;h) × STAIHeld-out only: Δh × BIS-motor
Early-stopped–1.391+0.10 (.475) / +0.16 (.246)−0.26 (.067) / −0.39 (.005)+0.15 (.301) / +0.21 (.135)−0.29 (.035) / −0.37 (.007)
250 steps1.3461.432−0.35 (.012) / −0.32 (.023)−0.32 (.022) / −0.42 (.002)−0.26 (.061) / −0.22 (.117)−0.27 (.049) / −0.39 (.004)
500 steps1.3881.486−0.40 (.003) / −0.37 (.007)−0.23 (.095) / −0.33 (.018)−0.44 (.001) / −0.39 (.004)−0.24 (.089) / −0.33 (.018)
1,000 steps1.4841.585−0.43 (.002) / −0.40 (.003)−0.27 (.054) / −0.33 (.016)−0.44 (.001) / −0.38 (.006)−0.29 (.036) / −0.34 (.013)
2,500 steps1.6561.787−0.42 (.002) / −0.39 (.005)−0.31 (.026) / −0.39 (.005)−0.36 (.008) / −0.30 (.028)−0.26 (.066) / −0.30 (.029)
5,000 steps1.8081.970−0.39 (.005) / −0.36 (.009)−0.33 (.018) / −0.39 (.004)−0.36 (.010) / −0.32 (.022)−0.24 (.085) / −0.30 (.033)
10,000 steps1.9712.153−0.42 (.002) / −0.39 (.004)−0.32 (.019) / −0.39 (.004)−0.39 (.005) / −0.33 (.015)−0.26 (.062) / −0.31 (.027)
Uniform mixture1.4251.504––––

Router recovery. With the per-depth policies fixed, the 500-step router yields stable estimates of both depth measures: refitting a 500-step router to behavior generated from each participant's router, with as many episodes as the participant, recovers the participant ranking of Δh\Delta h (ρ=0.81\rho = 0.81) and I(s;h)I(s;h) (ρ=0.75\rho = 0.75).

We will incorporate these changes into the revised manuscript.

Reviewer_GEFpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors study an extension of adversarial inverse reinforcement learning (AIRL) Fu et al., 2018. The method specifies a finite set H of truncation horizons. For each horizon h in H, one policy is trained.

  • The advantage used in the policy update is computed by generalised advantage estimation Schulman et al., 2016. GAE estimates the advantage of an action as an exponentially weighted sum of one-step temporal-difference errors.
  • The authors truncate this sum after h steps. Rewards within the next h steps therefore enter the estimate directly, while returns beyond step h enter only through a value network trained separately for each horizon. The submission truncates this sum after h steps, so explicit reward enters over at most h steps. A horizon-specific value network estimates the return beyond the window.
  • All policies share a single reward network trained with the AIRL discriminator, with the potential-based shaping term removed Φ≡0\Phi \equiv 0. During this stage, the policies are mixed with equal weights.

After training, the policies and reward are frozen. The agent’s policy is then defined as a state-dependent weighted average of the per-horizon policies. The weights are produced by a neural network that maps the state to a softmax distribution over H. Its parameters are fitted by maximising the log-likelihood of the expert’s actions under the resulting mixture.

Critical strengths
  • The question of whether the horizon of human foresight varies with the state, and whether such variation relates to individual traits, is well motivated.
  • Bounded foresight is represented as truncation of the explicit reward sum rather than as a change of discount rate, and the horizon weights may vary within a trajectory.
  • The paper is clearly written. It includes multiple training seeds and paired per-participant tests.
Critical weaknesses
  • Removing the shaping term places the demonstrator's foresight inside the reward, which runs against the paper's aim. At the AIRL equilibrium, the discriminator output equals the expert's soft advantage, log⁡πE(a∣s)=Q⋆(s,a)−V⋆(s)\log \pi_E(a \mid s) = Q^{\star}(s,a) - V^{\star}(s), which already encodes the expert's evaluation of the future (Fu et al., 2018). The shaping potential Φ\Phi is the component that would separate a one-step reward from this value term. With Φ≡0\Phi \equiv 0, the reward network must represent the advantage itself. In the family of rewards consistent with a policy, r(s,a)=log⁡π(a∣s)+v(s)−γE[v(s′)∣s,a]r(s,a) = \log \pi(a \mid s) + v(s) - \gamma \mathbb{E}[v(s') \mid s,a] for an arbitrary vv (Cao, Cohen and Szpruch, "Identifiability in inverse reinforcement learning", NeurIPS 2021), this selects v≡0v \equiv 0: the reward under which a one-step (γ=0\gamma = 0) agent reproduces the expert's policy. The participant's horizon is thus absorbed into rθr_{\theta}, and the per-horizon policies sum an advantage-like quantity over further steps. The inferred horizon therefore cannot be interpreted as the participant's foresight. This contradicts the stated rationale that rθr_\theta carries only horizon-invariant signal (Section 4.2).

  • The reward and the horizon weights are not identified. A single policy determines the reward only up to the arbitrary value function vv, and identification requires observing policies separately under at least two discount factors or transition dynamics (Cao et al., 2021). The method observes only the mixture πmix(a∣s)=∑h∈Hpϕ(h∣s)πh(a∣s)\pi_{\mathrm{mix}}(a \mid s) = \sum_{h \in H} p_\phi(h \mid s) \pi^h(a \mid s), so rθr_\theta, pϕ(h∣s)p_\phi(h \mid s) and the value networks VhV^h are jointly underdetermined.

  • The per-horizon policies do not implement bounded foresight. Each πh\pi^h is updated with the truncated advantage At(h)=∑k=0h−1(γλ)kδt+kA_t^{(h)} = \sum_{k=0}^{h-1} (\gamma\lambda)^k \delta_{t+k}, where δt=rθ(st,at)+γVh(st+1)−Vh(st)\delta_t = r_\theta(s_t,a_t) + \gamma V^h(s_{t+1}) - V^h(s_t) and VhV^h is trained to predict the full discounted return of πh\pi^h (γ=0.99\gamma = 0.99). As the authors note (Appendix B), if VhV^h equals the on-policy value, then E[At(h)∣st,at]\mathbb{E}[A_t^{(h)} \mid s_t,a_t] equals the full-horizon advantage for every hh. All πh\pi^h therefore optimise the same infinite-horizon objective; hh changes only how the advantage is estimated, and differences between the πh\pi^h arise from errors in VhV^h, not from different foresight. Proposition 1 concerns the weights inside the estimator, not the objective it targets, and does not address this.

  • The theoretical justification applies Fu et al. (2018) outside its conditions. That result recovers the reward up to shaping only for a state-only reward r(s)r(s) under a decomposability assumption with deterministic dynamics; here rθ(s,a)r_\theta(s,a) depends on the action and the dynamics are stochastic.

  • The policy updates add the environment's terminal reward to rθr_\theta (Appendix D), so behaviour is not modelled by the inferred reward alone. Part of the incentive is the known task score propagated over the truncated horizon, which confounds the inferred horizon with the task's reward structure.

Questions for authors
  1. With Φ≡0\Phi \equiv 0, the reward network must represent the expert's soft advantage log⁡πE(a∣s)=Q⋆(s,a)−V⋆(s)\log \pi_E(a \mid s) = Q^{\star}(s,a) - V^{\star}(s), which already contains the expert's evaluation of the future. How can a horizon inferred from policies trained on this quantity be interpreted as the participant's foresight? How do the results change if Φ\Phi is retained?

  2. Since E[At(h)∣st,at]\mathbb{E}[A_t^{(h)} \mid s_t,a_t] equals the full-horizon advantage for every hh when VhV^h is accurate, what distinguishes the policies πh\pi^h other than errors in VhV^h? How do the πh\pi^h and the fitted weights change as VhV^h is trained to convergence, and how do the results compare with policies that maximise only the hh-step return ∑k=0h−1γkrθ(st+k,at+k)\sum_{k=0}^{h-1} \gamma^k r_\theta(s_{t+k},a_{t+k})?

  3. Under which conditions do the authors expect the result of Fu et al. (2018) to apply, given that rθ(s,a)r_\theta(s,a) depends on the action and the pedestrian dynamics are stochastic?

2 comments
AuthorsOpenReview ↗

Response to Reviewer GEFp (1/2)

We address your questions below and appreciate the chance to strengthen the theoretical basis and interpretation of our work.

Q2 (W3). What distinguishes the policies (πh\pi^h) other than errors in VhV^h? What are the results when the policies maximise only the h-step return?

A. We revised the model as suggested, replacing the bootstrapped GAE-h objective with the explicitly truncated hh-step return (Eq. 1): each policy (πh\pi^h) now maximizes Gt(h)G_t^{(h)}, and VhV^h, which is fitted to Gt(h)G_t^{(h)}, serves only as a baseline. On held-out pedestrian episodes, the revised SC-AIRL (seed 42) maintains higher imitation fidelity than the other adversarial imitation learning baselines (Top-1 accuracy 0.650 vs. 0.614 for Episode-AIRL on the same heads and 0.549 for Standard AIRL, a three-seed mean). The associations between depth measures and psychometric measures remain almost identical (I(s;h)I(s;h) × STAI rs=−.40r_s = -.40, p=.003p = .003; Δh\Delta h × BIS-motor rs=−.23r_s = -.23, p=.095p = .095, or rs=−.33r_s = -.33, p=.018p = .018 after controlling for task score).

G_t^{(h)}=\sum_{k=0}^{h-1}\gamma^k r_\theta(s_{t+k},a_{t+k}),\qquad A_t^{(h)} = G_t^{(h)} - V^h(s_t) \tag{1}

The change in the participant ranking of depth measures between the submitted and revised models is comparable to that between two training seeds of the submitted model (rank agreement 0.51 vs. 0.53 for Δh\Delta h, 0.48 vs. 0.45 for I(s;h)I(s;h)). We are training the revised SC-AIRL on the highway task and will post the results in this thread when training finishes.

In the revised model, the per-depth policies act more cautiously as hh increases, which supports interpreting the router's weights over them as a proxy for planning depth. In held-out states, deeper policies move forward less (UP 0.615, 0.599, 0.504, and 0.470 for h=2,3,5,h = 2, 3, 5, and 1010) and step back or sideways more (DOWN: 0.003, 0.002, 0.028, 0.043, RIGHT 0.087, 0.083, 0.133, 0.155, and LEFT 0.093, 0.089, 0.122, 0.137 for h=2,3,5,h = 2, 3, 5, and 1010). In rollouts of each policy, deeper policies succeed more often (0.370, 0.345, 0.618, 0.747) and collide less (0.623, 0.647, 0.372, 0.229 for h=2,3,5,h = 2, 3, 5, and 1010).

Pedestrian, held-out, N = 52NOTHINGUPDOWNRIGHTLEFTSuccessCollisionTimeout
h=2h=20.2010.6150.0030.0870.0930.3700.6230.007
h=3h=30.2280.5990.0020.0830.0890.3450.6470.008
h=5h=50.2130.5040.0280.1330.1220.6180.3720.010
h=10h=100.1960.4700.0430.1550.1370.7470.2290.024
Mixture (router)0.2320.5200.0190.1150.1140.6020.3960.002

W3. Proposition 1 concerns the weights inside the estimator, not the objective it targets, and does not address this.

A. Each policy now maximizes the truncated h-step return, so Proposition 1 now characterizes the reward weighting of the policy objective itself.

AuthorsOpenReview ↗

Response to Reviewer GEFp (2/2)

Q1 (W1). How can the inferred horizon be interpreted as foresight when the reward may already encode future evaluation? With Φ=0\Phi = 0, the reward network must represent the expert's soft advantage. How do the results change if Φ\Phi is retained?

A. Interpretation of the inferred depth. We interpret the inferred depth as a proxy for foresight because all policies share the same reward function and architecture, differing only in how far ahead they optimize the reward (hh; Eq. 1). This design attributes state-dependent behavioral variation to differences in how far ahead the shared reward is accumulated, rather than to different rewards.

Reward alone vs. soft advantage. The learned reward does not simply encode the expert's action preferences as a soft advantage would. If rθr_\theta equaled the soft advantage, softmax(rθ(s,⋅))\text{softmax}(r_\theta(s,\cdot)) would reproduce the participants' choices; instead its Top-1 is lower than that of the policy (0.549 vs. 0.650; 50 of 52 participants).

Pedestrian, held-out, N = 52NLL ↓Top-1 ↑KL ↓Cells agreeing with participants / with the policy (of 10)
SC-AIRL (revised)reward alone1.2560.5490.1856 / 8
policy1.4860.6500.0848 / –

The reward alone has the lower mean NLL because of the 12% of steps in which either the policy or the reward gives the expert's action a probability below 1%; the policy has the lower NLL on the remaining steps (0.676 vs. 0.908; 50 of 52 participants) and the lower KL (0.084 vs. 0.185). Still, these results do not exclude the possibility that rθr_\theta also encodes part of the future evaluation, so we treat the inferred depth as a proxy rather than a direct measure of foresight.

Retaining Φ\Phi. Retaining the shaping term of Fu et al. (2018) leaves imitation comparable and weakens the psychometric associations in the same direction: Top-1 .650 in both, NLL 1.602 vs. 1.486, and I(s;h)I(s;h) × STAI rs=−.25r_s = -.25 vs. −.40-.40 (held-out pedestrian episodes, seed 42). A stationary Φ\Phi can absorb part of the state-dependent foresight that the router captures, because the potential form γΦ(s′)−Φ(s)\gamma\Phi(s')-\Phi(s) matches a demonstrator optimal under exponential discounting (Fu et al., 2018), not a bounded-horizon planner (Skalse & Abate, 2025). The reward itself barely changes: its correlation with the reward learned without Φ\Phi matches that between two training seeds (r=0.78r = 0.78 vs. 0.80). We used

f(s,a,s')=r_\theta(s,a)+\gamma(1-d)\,\Phi(s')-\Phi(s), \tag{2}

where dd marks episode termination.

Δh\Delta h × BIS-motor, raw / controlledI(s;h)I(s;h) × STAI, raw / controlled
SC-AIRL (revised)−0.23 (p = .095) / −0.33 (p = .018)−0.40 (p = .003) / −0.37 (p = .007)
SC-AIRL (revised) + Φ\Phi−0.19 (p = .181) / −0.21 (p = .136)−0.25 (p = .071) / −0.21 (p = .144)
Rank agreement between the two models0.460.31
Rank agreement between two training seeds (submitted model)0.530.45

Future direction. We therefore propose a horizon-dependent potential for future work:

f_h(s,a,s')=r_\theta(s,a)+\gamma(1-d)\,\Phi_{h-1}(s')-\Phi_h(s), \tag{3}

where Φk\Phi_k is shared across heads and indexed by the number of steps remaining in the planning horizon.

Q3 (W4). Under which conditions do the authors expect the result of Fu et al. (2018) to apply, given that rθ(s,a)r_{\theta}(s,a) depends on the action and the pedestrian dynamics are stochastic?

A. Given the differences between our settings and those of Fu et al. (2018), we do not assume that all of their theoretical results apply directly to ours. In future work, we plan to investigate a state-only reward formulation, r(s)r(s), using alternative datasets with deterministic dynamics to assess whether the theoretical results of Fu et al. (2018) could be applied.

W2. The reward and the horizon weights are not identified.

A. Although the reward and horizon weights are not uniquely identified from the mixed policy, we interpret the inferred depth as a model-derived proxy for foresight (see Q1). Its associations with psychometric measures further support this interpretation (Q2 above; Sec. 6.3). Formal identification is future work.

W5. The policy updates add the environment's terminal reward to rθr_{\theta}, so behaviour is not modelled by the inferred reward alone. Part of the incentive is the known task score propagated over the truncated horizon, which confounds the inferred horizon with the task's reward structure.

A. We thank the reviewer for pointing out this limitation. We will include it as a limitation of our study. We have included the terminal reward as it is computed from two task variables (the remaining time on the display and the collision severity indicated by vehicle color) that are included in the state used for reward inference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.