- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SC-AIRL, an extension of AIRL that incorporates state-dependent planning depth through a mixture-of-experts architecture. Evaluated on held-out episodes from pedestrian-crossing and highway-overtaking tasks, SC-AIRL improves over baselines.
Critical strengths
- MoE + AIRL seems to be interesting
Critical weaknesses
- limited methodological novelty
- missing motivation and necessity of MoE
- missing enough justification of assumptions
- limited evaluations
Questions for authors
Limited methodological novelty.
SC-AIRL appears to be a relatively straightforward extension of AIRL by incorporating a state-conditioned routing architecture.
A state-conditional router mixes them according to the current state.
- Could the authors clarify the methodological novelty compared with existing approaches, particularly InfoGAIL, Option-GAIL, and other multi-policy imitation learning methods?
1.1 What are the major theoretical contributions of the proposed framework? How do they differ from those of existing GAIL or AIRL approaches?
4.3 MULTI-POLICY ARCHITECTURE Under Assumption (A2) ....
1.2 GAE truncation primarily controls temporal credit assignment during policy optimization and does not necessarily correspond to human planning depth.
Could the authors clarify the theoretical justification for interpreting different GAE truncation horizons as different levels of human foresight? What empirical evidence supports this interpretation?
- Routing / MoE architectures have been extensively explored in LLMs. What specifically motivates the use of MoE in the proposed framework, and why is state-conditioned routing necessary for the two considered tasks? Could similar improvements be achieved using a single policy with state-dependent planning parameters or a simpler multi-policy architecture?
missing enough justification of assumptions
3.1 ASSUMPTIONS (A1) Shared reward. A set of depth-indexed policies ... shares a single reward ... (A2) Bounded foresight. (A3) State-conditional depth.
- Insufficient justification of assumptions. The assumptions in Section 3.1 (A1–A3) require further theoretical or empirical justification.
In particular, 3.1 Why should different planning depths share an identical reward function?
3.2 How can the inferred routing weights be interpreted as human planning depth rather than general policy specialization? What is the relationship between routing weights and human planning?
Evaluations are limited
- The experiments are restricted to two simulator-based tasks involving pedestrian crossing and highway overtaking. Could the authors provide evaluations on additional environments, such as established robotic control (e.g., Pendulum, Ant, Swimmer, Half-Cheetah, the ones evaluated in GAIL [1] or AIRL[2] ) or sequential decision-making benchmarks, to demonstrate the generalizability of SC-AIRL?
Option-GAIL adds ... Single-h AIRL uses ... Episode-AIRL uses the same per-depth policies ...
SC-AIRL has 14,101 parameters versus 3,472 for Standard AIRL on the highway task (Appendix G, Table 14)
- Missing important ablation studies. Although the paper includes comparisons with Single-h AIRL, Episode-AIRL, further controlled experiments would help distinguish the effects of state-dependent foresight from increased model capacity and policy diversity.
In particular, how does SC-AIRL compare with a parameter-matched MoE-AIRL model in which the expert policies use the same planning horizon? How can the authors demonstrate that the improvements specifically result from modeling different planning depths rather than simply combining multiple independently trained policies?
Furthermore, SC-AIRL has a substantially larger total parameter count than Standard AIRL. Could the authors clarify how this difference affects the performance comparison?
Table 2: Held-out imitation fidelity on both tasks.
- Table 2 reports results averaged over a small number of training runs, while some baselines are evaluated using only a single run. Why? Could the authors provide more extensive evaluations across random seeds and report consistent uncertainty estimates, such as standard deviations or confidence intervals for the main performance improvements, to better establish their statistical robustness?
[1] "Generative adversarial imitation learning." Advances in neural information processing systems 29 (2016).
[2] "Learning robust rewards with adversarial inverse reinforcement learning." arXiv preprint arXiv:1710.11248 (2017).
4 comments
Response to Reviewer AFgk (2/2)
Evaluation
Q2. Could a simpler multi-policy architecture achieve similar improvements?
A. As seen in the table below, state-conditional weights outperform simpler multi-policy alternatives on most metrics: achieving the highest Top-1 and the lowest KL, mean |ΔAction|, and outcome gap under the same shared reward and per-depth policies. State-independent weights perform better only on NLL. This comes from the 10.5% of held-out steps in which either model assigns the expert's action a probability below 1% (SC-AIRL 8.32 vs. 7.79), and on the remaining 89.5% of steps the two are on par (SC-AIRL 0.704 vs. 0.711).
| Pedestrian, held-out, N = 52 | in | NLL ↓ | Top-1 ↑ | KL ↓ | Mean |ΔAction| ↓ | Outcome gap ↓ | SC-AIRL better (of 52) |
|---|---|---|---|---|---|---|---|
| Uniform | 1.504 | 0.633 | 0.111 | 0.055 | 0.197 | 32 / 38 / 48 / 46 / 29 | |
| State-independent | 1.429 | 0.634 | 0.109 | 0.054 | 0.197 | 9 / 42 / 47 / 47 / 32 | |
| One depth per participant | 1.895 | 0.608 | 0.113 | 0.041 | 0.198 | 52 / 50 / 40 / 38 / 31 | |
| Fixed per episode (Episode-AIRL) | 1.574 | 0.614 | 0.114 | 0.046 | 0.215 | 41 / 50 / 43 / 44 / 33 | |
| State-conditional (SC-AIRL) | 1.486 | 0.650 | 0.084 | 0.033 | 0.183 | – |
Mean |ΔAction| and outcome gap come from rollouts (Table 2 protocol); outcome gap is the total variation distance between the model's and the participant's success, collision and timeout rates.
Q5. How does SC-AIRL compare with a parameter-matched MoE-AIRL model in which the expert policies use the same planning horizon?
A. SC-AIRL fits better than MoE-AIRL for 52/52 (NLL) and 50/52 (Top-1) participants. We trained a parameter-matched MoE-AIRL in which all four expert policies use the same, full planning horizon under the Standard AIRL objective (MoE-AIRL 52,166 vs. SC-AIRL 51,173 parameters).
| Pedestrian, held-out, N = 52 | Difference from SC-AIRL | NLL ↓ | Top-1 ↑ | SC-AIRL better (NLL / Top-1) |
|---|---|---|---|---|
| SC-AIRL (revised) | – | 1.486 | 0.650 | – |
| MoE-AIRL | Same planning horizon for all experts | 6.461 | 0.594 | 52 / 50 |
Q6. Could the authors provide more extensive evaluations across random seeds and report consistent uncertainty estimates?
A. The performance improvements of SC-AIRL exceed the variability observed across random seeds in the adversarial baselines, with participant-level 95% confidence intervals (CIs) for the performance differences excluding zero for all baselines. We retrained GAIL and Option-GAIL with three seeds and report the SD across training runs below.
| Held-out, mean ± SD over training runs | Pedestrian NLL | Pedestrian Top-1 | Highway NLL | Highway Top-1 |
|---|---|---|---|---|
| SC-AIRL (revised)* | 1.486 | 0.650 | – | – |
| SC-AIRL (submitted) | 1.579 ± 0.026 | 0.647 ± 0.004 | 1.505 ± 0.068 | 0.718 ± 0.002 |
| Episode-AIRL (submitted heads) | 1.656 ± 0.027 | 0.617 ± 0.003 | 1.742 ± 0.059 | 0.675 ± 0.001 |
| Single- | 2.109 ± 0.052 | 0.610 ± 0.002 | 2.681 ± 0.016 | 0.654 ± 0.005 |
| Standard AIRL | 15.67 ± 1.00 | 0.549 ± 0.005 | 4.422 ± 0.052 | 0.629 ± 0.006 |
| GAIL | 14.22 ± 1.41 | 0.460 ± 0.004 | 5.761 ± 0.107 | 0.611 ± 0.003 |
| Option-GAIL | 15.54 ± 1.67 | 0.363 ± 0.009 | 15.35 ± 0.42 | 0.172 ± 0.007 |
*One training run. Highway Standard AIRL has two runs and every other row has three. Both SC-AIRL rows use the 500-step router.
On the highway task, the Option-GAIL policy selects IDLE, the action participants choose most often, in 14% of steps (GAIL 63%), which places its Top-1 below 1/5.
| Pedestrian, revised SC-AIRL − baseline (95% CI over participants) | NLL | Top-1 |
|---|---|---|
| Episode-AIRL | −0.170 [−0.213, −0.127] | 0.033 [0.025, 0.041] |
| Single- | −0.623 [−0.694, −0.556] | 0.040 [0.032, 0.048] |
| Standard AIRL | −14.19 [−16.73, −11.76] | 0.100 [0.089, 0.111] |
| GAIL | −12.73 [−15.68, −9.99] | 0.190 [0.179, 0.201] |
| Option-GAIL | −14.06 [−15.89, −12.38] | 0.287 [0.272, 0.303] |
Q4. Could the authors provide evaluations on additional environments, such as established robotic control or sequential decision-making benchmarks, to demonstrate the generalizability of SC-AIRL?
A. To our knowledge, we identified only two publicly available datasets containing both human demonstrations and psychometric measures (e.g., STAI and BIS), which are required for validating the planning depths that SC-AIRL infers. In future work, we plan to collect human demonstrations along with participants' psychometric measures in additional environments, including sequential decision-making benchmarks, to further evaluate the generalizability of SC-AIRL.
Thank you for the detailed response and clarifications. However, I am still not convinced that the paper’s contribution is sufficiently novel. It is still unclear why "routing weights can be interpreted as planning depth." Additionally, the evaluations are still limited.
I’ll keep my current score for now and consider the other reviewers’ feedback and the discussion before finalizing my rating.
Follow-up Response to Reviewer AFgk
We sincerely thank the reviewer for the thoughtful response and continued consideration of our work. We would be grateful if the reviewer could consider the following two points during the final discussion and evaluation.
Applicational novelty. SC-AIRL enables us to model how individuals dynamically adjust their planning horizons according to the current state. This allows us to characterize how these state-dependent adaptations differ across individuals in naturalistic decision-making tasks and how they relate to individual psychological traits.
Interpretation of planning depth. As shown in the table in our Response to Reviewer GEFp (1/2), the horizon-specific policies exhibit intuitive differences in action selection: longer-horizon policies act more cautiously, moving forward less frequently and moving backward or sideways more frequently, with lower collision rates and higher success rates in rollouts. The fitted router identifies the state-dependent combination of these policies that best explains each participant’s behavior. We therefore interpret the routing weights as a model-based proxy for dynamic planning depth.
We are very grateful for the reviewer’s constructive feedback, which has helped us strengthen our work.
Response to Reviewer AFgk (1/2)
We thank the reviewer for the insightful and comprehensive feedback. The comments have been very helpful in thoroughly revisiting the theoretical foundations of our work and refining our methodology.
Assumptions
Q1.2. GAE truncation does not necessarily correspond to human planning depth.
A. We thank the reviewer for this important comment. In response, we revised the model so that h better represents a bounded planning horizon. Now, each policy explicitly optimizes an h-step return (Eq. 1) rather than GAE truncated at h steps (Eq. 2).
G_t^{(h)}=\sum_{k=0}^{h-1}\gamma^k r_\theta(s_{t+k},a_{t+k}),\qquad A_t^{(h)} = G_t^{(h)} - V^h(s_t) \tag{1}
\delta_t^{(h)} = r_\theta(s_t,a_t) + \gamma V^h(s_{t+1}) - V^h(s_t),\qquad A_t^{(h)} = \sum_{k=0}^{\min(h,T-t)-1}(\gamma\lambda)^k\delta_{t+k}^{(h)} \tag{2}
Q3. Justify assumptions in Section 3.1 (A1-A3)
Q2. Why are MoE and state-conditioned routing necessary?
A. The motivation and assumptions underlying SC-AIRL are grounded in prior work on human decision-making and bounded rationality. We have cited relevant references to support each assumption.
Q3.2. Justify interpretation of the inferred routing weights as human planning depth.
A. The heads share the same reward and architecture and differ only in their planning horizons (h). Thus, we believe the routing weights can be interpreted as proxies for planning depth.
Novelty
Q1. Clarify the methodological novelty compared with existing approaches, particularly InfoGAIL, Option-GAIL, and other multi-policy imitation learning methods.
Q1.1. What are the major theoretical contributions of the proposed framework? How do they differ from those of existing GAIL or AIRL approaches?
A. SC-AIRL’s key contribution is to explicitly represent individual differences in planning depth within an inverse reinforcement learning framework, guided by principles of human bounded rationality from the assumptions above.
[1] "Adaptive integration of habits into depth-limited planning defines a habitual-goal-directed spectrum." Proceedings of the National Academy of Sciences 113.45 (2016): 12868-12873.
[2] "Bonsai trees in your head: How the Pavlovian system sculpts goal-directed choices by pruning decision trees." PLoS Computational Biology 8.3 (2012): e1002410.
[3] "Planning in the brain." Neuron 110.6 (2022): 914-934.
[4] "Multi-task hierarchical adversarial inverse reinforcement learning." Proceedings of the 40th International Conference on Machine Learning, PMLR 202 (2023): 4895-4920.
[5] "Inverse reinforcement learning with multiple planning horizons." Reinforcement Learning Journal 3 (2024).
[6] "Measuring individual differences in the depth of planning." PsyArXiv preprint (2022).