- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies inverse reward learning in two-player Stackelberg games, focusing on recovering rewards that support effective downstream strategic decision-making rather than exact reward identification. The authors first establish the limitations of learning from a fixed policy and derive a minimax lower bound on Stackelberg sub-optimality. They then propose an active IRL algorithm that combines maximum-likelihood reward estimation with Fisher-information-based experimental design. Under linear reward parametrization and entropy-regularized follower responses, the algorithm achieves a finite-sample sub-optimality guarantee with an optimal dependence on the number of online interactions. Experiments on an anti-poaching benchmark and synthetic games demonstrate improvements in reward learning and downstream strategic performance.
Critical strengths
S1. Well-motivated problem formulation. The paper focuses on learning rewards that preserve strategic performance rather than exactly recovering the true rewards. The impossibility result for fixed-policy observations also provides a clear motivation for active exploration.
S2. Solid theoretical contributions. The paper connects reward estimation, Fisher information, and Stackelberg sub-optimality through the reward transfer lemma. The analysis is technically interesting, particularly the use of self-concordance and experimental design.
S3. Finite-sample guarantees. The authors establish both upper and lower bounds on Stackelberg sub-optimality, matching the rate up to logarithmic factors. This provides theoretical support for the proposed active learning approach.
S4. Encouraging experimental results. The experiments show improvements over random exploration baselines, especially when transferring the learned rewards to different leader objectives. The additional synthetic experiments help illustrate the benefits of active exploration.
Critical weaknesses
W1. Gap between theoretical bounds and practical sample complexity. The upper and lower bounds match in their dependence on , but not on the dimension . Moreover, the theoretical warm-up requires exponentially many samples in , while the experiments use much smaller warm-up budgets. Can the authors clarify the total sample complexity, including the warm-up phase? Is the exponential dependence on necessary, and can the dimensional gap between the upper and lower bounds be improved?
W2. Computational complexity. The algorithm requires optimizing the Fisher-information-based objective over leader policies at each iteration. Since this optimization is generally nonconvex, it is unclear how efficiently it can be solved and whether approximate solutions preserve the theoretical guarantees. How is the online information-maximization problem solved in practice? Can the theoretical guarantees accommodate approximate optimization?
W3. Strong modeling assumptions. The analysis assumes known linear reward features and exact entropy-regularized follower responses. The experiments also follow these assumptions, leaving the robustness of the method under model misspecification unclear. How sensitive is the method to misspecification of the reward features or the follower's softmax response model?
W4. Limited comparisons with informed exploration methods. The experiments mainly compare the proposed method with random exploration and fixed-expert baselines. Comparisons with other active exploration strategies would better establish the advantage of the proposed information criterion. How does the proposed exploration criterion compare with existing active inverse learning approaches, particularly Chen et al. (2025)? Is the main novelty methodological or primarily theoretical?
W5. Ambiguity in the experimental setup. In Appendix H.1, the source leader reward is defined as . However, the Stackelberg planning problem ordinarily assumes that the leader's own reward is known. If this reward and the zero-sum relation are both available to the learner, the follower's reward would already be determined. What information about the leader's reward and the zero-sum relationship is available to each algorithm in the experiments? How is the unknown follower reward assumption maintained?
Questions for authors
Please addresses my questions in the weaknesses part.
3 comments
Response to the reviewer 2/2
W4: Comparison with informed exploration methods, and nature of the novelty.
The novelty is both methodological and theoretical, the main part being theoretical. Compared to Chen et al. (2025), the closest work in terms of exploration:
- Algorithm. Their rule selects the action maximizing a design criterion of the Fisher information at the current MLE. Ours weights the Fisher information by the cumulated information , hence accounts for the directions already explored, and is preceded by an optimal design warm-up.
- Theory. Their guarantees are asymptotic (asymptotic normality of the MLE [4]) and require exact identifiability of the reward and a positive definite Fisher information. Ours are finite-sample, which Chen et al. (2025) left as an open question, and require neither of the aforementioned assumptions: we only need strategic identifiability, and bound the sub-optimality of the leader over a range of downstream tasks, with a lower bound matching in .
A comparison with other works, including Wu et al. (2022) and Chen et al. (2023), is given in our answer to Reviewer 7s1s (Q1). Additionally, the revision of the paper added new comparisons of our algorithm with baselines from Wu et al. (2022) and Chen et al. (2025), in Appendix H.1.1.
W5: Experimental setup and the zero-sum relation.
We emphasize that the follower's reward is unknown to all algorithms in our benchmarks, and our algorithm uses neither the leader's reward nor the zero-sum relation during training: its selection rule only depends on the Fisher information of the follower's responses. The relation is used for two purposes only:
- to define the expert of the fixed-expert baselines, and to allow the
Zero-sum OLSbaseline (Liao et al., 2025), which is the only method that exploits the zero-sum structure; - to define the target task at evaluation, where the leader is optimized for , with given as an arbitrary leader objective.
Our plots should therefore read as: given the estimate , what sub-optimality does the leader incur when optimizing for a given task. The transfer metrics (Figure 3, Appendix H) use other leader objectives, for which the game is not zero-sum. We clarified this in Appendix H.1 of the new revision.
We thank the reviewer again for these comments, and are happy to discuss further.
[1] Faury, Abeille, Calauzènes, Fercoq, Improved optimistic algorithms for logistic bandits, ICML 2020.
[2] Abeille, Faury, Calauzènes, Instance-wise minimax-optimal algorithms for logistic bandits, AISTATS 2021.
[3] Lee, Yun, Jun, A unified confidence sequence for generalized linear models, with applications to bandits, NeurIPS 2024.
[4] Hoadley, Asymptotic properties of maximum likelihood estimators for the independent not identically distributed case, The Annals of Mathematical Statistics, 1971.
Thanks, my questions are well-addressed, and I choose to raise my score to 4.
Response to the reviewer 1/2
We thank the reviewer for the time and effort spent on our paper, and for the positive assessment of the problem formulation, of the reward transfer analysis and of the experiments. We answer each weakness below. Some points were also raised by the other reviewers, and we refer to the corresponding answers where relevant.
W1: Gap between the bounds, and sample complexity.
Dimension. The reviewer is right: our bounds match in K, not in d. The upper bound scales as d⋄d, where d⋄≤d is the dimension of the reward directions the follower can distinguish (Appendix B.1), against d in Theorem 1. Reviewer 7s1s raised the same point, and we have restricted our optimality claim to the rate in K (see our answer to their Q2). We do not know yet which of the lower and upper bounds is loose.
Total sample complexity. Including the warm-up, reaching SubOpt≤ϵ requires
N=nwarm+K=O~(ηSe2ηS∣B∣d2logδ−1)+O~(ϵ2Rmax2d⋄(d(1+ηS)+logδ−1))
interactions. The exponential term is a burn-in that does not depend on ϵ: this is what our two-phase design buys, compared for instance to Wu et al. (2022), where the analogous factor multiplies ϵ−2 (see our answer to Reviewer 7s1s, Q1).
Is the exponential necessary? Such a dependence is standard when inverting a logistic model, and corresponds to the constant κ of the logistic and GLM bandit literature [1-3]. We do not have a lower bound showing that it is necessary in our setting. In practice it is conservative: the theorem covers all rewards with ∥ω∥2≤S, while Figure 2 shows that our algorithm performs well with a much smaller warm-up, or none (using λ>0).
W2: Computational complexity and approximate optimization.
The warm-up design is a convex problem, solved by Frank-Wolfe up to a certified factor 1+ϵ that is already accounted for in our analysis (Appendix E). The online selection rule is indeed non-convex in general. In our experiments, we solve it with SLSQP up to a tolerance.
Our guarantees extend to approximate solutions. Let Explorek(μ):=Trace(Gk−1†(ω^k−1on)Iμ(ω^k−1on))
be the objective at iteration k. The selection rule is used at a single step of the proof (l. 1972), as Explorek(μ~)≤Explorek(μkon). Therefore:
W3: Modeling assumptions and misspecification.
Linear rewards with known features and entropy-regularized responses are common assumptions in this literature, including in the works discussed in W4. Our analysis does not cover misspecification, we can however provide two partial answers:
To test the robustness of our interactive protocol with misspecification, we added a subsection H.4 to the Appendix of the new revision of the paper, testing the senstivity of our algorithm to misspecifications of the follower's response.