- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper investigate the benefits of combining reward and policy regularization in adversarial imitation learning. The main theoretical contribution is a fast-rate convergence guarantee for a jointly regularized AIL objective under general function approximation. The result appears technically nontrivial, particularly in its reward-learning analysis. The paper does not provide experiments and does not demonstrate the applicability of the guarantees.
Critical strengths
- Good theoretical contributions in particular within the reward-side analysis. I could not identify clear errors in the mathematical formulations and proofs.
Critical weaknesses
- Absence of practical evidences. Compared to other theoretical papers on this topic, such as [1], [2] and [3], this paper does not contain any experimental evaluation and does not demonstrate applicability of the provided results.
[1] Li, Shangzhe, Dongruo Zhou, and Weitong Zhang. "Near-optimal second-order guarantees for model-based adversarial imitation learning." International Conference on Learning Representations. Vol. 2026. 2026.
[2] Xu, Tian, et al. "Provably and practically efficient adversarial imitation learning with general function approximation." Advances in Neural Information Processing Systems 37 (2024): 66108-66146.
[3] Chen, Y, Giammarino, V, Queeney, J, and Paschalidis, IC. Provably Efficient Off-Policy Adversarial Imitation Learning with Convergence Guarantees. Retrieved from https://par.nsf.gov/biblio/10692547. Transactions on Machine Learning Research
Questions for authors
-
Would be possible to show that the theoretical strengths of the proposed algorithm translated in empirical advantages?
-
Do the improved rates of convergence imply better imitation performance?
2 comments
Dear Reviewer jk8i,
Thank you again for your detailed and constructive review. As we are about halfway through the discussion period, we would like to kindly follow up.
We would like to briefly summarize our main clarifications and additional experimental results:
-
Empirical validation of convergence rates (W1 & Q1). We add tabular MDP experiments with a stochastic expert across 20 seeds, where the regularized dual gap can be computed exactly. The log–log plots show fitted slopes of and with respect to online interactions and expert demonstrations , respectively, broadly consistent with the predicted and scaling (Appendix G).
-
Comparison with unregularized AIL (W1 & Q1). Under the same regularized evaluation criterion and sample budget, our method achieves substantially smaller dual gaps than unregularized AIL. We also provide ablations showing how and affect convergence and the regularized objective, offering additional empirical support for the role of regularization (Appendix G).
-
Imitation performance and regularization tradeoff (Q2). We clarify that our sample-complexity guarantees concern convergence to the regularized optimum, which may differ from the expert policy. We further derive an explicit bound on the unregularized imitation gap, with an additional bias term of , thereby quantifying the tradeoff between faster convergence and imitation accuracy (Appendix F).
We would greatly appreciate it if you could let us know whether our responses address your concerns, and we would be happy to clarify any remaining questions.
We thank the reviewer for the valuable comments and the support for acceptance of our manuscript. We provide our response to the reviewer's concerns below.
W1&Q1&Q2: The paper lacks empirical validation demonstrating whether its improved theoretical convergence rates translate into practical advantages and better imitation performance.
A: We thank the reviewer for this suggestion. We have added experiments on a small tabular MDP with a stochastic expert in Appendix G of the revised manuscript, where the regularized dual gap can be computed exactly. Across 20 seeds, log–log plots against K and N show sustained decay with online interactions and approximately inverse-N scaling, with fitted slopes of −0.825 and −1.023, respectively. At the same sample budget, our method achieves a substantially smaller gap than unregularized AIL under the common regularized evaluation objective. Ablations over α and τ further show that stronger regularization reduces the gap under the corresponding objective within the tested range.
Regarding imitation performance, our regularized formulation and the unregularized formulations in [1–3] optimize different objectives and may therefore have different optimal solutions. Our approach trades potential regularization bias for faster convergence to the regularized optimum: for fixed positive regularization parameters, we achieve O(1/ϵ) sample complexity in both expert demonstrations N and online interactions K for deterministic and stochastic experts under our assumptions. Such a simultaneous guarantee cannot hold uniformly for unregularized imitation learning for deterministic and stochastic experts. In particular, [1] establishes an Ω(1/ϵ2) expert-sample lower bound for stochastic instances with nonvanishing variance. Our faster rates thus characterize the benefit of regularization in exchange for the bias introduced by leveraging the regularized objective.