acceptodds
Under review as a conference paper at ICLR 2027

Diffusion-Guided Adversarial Imitation Learning

Abstract

Diffusion models have emerged as a powerful tool for learning and representing distributions, yet their role in adversarial imitation learning (AIL) has remained limited to acting as policy or reward generators. We propose **Diffusion-Guided Adversarial Imitation Learning (DGAIL)**, a new framework that leverages the distribution-matching capability of diffusion models for adversarial imitation learning by training a vector-field discriminator under the guidance of a pre-trained expert score network. Our key insight is an adversarial form of the Fisher divergence underlying diffusion models, which brings the explicit score signal into the adversarial imitation framework. Theoretically, we show that the game value attained by any policy certifies a bound on its true divergence under restricted discriminator classes, and that this bound transfers to an instance-dependent imitation gap guarantee under any bounded evaluation cost. We evaluate our method on MuJoCo continuous control tasks. Experimental results demonstrate that DGAIL matches or outperforms strong AIL baselines.

Primary area:
reinforcement learning
Keywords:
imitation learning, diffusion models, adversarial imitation learning, score-based models, distribution matching, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_v3G9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Diffusion-Guided Adversarial Imitation Learning (DGAIL), which matches learner and expert state occupancies using a diffusion Fisher divergence. The authors identify an additional gradient term arising when the learner-dependent score discrepancy is treated as a fixed policy reward. They then use a variational representation and integration by parts to obtain an objective involving a fixed expert score and a learned vector field, without explicitly fitting a learner-score network. The vector-field divergence is estimated using Hutchinson’s estimator, and discriminator and policy updates alternate.

The paper provides approximation-error bounds and an instance-dependent imitation-gap guarantee. Experiments on five MuJoCo tasks show competitive performance against several imitation-learning baselines, including DiffAIL and DRAIL.

The paper presents a coherent formulation and encouraging empirical results. My main reservations concern its insufficiently resolved relationship to the closest prior method, the absence of evidence isolating the resulting practical advantage, and the unquantified discrete-to-continuous step in the imitation-gap analysis.

A precise account of its relationship to SMILING, a controlled comparison demonstrating its benefits, and a rigorous treatment of the diffusion-time conversion could materially change my assessment.

Critical strengths

The integration-by-parts construction gives a clear way to replace the learner-score term with the divergence of a vector field. For a fixed discriminator, the resulting reward avoids the explicit policy dependence identified in the naive score-discrepancy objective. This provides a useful computational formulation whose practical properties merit investigation.

The main experiments include relevant adversarial, diffusion-based, and Q-based imitation-learning baselines, with results across five seeds. The additional comparisons between Hutchinson estimates and exact Jacobian traces, together with probe-budget experiments, help assess a central computational component. The theoretical error decomposition also distinguishes several relevant approximation sources.

Critical weaknesses
  1. The relationship to the closest prior method is insufficiently characterized, and the distinctive benefit is not isolated.

The most relevant comparison is Wu et al. (2025), Diffusing States and Matching Scores: A New Framework for Imitation Learning (SMILING). Although the introduction briefly discusses this work, describing it as departing from the adversarial framework altogether obscures an important connection: its Section 4 explicitly formulates a two-player zero-sum optimization problem using a difference of denoising losses.

My reading suggests a direct population-level relationship. Write the fixed expert score as sψs_\psi, the learner-score estimator as gg, and the conditional forward score as ut=∇xlog⁡qt(x∣s0)u_t=\nabla_x\log q_t(x\mid s_0). Under the change of variables f=sψ−gf=s_\psi-g, the payoff

E[∥sψ−ut∥2−∥g−ut∥2]\mathbb E[\|s_\psi-u_t\|^2-\|g-u_t\|^2]

becomes, by conditional expectation and integration by parts,

E[2f⊤sψ+2∇⋅f−∥f∥2],\mathbb E[2f^\top s_\psi+2\nabla\cdot f-\|f\|^2],

which is DGAIL’s Eq. (21). This identity holds under matched forward diffusion kernels and diffusion-time weighting, the translated function classes F={sψ−g:g∈G}\mathcal F=\{s_\psi-g:g\in\mathcal G\}, and the required integrability and boundary conditions.

This does not establish equivalence of the implemented algorithms: parameterization, stochastic estimation, data aggregation, and optimization can differ substantially. However, the paper should explain this relationship and identify precisely what its formulation adds. The current discussion does not sufficiently distinguish a new population objective from a different representation and estimator of a closely related objective.

Moreover, SMILING also holds its learned score function fixed during each policy update, so the dependence-term argument for the naive score-discrepancy reward does not by itself establish an advantage over SMILING's policy update.

SMILING is also absent from the experiments. Although DiffAIL and DRAIL are included, the existing comparisons do not isolate whether the benefits arise from the proposed representation, score prior, function class, or optimization choices. A controlled comparison with SMILING would be particularly informative.

  1. The connection between the theoretical guarantees and practical training requires clarification.

Theorem 2 bounds the divergence using the supremized discriminator objective. At the population level, evaluating the objective at the current discriminator gives only a lower bound on that supremum; an empirical logged value additionally incurs evaluation error. Moreover, the score error, discriminator approximation error, magnitude bound, and optimization residual in Eqs. (31)–(34) are population or oracle quantities that are not estimated in the experiments.

Remarks 1–2 appropriately acknowledge important parts of this limitation. Nevertheless, the prominent “training-time certificate” language suggests a more directly evaluable guarantee than is established. The results provide useful conditional error decompositions, but reading a certificate from training logs additionally requires control of the inner-maximization gap and empirical evaluation error. The comparator term in Eq. (34) is a legitimate approximation term, rather than an error, but further limits direct computability.

There is also an unresolved step in Theorem 3. Eq. (19) defines a discrete average over noise levels, whereas Appendix D uses a continuous-time reverse-diffusion argument. Lines 914–917 state that the Riemann discretization error is absorbed into constants, and Eq. (64) subsequently identifies the continuous integral with TDFT D_F without an explicit residual. The stated spatial Lipschitz and time-integrability conditions do not themselves provide the required quadrature bound.

A DDPM variance-preserving schedule can be embedded in an Ornstein–Uhlenbeck process through a time change, but this does not by itself identify the uniform average over DDPM indices in Eq. (19) with the continuous-time integral in Eq. (64). The analysis should distinguish the number of noise levels from the continuous diffusion horizon and account for the induced quadrature weights and discretization residual. An explicit conversion bound is needed to establish the stated connection between the implemented objective and the imitation-gap guarantee.

Questions for authors

Does the population-payoff relationship described above hold under your assumptions? If so, what specific advantage follows from the vector-field representation and divergence estimator? Please clarify the corresponding function classes and distinguish the proposed contribution from the existing score-based minimax formulation.

Can you compare DGAIL with SMILING under matched expert-data access, comparable model capacity, and comparable interaction and computational budgets, sharing the expert score prior where applicable? A small problem with analytically available scores could additionally quantify the gradient bias and optimization consequences of omitting gdepg_{\mathrm{dep}}. This diagnostic should be distinguished from evaluating SMILING itself.

Can you state an explicit discrete-to-continuous bound, including the required temporal assumptions, schedule-dependent weights, and discretization residual in Theorem 3? For the certificate claim, can you either provide the additional controls needed to use logged discriminator values or consistently qualify the result as a population-level conditional guarantee?

Please specify which methods access expert actions and which use only states. Also, Appendix E.3 states that the pretrained score network is kept unchanged when varying the number of expert trajectories. Does this mean its architecture and training protocol are unchanged, or that the same pretrained weights are reused? To interpret these results as demonstration-budget comparisons, the score network should be trained using only the demonstrations allowed at each budget.

3 comments
AuthorsOpenReview ↗

response to reviewer v3G9, part 1

We are very grateful to the reviewer for the meticulous review. The insights and the controlled comparison approach you provided greatly inspired us and were of tremendous help in optimizing this work. We sincerely thank you.

W1 Q1

We thank the reviewer for the careful derivation. Since both methods minimize the diffusion Fisher divergence, a suitable change of variables does establish a mathematical correspondence between the two objectives at the population level. We will state this in the revision and correct our earlier description of SMILING as departing from the adversarial framework, which was inaccurate.

However, as the reviewer notes, this does not make the two algorithms the same. They parameterize different objects. DGAIL parameterizes the score difference fϕf_\phi directly, whose optimum is the gradient of the log density ratio, ∇xlog⁡(ρE,t/ρπ,t)\nabla_x\log(\rho_{E,t}/\rho_{\pi,t}), whereas SMILING parameterizes the learner score gg. The class {sψ−gθ}\{s_\psi-g_\theta\} and a directly trained class {fϕ}\{f_\phi\} do not coincide in general, and the two methods represent different sets of functions. Beyond this, our formulation offers two advantages.

(1) No conditional-score target. SMILING's payoff is a difference of denoising losses and uses the conditional score ut=∇xtlog⁡qt(xt∣s0)=−ε/σtu_t=\nabla_{x_t}\log q_t(x_t\mid s_0)=-\varepsilon/\sigma_t as its regression target, which requires paired clean states and noise. DGAIL removes this term through integration by parts. The discriminator objective depends only on the diffused state xtx_t, and the divergence term is estimated with Hutchinson's estimator, whose error is controlled by the number of probes (Lemma 5 and the probe-budget experiments in the appendix). In addition, DGAIL does not fit a separate learner-score network and does not aggregate states over past policies via Follow-the-Leader. The discriminator and the policy are simply updated in alternation.

(2) Different assumptions in the analysis. This is the most substantive theoretical difference between the two works. SMILING's analysis assumes that the score class is realizable, i.e., that the diffused score of every policy lies in G\mathcal G; it permits misspecification of the policy class and bounds the mixture of the iterates. Our Theorem 2 holds for any single policy and does not assume realizability. Misspecification of the discriminator class enters the bound explicitly as ϵF(π)2\epsilon_{\mathcal F}(\pi)^2 through the exact identity J(π,f)=DF(ρπ∥ρE)−E∥f−fπ⋆∥2\mathcal J(\pi,f)=D_F(\rho_\pi\|\rho_E)-\mathbb{E}\|f-f^\star_\pi\|^2 (Eq. (50)). The bound therefore remains meaningful in the practical regime where a neural network cannot represent the optimal discriminator exactly, and it applies directly to the returned policy. We will add these analysis in our revised paper.

Empirical comparison. To test these differences empirically, we compare DGAIL with SMILING under matched conditions: the same state-only expert demonstrations, a shared pretrained expert score network sψs_\psi, the same MLP architecture for SMILING's learner-score network and for fϕf_\phi, and the same SAC backbone, diffusion schedule, and number of environment interactions.

TaskSMILINGDGAILExpert
HalfCheetah453452105330
Hopper229231763194
Walker2d412248755044
Ant440651915214
Humanoid359349935308

These results demonstrate that, under controlled comparison, our method achieves better empirical performance than SMILING. In the revised paper we will add the training curves of SMILING as an important baseline

AuthorsOpenReview ↗

response to reviewer v3G9, part 2

Q2

Following the reviewer's suggestion, we construct a small diagnostic problem in which all scores are available in closed form, isolating the gradient structure of gdepg_{\mathrm{dep}} from the RL backbone. We consider a 2-D distribution-matching task in which the policy directly parameterizes a Gaussian N(μθ,Σθ)\mathcal{N}(\mu_\theta, \Sigma_\theta) whose occupancy measure is the policy distribution itself (no dynamics, hence no additional noise from RL estimation), the expert occupancy is a bimodal Gaussian mixture, and diffusion follows the discrete DDPM schedule with T=50T = 50 noise levels. In this setting every relevant quantity is analytic: the Gaussian forward kernel preserves the distribution family with closed-form means and covariances at every level tt as functions of θ\theta. Consequently, the expert score sE,ts_{E,t}, the learner score gopt=sπ,tg_{\mathrm{opt}} = s_{\pi,t}, and even the parameter–state mixed derivative ∇θsπ,t\nabla_\theta s_{\pi,t} as well as the objective DF(θ)D_F(\theta) itself can be written in closed form at every level; Monte Carlo is used only to take expectations of these analytic quantities. The exact Fisher gradient g∗=∇θ(−DF)g^* = \nabla_\theta(-D_F) can thus be computed directly and is validated against finite differences. We also verify numerically the population identity Jψ(π,f⋆)=DFJ_\psi(\pi, f^\star) = D_F established in our submission, where f∗=sE,t−sπ,tf^* = s_{E,t} - s_{\pi,t} is the pointwise maximizer of the variational form.

Results and analysis. At several policy snapshots we compare two gradient estimators against g⋆g^\star in terms of systematic bias (the relative norm ∥g^‾−g⋆∥/∥g⋆∥\|\overline{\hat g} - g^\star\| / \|g^\star\| of the mean offset over 16 independent estimates) and cosine similarity with g⋆g^\star.

(i) The naive estimator: standard policy gradient on the exact score-discrepancy reward r=−∥sπ,t−sE,t∥2r = -\|s_{\pi,t} - s_{E,t}\|^2, i.e., keeping only gPGg_{\mathrm{PG}} while dropping the policy-dependent term gdepg_{\mathrm{dep}} that involves goptg_{\mathrm{opt}}. Its systematic bias reaches 0.71–1.17, and at its own converged point its gradient direction is anti-aligned with the true Fisher gradient (cosine −0.58-0.58); the final DFD_F is 21× the attainable optimum (5.90 vs. 0.276). This directly quantifies the gradient bias incurred by omitting goptg_{\mathrm{opt}} and its optimization consequences.

(ii) The population limit of DGAIL: standard policy gradient on the dual reward r=−Tf⋆r = -T_{f^\star} constructed from the analytic f⋆f^\star with exact divergence; This estimator should be unbiased. Its measured systematic bias is only 0.11–0.16, with cosine similarity 0.99, and its final DF=0.278D_F = 0.278 matches exact-gradient training (0.276). In the implemented version, where fϕf_\phi is learned by the discriminator, the bias decreases monotonically with the accuracy of the inner maximization (0.48 → 0.15): the residual bias is governed entirely by the inner-maximization gap, and the objective itself introduces no additional gradient bias. This diagnostic is a component-level analysis of the error terms of our own objective; it does not evaluate SMILING. This toy experiment quantitatively illustrates the effect of gdepg_{\mathrm{dep}} , and we will include this experiment in the appendix of the revised manuscript.

AuthorsOpenReview ↗

response to reviewer v3G9, part 3

W2 Q3

The reviewer is right that Eq. (64) identifies a continuous-time integral with the discrete average of Eq. (19) without accounting for the residual of this identification. We make this residual explicit below; only this step of the proof is replaced, and the remaining stepsare unchanged, includingthe Girsanov estimate, the mixing argument, and the final combination.

Clock. For the VP schedule with σk2=1−αk2\sigma_k^2=1-\alpha_k^2, α0=1\alpha_0=1, set uk=−log⁡αku_k=-\log\alpha_k, Δuk=uk−uk−1\Delta u_k=u_k-u_{k-1}, U=uTU=u_T. Level kk is the time-uku_k marginal of the OU process dxu=−xu du+2 dBu\mathrm dx_u=-x_u\thinspace \mathrm du+\sqrt2\thinspace \mathrm dB_u. Thus TT is the number of noise levels and UU the continuous horizon on which Eq. (64) is proved. With ϕ(u)=Eρπ^,u∥sπ^,u−sE,u∥2\phi(u)=\mathbb{E}_{\rho_{\hat\pi,u}}\Vert s_{\hat\pi,u}-s_{E,u}\Vert ^2, Eq. (19) reads DF=1T∑kϕ(uk)D_F=\frac1T\sum_k\phi(u_k).

Assumption. ϕ\phi is integrable on (0,U](0,U] and LϕL_\phi-Lipschitz on [umin⁡,U][u_{\min},U], umin⁡=uk0u_{\min}=u_{k_0}. The temporal Lipschitz condition is additional: spatial Lipschitz continuity of the scores does not imply it.

Proposition. With κ=Tmax⁡kΔuk/U\kappa=T\max_k\Delta u_k/U, ∫0Uϕ du≤κU DF+Lϕ2∑k>k0(Δuk)2+∫0umin⁡ϕ du.\int_0^U\phi\thinspace \mathrm du\le\kappa U\thinspace D_F+\frac{L_\phi}{2}\sum_{k>k_0}(\Delta u_k)^2+\int_0^{u_{\min}}\phi\thinspace \mathrm du. Proof. On each bin [uk−1,uk][u_{k-1},u_k], k>k0k>k_0, Lipschitz continuity gives ∫ϕ≤Δukϕ(uk)+Lϕ2(Δuk)2\int\phi\le\Delta u_k\phi(u_k)+\frac{L_\phi}{2}(\Delta u_k)^2. Sum, bound ∑k>k0Δukϕ(uk)≤max⁡kΔuk∑k=1Tϕ(uk)=κUDF\sum_{k>k_0}\Delta u_k\phi(u_k)\le\max_k\Delta u_k\sum_{k=1}^T\phi(u_k)=\kappa U D_F using ϕ≥0\phi\ge0, and add the integral over (0,umin⁡](0,u_{\min}]. □\square

Effect on Theorem 3. Substituting into the Hellinger estimate following Eq. (64), on the event DF≤ϵDGAILD_F\le\epsilon_{\mathrm{DGAIL}}, DH2(ρπ^,ρE)≤2κU ϵDGAIL+Lϕ∑k>k0(Δuk)2+2∫0umin⁡ϕ⏟ϵtime+Cmixe−U.D_H^2(\rho_{\hat\pi},\rho_E)\le 2\kappa U\thinspace \epsilon_{\mathrm{DGAIL}}+\underbrace{L_\phi\textstyle\sum_{k>k_0}(\Delta u_k)^2+2\int_0^{u_{\min}}\phi}_{\epsilon_{\mathrm{time}}}+C_{\mathrm{mix}}e^{-U}. The schedule-dependent weight is κU\kappa U; the discretization residual is ϵtime\epsilon_{\mathrm{time}}, which vanishes as max⁡kΔuk→0\max_k\Delta u_k\to0 at fixed UU and umin⁡→0u_{\min}\to0. For our schedule (T=1000T=1000, β∈[10−4,0.02]\beta\in[10^{-4},0.02]): U≈5.06U\approx5.06, κ≈2.00\kappa\approx2.00, ∑k(Δuk)2≈0.034\sum_k(\Delta u_k)^2\approx0.034. Choosing U=Θ(log⁡(1/ϵDGAIL))U=\Theta(\log(1/\epsilon_{\mathrm{DGAIL}})) and the grid so that ϵtime=O(ϵDGAIL)\epsilon_{\mathrm{time}}=O(\epsilon_{\mathrm{DGAIL}}) leaves the imitation gap at O~(ϵDGAIL)\tilde O(\sqrt{\epsilon_{\mathrm{DGAIL}}}). We will give full statement and proof in the revised Appendix D.

On the certificate language, we agree with the reviewer. Theorem 2 bounds DF(ρπ∥ρE)D_F(\rho_\pi\|\rho_E) by the supremum of the discriminator objective over F\mathcal F, whereas a value logged at the current discriminator only lower-bounds that supremum, and its empirical evaluation adds a further error. The quantities in Eqs. (31)–(34) are, moreover, population or oracle quantities. Theorem 2 is therefore a population-level conditional guarantee, as Remarks 1–2 indicate, rather than a bound that can be read off training logs. In the revision we replace "training-time certificate" with "population-level conditional guarantee" throughout, and we add to Remark 2 that reading a bound from logged values would additionally require control of the inner-maximization gap and of the empirical evaluation error.

Q4

Expert data DGAIL uses expert states only, and sψs_\psi is trained on state-only demonstrations. The adversarial baselines (GAIL, DAC, DiffAIL, DRAIL) are run in their state-only form, so their discriminators receive exactly the same information as DGAIL; their returns may therefore differ from those reported with state–action discriminators in the original papers. BC, ValueDICE, IQ-Learn, Reg IQ-Learn, and LS-IQ use expert state–action pairs, as BC requires actions by construction and the Q-based methods are formulated on state–action occupancies. These baselines thus receive strictly more information than DGAIL, so the comparison against them is conservative for DGAIL. We will give a clear clarification in the revised manuscript.

Score network In Appendix E.3, "kept unchanged" refers to the architecture and training protocol of sψs_\psi, not to its weights. For each budget in {1,3,5}\{1,3,5\}, sψs_\psi is retrained from scratch using only the trajectories allowed at that budget, with the protocol of Table 5. The comparison is therefore a demonstration-budget comparison, and Appendix E.3 now states this explicitly.

Thank you again for your careful review and valuable suggestions. We look forward to further discussion with you.

Reviewer_sTkgOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

The primary objective of score matching is to minimize the Fisher divergence between the expert and learner distributions: min⁡πEρπ,t[∥sE,t−sπ,t∥2].\min_\pi \mathbb{E}_{\rho_{\pi,t}} \left[\|s_{E,t}-s_{\pi,t}\|^2\right].

However, this objective presents two key challenges: (1) estimating the learner's score function, sπ,t=∇log⁡ρπ,ts_{\pi,t}=\nabla\log\rho_{\pi,t}, and (2) the inability to directly apply standard policy gradient methods, since sπ,ts_{\pi,t} depends on the policy π\pi.

To address these challenges, DGAIL reformulates the objective as the following minimax optimization problem: min⁡πmax⁡fEρπ,t[2f⊤sE,t+2∇⋅f−∥f∥2].\min_\pi\max_f \mathbb{E}_{\rho_{\pi,t}} \left[ 2f^\top s_{E,t} +2\nabla\cdot f-\|f\|^2\right].

The optimal vector field is given by f∗(xt,t)=sE,t(xt)−sπ,t(xt)f^*(x_t,t)=s_{E,t}(x_t)-s_{\pi,t}(x_t), indicating that ff implicitly learns the score difference between the expert and learner distributions. This reformulation eliminates the need to explicitly estimate the learner's score and enables policy optimization using standard policy gradient methods.

Critical strengths

Novel and convincing optimization formulation. The paper reformulates the original score-matching objective into a minimax optimization problem, eliminating the need to explicitly estimate the learner's score function while enabling policy optimization using standard policy gradient method. Although the underlying variational techniques are well established, their application to diffusion-based adversarial imitation learning provides a novel and well-motivated optimization framework.

Critical weaknesses
  1. Unconvincing motivation for using score-based rewards.

Although the proposed reformulation is well motivated, the necessity of using score-based rewards remains unclear. Both conventional discriminator-based rewards and the proposed score-based rewards aim to match the expert and learner occupancy distributions. However, the paper does not sufficiently explain why score-based rewards are preferable to conventional adversarial rewards or what specific advantages they provide for policy optimization. A clearer justification for adopting score-based rewards would strengthen the motivation of the proposed method.

  1. Computational overhead in the minimax optimization (min⁡πmax⁡fEρπ,t[2f⊤sE,t+2∇⋅f−∥f∥2]\min_\pi\max_f \mathbb{E}_{\rho_{\pi,t}} \left[ 2f^\top s_{E,t} +2\nabla\cdot f-\|f\|^2\right]).

Although DGAIL avoids explicit learner-score estimation, it introduces an additional adversarial optimization problem. It remains unclear whether this reformulation reduces the overall optimization difficulty, particularly considering the convergence of the inner maximization and the additional computational overhead.

  1. Limited experimental evaluation.

The experimental evaluation is limited to standard MuJoCo continuous control tasks, which are relatively simple and well-established benchmarks. Although the results demonstrate promising performance, it remains unclear whether the proposed method can generalize to more challenging and complex imitation learning scenarios. Experiments on a broader range of tasks would strengthen the empirical validation of the proposed method.

I expected a simpler optimization approach that bypasses the estimation of the learner's score function. However, the proposed method appears to introduce additional complexity by replacing score esimation with a minimax optimization problem, which can itself be complex. Given this added complexity, I am not fully convinced that using score-based reward is sufficiently justified over conventional adversarial imitation learning methods.

Questions for authors

See weaknesses

3 comments

We sincerely thank the reviewer for the careful reading and for recognizing that our optimization formulation is novel and convincing. We address the three concerns below.

Weakness 1 The motivation for adopting score-based rewards instead of JS-based adversarial rewards is that score-based rewards can provide a non-saturating signal of distributional discrepancy when the expert and learner distributions have limited overlap. The JS divergence is bounded above by log⁡2\log 2 and approaches this upper bound as the two distributions become increasingly distinguishable. In this regime, the optimal GAIL discriminator can become highly confident, producing nearly constant rewards over regions predominantly occupied by learner samples. In contrast, Fisher divergence measures the squared discrepancy between the score functions of two distributions and is not inherently bounded.

For example, consider two Gaussian distributions with the same variance, p=N(0,σ2)p=\mathcal{N}(0,\sigma^2) and q=N(Δ,σ2)q=\mathcal{N}(\Delta,\sigma^2). Their Fisher divergence is DF(q∥p)=Δ2/σ4D_F(q\| p)=\Delta^2/\sigma^4, which increases quadratically with the separation Δ\Delta, whereas their JS divergence approaches log⁡2\log 2 as Δ→∞\Delta \rightarrow \infty. This example illustrates that Fisher divergence can retain magnitude-sensitive information about distributional mismatch even when the two distributions are substantially separated. Thus, compared with the bounded JS-based objective, score-based distribution matching offers a potentially more informative measure of distributional discrepancy in the low-overlap regime, which motivates its use in our imitation learning framework.

These advantages are corroborated in the experiments. DGAIL outperforms GAIL and DiffAIL, which relies on JS-based adversarial rewards. We will incorporate this motivation more explicitly into the revised manuscript, and we thank the reviewer for pointing out this deficiency of our paper.

Weakness 2 Compared with previous imitation learning methods, DGAIL does not introduce significant computational overhead. The computation of pretraining the expert score network is a one-time cost. When optimizing the inner maximization, by leveraging the Hutchinson estimator, the divergence is computed cheaply without evaluating the full Jacobian. The neural network optimization itself introduces no computation more complex than GAIL's discriminator optimization. To demonstrate this quantitatively, we measured the wall-clock overhead directly. On a standard 50-epoch training run on HalfCheetah-v2, the total training time was 3165.9 s, of which the inner maximization training accounted for only 42.56 s.

Weakness 3 We thank the reviewer for raising this valid concern. We agree that our original evaluation is indeed not sufficiently complex . To address this, we have run new experiments on a substantially more challenging setting of vision-based MetaWorld manipulation. The observations are 64×64 RGB pixels, and the tasks are robotic manipulation rather than simple locomotion. In these tasks, imitation learning is considerably hard under such large observation space.

We design our SAC agent with a DrQ-style convolutional encoder learned end-to-end from pixels. The imitation reward uses a frozen DINOv2 (ViT-S/14) encoder to extract representation-space features, with expert demonstrations encoded by the same encoder. We report the success rate (median over evaluation episodes with 9 fixed evaluation seeds, computed over the final 100k training steps) and the median episode reward over the same window:

TaskExpertDGAIL (ours)BCGAILDiffAIL
plate-slide1.00 / 44781.00 / 42610.81 / 36550.75 / 32810.32 / 2187
hammer1.00 / 19270.94 / 20890.67 / 14360.44 / 7340.67 / 1775
drawer-open1.00 / 40501.00 / 40790.65 / 39910.89 / 34430.22 / 1885

DGAIL attains the highest success rate on all three tasks, matching or approaching expert performance. These results extend the empirical validation from vector-state MuJoCo locomotion to robotic manipulation with high-dimensional pixel observations. They confirm that the method generalizes to more challenging imitation learning scenarios.

We will include the full details and learning curves of this experiment in the revised manuscript. We believe this addition genuinely strengthens the persuasiveness of our paper, and we thank the reviewer again for this valuable suggestion.

We hope these clarifications and the new experiments address the reviewer's concerns, and we would be happy to discuss further.

Thank you for the clarifications and additional experiments. However, several of my concerns remain.

First, it remains unclear whether the experiments explicitly test the low-overlap regime used to motivate the advantage of score-based rewards, where the expert and learner occupancy distributions are substantially separated. It would be helpful to visualize how the learend reward values change as the learner distribution approaches the expert distribution, and to compare this behavior with GAIL acorss different degrees of overlap. Such an analysis would provide more direct evidence that the proposed reward remains informative in the regime where conventional adversarial rewards are claimed to saturate. Improvements in policy performance alone do not establish this explanation.

Second, my original expectation was a simpler and more direct way to construct the imitation reward, ideally through a closed-form solution that avoids estimating the learner’s score. Although the variational formulation is interesting, replacing score estimation with a minimax optimization problem does not fully address my concern about optimization complexity. I am not yet convinced that the demonstrated benefits sufficiently justify this formulation.

Finally, I appreciate the addition of image-based manipulation experiments. Nevertheless, evaluation on only three tasks remains limited, even with image observations, and the comparisons still do not cover the full set of baselines used in the main experiments.

For these reasons, I will maintain my current score.

Thank you for reading our previous response and for stating so clearly what remains open. We are grateful for the time you gave the paper across both rounds. We answer your three points below.

1.Learned rewards in the low-overlap regime You pointed out that final policy return does not show that the reward stays informative when the overlap is low. On MuJoCo the gap between the learner and the expert is entangled with policy optimization and the dynamics, and cannot be specified on its own, so we turn the unit-Gaussian pair from our previous response into a learning experiment. Here that gap is exactly the mean shift �0��0�, so the change in the reward as the gap varies can be read off directly. The expert is �1��1� and the learner is �2��2�, with �3��3�, and the policy is held fixed. At each �4��4� we train a GAIL discriminator and a DGAIL vector field on online samples, using a two-layer SiLU MLP and five random seeds. The GAIL reward is �5��5�. DGAIL uses the analytic expert score of this Gaussian, �6��6�, and the vector field is trained on learner samples only.

The entries are the reward at the learner mean and its slope with respect to the state, averaged over the five seeds. A negative slope means that moving the state toward the expert increases the reward.

Δ\DeltaGAIL rewardGAIL slopeDGAIL rewardDGAIL slope
00.6920.0000.003−0.006-0.006
10.474−0.376-0.376−1.001-1.001−2.009-2.009
20.123−0.237-0.237−4.002-4.002−3.996-3.996
30.012−0.032-0.032−8.995-8.995−5.984-5.984
40.0005−0.0017-0.0017−16.02-16.02−7.94-7.94

2. Why the reward is a minimax problem

We share the wish for an imitation reward that does not require the learner's score. The direct expression of the Fisher divergence is �21��21�, which still contains sπs_\pi. Section 4.1 shows that treating this reward as independent of the current policy and taking a policy gradient drops a nonzero term, because sπs_\pi depends on the policy. The variational form is the identity that removes sπs_\pi by integration by parts. The inner objective is quadratic and strongly concave in the vector field, and its unique maximizer f⋆=sE−sπf^\star=s_E-s_\pi is available in closed form. The network is trained because sπs_\pi is unknown. The inner gradient grows as the two occupancies separate.

What this form removes is the learner score, together with the biased policy gradient that the score induces. Whether the form is worth using depends on whether the reward still has a direction when the overlap is low. The experiment in point 1 is that comparison.

3. Scope of the pixel experiments In the supplementary experiments, we included additional comparison baselines. The pixel comparison now includes behavioral cloning and every adversarial baseline from the main experiments (GAIL, DAC, DiffAIL, and DRAIL), together with IQ-Learn and ValueDICE. DRAIL uses the same SAC backbone as DGAIL. Reg IQ-Learn and LS-IQ are not part of this comparison. Each entry is the success rate / episode return, both medians over the final 100k steps.

TaskExpertDGAILBCGAILDiffAILDACDRAILIQ-LearnValueDICE
plate-slide1.00 / 44781.00 / 42610.81 / 36550.75 / 32810.32 / 21870.33 / 17660.11 / 10760.00 / 1891.00 / 4538
hammer1.00 / 19270.94 / 20890.67 / 14360.44 / 7340.67 / 17750.00 / 6290.00 / 5340.00 / 4670.17 / 600
drawer-open1.00 / 40501.00 / 40790.65 / 39910.89 / 34430.22 / 18850.17 / 18190.00 / 12210.00 / 6520.72 / 1421

ValueDICE matches DGAIL's success rate on plate-slide and records a higher return there (4538 against 4261). On hammer and drawer-open its success rates are 0.17 and 0.72, against 0.94 and 1.00. DGAIL is the only method whose success rate is at least 0.94 on all three tasks. We will put the combined table in the appendix.

Thank you again for the time you spent on the paper and on both rounds of discussion.

Reviewer_JHxbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DGAIL, an adversarial imitation learning method that uses a diffusion model trained on expert demonstrations to guide policy learning. Unlike previous method training a diffusion model on-the-fly and return a scaler as a reward function, the author uses a frozen diffusion model provides information about the expert state distribution, while a discriminator compares it with the learner’s distribution and provides rewards to train the policy. Experiments on 5 MuJoCo tasks show that DGAIL matches or outperforms several imitation learning baselines.

Critical strengths
  1. The score-based formulation is novel and technically meaningful. It converts the Fisher divergence into an adversarial objective that avoids explicitly estimating the learner score and remains compatible with standard policy-gradient methods.
  2. DGAIL shows promising data efficiency. It reaches near-expert performance and generally outperforms several strong baselines across five MuJoCo tasks, including when only a few expert trajectories are available.
Critical weaknesses

Despite the thorough theoretical discussion, the experiments are limited and may not fully demonstrate the method’s practical effectiveness:

  1. The fairness and breadth of the experimental comparison are limited. DGAIL uses SAC and a different network architecture, while baselines such as DRAIL use their original PPO-based pipelines. It is therefore unclear whether the improvement comes from DGAIL or from the different RL backbone. The evaluation is also limited to vector-state MuJoCo locomotion tasks, without image-based states or other domains.
  2. The reliability of the pretrained score model is not sufficiently evaluated. It is trained only on expert demonstrations but later queried on learner states that may be outside the demonstrated distribution. The paper does not study how the amount of pretraining balances fitting the limited expert data against generalizing to unseen learner states: insufficient training may provide a weak diffusion signal, while excessive fitting may reduce out-of-distribution reliability. This is important because the score model is frozen and directly guides policy learning.
Questions for authors
  1. Do DGAIL’s improvements remain when DGAIL is trained with PPO/GRPO, or when DRAIL and the other baselines are trained with the same SAC backbone and matched policy capacity?
  2. Since the frozen score model is trained only on expert states but queried on learner states, how is its accuracy outside the expert occupancy evaluated? How sensitive is DGAIL to the pretraining duration or checkpoint, particularly under shifted initial states or goals?
5 comments

We thank the reviewer for the constructive review. We address the two concerns below

W1, Q1 Among our AIL baselines, the standard implementations of GAIL, DAC and DiffAIL that we use are built on SAC, as is DGAIL; DRAIL is the only baseline whose official pipeline uses PPO. This indeed leads readers to consider whether the gap in experimental performance stems from the choice of RL backbone rather than from the imitation learning method. To address this concern, we instead put DRAIL on the same SAC backbone as DGAIL.

DRAIL-SAC retains DRAIL’s diffusion discriminator, its reward, and its official hyperparameters, and replaces only the policy optimizer with SAC configured in the same way as in our method. Each experiment evaluated four times independently over two seeds.

TaskExpertDRAIL-SACDRAIL-PPODGAIL (ours)
Ant-v25214320531075191
HalfCheetah-v25330430340615210
Hopper-v23194314331193176
Walker2d-v25044395040314875
Humanoid-v25308851104993

We can see from this experiment that the backbone has little effect on DRAIL. Across all tasks, the performance of DRAIL-SAC is comparable to that of the standard PPO-based DRAIL implementation. Under the same RL backbone, DGAIL still exceeds DRAIL-SAC under most tasks. The performance improvement therefore comes from the DGAIL objective rather than from the RL backbone. We thank the reviewer for pointing out the backbone network as a confounding factor. Unifying the backbone yields a fairer comparison, and we will incorporate this update into the revised manuscript.

W2, Q2

Reliability of the frozen score model

To test the trade-off the reviewer describes between under-fitting the expert data and over-fitting it, we retrain sψs_\psi on Ant-v2 from 0.57× to 1.43× the default epochs, keeping everything else fixed, and report the return of DGAIL.

Pretraining epochs relative to defaultAverage return (relative to default)
0.57×5020 (96.7%)
0.71×5172 (99.6%)
0.86×5238 (101.0%)
1.00×5191 (100%)
1.29×5170 (99.6%)
1.43×4947 (95.3%)

The shortest run, which is the most under-fitted, loses 3.3%; the longest, which is the most over-fitted, loses 4.7%. Therefore, in practical implementations, the algorithm exhibits a certain degree of robustness to moderate underfitting and overfitting in the training of the expert score network. We will incorporate this discussion into the manuscript.

Shifted initial states or goals The MuJoCo locomotion tasks we study are not goal-conditioned, and in state-only imitation the task is specified by the expert's state distribution itself. A goal that never appears in the demonstrations therefore defines a different task. This lies outside the problem setting shared by DGAIL and all the baselines. We thank the reviewer for suggesting this meaningful future work direction: investigating the goal-conditioning of the score model and the policy in our method.

We hope these clarifications and the new experiments address the reviewer's concerns, and we would be happy to discuss further.

Thank you to the authors for the thorough clarifications and additional experiments. The additional DRAIL-SAC baseline addresses W1. For W2, the results demonstrate the robustness of DGAIL across different pretraining levels. However, we can still see that performance drops progressively as the number of epochs diverges from the optimal value. It would be more interesting to push the training to extremes, like going as low as 0.1x and as high as 5x, to see the full trend. This is a practical issue when applying DGAIL to a new task or environment other than locomotion, as it is difficult to select the optimal number of pretraining steps in advance.

Also, I agree with Reviewer sTkg that the tasks originally selected in the paper are relatively simple, and I appreciate the authors reporting new results on vision-based MetaWorld manipulation in the rebuttal. However, some baselines appear to be missing from that table. Could the authors also include other baselines (or at least one of the strongest baselines from the earlier experiments, such as DRAIL) for comparison?

I'm willing to raise my score accordingly once these concerns are addressed.

Pretraining duration, extended to 0.1×–5.7×. It is indeed interesting to push the training to extremes. We extend the sweep on Ant-v2 to 0.1× and 5.71× the default number of pretraining epochs, keeping everything else fixed.

Pretraining epochs (relative to default)Average return (relative to default)
0.10×4907 (94.5%)
0.23×4980 (95.9%)
0.34×4958 (95.5%)
1.00×5191 (100%)
2.26×4896 (94.3%)
3.43×4985 (96.0%)
4.57×4969 (95.7%)
5.71×4864 (93.7%)

Over a 57-fold range of pretraining durations, the return stays within 93.7%–100.9% of the default. The decrease the reviewer noted does not continue: beyond roughly ±0.5× the default, the return settles at 94%–96% on both sides, from 0.1× to 0.34× and from 1.43× to 5.71×, instead of declining further.

The pretraining curves of sψs_\psi explain both flat ends of the table. At 0.1× the training denoising loss has already completed 87% of its total decrease, and the last 20% of the schedule is a noise plateau, so stopping early changes sψs_\psi little. Over the full schedule, we did not observe the validation loss rises while the training loss decreases, so no overfitting is detected and training longer does not degrade sψs_\psi.

Transferring the pretraining hyperparameters to a new task. We use the same pretraining hyperparameters across different experiments including mujoco tasks and visual based MetaWorld tasks. On the pixel-based MetaWorld window-close task, the training loss has completed 83% of its decrease at 0.1×, and the validation loss again shows no rise over the full schedule. Therefore, empirically, even across tasks of varying difficulty and type, our algorithm does not require re-tuning the pretraining hyperparameters.

We will add the extended sweep and the pretraining curves of both tasks into our appendix.

Additional baselines on MetaWorld. Thank you for your suggestion. We add DAC, DRAIL, IQ-Learn and ValueDICE to the pixel-based MetaWorld comparison, under the same protocol as the other methods: 64×64 pixel observations, 0.5M environment steps, the same demonstrations, and the same evaluation. All methods share the same visual pipeline: the policy uses the same DrQ-style convolutional encoder, and the discriminator or critic of every method operates on the same frozen DINOv2 latents, so the comparison differs only in the imitation objective. Each entry is the success rate / episode reward, both taken as medians over the final 100k training steps. In particular, for this baseline supplementation, we used DRAIL-SAC instead of DRAIL-PPO to ensure the fairness of the RL backbone.

TaskExpertDGAIL (ours)DACDRAILIQ-LearnValueDICE
plate-slide1.00 / 44781.00 / 42610.33 / 17660.11 / 10760.00 / 1891.00 / 4538
hammer1.00 / 19270.94 / 20890.00 / 6290.00 / 5340.00 / 4670.17 / 600
drawer-open1.00 / 40501.00 / 40790.17 / 18190.00 / 12210.00 / 6520.72 / 1421

The strongest baseline here is ValueDICE, which matches DGAIL on plate-slide but falls far behind on hammer (0.17 vs. 0.94) and drawer-open (0.72 vs. 1.00). DGAIL is the only method that reaches a success rate of at least 0.94 on all three tasks, across baselines spanning behavioral cloning, adversarial imitation and Q-based imitation. We will include the full details and learning curves of this experiment in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.