- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Diffusion-Guided Adversarial Imitation Learning (DGAIL), which matches learner and expert state occupancies using a diffusion Fisher divergence. The authors identify an additional gradient term arising when the learner-dependent score discrepancy is treated as a fixed policy reward. They then use a variational representation and integration by parts to obtain an objective involving a fixed expert score and a learned vector field, without explicitly fitting a learner-score network. The vector-field divergence is estimated using Hutchinson’s estimator, and discriminator and policy updates alternate.
The paper provides approximation-error bounds and an instance-dependent imitation-gap guarantee. Experiments on five MuJoCo tasks show competitive performance against several imitation-learning baselines, including DiffAIL and DRAIL.
The paper presents a coherent formulation and encouraging empirical results. My main reservations concern its insufficiently resolved relationship to the closest prior method, the absence of evidence isolating the resulting practical advantage, and the unquantified discrete-to-continuous step in the imitation-gap analysis.
A precise account of its relationship to SMILING, a controlled comparison demonstrating its benefits, and a rigorous treatment of the diffusion-time conversion could materially change my assessment.
Critical strengths
The integration-by-parts construction gives a clear way to replace the learner-score term with the divergence of a vector field. For a fixed discriminator, the resulting reward avoids the explicit policy dependence identified in the naive score-discrepancy objective. This provides a useful computational formulation whose practical properties merit investigation.
The main experiments include relevant adversarial, diffusion-based, and Q-based imitation-learning baselines, with results across five seeds. The additional comparisons between Hutchinson estimates and exact Jacobian traces, together with probe-budget experiments, help assess a central computational component. The theoretical error decomposition also distinguishes several relevant approximation sources.
Critical weaknesses
- The relationship to the closest prior method is insufficiently characterized, and the distinctive benefit is not isolated.
The most relevant comparison is Wu et al. (2025), Diffusing States and Matching Scores: A New Framework for Imitation Learning (SMILING). Although the introduction briefly discusses this work, describing it as departing from the adversarial framework altogether obscures an important connection: its Section 4 explicitly formulates a two-player zero-sum optimization problem using a difference of denoising losses.
My reading suggests a direct population-level relationship. Write the fixed expert score as , the learner-score estimator as , and the conditional forward score as . Under the change of variables , the payoff
becomes, by conditional expectation and integration by parts,
which is DGAIL’s Eq. (21). This identity holds under matched forward diffusion kernels and diffusion-time weighting, the translated function classes , and the required integrability and boundary conditions.
This does not establish equivalence of the implemented algorithms: parameterization, stochastic estimation, data aggregation, and optimization can differ substantially. However, the paper should explain this relationship and identify precisely what its formulation adds. The current discussion does not sufficiently distinguish a new population objective from a different representation and estimator of a closely related objective.
Moreover, SMILING also holds its learned score function fixed during each policy update, so the dependence-term argument for the naive score-discrepancy reward does not by itself establish an advantage over SMILING's policy update.
SMILING is also absent from the experiments. Although DiffAIL and DRAIL are included, the existing comparisons do not isolate whether the benefits arise from the proposed representation, score prior, function class, or optimization choices. A controlled comparison with SMILING would be particularly informative.
- The connection between the theoretical guarantees and practical training requires clarification.
Theorem 2 bounds the divergence using the supremized discriminator objective. At the population level, evaluating the objective at the current discriminator gives only a lower bound on that supremum; an empirical logged value additionally incurs evaluation error. Moreover, the score error, discriminator approximation error, magnitude bound, and optimization residual in Eqs. (31)–(34) are population or oracle quantities that are not estimated in the experiments.
Remarks 1–2 appropriately acknowledge important parts of this limitation. Nevertheless, the prominent “training-time certificate” language suggests a more directly evaluable guarantee than is established. The results provide useful conditional error decompositions, but reading a certificate from training logs additionally requires control of the inner-maximization gap and empirical evaluation error. The comparator term in Eq. (34) is a legitimate approximation term, rather than an error, but further limits direct computability.
There is also an unresolved step in Theorem 3. Eq. (19) defines a discrete average over noise levels, whereas Appendix D uses a continuous-time reverse-diffusion argument. Lines 914–917 state that the Riemann discretization error is absorbed into constants, and Eq. (64) subsequently identifies the continuous integral with without an explicit residual. The stated spatial Lipschitz and time-integrability conditions do not themselves provide the required quadrature bound.
A DDPM variance-preserving schedule can be embedded in an Ornstein–Uhlenbeck process through a time change, but this does not by itself identify the uniform average over DDPM indices in Eq. (19) with the continuous-time integral in Eq. (64). The analysis should distinguish the number of noise levels from the continuous diffusion horizon and account for the induced quadrature weights and discretization residual. An explicit conversion bound is needed to establish the stated connection between the implemented objective and the imitation-gap guarantee.
Questions for authors
Does the population-payoff relationship described above hold under your assumptions? If so, what specific advantage follows from the vector-field representation and divergence estimator? Please clarify the corresponding function classes and distinguish the proposed contribution from the existing score-based minimax formulation.
Can you compare DGAIL with SMILING under matched expert-data access, comparable model capacity, and comparable interaction and computational budgets, sharing the expert score prior where applicable? A small problem with analytically available scores could additionally quantify the gradient bias and optimization consequences of omitting . This diagnostic should be distinguished from evaluating SMILING itself.
Can you state an explicit discrete-to-continuous bound, including the required temporal assumptions, schedule-dependent weights, and discretization residual in Theorem 3? For the certificate claim, can you either provide the additional controls needed to use logged discriminator values or consistently qualify the result as a population-level conditional guarantee?
Please specify which methods access expert actions and which use only states. Also, Appendix E.3 states that the pretrained score network is kept unchanged when varying the number of expert trajectories. Does this mean its architecture and training protocol are unchanged, or that the same pretrained weights are reused? To interpret these results as demonstration-budget comparisons, the score network should be trained using only the demonstrations allowed at each budget.
3 comments
response to reviewer v3G9, part 2
Q2
Following the reviewer's suggestion, we construct a small diagnostic problem in which all scores are available in closed form, isolating the gradient structure of from the RL backbone. We consider a 2-D distribution-matching task in which the policy directly parameterizes a Gaussian whose occupancy measure is the policy distribution itself (no dynamics, hence no additional noise from RL estimation), the expert occupancy is a bimodal Gaussian mixture, and diffusion follows the discrete DDPM schedule with noise levels. In this setting every relevant quantity is analytic: the Gaussian forward kernel preserves the distribution family with closed-form means and covariances at every level as functions of . Consequently, the expert score , the learner score , and even the parameter–state mixed derivative as well as the objective itself can be written in closed form at every level; Monte Carlo is used only to take expectations of these analytic quantities. The exact Fisher gradient can thus be computed directly and is validated against finite differences. We also verify numerically the population identity established in our submission, where is the pointwise maximizer of the variational form.
Results and analysis. At several policy snapshots we compare two gradient estimators against in terms of systematic bias (the relative norm of the mean offset over 16 independent estimates) and cosine similarity with .
(i) The naive estimator: standard policy gradient on the exact score-discrepancy reward , i.e., keeping only while dropping the policy-dependent term that involves . Its systematic bias reaches 0.71–1.17, and at its own converged point its gradient direction is anti-aligned with the true Fisher gradient (cosine ); the final is 21× the attainable optimum (5.90 vs. 0.276). This directly quantifies the gradient bias incurred by omitting and its optimization consequences.
(ii) The population limit of DGAIL: standard policy gradient on the dual reward constructed from the analytic with exact divergence; This estimator should be unbiased. Its measured systematic bias is only 0.11–0.16, with cosine similarity 0.99, and its final matches exact-gradient training (0.276). In the implemented version, where is learned by the discriminator, the bias decreases monotonically with the accuracy of the inner maximization (0.48 → 0.15): the residual bias is governed entirely by the inner-maximization gap, and the objective itself introduces no additional gradient bias. This diagnostic is a component-level analysis of the error terms of our own objective; it does not evaluate SMILING. This toy experiment quantitatively illustrates the effect of , and we will include this experiment in the appendix of the revised manuscript.
response to reviewer v3G9, part 3
W2 Q3
The reviewer is right that Eq. (64) identifies a continuous-time integral with the discrete average of Eq. (19) without accounting for the residual of this identification. We make this residual explicit below; only this step of the proof is replaced, and the remaining stepsare unchanged, includingthe Girsanov estimate, the mixing argument, and the final combination.
Clock. For the VP schedule with , , set , , . Level is the time- marginal of the OU process . Thus is the number of noise levels and the continuous horizon on which Eq. (64) is proved. With , Eq. (19) reads .
Assumption. is integrable on and -Lipschitz on , . The temporal Lipschitz condition is additional: spatial Lipschitz continuity of the scores does not imply it.
Proposition. With , Proof. On each bin , , Lipschitz continuity gives . Sum, bound using , and add the integral over .
Effect on Theorem 3. Substituting into the Hellinger estimate following Eq. (64), on the event , The schedule-dependent weight is ; the discretization residual is , which vanishes as at fixed and . For our schedule (, ): , , . Choosing and the grid so that leaves the imitation gap at . We will give full statement and proof in the revised Appendix D.
On the certificate language, we agree with the reviewer. Theorem 2 bounds by the supremum of the discriminator objective over , whereas a value logged at the current discriminator only lower-bounds that supremum, and its empirical evaluation adds a further error. The quantities in Eqs. (31)–(34) are, moreover, population or oracle quantities. Theorem 2 is therefore a population-level conditional guarantee, as Remarks 1–2 indicate, rather than a bound that can be read off training logs. In the revision we replace "training-time certificate" with "population-level conditional guarantee" throughout, and we add to Remark 2 that reading a bound from logged values would additionally require control of the inner-maximization gap and of the empirical evaluation error.
Q4
Expert data DGAIL uses expert states only, and is trained on state-only demonstrations. The adversarial baselines (GAIL, DAC, DiffAIL, DRAIL) are run in their state-only form, so their discriminators receive exactly the same information as DGAIL; their returns may therefore differ from those reported with state–action discriminators in the original papers. BC, ValueDICE, IQ-Learn, Reg IQ-Learn, and LS-IQ use expert state–action pairs, as BC requires actions by construction and the Q-based methods are formulated on state–action occupancies. These baselines thus receive strictly more information than DGAIL, so the comparison against them is conservative for DGAIL. We will give a clear clarification in the revised manuscript.
Score network In Appendix E.3, "kept unchanged" refers to the architecture and training protocol of , not to its weights. For each budget in , is retrained from scratch using only the trajectories allowed at that budget, with the protocol of Table 5. The comparison is therefore a demonstration-budget comparison, and Appendix E.3 now states this explicitly.
Thank you again for your careful review and valuable suggestions. We look forward to further discussion with you.
response to reviewer v3G9, part 1
We are very grateful to the reviewer for the meticulous review. The insights and the controlled comparison approach you provided greatly inspired us and were of tremendous help in optimizing this work. We sincerely thank you.
W1 Q1
We thank the reviewer for the careful derivation. Since both methods minimize the diffusion Fisher divergence, a suitable change of variables does establish a mathematical correspondence between the two objectives at the population level. We will state this in the revision and correct our earlier description of SMILING as departing from the adversarial framework, which was inaccurate.
However, as the reviewer notes, this does not make the two algorithms the same. They parameterize different objects. DGAIL parameterizes the score difference fϕ directly, whose optimum is the gradient of the log density ratio, ∇xlog(ρE,t/ρπ,t), whereas SMILING parameterizes the learner score g. The class {sψ−gθ} and a directly trained class {fϕ} do not coincide in general, and the two methods represent different sets of functions. Beyond this, our formulation offers two advantages.
(1) No conditional-score target. SMILING's payoff is a difference of denoising losses and uses the conditional score ut=∇xtlogqt(xt∣s0)=−ε/σt as its regression target, which requires paired clean states and noise. DGAIL removes this term through integration by parts. The discriminator objective depends only on the diffused state xt, and the divergence term is estimated with Hutchinson's estimator, whose error is controlled by the number of probes (Lemma 5 and the probe-budget experiments in the appendix). In addition, DGAIL does not fit a separate learner-score network and does not aggregate states over past policies via Follow-the-Leader. The discriminator and the policy are simply updated in alternation.
(2) Different assumptions in the analysis. This is the most substantive theoretical difference between the two works. SMILING's analysis assumes that the score class is realizable, i.e., that the diffused score of every policy lies in G; it permits misspecification of the policy class and bounds the mixture of the iterates. Our Theorem 2 holds for any single policy and does not assume realizability. Misspecification of the discriminator class enters the bound explicitly as ϵF(π)2 through the exact identity J(π,f)=DF(ρπ∥ρE)−E∥f−fπ⋆∥2 (Eq. (50)). The bound therefore remains meaningful in the practical regime where a neural network cannot represent the optimal discriminator exactly, and it applies directly to the returned policy. We will add these analysis in our revised paper.
Empirical comparison. To test these differences empirically, we compare DGAIL with SMILING under matched conditions: the same state-only expert demonstrations, a shared pretrained expert score network sψ, the same MLP architecture for SMILING's learner-score network and for fϕ, and the same SAC backbone, diffusion schedule, and number of environment interactions.
These results demonstrate that, under controlled comparison, our method achieves better empirical performance than SMILING. In the revised paper we will add the training curves of SMILING as an important baseline