- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on the setting of interactive imitation learning, reformulating weighted dataset aggregation as numerical quadrature over a fixed expert–learner policy-mixture path. It investigates how to select mixture coefficients and aggregation weights to approximate the path-average state occupancy distribution, deriving theoretical convergence rates and finite-sample error bounds for different collection schedules. Experiments on synthetic MDPs, LQR systems, and continuous-control environments demonstrate improved occupancy approximation and reduced reference-path imitation loss, although improvements in policy return remain limited and inconsistent.
Critical strengths
- dataset aggregation as numerical quadrature
- convergence about different schedules and occupancy distributions
Critical weaknesses
- limited demonstrated practical impact
- limited theoretical novelty
Questions for authors
Limited theoretical novelty
Theorem 1 (Coverage properties of the path average), Theorem 5 (Schedule-dependent approximation rates)
- The theoretical novelty of the proposed framework appears limited. Theorem 1 largely follows from standard properties of distributional averages, the triangle inequality, and total-variation bounds, while Theorem 5 applies established numerical quadrature convergence results to occupancy distributions. In particular, the convergence rates of classical quadrature methods are well established in numerical analysis [1].
Could the authors more clearly distinguish their original theoretical contributions from applications of existing mathematical results? In particular, how do Theorems 1, 4, and 5 advance the theoretical understanding of interactive imitation learning beyond characterizing the approximation error of a predefined occupancy target?
- Connection between occupancy approximation and imitation learning performance. Theorems 1, 5, and 9 primarily characterize coverage properties and approximation errors along a fixed expert–learner policy-mixture path. However, the relationship between these theoretical results and the performance of the learned policy remains unclear.
What theoretical or empirical evidence establishes that reducing path-average occupancy approximation error leads to improved imitation learning performance, particularly in terms of on-policy imitation loss and policy return? Could the authors further clarify whether these guarantees can be extended to iterative policy retraining, where the collection policies no longer follow the fixed reference path?
2.1 The uniform path-average occupancy target is not sufficiently justified. Section 3 defines using auniform distribution. However, this is a design choice rather than a demonstrated optimal target for imitation learning. Why is that? Could alternative weighting schemes yield better task performance, particularly when different regions of the policy-mixture path have different relevance to the learned policy?
Limited practical usage
the mixture path, Eqs. (1), (7), (8)
- The proposed framework requires access to expert actions to execute mixed expert–learner policies and obtain training labels. This assumption may limit its applicability in real-world settings where only offline expert demonstrations are available or where expert actions cannot be queried or executed on demand.
Could the authors clarify the expert-access requirements of the proposed framework and discuss its applicability under more constrained supervision settings, including limited expert-query budgets or offline demonstrations? How do these requirements affect its practical advantages over conventional DAgger?
3.1 Any proofs / results on expert sample efficiency?
- How does the proposed framework perform when the expert policy is suboptimal, noisy, or inconsistent, as may occur in real-world applications [2][3]? It remains unclear whether improved occupancy approximation provides meaningful benefits when expert supervision is imperfect. Could the authors investigate the sensitivity of the proposed methods to expert quality and label availability?
Missing ablations
-
Could the authors provide ablation studies that separately evaluate the contributions of quadrature-node placement and aggregation weights? In particular, comparisons with simpler interior-mixture schedules, such as midpoint or uniformly spaced nodes, would help establish whether the observed improvements arise from higher-order quadrature or merely from broader state coverage.
-
Although the frozen-path comparisons in Section 6.4 partially isolate the effects of collection schedules, the main continuous-control methods differ in retraining frequency, policy-fitting procedures, and environment-interaction budgets (Table 9).
Could the authors further evaluate quadrature-based schedules under iterative policy retraining and matched environment-interaction budgets?
[1] "Is gauss quadrature better than clenshaw–curtis?." SIAM review 50.1 (2008): 67-87.
[2] "Causal imitation for markov decision processes: A partial identification approach." Advances in neural information processing systems 37 (2024): 87592-87620.
[3] "Imitation learning from imperfect demonstration." International Conference on Machine Learning. PMLR, 2019.
Post-rebuttal update
Thanks for your detailed response. Your clarifications have addressed most of my concerns, and I will raise my score. My main remaining concern is how readily the proposed approach can be used in practice.
As you proposed, please expand the related work section to discuss the papers mentioned above. I also encourage you to release the code to help others reproduce your results.
3 comments
Response to Reviewer joqP (2/3): Q2.1, Q3, Q3.1, and Q4
Q2.1. Choice of the uniform target.
We chose the uniform target because it gives equal weight to equal-width intervals of , covering the full range of per-step expert probabilities, and Theorem 1 characterises the resulting occupancy as the squared- barycentre of the path. It is a design choice rather than a proven optimum, but in a new test neither alternative weighting gave a significant return gain, and the expert-heavy one had higher own-state loss. The framework permits other weights, including Gauss–Jacobi rules for Beta targets.
New experiment: we tested an expert-heavy Beta(2, 1) design and a learner-heavy Beta(1, 2) design on the submitted collector. All eight return comparisons with GL were inconclusive. The expert-heavy design increased own-state loss on Hopper and Walker2d by 0.163 and 0.171 of the behaviour-cloning loss. Selecting a weighting that optimises task return remains open.
Q3. Expert-access requirements.
Our interior-mixture schedules need the same expert access as -mixing DAgger: an expert that can act during collection and label the visited states. With offline demonstrations alone, the available data support behaviour cloning. If the expert can label visited states but cannot act during new rollouts, we can start from demonstrations and collect new data under the learner (). The collection procedure uses only rollouts and labels; the transition matrices in Eqs. (7)–(8) enter the analysis only.
Compared with DAgger-P's schedule, interior mixing costs more expert calls, and it still lowered when expert use was matched on average. In the new MuJoCo experiments at 5,000 steps and 2,000 labels, schedules with 50% average expert participation used about 4,500 expert calls, counting executed expert actions and training labels, against 3,000 for DAgger-P's nodes. The endpoint control matches the single mixture's average participation, and the single mixture lowered relative to it in three of four environments and raised Hopper return by 0.222 of the gap [0.166, 0.280].
Q3.1. Expert sample efficiency.
Yes. Theorem 9 gives a sufficient label budget, and Corollary 6 gives the order of the number of collection policies each schedule needs. Suppose the state space has size and the samples are independent. If the quadrature bias plus the weight-mismatch term from label allocation is at most , then
labels suffice for total variation error at most with probability at least .
The label budget is shared across policies once this condition is met. With controlled allocation error, the sufficient number of collection policies depends on the quadrature rule. For a fixed path, Corollary 6 gives order for endpoint Riemann, for midpoint and for GL. At a fixed number of policies, lower bias leaves more of the error budget for sampling. This is an occupancy approximation result under independent sampling. Q2 explains the implications for the fitted policy.
Q4. Imperfect experts and label availability.
New experiment: GL lowered relative to the endpoint control in all four environments under each of three imperfect-expert conditions, a degraded expert and label noise at two levels. The degraded expert acted and labelled, with return 62–80% of the way from the random policy to the clean expert. The label-noise conditions added per-query Gaussian noise at or , a model of an inconsistent expert, with imitation evaluated against the clean expert.
With the degraded expert, GL lowered own-state loss in all four environments and increased return on HalfCheetah ( of the degraded expert–initial learner gap) and Walker2d (). Under label noise, GL lowered own-state loss on Hopper, with return gains of and of the gap at the two noise levels. The remaining return comparisons were inconclusive or reported descriptively where the expert's own return was highly variable. Under the degraded expert, both imitation losses measure agreement with that weaker policy. In your references [2] and [3], a confounded or suboptimal expert changes the imitation target. Here we keep the target fixed and test whether the ordering of designs survives an imperfect expert. We will discuss both papers in the related work.
For label availability, new descriptive refits on a 500-label subset of the collected data showed the single mixture lowering relative to both endpoint designs in three environments. Increasing the budget to 10,000 labels and 25,000 steps increased GL's lead over the single mixture on Hopper and Walker2d.
Response to Reviewer joqP (3/3): Q5 and Q6
Q5. Placement and aggregation weights.
In MuJoCo at 2,000 labels, the improvements came mainly from broader state coverage, which is what the path-average target asks for: a single 50/50 mixture gave most of the gain over DAgger-P's nodes, collecting at several levels added more on Hopper and Walker2d, and placement and weighting effects were inconclusive. Higher-order quadrature shows where coverage error is measured: on the chain, GL's nodes and weights together cut midpoint's error by more than half, while neither helped on its own; and in the trajectory study, GL needed fewer distinct policies than midpoint.
New experiment: we added midpoint and stratified uniform to MuJoCo and crossed node placement with aggregation weights. At 5,000 steps and 2,000 labels, midpoint, GL and stratified uniform each lowered relative to one 50/50 mixture on Hopper and Walker2d. The comparisons between GL and midpoint, between equal-weight GL nodes and midpoint, and between alternative weights on identical GL-node data were inconclusive in all four environments. On Hopper, allocating labels in proportion to GL weights lowered by 0.008 of the behaviour-cloning loss relative to collecting equal label counts and reweighting them, consistent with the allocation analysis in Lemma 6.
The submitted exact and sampling experiments included midpoint (Tables 12 and 2). A new exact calculation on the , chain gives these total variation errors with five nodes:
| Nodes | GL weights | Equal weights |
|---|---|---|
| GL | 0.0093 | 0.0590 |
| Midpoint | 0.0468 | 0.0235 |
Both crossed designs have higher error than either paired rule. This chain is a polynomial path, a setting favourable to GL. The trajectory study also included four nonpolynomial paths, where GL needed fewer policies than midpoint at every tested budget from 32,000 steps.
Q6. Retraining at equal budgets.
Yes, in a new experiment. With matched steps, labels and fits, retrained GL lowered relative to retrained DAgger-P and the endpoint control on Hopper and Walker2d, by 0.12–0.26 of the behaviour-cloning loss, and its comparison with midpoint was inconclusive on all three outcomes in every environment. On Hopper, it also lowered own-state loss relative to both ( [, ] and [, ]). Every retrained schedule used 5,000 environment steps, 2,000 labels and five fits, one after each collection block. GL reweighted the accumulated blocks using its quadrature weights, renormalised at each fit.
Retrained DAgger-G had lower than retrained GL on Hopper. Retrained GL increased return relative to retrained DAgger-P on Hopper ( of the expert–initial learner gap) and relative to the endpoint control on Hopper () and HalfCheetah (). The reduction in distinct policies seen in the trajectory study offers a potential saving when each new collection policy has a setup cost.
Because every block here is followed by a refit, each stage collects at a single node, so these results are empirical rather than covered by the stage-wise guarantees of Q2.
Response to Reviewer joqP (1/3): Q1 and Q2
Thank you for the detailed review. Our contribution is to explain and quantify how collection schedules shape the training-state distribution along a fixed expert–learner path.
Practical impact.
At equal step and label budgets, collecting at an interior mixture lowered the fitted policy's imitation loss along the path and improved return on Hopper, and in a new trajectory study GL needed fewer distinct collection policies than midpoint to meet a fixed coverage criterion. In new MuJoCo experiments with a fixed initial learner, a single 50/50 mixture lowered reference-path imitation loss (J) relative to DAgger-P's endpoint nodes in three of four environments. On 80 fresh Hopper seeds, return increased by 0.249 of the expert–initial learner gap [0.205, 0.296]. Both gains persist against an endpoint control with the same average expert participation: J was lower in three environments, and on the main seeds Hopper return was higher by 0.222 [0.166, 0.280].
Q1. Theoretical contribution.
Our original contribution is the analysis of collection schedules as quadrature rules on the occupancy path, which gives results about DAgger schedules that the classical tools alone do not:
The mean mixing level does not determine coverage. In a new exact calculation on the K=100, γ=0.999 chain, the endpoint control and the single 50/50 mixture both have mean α=0.5, yet their coverage errors are 0.457 and 0.137. New analysis adds two results for H-step paths: the path average is at least 1/H of every mixture's occupancy, and ⌈H/2⌉ GL nodes give exact coverage, because H-step occupancies have degree at most H−1 and n-node GL is exact to degree 2n−1. On a chain, fewer nodes cannot be exact.
The tools themselves are classical: the barycentre argument in Theorem 1, the perturbation tools behind Theorem 4 and the quadrature rates in Theorem 5, including the Bernstein-ellipse argument in your reference [1]. We will state this distinction explicitly in the introduction.
Q2. Coverage and policy performance.
In theory, coverage error controls the fitted policy's imitation loss on the expert–learner path, but not on states the fitted policy reaches off that path, so we also test own-state loss and return directly. For iterative retraining, the guarantees hold for each stage that collects with a fixed learner, but not yet across refits (Q6).
For an imitation loss between 0 and M, the fitted policy's expected loss on the path average is at most its expected loss on the aggregate plus M times the total variation error ε, by the submission's bounded-loss certificate, eq. (22). Hence a policy within ξ of the best in its class on the aggregate is within 2Mε+ξ of the best on the path average. On H-step paths, the loss at each mixture is at most H times the first bound. Off the path, with a misspecified policy class, no ordering of designs by coverage error carries over to return: in a new six-state example, the minimisers for exact coverage, for DAgger-P's fixed-path nodes and for a single 50/50 mixture never reach the goal, while those for behaviour cloning and a single node at α=0.9 do.
Empirically, in a new tabular demonstration with exact losses and values, policies fitted to five-node GL data had lower on-policy loss than those fitted to the single mixture in all 12 settings, and in 5 of them the median GL policy reached the goal, which the median single-mixture policy never did. In MuJoCo, in descriptive comparisons at 10,000 labels, GL improved on the single mixture in own-state loss in three of four environments and in HalfCheetah return (+0.059 [0.021, 0.098]); under retraining, it lowered Hopper own-state loss (Q6). Not every contrast favoured broader coverage: on HalfCheetah, the single mixture had higher own-state loss than DAgger-P's nodes (+0.189 [0.095, 0.293]), though midpoint and uniform improved on the single mixture there; on fresh Hopper seeds, its own-state difference from DAgger-P's nodes, −0.041 [−0.090, +0.006], included zero; and at 2,000 labels it had higher Hopper return than uniform on the original seeds but not on fresh ones. MuJoCo does not measure occupancy error, and J is unclipped, so these tests are empirical.