acceptodds
Under review as a conference paper at ICLR 2027

Where to Query: Dataset Aggregation as Quadrature in Interactive Imitation Learning

Abstract

Interactive imitation learning often collects data under mixtures of expert and learner behaviour to mitigate covariate shift, but mixture points and weights are typically specified as a training schedule without an explicit approximation target. We show that, along a fixed expert–learner path, weighted dataset aggregation is a numerical quadrature rule for the path-average occupancy. This target assigns positive mass to every state reachable under some mixture on the discounted path, whereas aggregates supported only on the endpoint policies can miss states that occur under interior mixtures. For finite-state Markov Decision Processes (MDPs), classical quadrature theory yields schedule-dependent rates, including an endpoint-Riemann bound and geometric convergence for Gauss–Legendre, with constants that grow with the effective horizon or mixing time. Exact evaluation reproduces the rate hierarchy and, on Garnets, the predicted endpoint limit. Under independent finite sampling, Gauss–Legendre and midpoint reach an error no more than 20% above that of direct path-average sampling with the same budget in at most three rounds at every tested budget for the median instance, whereas endpoint Riemann, DAgger, and random-uniform schedules require many more rounds or fail to reach this level as the budget grows. In continuous control, with retraining held fixed, path-covering node sets lower reference-path imitation loss relative to DAgger-P's endpoint nodes on three of four environments but improve return significantly on at most two.

Primary area:
reinforcement learning
Keywords:
Imitation Learning, Dataset Aggregation, Dagger, Numerical Quadrature, Occupancy Measures, Covariate Shift

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_joqPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the setting of interactive imitation learning, reformulating weighted dataset aggregation as numerical quadrature over a fixed expert–learner policy-mixture path. It investigates how to select mixture coefficients and aggregation weights to approximate the path-average state occupancy distribution, deriving theoretical convergence rates and finite-sample error bounds for different collection schedules. Experiments on synthetic MDPs, LQR systems, and continuous-control environments demonstrate improved occupancy approximation and reduced reference-path imitation loss, although improvements in policy return remain limited and inconsistent.

Critical strengths
  • dataset aggregation as numerical quadrature
  • convergence about different schedules and occupancy distributions
Critical weaknesses
  • limited demonstrated practical impact
  • limited theoretical novelty
Questions for authors

Limited theoretical novelty

Theorem 1 (Coverage properties of the path average), Theorem 5 (Schedule-dependent approximation rates)

  1. The theoretical novelty of the proposed framework appears limited. Theorem 1 largely follows from standard properties of distributional averages, the triangle inequality, and total-variation bounds, while Theorem 5 applies established numerical quadrature convergence results to occupancy distributions. In particular, the convergence rates of classical quadrature methods are well established in numerical analysis [1].

Could the authors more clearly distinguish their original theoretical contributions from applications of existing mathematical results? In particular, how do Theorems 1, 4, and 5 advance the theoretical understanding of interactive imitation learning beyond characterizing the approximation error of a predefined occupancy target?

  1. Connection between occupancy approximation and imitation learning performance. Theorems 1, 5, and 9 primarily characterize coverage properties and approximation errors along a fixed expert–learner policy-mixture path. However, the relationship between these theoretical results and the performance of the learned policy remains unclear.

What theoretical or empirical evidence establishes that reducing path-average occupancy approximation error leads to improved imitation learning performance, particularly in terms of on-policy imitation loss and policy return? Could the authors further clarify whether these guarantees can be extended to iterative policy retraining, where the collection policies no longer follow the fixed reference path?

2.1 The uniform path-average occupancy target is not sufficiently justified. Section 3 defines ρˉ\bar{\rho} using auniform distribution. However, this is a design choice rather than a demonstrated optimal target for imitation learning. Why is that? Could alternative weighting schemes yield better task performance, particularly when different regions of the policy-mixture path have different relevance to the learned policy?

Limited practical usage

the mixture path, Eqs. (1), (7), (8)

  1. The proposed framework requires access to expert actions to execute mixed expert–learner policies and obtain training labels. This assumption may limit its applicability in real-world settings where only offline expert demonstrations are available or where expert actions cannot be queried or executed on demand.

Could the authors clarify the expert-access requirements of the proposed framework and discuss its applicability under more constrained supervision settings, including limited expert-query budgets or offline demonstrations? How do these requirements affect its practical advantages over conventional DAgger?

3.1 Any proofs / results on expert sample efficiency?

  1. How does the proposed framework perform when the expert policy is suboptimal, noisy, or inconsistent, as may occur in real-world applications [2][3]? It remains unclear whether improved occupancy approximation provides meaningful benefits when expert supervision is imperfect. Could the authors investigate the sensitivity of the proposed methods to expert quality and label availability?

Missing ablations

  1. Could the authors provide ablation studies that separately evaluate the contributions of quadrature-node placement and aggregation weights? In particular, comparisons with simpler interior-mixture schedules, such as midpoint or uniformly spaced nodes, would help establish whether the observed improvements arise from higher-order quadrature or merely from broader state coverage.

  2. Although the frozen-path comparisons in Section 6.4 partially isolate the effects of collection schedules, the main continuous-control methods differ in retraining frequency, policy-fitting procedures, and environment-interaction budgets (Table 9).

Could the authors further evaluate quadrature-based schedules under iterative policy retraining and matched environment-interaction budgets?

[1] "Is gauss quadrature better than clenshaw–curtis?." SIAM review 50.1 (2008): 67-87.

[2] "Causal imitation for markov decision processes: A partial identification approach." Advances in neural information processing systems 37 (2024): 87592-87620.

[3] "Imitation learning from imperfect demonstration." International Conference on Machine Learning. PMLR, 2019.


Post-rebuttal update

Thanks for your detailed response. Your clarifications have addressed most of my concerns, and I will raise my score. My main remaining concern is how readily the proposed approach can be used in practice.

As you proposed, please expand the related work section to discuss the papers mentioned above. I also encourage you to release the code to help others reproduce your results.

3 comments
AuthorsOpenReview ↗

Response to Reviewer joqP (1/3): Q1 and Q2

Thank you for the detailed review. Our contribution is to explain and quantify how collection schedules shape the training-state distribution along a fixed expert–learner path.

Practical impact.

At equal step and label budgets, collecting at an interior mixture lowered the fitted policy's imitation loss along the path and improved return on Hopper, and in a new trajectory study GL needed fewer distinct collection policies than midpoint to meet a fixed coverage criterion. In new MuJoCo experiments with a fixed initial learner, a single 50/50 mixture lowered reference-path imitation loss (JJ) relative to DAgger-P's endpoint nodes in three of four environments. On 80 fresh Hopper seeds, return increased by 0.249 of the expert–initial learner gap [0.205, 0.296]. Both gains persist against an endpoint control with the same average expert participation: JJ was lower in three environments, and on the main seeds Hopper return was higher by 0.222 [0.166, 0.280].

Q1. Theoretical contribution.

Our original contribution is the analysis of collection schedules as quadrature rules on the occupancy path, which gives results about DAgger schedules that the classical tools alone do not:

  • Endpoint-only designs can omit states reached by per-step mixing (Proposition 2).
  • DAgger-L's linear schedule has first-order coverage error when the endpoint occupancies differ (Proposition 7), and fixed-path DAgger-P and DAgger-G converge to the learner's occupancy, not the path average (Proposition 8).
  • Theorem 4 shows that occupancy paths are regular enough for quadrature: discounted occupancies are analytic, stationary ones are analytic under the stated uniqueness condition, and HH-step ones are polynomials of degree at most H−1H-1. Corollary 6 makes the dependence on horizon or mixing time and expert–learner disagreement explicit.
  • Theorem 9 separates schedule bias, weight mismatch and sampling error.

The mean mixing level does not determine coverage. In a new exact calculation on the K=100K=100, γ=0.999γ=0.999 chain, the endpoint control and the single 50/50 mixture both have mean α=0.5α=0.5, yet their coverage errors are 0.457 and 0.137. New analysis adds two results for HH-step paths: the path average is at least 1/H1/H of every mixture's occupancy, and ⌈H/2⌉⌈H/2⌉ GL nodes give exact coverage, because HH-step occupancies have degree at most H−1H-1 and nn-node GL is exact to degree 2n−12n-1. On a chain, fewer nodes cannot be exact.

The tools themselves are classical: the barycentre argument in Theorem 1, the perturbation tools behind Theorem 4 and the quadrature rates in Theorem 5, including the Bernstein-ellipse argument in your reference [1]. We will state this distinction explicitly in the introduction.

Q2. Coverage and policy performance.

In theory, coverage error controls the fitted policy's imitation loss on the expert–learner path, but not on states the fitted policy reaches off that path, so we also test own-state loss and return directly. For iterative retraining, the guarantees hold for each stage that collects with a fixed learner, but not yet across refits (Q6).

For an imitation loss between 00 and MM, the fitted policy's expected loss on the path average is at most its expected loss on the aggregate plus MM times the total variation error εε, by the submission's bounded-loss certificate, eq. (22). Hence a policy within ξξ of the best in its class on the aggregate is within 2Mε+ξ2Mε+ξ of the best on the path average. On HH-step paths, the loss at each mixture is at most HH times the first bound. Off the path, with a misspecified policy class, no ordering of designs by coverage error carries over to return: in a new six-state example, the minimisers for exact coverage, for DAgger-P's fixed-path nodes and for a single 50/50 mixture never reach the goal, while those for behaviour cloning and a single node at α=0.9α=0.9 do.

Empirically, in a new tabular demonstration with exact losses and values, policies fitted to five-node GL data had lower on-policy loss than those fitted to the single mixture in all 12 settings, and in 5 of them the median GL policy reached the goal, which the median single-mixture policy never did. In MuJoCo, in descriptive comparisons at 10,000 labels, GL improved on the single mixture in own-state loss in three of four environments and in HalfCheetah return (+0.059 [0.021, 0.098]); under retraining, it lowered Hopper own-state loss (Q6). Not every contrast favoured broader coverage: on HalfCheetah, the single mixture had higher own-state loss than DAgger-P's nodes (+0.189 [0.095, 0.293]), though midpoint and uniform improved on the single mixture there; on fresh Hopper seeds, its own-state difference from DAgger-P's nodes, −0.041 [−0.090, +0.006], included zero; and at 2,000 labels it had higher Hopper return than uniform on the original seeds but not on fresh ones. MuJoCo does not measure occupancy error, and JJ is unclipped, so these tests are empirical.

AuthorsOpenReview ↗

Response to Reviewer joqP (2/3): Q2.1, Q3, Q3.1, and Q4

Q2.1. Choice of the uniform target.

We chose the uniform target because it gives equal weight to equal-width intervals of α\alpha, covering the full range of per-step expert probabilities, and Theorem 1 characterises the resulting occupancy as the squared-ℓ2\ell_2 barycentre of the path. It is a design choice rather than a proven optimum, but in a new test neither alternative weighting gave a significant return gain, and the expert-heavy one had higher own-state loss. The framework permits other weights, including Gauss–Jacobi rules for Beta targets.

New experiment: we tested an expert-heavy Beta(2, 1) design and a learner-heavy Beta(1, 2) design on the submitted collector. All eight return comparisons with GL were inconclusive. The expert-heavy design increased own-state loss on Hopper and Walker2d by 0.163 and 0.171 of the behaviour-cloning loss. Selecting a weighting that optimises task return remains open.

Q3. Expert-access requirements.

Our interior-mixture schedules need the same expert access as β\beta-mixing DAgger: an expert that can act during collection and label the visited states. With offline demonstrations alone, the available data support behaviour cloning. If the expert can label visited states but cannot act during new rollouts, we can start from demonstrations and collect new data under the learner (α=0\alpha=0). The collection procedure uses only rollouts and labels; the transition matrices in Eqs. (7)–(8) enter the analysis only.

Compared with DAgger-P's schedule, interior mixing costs more expert calls, and it still lowered JJ when expert use was matched on average. In the new MuJoCo experiments at 5,000 steps and 2,000 labels, schedules with 50% average expert participation used about 4,500 expert calls, counting executed expert actions and training labels, against 3,000 for DAgger-P's nodes. The endpoint control matches the single mixture's average participation, and the single mixture lowered JJ relative to it in three of four environments and raised Hopper return by 0.222 of the gap [0.166, 0.280].

Q3.1. Expert sample efficiency.

Yes. Theorem 9 gives a sufficient label budget, and Corollary 6 gives the order of the number of collection policies each schedule needs. Suppose the state space has size dd and the samples are independent. If the quadrature bias plus the weight-mismatch term from label allocation is at most ε/2\varepsilon/2, then

B≥(d+2log⁡(1/δ))2ε2B \geq \frac{\bigl(\sqrt{d}+\sqrt{2\log(1/\delta)}\bigr)^{2}}{\varepsilon^{2}}

labels suffice for total variation error at most ε\varepsilon with probability at least 1−δ1-\delta.

The label budget is shared across policies once this condition is met. With controlled allocation error, the sufficient number of collection policies depends on the quadrature rule. For a fixed path, Corollary 6 gives order 1/ε1/\varepsilon for endpoint Riemann, 1/ε1/\sqrt{\varepsilon} for midpoint and log⁡(1/ε)\log(1/\varepsilon) for GL. At a fixed number of policies, lower bias leaves more of the error budget for sampling. This is an occupancy approximation result under independent sampling. Q2 explains the implications for the fitted policy.

Q4. Imperfect experts and label availability.

New experiment: GL lowered JJ relative to the endpoint control in all four environments under each of three imperfect-expert conditions, a degraded expert and label noise at two levels. The degraded expert acted and labelled, with return 62–80% of the way from the random policy to the clean expert. The label-noise conditions added per-query Gaussian noise at σ=0.1\sigma=0.1 or 0.30.3, a model of an inconsistent expert, with imitation evaluated against the clean expert.

With the degraded expert, GL lowered own-state loss in all four environments and increased return on HalfCheetah (+0.086+0.086 of the degraded expert–initial learner gap) and Walker2d (+0.042+0.042). Under label noise, GL lowered own-state loss on Hopper, with return gains of +0.151+0.151 and +0.107+0.107 of the gap at the two noise levels. The remaining return comparisons were inconclusive or reported descriptively where the expert's own return was highly variable. Under the degraded expert, both imitation losses measure agreement with that weaker policy. In your references [2] and [3], a confounded or suboptimal expert changes the imitation target. Here we keep the target fixed and test whether the ordering of designs survives an imperfect expert. We will discuss both papers in the related work.

For label availability, new descriptive refits on a 500-label subset of the collected data showed the single mixture lowering JJ relative to both endpoint designs in three environments. Increasing the budget to 10,000 labels and 25,000 steps increased GL's JJ lead over the single mixture on Hopper and Walker2d.

AuthorsOpenReview ↗

Response to Reviewer joqP (3/3): Q5 and Q6

Q5. Placement and aggregation weights.

In MuJoCo at 2,000 labels, the improvements came mainly from broader state coverage, which is what the path-average target asks for: a single 50/50 mixture gave most of the gain over DAgger-P's nodes, collecting at several levels added more on Hopper and Walker2d, and placement and weighting effects were inconclusive. Higher-order quadrature shows where coverage error is measured: on the K=100K=100 chain, GL's nodes and weights together cut midpoint's error by more than half, while neither helped on its own; and in the trajectory study, GL needed fewer distinct policies than midpoint.

New experiment: we added midpoint and stratified uniform to MuJoCo and crossed node placement with aggregation weights. At 5,000 steps and 2,000 labels, midpoint, GL and stratified uniform each lowered JJ relative to one 50/50 mixture on Hopper and Walker2d. The JJ comparisons between GL and midpoint, between equal-weight GL nodes and midpoint, and between alternative weights on identical GL-node data were inconclusive in all four environments. On Hopper, allocating labels in proportion to GL weights lowered JJ by 0.008 of the behaviour-cloning loss relative to collecting equal label counts and reweighting them, consistent with the allocation analysis in Lemma 6.

The submitted exact and sampling experiments included midpoint (Tables 12 and 2). A new exact calculation on the K=100K=100, γ=0.999\gamma=0.999 chain gives these total variation errors with five nodes:

NodesGL weightsEqual weights
GL0.00930.0590
Midpoint0.04680.0235

Both crossed designs have higher error than either paired rule. This chain is a polynomial path, a setting favourable to GL. The trajectory study also included four nonpolynomial paths, where GL needed fewer policies than midpoint at every tested budget from 32,000 steps.

Q6. Retraining at equal budgets.

Yes, in a new experiment. With matched steps, labels and fits, retrained GL lowered JJ relative to retrained DAgger-P and the endpoint control on Hopper and Walker2d, by 0.12–0.26 of the behaviour-cloning loss, and its comparison with midpoint was inconclusive on all three outcomes in every environment. On Hopper, it also lowered own-state loss relative to both (−0.091-0.091 [−0.135-0.135, −0.048-0.048] and −0.167-0.167 [−0.229-0.229, −0.110-0.110]). Every retrained schedule used 5,000 environment steps, 2,000 labels and five fits, one after each collection block. GL reweighted the accumulated blocks using its quadrature weights, renormalised at each fit.

Retrained DAgger-G had lower JJ than retrained GL on Hopper. Retrained GL increased return relative to retrained DAgger-P on Hopper (+0.117+0.117 of the expert–initial learner gap) and relative to the endpoint control on Hopper (+0.125+0.125) and HalfCheetah (+0.042+0.042). The reduction in distinct policies seen in the trajectory study offers a potential saving when each new collection policy has a setup cost.

Because every block here is followed by a refit, each stage collects at a single node, so these results are empirical rather than covered by the stage-wise guarantees of Q2.

Reviewer_x4Y4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper considers the interactive imitation learning setup where algorithms like DAGGER achieve strong performance in theory and practice. For these class of methods, there is a mixing schedule relating to when the actions are sampled from the expert and/or the policy. This paper, considering a fixed learner-policy path, shows that the mixing schedule is a numerical quadrature rule for path average occupancy. Such a perspective, under certain assumptions shows surprising sub-optimal results for classes of mixing schemes used within DAGGER. The paper also presents emprirical results on discrete setups all the way leading upto mujoco.

Critical strengths
  1. I think the connection of mixture points and aggregation weights as numerical quadrature seems original to my knowledge, and has been given a rigorous mathematical treatment.
  2. The paper, despite its theoretical rigor has been reasonably well written.
  3. The results presented (e.g. theorem 9) highlight clear bias variance tradeoffs which shed light on failure modes etc, all of which are pretty illustrative and interesting.
Critical weaknesses
  1. It is still worth noting the power of interactive imitation learning does come from retraining, and with this framing the paper removes this critical component to something like openloop multi policy offline data collection plus something like BC; this is something the paper's theory cannot quite account for.
  2. Despite what theory suggests, it seems like much of the gains come from sampling from interior mixtures (e.g. alpha=0.5) and not just sampling from the ends, and much of what something like Gauss-Legendre style schemes offer appear to disappear owing to function approximation and related noise.
Questions for authors

None apart from the weaknesses noted above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer x4Y4

Thank you for the review and identifying the two limits of the interpretation. We ran new experiments on both.

W1. Retraining.

We agree that retraining is central and that our theory does not cover the full retraining loop. It covers each collection stage with a fixed learner, such as a stage of batch DAgger that collects at several mixtures. Within such a stage, for a loss between 00 and MM, the fitted policy's expected loss on the path average is at most its expected loss on the aggregate training distribution plus MM times the total variation error that the schedule controls. Refitting changes the path, so extending the guarantee across refits requires further analysis.

New experiment: the schedule still matters with retraining. In the submission, retraining at DAgger-P's nodes lowered imitation loss on the fitted policy's own states in all four environments, but with different fit counts and step budgets (§6.4). We therefore retrained every schedule with the same 5,000 steps, 2,000 labels and five fits, one after each block. Each stage then has a single node, so these results are empirical.

  • Retrained GL lowered JJ relative to retrained DAgger-P and the endpoint control on Hopper and Walker2d, two of the four environments. Retrained DAgger-G had lower JJ than retrained GL on Hopper.
  • On Hopper, retrained GL also lowered own-state loss relative to retrained DAgger-P (−0.091-0.091 of the behaviour-cloning loss [−0.135-0.135, −0.048-0.048]) and the endpoint control (−0.167-0.167 [−0.229-0.229, −0.110-0.110]).
  • Retrained GL increased return over retrained DAgger-P on Hopper (+0.117+0.117 of the expert–initial learner gap) and over the endpoint control on Hopper (+0.125+0.125) and HalfCheetah (+0.042+0.042).
  • To isolate the feedback from retraining into collection, a GL control refit five times but kept collecting with the initial learner. Collecting with the updated learner lowered own-state loss on Hopper and raised HalfCheetah return, while raising JJ in three environments. So retraining can help the deployed policy while moving collection away from the reference path that JJ measures.

W2. Interior mixing and quadrature order.

Yes for MuJoCo at 2,000 labels: most of the gain came from interior mixing, and GL versus midpoint was inconclusive. The interior-mixing gain matches our coverage analysis: endpoint-only designs can miss states that per-step mixing reaches (Proposition 2), and in a new exact calculation on the K=100K=100, γ=0.999\gamma=0.999 chain, a single 50/50 mixture has less than a third of the endpoint control's coverage error at the same mean α\alpha (0.137 versus 0.457). In MuJoCo, GL's added value for the fitted policy remains open; quadrature order shows instead in the number of distinct collection policies.

  • At 2,000 labels, a single 50/50 mixture lowered JJ relative to the endpoint control in three of four environments at the same average expert participation. GL versus midpoint was inconclusive on JJ, own-state loss and return in all four environments.
  • Collecting at several mixture levels, as the path-average target asks, lowered JJ further than a single mixture on Hopper and Walker2d. At 10,000 labels and 25,000 steps, GL's JJ lead over the single mixture grew there, and GL lowered own-state loss in three environments.

In the new trajectory study, GL needed fewer distinct policies than midpoint on all 13 hard instances at each tested budget from 128,000 steps, including four nonpolynomial paths. Theorem 9 explains this bias–sampling trade-off for independent samples in finite-state systems.

Reviewer_x4Y4OpenReview ↗

response to author rebuttal

Thank you for your comments. I liked reading the paper and think the idea / direction is interesting but I will keep my score as is given the scope of the contributions.

Reviewer_f5BaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to collect and combine training data using mixtures of an expert and a fixed learner. In the main control comparison, the learner is first trained on 10 expert episodes and then held fixed while additional data is collected. At each step, a mixture follows the expert with a chosen probability and the learner otherwise. The authors use numerical integration rules to approximate the state distribution obtained by averaging equally over these probabilities.

  • Under the paper's finite-state assumptions, the state visitation probabilities change smoothly with the probability of following the expert. This lets the authors apply standard integration rules, including Gauss–Legendre, and bound how the approximation error falls as more mixtures are added.
  • Table 2 tests how many mixtures are needed at a fixed total sample count. For the median Garnet problem, Gauss–Legendre and the simpler midpoint rule need at most three to get within 20% of the error of sampling directly from the target distribution. Midpoint uses evenly spaced mixture levels.
  • With the learner and training procedure held fixed, collecting from mixtures that use both policies lowers imitation error on this averaged state distribution in three of four MuJoCo tasks. Improvements in task reward are less consistent (Table 3).
Critical strengths

Exploration is an age old question, and this paper studies this in an interesting setting. One of the the key insights is that switching between the expert and learner can reach states that neither visits alone. The paper gives a mathematical account of this effect.

  • Proposition 2 makes this coverage benefit precise - "multiplexing" between expert and learner at every step can reach new states. Choosing one policy for an entire episode only combines the two policies' existing state distributions. The distinction matters for the paper's main idea.
  • The theory shows when standard integration rules apply, and how their error depends on episode length or the time needed for state frequencies to settle. The experiments with exactly computed distributions support the predictions. The harder examples in Table 12 show a clear advantage for Gauss–Legendre over simpler rules.
  • The follow-up that holds the learner fixed uses the same data-collection procedure across methods and trains each final policy once. This helps isolate the effect of choosing different mixtures. The paper also reports uncertainty, adjusts for multiple comparisons, and documents unsuccessful predictions and revised experiments.
Critical weaknesses

My main concern is whether training on an equal average of the state distributions generated by all mixtures of the fixed learner and expert leads to a better policy. The authors give a mathematical motivation for that average, but its practical benefit for learning is less established.

  • The average is a sensible way to summarize the mixtures, and Theorem 1 gives it a mathematical justification. What is less clear is when matching it gets better reward. In the discounted setting, a single 50/50 mixture can already reach all the same states (Section 3), (although probabilities can differ greatly, of course). A comparison with that simpler training distribution would help establish the benefit of approximating the full average. The authors acknowledge that their fixed-learner analysis does not guarantee the final policy's performance.
  • The practical advantage of Gauss–Legendre over simpler rules is not yet clear. Midpoint is nearly as effective in Table 2, but is absent from the control experiments. The main comparison using the same collection procedure does not establish an imitation-error difference between Gauss–Legendre and trapezoid (Table 14). Table 5 gives some suggestive evidence for Gauss–Legendre at a larger data budget. I would like to see midpoint tested under the same collection and training procedure, particularly on task reward and states visited by the trained policy. This would be useful to validate the main theory.
  • The reduction in the number of mixtures has not yet been shown to reduce collection cost on the hard examples. Table 2 draws states independently from known state distributions, and Table 12 uses exact calculations. The control experiments do collect trajectories and report environment steps (Table 9), which is useful. But those results do not establish whether the hard-example advantage survives the time needed to collect trajectories and the dependence between consecutive observations, at the same error in approximating the target distribution.
Questions for authors

One thing I would like to see is a clearer understanding of how the chosen average improves learning and whether fewer mixtures save real collection work. These checks would help judge the practical value of the mathematical results.

  • In what settings does averaging over all mixture levels produce a better trained policy than using a single 50/50 mixture?
  • How does midpoint compare with Gauss–Legendre using the same collection procedure and number of expert labels? I would be most interested in task reward and imitation error on states visited by the trained policy.
  • On the hard examples, how many environment steps do Gauss–Legendre and midpoint need to reach the same state-distribution error when the data comes from trajectories?
1 comment
AuthorsOpenReview ↗

Response to Reviewer f5Ba

Thank you for the review and here are answers to your three questions. We answered each with new experiments: single-mixture and midpoint controls in MuJoCo at equal budgets, and trajectory collection on all 13 hard instances.

Q1. Averaging versus a single 50/50 mixture.

Averaging over mixture levels gave a better trained policy than the single mixture at the larger budget we tested, 10,000 labels and 25,000 steps, with lower imitation loss on the policy's own states in three of four environments and higher HalfCheetah return. At 2,000 labels, it lowered reference-path loss on Hopper and Walker2d, while return differences from the single mixture were mostly inconclusive.

  • At 10,000 labels and 25,000 steps, in descriptive comparisons, GL lowered JJ relative to the single mixture in three of four environments, and own-state loss fell by 0.089–0.108 of the behaviour-cloning loss in three of four. HalfCheetah return increased by +0.059+0.059 [0.021, 0.098]. Hopper's +0.116+0.116 [0.020, 0.208] was inconclusive after Holm correction (adjusted p=0.052p=0.052).
  • At 2,000 labels and 5,000 steps, midpoint, GL and stratified uniform each lowered JJ on Hopper and Walker2d by 0.067–0.096 of the behaviour-cloning loss, and midpoint and uniform lowered own-state loss on HalfCheetah. On Hopper, however, the single mixture had higher return than uniform on the original 40 seeds (uniform minus single −0.110-0.110 of the expert–initial learner gap). The comparison on 80 fresh Hopper seeds was inconclusive (−0.057-0.057 [−0.115-0.115, +0.002+0.002]). On the submitted collector, five random nodes also had lower Hopper return than the single mixture (−0.181-0.181).

Our explanation is coverage: a single mixture may visit states the task needs too rarely to give enough training examples. For finite-state HH-step blocks from a reset, the uniform path average assigns each state at least 1/H1/H of the occupancy assigned by any individual mixture. Each trajectory's probability is a nonnegative combination of terms αk(1−α)t−k\alpha^k(1-\alpha)^{t-k} with t<Ht < H, and each such term is at most t+1t+1 times its average over α\alpha, the bound behind Laplace's rule of succession. In a truncated reset chain, the expert assigns 2H−12^{H-1} times as much occupancy as the 50/50 mixture to the state requiring H−1H-1 consecutive expert actions. A new tabular demonstration on constructed discounted chains illustrates this: policies trained on five-node GL data had lower exact on-policy loss than those trained on the single mixture in all 12 settings, and in 5 settings the median GL-trained policy attained the expert's value, while the median single-mixture policy had zero value in all 12.

Q2. Midpoint versus GL.

With the same collection procedure and 2,000 labels, the differences between midpoint and GL in task reward, own-state imitation loss and JJ were inconclusive in all four environments, with fixed collection or with retraining. On fresh seeds, GL minus midpoint return was +0.036+0.036 [−0.015-0.015, +0.090+0.090] on Hopper and +0.065+0.065 [−0.051-0.051, +0.175+0.175] on Ant. The comparison on the submitted collector was also inconclusive. Descriptive refits on 500 of the collected labels favoured GL for Hopper return (+0.078+0.078 [0.029, 0.133]), with the other contrasts inconclusive. GL's advantage appears in the number of collection policies (Q3).

Q3. Environment steps with trajectory data.

On our doubling budget grid, GL and midpoint reached median total variation error 10−210^{-2} within one doubling of each other on all 13 hard instances: with the same number of steps on nine, and GL with half as many on the other four. The median reached 10−210^{-2} at 1,000–16,000 steps for GL and 1,000–32,000 for midpoint. Each node supplied one trajectory, with burn-in included in the budget. We report the 10−210^{-2} target because direct path-average sampling stayed above 5×10−45\times10^{-4} on every instance.

In reliability checks with 40 fresh replicates per rule, at budgets chosen in a separate pilot, at least 38 of 40 replicates reached the target on 10 instances for GL and 7 for midpoint. Where both rules met this criterion, budgets were equal on five instances and half as large for midpoint on two. The remaining rule–instance pairs met the criterion at larger budgets.

The number of distinct policies is where GL differs. At a fixed step budget, we counted the policies needed to come within 20% of direct path-average sampling error. GL's median count was lower at each tested budget from 8,000 steps. At 256,000 steps, the medians were 3 for GL, 4 for midpoint and 7 for trapezoid. GL needed fewer policies on all 13 instances at every tested budget from 128,000 steps, including the four nonpolynomial paths.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviews (1/2): Contribution and new experiments

Thank you for the constructive reviews. In this general response, we summarize our core contribution, outline the new experiments conducted to address your shared questions (testing baseline controls, iterative retraining, and practical robustness) and present our main empirical findings and experimental setup.

Core contribution and overview. Our contribution is to formulate data collection along a fixed expert-learner path as an occupancy approximation problem, in which mixture coefficients and aggregation weights are quadrature nodes and weights. This explains how per-step mixing expands coverage, which occupancy fixed-path DAgger schedules converge to, and how schedule bias, weight mismatch and sampling error add up. To test the practical impact of this formulation under realistic constraints, we ran new experiments that support two primary takeaways: at equal step and label budgets, collecting at an interior mixture lowered the fitted policy's imitation loss along the path in three of four MuJoCo environments, and Gauss-Legendre (GL) reduced the number of distinct collection policies needed. Every result below is new unless marked as from the submission, and we will incorporate all of them into the revised paper.

New experiments and analysis. To directly address your questions regarding simpler mixing baselines, iterative retraining, budget scaling, and relaxed assumptions, we expanded our evaluation with seven new sets of experiments and theoretical analyses:

  • MuJoCo controls at equal budgets of 5,000 steps and 2,000 labels: a single 50/5050/50 mixture, an endpoint control with the same average expert participation, midpoint, stratified uniform and GL.
  • Retraining: every schedule refit five times within the same step and label budget.
  • A larger budget of 10,000 labels and 25,000 steps.
  • Imperfect experts: a degraded expert, and noisy labels.
  • Expert-heavy and learner-heavy Beta weightings in place of the uniform target.
  • Trajectory collection on the 13 hard finite-state instances.
  • Exact calculations and analysis: node placement crossed with weights, a two-state example with a nonpolynomial path, a coverage bound and an exact GL node count for HH-step paths, and a six-state example on coverage and return.

Main findings. Across these new evaluations, our results clarify when interior mixing improves policy performance, where higher-order quadrature (GL) provides a distinct advantage, and how these schedules behave under iterative retraining:

  • Interior mixing vs. endpoints: With the initial learner fixed during collection, the single 50/5050/50 mixture lowered reference-path imitation loss (JJ) relative to both DAgger-P's endpoint nodes and the endpoint control on Hopper, HalfCheetah and Walker2d. On 80 fresh Hopper seeds, it improved return over DAgger-P's nodes by 0.249 of the expert-initial learner return gap [0.205,0.296][0.205, 0.296]. On the main seeds, it also improved Hopper return over the endpoint control, which has the same average expert participation, by 0.222 [0.166,0.280][0.166, 0.280].
  • GL policy efficiency: GL's primary benefit is reducing the number of distinct collection policies. In the exact two-state example, GL reached total variation error 10−310^{-3} with 5 policies versus 19 for midpoint. With trajectory data, GL needed fewer policies than midpoint to come within 20% of direct path-average sampling error on the median hard instance at each tested budget from 8,000 steps, and on each of the four nonpolynomial instances from 32,000 steps. With independent sampling, the submission's Table 2 shows the same order (2 GL policies versus 3 for midpoint on the median held-out Garnet at the largest budget).
  • Quadrature rules and retraining in MuJoCo: At 2,000 MuJoCo labels, GL versus midpoint was inconclusive on JJ, imitation loss on the fitted policy's own states, and return. With retraining, GL lowered JJ relative to both DAgger-P and the endpoint control on Hopper and Walker2d, and retrained DAgger-G had lower JJ than retrained GL on Hopper. While our quadrature guarantees apply to each collection stage with a fixed learner, these retraining results are empirical.

The experimental protocol and statistical methods are in part 2/2.

AuthorsOpenReview ↗

General response to all reviews (2/2): Experimental details

Experimental details. To ensure fair, controlled comparisons across all new MuJoCo runs, we standardized our collection blocks, label allocations, and statistical testing protocols as follows. The control comparisons use five 1,000-step blocks, each starting from a reset with its mixing coefficient held fixed. Actions are mixed independently at each step. The interior designs use one 50/5050/50 mixture, five midpoint nodes, five GL nodes, or one uniformly sampled node in each fifth of the interval. DAgger-P's nodes give one expert block and four learner blocks. The endpoint control gives two of each plus one extra expert or learner block, so its expert participation is 40% or 60% per seed and 50% across seeds. Fixed-path methods fit once after collection, with GL label quotas approximately proportional to its weights. Retrained methods fit after every block, and retrained GL renormalises its weights over the blocks collected so far.

The main comparisons use 40 seeds on each of Hopper, HalfCheetah, Walker2d and Ant, with 80 fresh seeds per environment on Hopper and Ant. Intervals are paired percentile bootstraps with 10,000 resamples, and comparisons use Holm correction at 0.05 within each test family. Loss changes are scaled by the behaviour-cloning loss, and return changes by the expert-initial learner gap. We would be glad to answer follow-up questions during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.