acceptodds
Under review as a conference paper at ICLR 2027

Frozen Policy Mixtures: How Deployment Limits Feedback in Imitation Learning

Abstract

We study how deployment affects imitation learning from clean expert labels in finite policy classes containing the expert. For two stochastic policies with a uniform prior and Bayesian updates, frozen per-step mixing has worst-case cumulative squared trajectory Hellinger loss of iterated-logarithm order in episode length. Matching bounds identify a feedback mechanism: failure ends informative visits, limiting the next update. Sampling once per episode or updating after each label removes this horizon dependence. This law is specific to trajectory discrepancy; no cumulative return-gap bound uniform in rounds and stochastic classes holds even for one-step episodes. On our two-state construction, however, terminal-success rewards yield the same separation in cumulative return gaps: iterated-logarithmic for frozen mixing and constant for both alternatives. We also characterize batch updates and likelihood tempering. For deterministic classes, matching return-gap bounds resolve the proposed class-size improvement for Warm-Stagger; plurality voting achieves the proposed rates in expectation under its single-state query model. Controlled experiments validate the trajectory and return calculations, update choices, larger stochastic classes, and deterministic voting.

Primary area:
learning theory
Keywords:
imitation learning, policy mixtures, Bayesian learning, sequential decision making, Hellinger distance, learning theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_osfROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how deploying a mixture over candidate policies affects the expert feedback available for subsequent learning. It compares frozen per-step mixing (PS), sampling once per episode (PE), and updating after each label (WU), for finite realizable classes with clean expert labels.

For two stochastic policies with a uniform prior, worst-case cumulative squared trajectory Hellinger loss is Θ(log⁡⋆H)\Theta(\log^\star H) under PS, versus Θ(1)\Theta(1) under PE and WU. Likelihood tempering by α\alpha gives Θ(1/α+log⁡⋆H)\Theta(1/\alpha+\log^\star H), while refreshing every LL steps gives Θ(log⁡⋆L)\Theta(\log^\star L).

For deterministic classes, matching offline return-gap rates are Θ(min⁡1,AH(B−1)/(n+1))\Theta(\min\\{1,A_H(B-1)/(n+1)\\}) for PS and Θ(min⁡1,log⁡B/(n+1))\Theta(\min\\{1,\log B/(n+1)\\}) for PE. Corresponding hybrid bounds hold for the averaged iterate. A rare-context construction refutes the proposed class-size improvement for WARM-STAGGER as specified, while plurality voting achieves the proposed rates in expectation under single-state queries. The experiments are synthetic and mainly validate the theoretical calculations.

Critical strengths
  1. Sharp results for a clear question. The main result goes beyond showing that one deployment can fail: it provides matching upper and lower bounds in the stated two-policy model. The tower potential connects information collected along the learner's trajectory to future loss. I find the distinction between bounded and unbounded cumulative discrepancy meaningful, even though the iterated logarithm grows very slowly.

  2. A concrete resolution of an algorithmic conjecture. The deterministic AH(M)A_H(M) characterization is another valuable contribution. It quantifies the extra cost of per-step resampling, and the WARM-STAGGER counterexample addresses a specific proposed improvement. The positive plurality result makes this more useful than a negative result alone, although its guarantee is in expectation rather than high probability.

Critical weaknesses
  1. The scope limits the broader takeaway. I see this primarily as a theoretical contribution. The sharpest stochastic characterization concerns two policies under clean-label Bayesian updates and measures trajectory Hellinger loss rather than general cumulative return gaps. Larger-class bounds are provided, but the uniform order around M=HM=H remains unresolved. The authors acknowledge these limitations, so I do not view them as correctness issues, but they limit the broader applicability of the results.

  2. The experiments offer limited evidence beyond the constructions, and the presentation is dense. The stochastic experiments use informative labels that consistently reduce the wrong policy's posterior odds by a fixed factor. They do not test cases where labels can temporarily increase a wrong policy's weight. A small variation would help establish how robust the mechanism is. I would also move the two-state recurrence and a short explanation of the tower potential into the main text.

Overall, I lean toward acceptance because the matching bounds and the concrete conjecture refutation are worthwhile contributions. The limited experimental coverage and the scope of the stochastic results keep my recommendation at weak acceptance.

Questions for authors
  1. How sensitive is the finite-horizon separation to the uninformative absorbing state? For example, what happens when post-failure states provide some distinguishing labels, or there is a chance of returning to informative states? A small sensitivity analysis would help clarify the mechanism's scope.

  2. Proposition 8 is explicitly an expectation guarantee. Does its argument extend to high probability with logarithmic dependence on 1/δ1/\delta, or is there a particular obstacle? I regard this as a possible extension rather than a missing step in the stated result.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3

Thank you for suggesting tests of feedback sensitivity and posterior fluctuations.

Feedback after failure and returns to informative states

We evaluated two feedback perturbations while keeping per-step deployment frozen throughout each episode.

In the first, the failure state remains absorbing, but each label there has likelihood ratio 2^(−λ) in favor of the wrong policy relative to the expert. This is realized by keeping the expert distribution (1/2, 1/2, 0) and setting the wrong policy at the absorbing state to (2^(−λ)/2, 2^(−λ)/2, 1 − 2^(−λ)). If K_t counts active visits, the negative base-two log odds satisfy

b_(t+1) = b_t + K_t + λ(H − K_t).

In the second, labels at the failure state remain uninformative, but each step there returns to the active state with probability ρ. The update then uses all active visits, including revisits. To keep the return criterion fixed, reward is one only if no failure action occurred; a failure flag in the state records this event.

Summing the count recursions gives these cumulative expected terminal-success return gaps over all learning rounds:

Feedback modificationH = 128H = 1024
Original absorbing, uninformative failure state2.09512.3328
Distinguishing failure-state labels, λ = 1/1281.86141.2849
Distinguishing failure-state labels, λ = 1/161.05101.0000
Return probability ρ = 0.011.50231.0048
Return probability ρ = 0.11.00031.0000
Exactly H active-state labels after every episode1.00001.0000

The recursions have omitted-tail bounds below 10^(−15); the displayed values use ordinary floating-point evaluation. Recovery kernels were checked against exhaustive enumeration at small horizons, and zero recovery reproduces the original recursion.

For the first perturbation, we can also characterize the asymptotic change. Fix λ > 0 and write a = min{1, λ}. Every episode adds at least aH to b_t, hence

∑_(t≥1) E[Δ_t] ≤ 1 + H / [2(2^(aH) − 1)].

This is bounded uniformly in H for fixed λ and converges to one as H → ∞. Thus, a fixed positive amount of distinguishing information per post-failure step removes the asymptotic iterated-logarithm growth in this perturbed construction. Small amounts of feedback can still leave excess loss at finite horizons, as the table shows. The construction demonstrates the cost of losing informative visits; its asymptotic separation need not persist with continued feedback.

Labels that temporarily favor the wrong policy

Theorem 1 also permits such labels. Its proof uses conditional likelihood-ratio moments and the expected decrease of the tower potential, without assuming that posterior odds decrease on every update.

We also ran a variation with expert probabilities (1/2, 1/2, 0) and wrong-policy probabilities (0.6, 0.15, 0.25) at the active state. An expert label multiplies the wrong-policy odds by either 1.2 or 0.3, each with probability 1/2. The failure state remains absorbing and uninformative. Over 128 learning rounds and 100,000 independent histories per setting, the cumulative expected terminal-success return gaps were:

HorizonFrozen per-step mixingPer-episode sampling
1281.8251 ± 0.00480.6419 ± 0.0017
10242.0710 ± 0.00510.6434 ± 0.0017

Intervals are pointwise approximate 95% Monte Carlo confidence intervals across independent histories. We integrate conditional expected returns and sample the histories determining subsequent posteriors. For frozen per-step mixing, the first complete episode increased the wrong-policy odds in 11.46% and 11.23% of runs, respectively. The deployment gap persists with posterior fluctuations; these finite-round results establish neither a new asymptotic rate nor broad environmental coverage.

High probability and presentation

Proposition 8 extends to high probability under its existing assumptions. Each queried plurality mistake halves the version space, giving a deterministic bound on the number of observed mistakes. An exponential supermartingale converts this into a bound on their conditional probabilities. Combined with the offline finite-class bound, the result is

J(π^E) − J(π̂) ≤ O(min{
    R log(B/δ) / N_off,
    μ H log(B_bc/δ) / N_int
})

with probability at least 1 − δ. Omit the offline term without offline data. Reviewer 1's response gives constants and proof. No additional round logarithm is needed.

We will move the recurrence and a short explanation of the tower potential into the main text. We will also state the unresolved M ≈ H regime there and preserve the distinction between the general Hellinger theorem, the deterministic return guarantees, and the return separation for the terminal-success construction.

Thank you for the detailed response. The sensitivity analysis clarifies the role of lost feedback, and the experiment with fluctuating posterior odds addresses my concern about the original label model. The high-probability extension of Proposition 8 is also a welcome addition. Please include these results in the revision, particularly the finding that continued informative feedback can remove the asymptotic separation, along with the proposed presentation improvements. My concerns about the narrow stochastic setting and broader applicability remain, but I still find the theoretical contributions worthwhile and will maintain my weak acceptance.

Reviewer_ea7POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, different ways of selecting candidate policies are the main consideration. Different ways lead to different imitation learning effects. The major point is that, this paper considers whether we need to use per-step method or per-episode method. The authors finally showed that per-step method results in earlier failures that might end up failing and preventing learning useful expert feedback, and also slowing down further learning. In this paper, authors tried to cover the characteristics of this effect, showing that frozen per-step mixing method might lead to trajectory loss. But per-episode selection can deliver better effects (removes the dependencies on episode length). This paper also analyzes different update frequencies and strengths. Authors also tried to derive the performance bounds to guide deeper analysis. Experiments are basically theoretical.

Critical strengths
  1. Solid theoretical analysis. (like upper bound/lower bound that shows how different strategies affect learning).
  2. Extension to existing work. Timing and strengh are further analyzed, and a couterexample is provided.
Critical weaknesses
  1. Several unclear points. a) Limited conceptual novelty. The main topics are already well studied in this field in imitation learning. b) Unclear problem significance. The paper focuses on a specific setting rather than starting from a familiar setting/familiar challenge/familiar common sense and diving deeper. This paper main setting is set up artificially and may not make sense to some extend. c) Limited empirical influence. Synthetic analysis is not really enough as this paper targets on solving more realistic issues. Like as Figure 1 demonstrated, would that benefit real-world scenarios?
  2. Not easy to understand. There are too many unnatural logical flows, weird AI-generated words that are not quite standard in this field, and not enough why&how before mentioning something new.
Questions for authors

Please provide any evidence that shows this work has improved real-world imitation learning.

2 comments

Thank you for the comments on motivation and presentation. The paper is a theoretical study of deployment and feedback. We do not currently have evidence that these results improve performance in real-world imitation-learning deployments. Figure 1 measures the proposed mechanism in a controlled MDP, and we will make that purpose explicit.

Relation to established imitation-learning questions

The starting point is the familiar dependence of future training states on the learner's actions, which motivates DAgger. We ask a more specific question: with the same candidate policies and initial weights, how much does resampling a candidate before every action change the feedback available for learning?

This deployment rule is used in an existing algorithm. Li and Zhang's WARM-STAGGER deploys an each-step mixture and asks whether its offline guarantee can depend on log B rather than the logarithm of the step-wise completed class. Our deterministic construction satisfies the assumptions of that guarantee and disproves the proposed improvement for the specified algorithm. Taking H = m, B = m + 1, and N_off = m^2 gives a return gap of order 1/m, while the conjectured bound would be of order log(m)/m^2 at fixed confidence.

Our contribution is the quantitative characterization of this deployment cost. For deterministic classes, we identify the sharp factor A_H(B − 1) and its dependence on both horizon and class size. For two stochastic candidates, we prove that cumulative squared trajectory Hellinger loss has worst-case order 1 + log* H, although per-episode sampling and updating after each label have constant worst-case cumulative loss. The matching upper bound applies to every MDP under the stated protocol. The paper builds on established deployment rules and feedback models; the rates and the conjecture refutation are the claims of novelty.

What the experiments establish

The synthetic experiments check the recursions, compare update schedules, and examine the effect of informative feedback. They do not establish gains in robotics or continuous-control benchmarks. The additional sensitivity calculations reported in our response to Reviewer 3 show that feedback after failure can substantially reduce the effect. This limits how broadly Figure 1 should be interpreted.

The implication for deployment is conditional: under the analyzed models, resampling frequency and update timing can affect both execution loss and how much the next episode teaches the learner. Whether changing these choices improves a particular application requires evaluation in that application. We will keep this distinction explicit rather than presenting the synthetic results as evidence of practical gains.

Presentation changes

We will reorganize the introduction around the sequence of actions, visited states, labels, and updates. We will put the two-state recurrence in the main text before introducing the iterated logarithm, and explain the tower potential as a sum of terms controlling loss at successive scales of the posterior odds. We will also replace informal labels with standard terms such as posterior odds, informative labels, and update interval, and state the feedback protocol and loss metric next to each main result.

Thanks for your clarification and reply. I carefully revisited the paper and rebuttal, and I appreciate the clarifications. My main concerns hold:

  1. Presentation and motivation. The paper is very difficult to follow, with many important why and how questions left unanswered. For example, the introduction goes directly to Hellinger loss and iterated-logarithmic bounds without clearly motivating why this matters. Similarly, Sec. 3 to 7 switch between stochastic and deterministic settings, different feedback protocols, and evaluation metrics without clearly explaining how these results connect or what practical insight they offer. The whole logical flow is unclear.

  2. Not novel or significant. The central observation that policy deployment alters state visitation and feedback is already widely studied and understood, e.g., DAgger. The distinction between per-step and per-episode sampling is also established. The new bound derivation DOES NOT provide new insights.

  3. Restrictive assumptions and limited generality. The main results focus on two policies, clean labels, and specific Bayesian updates. Fig. 1 and experiments largely reproduce theoretical predictions on synthetic constructions.

I agree that the theoretical work does not require real-world experiments. However, the contribution, novelty, and insights are too limited for me to raise the score.

Reviewer_DAPROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how deployment affects imitation learning with clean expert labels and a finite policy class containing the expert. With two stochastic policies, a uniform prior, and untempered Bayesian updates, frozen per-step mixing incurs worst-case cumulative squared trajectory Hellinger loss of iterated-logarithmic order in the horizon. Sampling once per episode or updating after each label removes this dependence. The authors also analyze tempering and batch updates and derive tight return-gap rates for deterministic classes. They refute a WARM-STAGGER conjecture and give expected bounds for a plurality-voting alternative. Synthetic experiments compare the analytical recursions with simulated learning histories.

Critical strengths

The paper explains how deployment limits the evidence available for learning: failure ends informative visits, leaving fewer labels for the next posterior update. Theorems 1 through 4 give matching bounds and show how update timing and strength affect the loss. Lemmas 11 and 12 relate information collected on the learner's trajectory to a potential over posterior odds. The distinctions between trajectory discrepancy, return gaps, and the different feedback protocols are clear.

Critical weaknesses

1.The tight stochastic results have a narrow scope: two policies, clean labels at every visited state, and cumulative squared trajectory Hellinger loss. 2. Theorem 13 covers larger classes, though its upper and lower bounds do not match at M=H. 3. The general return bound still depends on the number of rounds. 4. The separation between constant and iterated-logarithmic cumulative return gaps holds for the specified terminal-success construction, and the experiments do not establish how broadly it occurs in other environments.

The authors acknowledge these limits, which do not undermine the stated theorems. My concern is how much the results tell us about larger policy ensembles and deployment choices based on expected returns.

Questions for authors
  1. In the two-state construction, could you compare the existing recursion with frozen per-step deployment that receives H independent expert labels from active states after each episode? The informative-label count before episode t would then be (t-1)H. This comparison would help isolate the additional cost of losing informative feedback.

  2. Can the expectation guarantee for plurality voting in Proposition 8 be extended to a high-probability bound under the same assumptions, matching the type of guarantee given for WARM-STAGGER? If not, what prevents the extension?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 1

Thank you. We agree that the distinction between trajectory discrepancy and expected returns should guide interpretation.

Scope, larger ensembles, and expected returns

The sharp stochastic result assumes two candidate policies, realizability, independent expert labels at every visited state, and Bayesian updates after each episode. Within these assumptions, its upper bound holds for every MDP and every two-policy stochastic class. The two-state construction gives the matching lower bound. The upper bound requires neither absorption nor the experimental label likelihoods.

Theorem 13 gives only a partial characterization for larger stochastic classes. At M = H, it leaves the gap

H/16 ≤ C_H(H) ≤ 56H(1 + log* H).

We will state this unresolved regime alongside the main stochastic result. For deterministic realizable classes, Theorems 5 and 6 do give matching expected-return rates for arbitrary finite class sizes: per-step mixing depends on A_H(B − 1), whereas per-episode sampling depends on log B, both after offline demonstrations and for the averaged iterate under full-trajectory interaction.

The round dependence in the general stochastic return bound is unavoidable under the stated assumptions. Appendix B gives a one-step example with expert action probability 1/2 + ε and alternative probability 1/2. Distinguishing them takes order ε^(−2) labels, during which the cumulative return gap is order ε^(−1). Choosing ε proportional to n^(−1/2) gives an Ω(√n) lower bound over this family. We therefore cannot infer a round-uniform return guarantee from the Hellinger theorem. The constant-versus-iterated-logarithm return separation applies to the stated terminal-success construction.

Frozen deployment with H informative labels per episode

The proposed control separates the effect of lost feedback from the effect of keeping weights fixed. Write b_t for the number of informative labels before episode t. In the original construction,

p(b) = 1 / [2(1 + 2^b)],
b_(t+1) = b_t + min{G_t, H},
G_t | b_t ~ Geom(p(b_t)).

The geometric variable includes the active-state visit on which failure occurs. The conditional trajectory Hellinger loss and terminal-success return gap are

g_H(b) = 1 − (1 − p(b))^(H/2),
Δ_H(b) = 1 − (1 − p(b))^H.

If each episode instead supplies exactly H independent active-state labels, then b_t = (t − 1)H. Deployment and update timing remain frozen within the episode. Since Δ_H(b) ≤ H p(b) ≤ H × 2^(−b) / 2,

∑_(t≥1) Δ_H((t − 1)H)
    ≤ 1 + H / [2(2^H − 1)] ≤ 3/2.

The cumulative Hellinger loss obeys the same upper bound because g_H ≤ Δ_H. Both cumulative quantities converge to one as H → ∞, with only the first episode contributing in the limit.

For example, at H = 128 and H = 1024, the original cumulative return gaps are 2.0951 and 2.3328; the proposed control gives 1.0000 at both horizons. The loss of informative feedback therefore accounts for the unbounded growth in this construction. The label budget is unchanged, but accessing the active states after leaving them requires additional query access.

A high-probability version of Proposition 8

Yes. Under the same assumptions, Proposition 8 extends to logarithmic dependence on 1/δ. Let B_bc be the number of candidates surviving the offline data, and set c = 1 − e^(−1). For N_off, N_int ≥ 1, with probability at least 1 − δ over the training data and interactive histories,

J(π^E) − J(π̂) ≤ min{
    R,
    2R log(2B/δ) / N_off,
    μ H [log_2(B_bc) + log(2/δ)] / (c N_int)
}.

When N_off = 0, omit the offline term and use B_bc = B. The output remains the uniform per-episode mixture of the plurality iterates. No additional labels are required.

For the offline term, a finite-class union bound ensures that every surviving candidate disagrees somewhere along an expert trajectory with probability at most log(2B/δ) / N_off. This event holds for every later subset obtained by consistent pruning. If the plurality policy deviates, at least half its surviving candidates disagree on that expert trajectory, giving the factor two.

For the online term, let X_t indicate a mistake at the queried state and let ε_t = E[X_t | F_(t−1)]. Each such mistake removes at least half the candidates, so ∑_t X_t ≤ log_2(B_bc) on every history. The process

exp(c ∑_(t≤k) ε_t − ∑_(t≤k) X_t)

is a nonnegative supermartingale. It yields c ∑_t ε_t ≤ log_2(B_bc) + log(2/δ) except on an event of probability δ/2. Recoverability gives J(π^E) − J(v_t) ≤ μ H ε_t. Combining the events strengthens Proposition 8 to the displayed high-probability bound.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.