acceptodds
Under review as a conference paper at ICLR 2027

Asymmetric On-Policy Distillation Learns In-Context Exploration

Abstract

Sequential decision-making under partial observability is difficult because agents must explore to infer hidden, task-relevant information before acting effectively. Learning such information-gathering behaviors tabula rasa via online reinforcement learning is sample-inefficient, whereas offline imitation fails to learn exploratory behaviors due to insufficient coverage of deployment-time histories. We focus on the setting of asymmetric student-teacher learning in partially observed settings. However, direct imitation of privileged experts fails because such experts never need to explore. In this work, we propose ASTEROID, a simple yet surprisingly effective online framework for learning test-time exploration. Our key idea is that history-conditioned student policies can learn to explore through online asymmetric distillation: the student first collects on-policy context under partial observability, and the privileged expert provides action labels conditioned on that context. Iteratively, as this context grows, the student learns to use past observations to infer hidden task information and act according to the inferred latent state, yielding exploration similar to efficient Bayesian posterior sampling. Across a diverse range of eleven different simulated environments, including discrete exploration benchmarks, robotic manipulation, navigation, and Procgen environments, ASTEROID achieves 2 higher returns under fixed exploration budgets and uses up to 100 fewer interactions than baselines. Finally, our method transfers policies trained entirely in simulation to vision-denied real-world robot manipulation, exhibiting efficient exploration and improving success by over baselines. Overall, it provides a simple yet surprisingly effective framework to learn efficient test-time exploration from privileged supervision.

Primary area:
reinforcement learning
Keywords:
in-context learning, exploration, robotics, sequential decision making

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FwKFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The submission addresses asymmetric teacher-student learning for partially observable control problems. In this asymmetric setting, the teacher and the student have access to different forms of information concerning the task:

  • the teacher is capable of observing the underlying state of the environment, and represents behavior that is performs well in the corresponding MDP.
  • the student is incapable of observing the underlying state of the environment, and is tasked to learn behavior that performs well in the corresponding POMDP.

Purely offline imitation-learning approaches suffer from a number of issues. This submission primarily addresses an issue concerning distribution shift, typical of any imitation learning approach, though the authors reframe it in this work in terms of exploration capabilities. The proposes solution is fundamentally a version of DAgger for this asymmetric scenario, i.e., the training dataset should be periodically repopulated with the student's on-policy sample experience, in order to have the training data maintain support of the student's visited contexts. The proposed approach will eventually approximate the teacher's posterior sampling distribution (conditioned on the student's context).

The evaluation is performed on a substantial number of environments.

Critical strengths

The evaluation is fairly extensive (though still limited to an unspecified subclass of POMDPs)

See below for a discussion of POMDP subclasses.

Critical weaknesses

The submission conflates the use of posterior sampling approaches for unknown MDPs and meta-RL, vs POMDPs

The authors claims that "Bayesian posterior sampling is a provably efficient exploration strategy". However, this is only applicable to learning under unknown MDPs via posterior sampling (PSRL) approaches, and has nothing to do with control in POMDPs. A notable difference is that the Bayesian posterior in PSRL is a posterior over possible MDP models, whereas the Bayesian posterior that is relevant for a POMDP is over the underlying state, i.e., the agent's belief. These two settings are not interchangeable.

There are various statements that are made about partially observalbe that are inaccurate, presumably due to actually concerning meta-RL rather than partially observable RL, e.g.: "This expert can teach the student how to act if the hidden state is known" is inaccurate. Even under the scenario where a partially observable agent is able to infer the hidden state in one timestep, the current optimal action may be wholly different from that prescribed by a fully observable expert, due to state uncertainties that may re-appear in the future.

The submission implicitly addresses a very narrow notion of exploration, and never addresses its limitations

As direct consequence of the above, the version of posterior sampling as discussed and employed by the authors has unbounded regret for a generic POMDP. There are common POMDPs and contexts hh, where the set of optimal fully observable actions and the set of optimal partially observable actions do not overlap. This has been sometimes been called "cross-observability optimality", e.g., in "Leveraging Fully Observable Policies for Learning under Partial Observability" (Nguyen, 2022) and "Leveraging Fully-Observable Solutions for Improved Partially-Observable Offline Reinforcement Learning" (Wijesundara, 2025).

The issue seems to be that the authors are addressing a limited form of exploration, where the information-gathering location and the task location are the same. However, there are many situations where the information must be acquired in one location, and the task must be performed in another. In these scenarios, the fully observable teacher will trivially fail to guide the student towards performing information-gathering.

Simple example: warehouse item retrieval. The task is to retrieve an item randomly placed in a large number of possible locations. The fully observable teacher knows exactly where the item is stored, and always chooses actions that go to the item. The partially observable student does not know where the item is stored, but should consult an inventory that is available in some corner of the room, far from any storage area. The fully observable teacher will never choose to check the inventory, so the student will at best learn to guess the item locations randomly, until the item is found. This can be arbirarily suboptimal.

Contrary to what the authors claim, this is why pure imitation-learning approaches are going to be limited to some unclear subset of PODMPs, and general solutions require the use of RL.

The approach lacks depth

It is a direct application of DAgger in the asymmetric learning setting.

Questions for authors

Clarifications:

  • How would you reconcile the differences between of meta-RL (fully observable) and POMDPs?

  • Would you be able to characterize the subclass of POMDPs where asymmetric imitation learning (without an RL component) would work well?

Suggestions for future work:

  • Decouple the use of meta-RL approaches and frameworks (including BAMDPs) from the use of POMDPs. It seems to me like the whole use of BAMP is not relevant to partially observable control (as the submission currently stands)
  • Address the suboptimality limitations of the posterior sampling approaches for POMDPs, at the very least acknowledging large sets of POMDPs where the current approach would be sub-optimal, and would not lead to correct exploration. Ideally, find a characterization of a subclass of POMDPs where the proposed approach does work well.
  • (minor) line 121: zz being fixed for K episodes is not a standard assumption, and does not represent a valid subset to the class of POMDPs. POMDPs cannot have memory that is shared across episodes. It is unclear why the assumption of zz being fixed across episodes is necessary.
  • (minor) line 129: implies observable rewards at test time, which is not the standard control setting
  • (minor) line 130: distribution over histories is not guaranteed to be well-defined without further assumptions that identify a marginal distribution over timesteps.
  • (minor) Figure 2 does not seem to fully demonstrate the issue discussed in the text, and it is unclear what it is trying to depict. The example mostly made sense for a single-timestep decision process. What are the various paths trying to depict? why are there multiple subfigures with different subsets of paths?
2 comments

We thank the reviewer for their feedback. We address all their concerns below, particularly clarifying the confusion regarding the setup of the problems and the exploration strategy learned.

1. Clarification about Bayes-optimal exploration

We agree that ASTEROID learns posterior sampling over hidden parameters, which is not Bayes-optimal exploration. However, posterior sampling is a scalable and simple alternative that trades off optimality with learning efficiency and strong regret guarantees [1]. We note this in our introduction (L97) where we state that this exploration has bounded regret and is very efficient to learn but is not optimal. We use widely established results on this exploration from prior work [2,3]. While the goal of our paper is not to claim optimal exploration, it is to show that it can learn more effective exploration than previous imitation learning methods, including DAgger (see Figure 9), while scaling to complex tasks, even real robots. We will make this contribution and limitation clearer in our final version.


2. Making the limitations clearer about the settings

Based on your feedback, we will include a detailed explanation of the failure case of this exploration (including citing the works you mentioned). This exploration is suboptimal precisely when the information to identify hidden parameters lies completely outside the expert's support, i.e., no expert behavior enables the policy to update the latent distribution (L479).

Formally, for any two latents z≠z′z \neq z' in the posterior support, some history the student visits must yield feedback that distinguishes zz from z′z', i.e., helps the student policy shrink the posterior over the hidden parameter. The denser the feedback, the more efficient the exploration. Notably, many practical settings satisfy this because observations give dense implicit feedback (e.g., real-world tactile manipulation). We will add this criterion to the paper and highlight this condition clearly.

We further discuss a potential extension to achieve optimal exploration, which we discuss in our future work section -- use residual RL to expand the space of actions beyond what the experts can do. E.g., ASTEROID could be used to pretrain a policy that does sampling-based exploration of different regions in the warehouse. This allows it to explore certain locations and achieve a high success rate. This downstream policy could be fine-tuned via RL [4] to achieve Bayes-optimal exploration, where it could learn to take actions off the experts' manifold. We are excited to explore this direction in future work.

We appreciate your suggestions for clarifying the nuances in the theoretical literature we attempt to draw a connection with and will incorporate them into the manuscript.


3. Additional baselines including COSIL

Based on your feedback, we include COSIL as a baseline in our experiments on two tasks: tactile cube pick and the procgen maze task. To verify the implementation, we first run the implementation with full observability – COSIL reaches 100% success rate on the cube pick task within ~1.7M env steps. We then train and eval (3 seeds) it under the same partial observability as ASTEROID:

TaskExpertASTEROIDCOSIL (full)COSIL (partial)
Tactile cube pick0.960.951.000.001
Procgen maze (3×3 view)1.000.99—0.30 ± 0.02

Under partial observability, COSIL fails on tactile cube pick even with 100x more environment steps than ASTEROID, and on the maze it performs on par with DPT (0.29). We observe that the student is unable to generate exploration behavior that enables expert imitation. Thus, the divergence to the expert does not decrease, and the imitation weight keeps growing. ASTEROID, on the other hand, uses expert imitation conditioned on student contexts and can learn exploration efficiently. We will add these results and implementation details to the paper. We also compare to the frontier model Opus 5.5 in response to reviewer VzMv.


4. Differences to DAgger

We include DAgger as a baseline in Figure 9 and show that as we run standard DAgger over multiple steps, the performance degrades. Here, this happens because DAgger rolls out a student to collect histories, relabels the student at each time step to generate action labels, i.e., DAgger generates (h,s,a*) – where h,s come from the student, and a* comes from the expert. This learns a policy that fails to commit to a particular mode of behavior and dithers (see Figure 9 and website videos for example behavior).

ASTEROID, on the other hand, uses student policy to generate histories only. It then uses expert rollouts to generate both query states and action targets i.e., we generate (h,s*,a*) - where h is from the student and s*,a* is from expert rollout. In the control experiments, this is equivalent to the expert taking over the control and generating expert behavior to success. This trains the policy to commit to and follow a mode of behavior (akin to action chunking in standard imitation learning), and this mode keeps adapting based on the online context. A benefit over DAgger is that we could use offline datasets or planners as the experts are not queried on arbitrary states the students visit but over expert visited states only. Regarding our method depth: Our goal is to show that simple changes to student-teacher distillation can enable learning exploration from experts that don't explore (reviewer VzMv)


5. Reconciling the settings

  • Thank you for this clarification; we will make the distinction explicit. Similar to prior work DPT[2], RL^2[3], we first present our method under a simple class of problems - BAMDPs where prior work makes two standard assumptions: the environment has a latent hidden parameter that is unchanged during episodes, and that the policy can interact with the environment over multiple episodes under the same parameter and has access to rewards (e.g., which of three doors leads to the goal). Sampling-based exploration means acting as if one consistent guess were true, then ruling it out. In direct comparison over benchmarks from prior works in Figure 6, we show that our method learns a more efficient exploration strategy.

  • In a POMDP, the uncertainty is the belief over the current state, which can change as the agent acts (e.g., in tactile cube pick Figure 8 and 13, the cube's pose is unknown and shifts when the gripper touches it). ASTEROID handles the POMDP case the same way: conditioned on its history, the student acts like the expert would in a state sampled from its current belief, and each contact or observation updates that belief. This is related to QMDP [5] which acts on a belief over fully observed experts and shares its limitation: it never takes actions only for gathering information. We will add this distinction and the QMDP connection to the paper.


6. Minor suggestions:

  • Distribution over histories: We would make it clear that the distribution over histories is defined over a marginal distribution of timesteps in the problem statement. We discuss this in informally in Section 4.3.
  • Figure 2: We are just trying to show that DPT assumes that you need to cover all possible paths via random context generation in your supervised learning dataset. Based on your feedback, we will update this figure and the caption to make this clear.

[1] (More) Efficient Reinforcement Learning via Posterior Sampling, Osband et al.

[2] Supervised Pretraining Can Learn In-Context Reinforcement Learning, Lee et al.

[3] RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning, Duan et al.

[4] Bayesian Residual Policy Optimization:Scalable Bayesian Reinforcement Learning with Clairvoyant Experts, Lee et al.

[5] Learning Policies for Partially Observable Environments: Scaling Up, Littman et al.

Reviewer_WJFUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose ASTEROID, an imitation-learning approach for POMDP/BAMDP settings where access to a privileged expert is provided.

Critical strengths

The paper presents extensive experimental results, including real-world experiments. The baselines it compares against are established and relevant choices.

Critical weaknesses

One of the main weaknesses of the paper my view is the structure and presentation. The paper spends much time with simple examples that require some polishing (see questions).

The main body of the paper is not self-contained, and many important aspects have been deferred to the appendix.

As far as I can see, the theoretical results only cover the BAMDP setting but not the POMDP setting, which is allegedly the main topic of the paper. Please justify how this extension can be made.

Questions for authors

The introduction of BAMDPs is somewhat confusing, particularly because Section 2 opens stating that POMDPs are studied here (BAMDPs only get mentioned afterwards). It's also unclear why this is done "for simplicity". Is this purely for simplicity of exposition, or is it also studied in the experiments?

Please reconcile the policy-conditioning notation in Eq. (1) and Fig. 3.

Line 175: \pi( \cdot | \cdot, z) has not been defined.

The sentence in lines 208/209 is confusing.

In Fig. 3, the context tokens repeat subscript 0; please clarify or correct the temporal/episode indexing.

In lines 211/212: "does not have to cover" instead of "should not cover"?

It would be useful to clarify where the term asymmetric comes from.

The caption of Figure 4 is hard to read, as it's too close to the main body text.

The text/figure describing iteration 1 is not clear to me.

The shorthand \tau is not used consistently throughout the paper.

Pseudocode in the main body would be useful to understand the algorithm better.

The paper makes claims about theoretical results, but these are only covered in the appendix (including main results).

1 comment

We thank the reviewer for their feedback and for providing suggestions to improve the presentation. We propose the changes below to clarify all their concerns:

1. BAMDPs and POMDPs

  • Similar to prior work DPT[1], RL^2[2], we first present our method under a simple class of problems - BAMDPs where prior work makes two standard assumptions: the environment has a latent hidden parameter that is unchanged during episodes, and that the policy can interact with the environment/ This enables us to explain why our proposed method works to learn exploration, i.e., learns to eliminate hidden states (say, goals) as it explores the environment. In direct comparison over benchmarks from prior works in Figure 6, we show that our method learns a more efficient exploration strategy.
  • Finally, unlike any prior work, we scale our method to POMDPs, i.e., the hidden state changing every time step (e.g., the robot experiments, including a task with no proprioception in Figure 13b), and show that our method can learn sampling-based exploration for high-dimensional complex tasks, including real-world robot manipulation. We include further clarification about this setting in response to the reviewer FwKF.

2. Please reconcile the policy-conditioning notation in Eq. (1) and Fig. 3.

We apologize for the difference. We will harmonize the equation and make sure to include the rewards from the environment as part of the observation (similar to [1,2]), which would make it clearer.


3. Line 175: \pi( \cdot | \cdot, z) has not been defined.

This represents the expert under the given hidden state in the environment. We defined expert as π∗(⋅∣st)\pi*(\cdot|s_t) and (ot,z)=st(o_t,z)=s_t. We will fix this notation in the manuscript.


4. The sentence in lines 208/209 is confusing.

To clarify: We imply that the bottleneck in learning a policy pi(|o_t, h_t), i.e., a history-conditioned policy that explores under partial observability, is generating supervision tuples that cover histories h_t that a policy would see on deployment. Our method shows that on-policy contexts are sufficient for learning such policies. We will update the lines with the description above to make it clear.


5. Figure 3

It represents episode indexing. We will fix the notation.


6. In lines 211/212: "does not have to cover" instead of "should not cover"?

Thank you for this. This is better wording, and we will include this in the final version.


7. Asymmetry definition

The asymmetry results from the difference in information received by the teacher and the student when acting. We will include some examples in the problem statement to expose this earlier in the paper. Such as in a household navigation problem, the oracle teacher can have access to a map, while the student has to rely on images from onboard cameras only.


8. Figure 4 caption and \tau notation and Iteration 1.

We will rearrange the space around Figure 4. For Iteration 1: we want to show the mechanism of how an in-context exploration policy uses its own history to eliminate the potential hidden states and converge to the goal over multiple iterations. \tau is used to represent the trajectories generated by the policy. We will ensure that the notation is consistently used across the paper.


9. The paper makes claims about theoretical results, but these are only covered in the appendix.

In the camera-ready version, we will use the updated space to include the algorithm and the main theoretical claims in the paper.


We hope that this addresses your concerns about the presentation. We would be happy to address further questions and presentation clarifications with concrete changes.

[1] Supervised Pretraining Can Learn In-Context Reinforcement Learning, Lee et al.

[2] RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning, Duan et al.

Reviewer_VzMvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ASTEROID, a method for learning exploration under partial observability through student-teacher distillation. The key idea is to collect on-policy histories from the student and use a privileged expert to provide training labels, so the student can learn to explore without RL. The approach is fairly simple, but seems to work well across a range of tasks, with much better sample efficiency than RL-based methods.

Critical strengths

Learning exploration from an expert that never explores is the interesting part of this paper. The tactile-pick experiment goes well beyond the three-door example: the student reaches about 92% success without proprioception, using only contact feedback and past actions. That is a difficult setting for imitation learning.

Critical weaknesses
  • The paper gives on-policy data much of the credit, but ASTEROID also replaces DPT’s random query observations with expert rollouts. These are different changes. There is a curriculum ablation in Appendix B.3, but I could not find a comparison that isolates the history source while keeping the expert supervision unchanged. The current results do not settle which change explains the gap to DPT.

  • The real-robot result is much more limited than the simulation study (6 successes versus 2 in just 10 trials per method), with the cube placed within a 10*2 cm region. This shows that transfer is possible, but the reported 40 pp gain should not carry much weight on its own.

Questions for authors

Appendix F.1 says that only successful trajectories are kept.

  • What happens to a student history when the teacher takes over and still fails?
  • Is that history dropped from training entirely?
3 comments

We thank the reviewer for their feedback and for acknowledging one of the main contributions: which is learning exploration from interactive imitation of experts that don't. We clarify all the questions below:

1. On-policy data vs expert queries

Thank you for pointing this out. We would like to make some clarifications:

  • In the discrete grid world environments in Figure 6, we compare DPT with random contexts and random queries against ASTEROID with on-policy context + expert queries. We observe that with the same test-time budget, ASTEROID learns a policy that is more efficient at exploration and converges to the expert performance faster.
  • In the continuous control experiments, in Figure 8 and Figure 12, the DPT version with random sampling of context and random query observations learns a policy with zero successes. We made DPT tractable in these experiments by using a noisy expert policy to provide coverage over histories and using expert rollouts to generate the queries (the original DPT code has a setting for this). Even with expert queries, we observe that DPT performance is below ASTEROID, implying that on-policy contexts are essential to learn in-context exploration in high-dimensional environments.
  • We will make these details about DPT implementation clear in Appendix G.

2. Real-world results + Comparing ASTEROID to frontier models

  • We acknowledge that the real-world experiments are limited, and transferring these results to the real world is future work we are excited to pursue. While we provide qualitative evidence of the behavior, we intend to scale this work to real-world tasks like retrieving objects from cluttered bins and bags.
  • To further strengthen our contribution, we include an additional experiment: comparing the exploration behavior with frontier models on the wrist-camera peg insertion task. We use Claude Opus 5.5 and use it for code as policies[1]. Like ASTEROID, it has access to the full Isaac Lab environment and the privileged expert policy to build its harness. Then, during each episode, the model receives the same observations and writes code multiple times based on the subtask and harness it created. We observe that ASTEROID policies achieve a higher success rate (∼ 78% vs. ∼ 50%) while being more ~2x efficient. We found that Opus writes aggressive search strategies that fail by either outputting unsafe actions or not finding the peg.

3. Only successful trajectories are kept

  • We discard the failed trajectories, as the policy is trained via supervised learning. The failure is a result of the experts being only ~90% successful, and we don't want to imitate suboptimal trajectories. We will make the explanation clearer in the paper.
  • As a result, we drop the context generated in such trajectories. But your feedback leads to an interesting idea: we could make efficient use of the student context by resampling expert behavior under the same context until it succeeds, which would further increase the sample efficiency of the method.

[1] Code as Policies: Language Model Programs for Embodied Control, Liang et al.

Thank the authors for the clarification. I appreciate the additional explanation regarding the experimental setup. However, I'm still not convinced by the contribution. I see ASTEROID is primarily a combination of existing ideas in DPT and interactive imitation learning, with limited methodological novelty.

Thank you for responding to our rebuttal.

In our paper, we compare ASTEROID to DPT, DAgger and seven other baselines across eleven different environments of varying complexity (including on real robots unlike prior work) and show that

  • simple changes to student-teacher distillation enables learning exploration which is efficient in terms of training samples required and test-time budget, and
  • the simplicity of this method allows it to scale to higher dimensional and complex tasks.

We believe that this provides strong evidence towards our core contribution - learning exploration from experts that don't explore. However, we would be happy to include any additional experiments or discussion to further strengthen our contributions and claims. We hope that this further clarifies our overall paper, and you would consider this in your evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.