- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The submission addresses asymmetric teacher-student learning for partially observable control problems. In this asymmetric setting, the teacher and the student have access to different forms of information concerning the task:
- the teacher is capable of observing the underlying state of the environment, and represents behavior that is performs well in the corresponding MDP.
- the student is incapable of observing the underlying state of the environment, and is tasked to learn behavior that performs well in the corresponding POMDP.
Purely offline imitation-learning approaches suffer from a number of issues. This submission primarily addresses an issue concerning distribution shift, typical of any imitation learning approach, though the authors reframe it in this work in terms of exploration capabilities. The proposes solution is fundamentally a version of DAgger for this asymmetric scenario, i.e., the training dataset should be periodically repopulated with the student's on-policy sample experience, in order to have the training data maintain support of the student's visited contexts. The proposed approach will eventually approximate the teacher's posterior sampling distribution (conditioned on the student's context).
The evaluation is performed on a substantial number of environments.
Critical strengths
The evaluation is fairly extensive (though still limited to an unspecified subclass of POMDPs)
See below for a discussion of POMDP subclasses.
Critical weaknesses
The submission conflates the use of posterior sampling approaches for unknown MDPs and meta-RL, vs POMDPs
The authors claims that "Bayesian posterior sampling is a provably efficient exploration strategy". However, this is only applicable to learning under unknown MDPs via posterior sampling (PSRL) approaches, and has nothing to do with control in POMDPs. A notable difference is that the Bayesian posterior in PSRL is a posterior over possible MDP models, whereas the Bayesian posterior that is relevant for a POMDP is over the underlying state, i.e., the agent's belief. These two settings are not interchangeable.
There are various statements that are made about partially observalbe that are inaccurate, presumably due to actually concerning meta-RL rather than partially observable RL, e.g.: "This expert can teach the student how to act if the hidden state is known" is inaccurate. Even under the scenario where a partially observable agent is able to infer the hidden state in one timestep, the current optimal action may be wholly different from that prescribed by a fully observable expert, due to state uncertainties that may re-appear in the future.
The submission implicitly addresses a very narrow notion of exploration, and never addresses its limitations
As direct consequence of the above, the version of posterior sampling as discussed and employed by the authors has unbounded regret for a generic POMDP. There are common POMDPs and contexts , where the set of optimal fully observable actions and the set of optimal partially observable actions do not overlap. This has been sometimes been called "cross-observability optimality", e.g., in "Leveraging Fully Observable Policies for Learning under Partial Observability" (Nguyen, 2022) and "Leveraging Fully-Observable Solutions for Improved Partially-Observable Offline Reinforcement Learning" (Wijesundara, 2025).
The issue seems to be that the authors are addressing a limited form of exploration, where the information-gathering location and the task location are the same. However, there are many situations where the information must be acquired in one location, and the task must be performed in another. In these scenarios, the fully observable teacher will trivially fail to guide the student towards performing information-gathering.
Simple example: warehouse item retrieval. The task is to retrieve an item randomly placed in a large number of possible locations. The fully observable teacher knows exactly where the item is stored, and always chooses actions that go to the item. The partially observable student does not know where the item is stored, but should consult an inventory that is available in some corner of the room, far from any storage area. The fully observable teacher will never choose to check the inventory, so the student will at best learn to guess the item locations randomly, until the item is found. This can be arbirarily suboptimal.
Contrary to what the authors claim, this is why pure imitation-learning approaches are going to be limited to some unclear subset of PODMPs, and general solutions require the use of RL.
The approach lacks depth
It is a direct application of DAgger in the asymmetric learning setting.
Questions for authors
Clarifications:
-
How would you reconcile the differences between of meta-RL (fully observable) and POMDPs?
-
Would you be able to characterize the subclass of POMDPs where asymmetric imitation learning (without an RL component) would work well?
Suggestions for future work:
- Decouple the use of meta-RL approaches and frameworks (including BAMDPs) from the use of POMDPs. It seems to me like the whole use of BAMP is not relevant to partially observable control (as the submission currently stands)
- Address the suboptimality limitations of the posterior sampling approaches for POMDPs, at the very least acknowledging large sets of POMDPs where the current approach would be sub-optimal, and would not lead to correct exploration. Ideally, find a characterization of a subclass of POMDPs where the proposed approach does work well.
- (minor) line 121: being fixed for K episodes is not a standard assumption, and does not represent a valid subset to the class of POMDPs. POMDPs cannot have memory that is shared across episodes. It is unclear why the assumption of being fixed across episodes is necessary.
- (minor) line 129: implies observable rewards at test time, which is not the standard control setting
- (minor) line 130: distribution over histories is not guaranteed to be well-defined without further assumptions that identify a marginal distribution over timesteps.
- (minor) Figure 2 does not seem to fully demonstrate the issue discussed in the text, and it is unclear what it is trying to depict. The example mostly made sense for a single-timestep decision process. What are the various paths trying to depict? why are there multiple subfigures with different subsets of paths?
2 comments
4. Differences to DAgger
We include DAgger as a baseline in Figure 9 and show that as we run standard DAgger over multiple steps, the performance degrades. Here, this happens because DAgger rolls out a student to collect histories, relabels the student at each time step to generate action labels, i.e., DAgger generates (h,s,a*) – where h,s come from the student, and a* comes from the expert. This learns a policy that fails to commit to a particular mode of behavior and dithers (see Figure 9 and website videos for example behavior).
ASTEROID, on the other hand, uses student policy to generate histories only. It then uses expert rollouts to generate both query states and action targets i.e., we generate (h,s*,a*) - where h is from the student and s*,a* is from expert rollout. In the control experiments, this is equivalent to the expert taking over the control and generating expert behavior to success. This trains the policy to commit to and follow a mode of behavior (akin to action chunking in standard imitation learning), and this mode keeps adapting based on the online context. A benefit over DAgger is that we could use offline datasets or planners as the experts are not queried on arbitrary states the students visit but over expert visited states only. Regarding our method depth: Our goal is to show that simple changes to student-teacher distillation can enable learning exploration from experts that don't explore (reviewer VzMv)
5. Reconciling the settings
-
Thank you for this clarification; we will make the distinction explicit. Similar to prior work DPT[2], RL^2[3], we first present our method under a simple class of problems - BAMDPs where prior work makes two standard assumptions: the environment has a latent hidden parameter that is unchanged during episodes, and that the policy can interact with the environment over multiple episodes under the same parameter and has access to rewards (e.g., which of three doors leads to the goal). Sampling-based exploration means acting as if one consistent guess were true, then ruling it out. In direct comparison over benchmarks from prior works in Figure 6, we show that our method learns a more efficient exploration strategy.
-
In a POMDP, the uncertainty is the belief over the current state, which can change as the agent acts (e.g., in tactile cube pick Figure 8 and 13, the cube's pose is unknown and shifts when the gripper touches it). ASTEROID handles the POMDP case the same way: conditioned on its history, the student acts like the expert would in a state sampled from its current belief, and each contact or observation updates that belief. This is related to QMDP [5] which acts on a belief over fully observed experts and shares its limitation: it never takes actions only for gathering information. We will add this distinction and the QMDP connection to the paper.
6. Minor suggestions:
- Distribution over histories: We would make it clear that the distribution over histories is defined over a marginal distribution of timesteps in the problem statement. We discuss this in informally in Section 4.3.
- Figure 2: We are just trying to show that DPT assumes that you need to cover all possible paths via random context generation in your supervised learning dataset. Based on your feedback, we will update this figure and the caption to make this clear.
[1] (More) Efficient Reinforcement Learning via Posterior Sampling, Osband et al.
[2] Supervised Pretraining Can Learn In-Context Reinforcement Learning, Lee et al.
[3] RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning, Duan et al.
[4] Bayesian Residual Policy Optimization:Scalable Bayesian Reinforcement Learning with Clairvoyant Experts, Lee et al.
[5] Learning Policies for Partially Observable Environments: Scaling Up, Littman et al.
Rebuttal
We thank the reviewer for their feedback. We address all their concerns below, particularly clarifying the confusion regarding the setup of the problems and the exploration strategy learned.
1. Clarification about Bayes-optimal exploration
We agree that ASTEROID learns posterior sampling over hidden parameters, which is not Bayes-optimal exploration. However, posterior sampling is a scalable and simple alternative that trades off optimality with learning efficiency and strong regret guarantees [1]. We note this in our introduction (L97) where we state that this exploration has bounded regret and is very efficient to learn but is not optimal. We use widely established results on this exploration from prior work [2,3]. While the goal of our paper is not to claim optimal exploration, it is to show that it can learn more effective exploration than previous imitation learning methods, including DAgger (see Figure 9), while scaling to complex tasks, even real robots. We will make this contribution and limitation clearer in our final version.
2. Making the limitations clearer about the settings
Based on your feedback, we will include a detailed explanation of the failure case of this exploration (including citing the works you mentioned). This exploration is suboptimal precisely when the information to identify hidden parameters lies completely outside the expert's support, i.e., no expert behavior enables the policy to update the latent distribution (L479).
Formally, for any two latents z=z′ in the posterior support, some history the student visits must yield feedback that distinguishes z from z′, i.e., helps the student policy shrink the posterior over the hidden parameter. The denser the feedback, the more efficient the exploration. Notably, many practical settings satisfy this because observations give dense implicit feedback (e.g., real-world tactile manipulation). We will add this criterion to the paper and highlight this condition clearly.
We further discuss a potential extension to achieve optimal exploration, which we discuss in our future work section -- use residual RL to expand the space of actions beyond what the experts can do. E.g., ASTEROID could be used to pretrain a policy that does sampling-based exploration of different regions in the warehouse. This allows it to explore certain locations and achieve a high success rate. This downstream policy could be fine-tuned via RL [4] to achieve Bayes-optimal exploration, where it could learn to take actions off the experts' manifold. We are excited to explore this direction in future work.
We appreciate your suggestions for clarifying the nuances in the theoretical literature we attempt to draw a connection with and will incorporate them into the manuscript.
3. Additional baselines including COSIL
Based on your feedback, we include COSIL as a baseline in our experiments on two tasks: tactile cube pick and the procgen maze task. To verify the implementation, we first run the implementation with full observability – COSIL reaches 100% success rate on the cube pick task within ~1.7M env steps. We then train and eval (3 seeds) it under the same partial observability as ASTEROID:
Under partial observability, COSIL fails on tactile cube pick even with 100x more environment steps than ASTEROID, and on the maze it performs on par with DPT (0.29). We observe that the student is unable to generate exploration behavior that enables expert imitation. Thus, the divergence to the expert does not decrease, and the imitation weight keeps growing. ASTEROID, on the other hand, uses expert imitation conditioned on student contexts and can learn exploration efficiently. We will add these results and implementation details to the paper. We also compare to the frontier model Opus 5.5 in response to reviewer VzMv.