- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Generative On-Policy Learners (GOLs), a method for few-shot policy adaptation through weight generation. Given an initial policy and a short period of trajectory collected from target domain, GOL generates policy parameters for target domain policy. The method first trains an autoencoder to encode policies into latent space, then trains an evolution module that predicts update for checkpoints. This paper reports strong performance on some out-of-distribution settings, especially the more dynamic tasks.
Critical strengths
- This paper studies a setting where target-domain rewards and domain parameters are unavailable. This problem formulation is practical since usually we have little information about target domain. The author also test their method on OOD setting and real world senarios, which proves its performance on target-domain.
Critical weaknesses
- The diversity of experiments is limited. Including real-world experiments, this paper test the proposed method on ant, legged, and drone racing environment, with OOD tests conducted on the same task families. The experimental scope is too narrow to support the main claim of the paper especially the method does not outperform other baselines by a large margin.
- The paper does not explain why GOL is mostly comparable to baselines in-distribution but outperforms them on some OOD tasks. Also, the paper introduce that previous weight generation method can be sensitive to errors. GOL can also be divided into this category, but the paper does not explain how they overcome this shortcoming.
Questions for authors
- Could the author evaluate GOL on additional experiments with different dynamics and embodiedments? The author may refer to DADP[1 ], Meta-DT[2] about the possible extensive experiments design.
- Could the author explain the design intuition of their proposed framework including network architecture and training recipe? Also, why GOL improves more on OOD settings rather than ID setting? What experiment results can support the claim? This is my biggest concerns, which determines whether this paper can be accepted.
[1]Wang, P., Liu, Q., Lin, H., Li, Y., Zhan, G., Tomizuka, M., & Wang, Y. (2026). DADP: Domain Adaptive Diffusion Policy. arXiv preprint arXiv:2602.04037. [2]Wang, Z., Zhang, L., Wu, W., Zhu, Y., Zhao, D., & Chen, C. (2024). Meta-DT: Offline meta-RL as conditional sequence modeling with world model disentanglement. Advances in Neural Information Processing Systems, 37, 44845-44870.
3 comments
Response to Reviewer HYtg (2/2)
Also, why GOL improves more on OOD settings rather than ID setting? What experiment results can support the claim?
We have performed a new analysis to show that the advantage of our approach out of distribution stems from to its training procedure. The optimization process of prior work focuses on reward optimization, i.e., maximizing the sum of total rewards (in distribution). In contrast, we train GOL to learn-to-learn, by predicting higher-performance policies from lesser-performant ones.
To ablate the effect of the optimization, we train GOL to achieve the highest reward in distribution. To do this, we directly train GOL with reinforcement learning. This mimics the optimization performed by DR-H, RMA, GENTLE, UNICORN + CERTAIN, and RAPTOR* baselines. The details of this process can be found in appendix G.2.
The results of this experiment are shown in the table below. We find that, when optimized for reward maximization alone, GOL exhibits the same failure case observed in the baselines: high performance in distribution and poor generalization out of distribution. Moreover, this ablation of GOL achieves a score OOD which closely resembles some of the baselines. Overall, this analysis justifies the claim that learning-to-learn is key to domain adaptation. We have added this discussion in Fig. 5 at line 426 in the manuscript.
| ID | OOD | |
|---|---|---|
| GOL | 113.55 | 94.32 |
| GOL-RL | 116.66 | 62.98 |
Follow-up questions and reply
I thank again to the authors for their effort on this work and rebuttal. However, my concerns about narrow evaluation still remains. I admit that Mujoco environment may seem to be simpler on embodiments and dynamics. But the current experiments with only three main evaluation environment constrains its generalization. Also, I note the explanation and follow up experiments provided by authors about training procedures. But it still can not explain why this difference on training setting can influence its performance. I am sorry to retain my score.
Response to Reviewer HYtg (1/2)
While we appreciate the concern of the reviewer about diversity we want to underline that our experimental design already targets three different embodiments and dynamics. In particular, legged robot and drone racing tasks have more complex dynamics (in term of contacts and mode instabilities) than the mujoco tasks proposed in the refered papers. We thus believe that including them is not necessary to support our paper’s conclusions.
Regarding OOD tests, we would like to clarify that the embodiment domain designs in our ant and legged robot tasks are in the same group as propopsed in the refered papers. However, our design choices use much smaller in-distribution ranges and larger out-of-distribution ranges compared to the refered papers, which makes our tasks harder in benchmarking the generalization abilities of different methods.
We introduced several design choices in our proposed framework and training recipes, which refer to several different previous work.
[1] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." CVPR 2022.
[2] Schürholt, Konstantin, et al. "Hyper-representations as generative models: Sampling unseen neural network weights." NeurIPS 2022.
[3] Liang, Yongyuan, et al. "Make-an-agent: A generalizable policy network generator with behavior-prompted diffusion." NeurIPS 2024.
[4] Konstantin Schürholt, Dimche Kostadinov, Damian Borth. "Hyper-Representations: Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction." NeurIPS 2021.
[5] Meynent, Léo, et al. "Structure is not enough: Leveraging behavior for neural network weight reconstruction." arXiv 2025.
[6] Ross, Stéphane, Geoffrey Gordon, and Drew Bagnell. "A reduction of imitation learning and structured prediction to no-regret online learning." AISTATS 2011.
[7] Peebles, William, et al. "Learning to learn with generative models of neural network checkpoints." arXiv 2022.