acceptodds
Under review as a conference paper at ICLR 2027

Learning on-Policy Learning to Specialize Robot Policies

Abstract

We introduce Generative On-Policy Learners (GOLs), generative models that quickly adapt policy parameters to an unknown target domain. Conditioned on an initial (potentially random) policy and a few interaction episodes in the target domain, GOLs learn to generate improved policy parameters, by mimicking the policy evolution in traditional on-policy reinforcement learning. Unlike prior approaches, GOLs do not require (1) knowledge about domain parameters and rewards, (2) a large number of target domain interactions, or (3) expensive online domain identification. Generation is performed in a behavior-aligned latent space. We evaluate GOLs on continuous control problems, where they outperform state-of-the-art baselines in few shot domain adaptation on the most dynamic tasks. In addition, we demonstrate that our weight generation approach can iteratively improve policies in a sim-to-real transfer task. Finally, we characterize the key properties of the training dataset required to effectively train the weight generator, offering insights for scaling generative policy synthesis.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
meta-learning, few-shot adaptation, continuous control with hypernetworks.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HYtgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Generative On-Policy Learners (GOLs), a method for few-shot policy adaptation through weight generation. Given an initial policy and a short period of trajectory collected from target domain, GOL generates policy parameters for target domain policy. The method first trains an autoencoder to encode policies into latent space, then trains an evolution module that predicts update for checkpoints. This paper reports strong performance on some out-of-distribution settings, especially the more dynamic tasks.

Critical strengths
  1. This paper studies a setting where target-domain rewards and domain parameters are unavailable. This problem formulation is practical since usually we have little information about target domain. The author also test their method on OOD setting and real world senarios, which proves its performance on target-domain.
Critical weaknesses
  1. The diversity of experiments is limited. Including real-world experiments, this paper test the proposed method on ant, legged, and drone racing environment, with OOD tests conducted on the same task families. The experimental scope is too narrow to support the main claim of the paper especially the method does not outperform other baselines by a large margin.
  2. The paper does not explain why GOL is mostly comparable to baselines in-distribution but outperforms them on some OOD tasks. Also, the paper introduce that previous weight generation method can be sensitive to errors. GOL can also be divided into this category, but the paper does not explain how they overcome this shortcoming.
Questions for authors
  1. Could the author evaluate GOL on additional experiments with different dynamics and embodiedments? The author may refer to DADP[1 ], Meta-DT[2] about the possible extensive experiments design.
  2. Could the author explain the design intuition of their proposed framework including network architecture and training recipe? Also, why GOL improves more on OOD settings rather than ID setting? What experiment results can support the claim? This is my biggest concerns, which determines whether this paper can be accepted.

[1]Wang, P., Liu, Q., Lin, H., Li, Y., Zhan, G., Tomizuka, M., & Wang, Y. (2026). DADP: Domain Adaptive Diffusion Policy. arXiv preprint arXiv:2602.04037. [2]Wang, Z., Zhang, L., Wu, W., Zhu, Y., Zhao, D., & Chen, C. (2024). Meta-DT: Offline meta-RL as conditional sequence modeling with world model disentanglement. Advances in Neural Information Processing Systems, 37, 44845-44870.

3 comments
AuthorsOpenReview ↗

Response to Reviewer HYtg (1/2)

The diversity of experiments is limited. Including real-world experiments, this paper test the proposed method on ant, legged, and drone racing environment, with OOD tests conducted on the same task families. The experimental scope is too narrow to support the main claim of the paper especially the method does not outperform other baselines by a large margin.

Could the author evaluate GOL on additional experiments with different dynamics and embodiedments? The author may refer to DADP[1], Meta-DT[2] about the possible extensive experiments design.

While we appreciate the concern of the reviewer about diversity we want to underline that our experimental design already targets three different embodiments and dynamics. In particular, legged robot and drone racing tasks have more complex dynamics (in term of contacts and mode instabilities) than the mujoco tasks proposed in the refered papers. We thus believe that including them is not necessary to support our paper’s conclusions.

Regarding OOD tests, we would like to clarify that the embodiment domain designs in our ant and legged robot tasks are in the same group as propopsed in the refered papers. However, our design choices use much smaller in-distribution ranges and larger out-of-distribution ranges compared to the refered papers, which makes our tasks harder in benchmarking the generalization abilities of different methods.

Could the author explain the design intuition of their proposed framework including network architecture and training recipe?

We introduced several design choices in our proposed framework and training recipes, which refer to several different previous work.

  1. Two-stage training: we’re inspired by prior work in two-stage training framework for image generation [1]. This idea was already applied to weight generation in prior work for image classification [2], showing that latent-weight representations are smooth and robust to sampling noise. These representations are effective to sample diverse and performant new weights, which showcases better generative modeling. Therefore, we propose to follow a similar approach and learn the mapping from weight space to latent space using an autoencoder, and then learn the weight evolution in the latent space. Furthermore, the two-stage training pipeline reduces the training wall time significantly while preserving the performance compared to weight-space training pipeline as detailed in Appendix G.
  2. Autoencoder: we use layerwise normalization and encoding for MLP weights as proposed in [2]. We’re inspired by [3] for trajectory-conditioning generation, and we decided to jointly encode the weights and trajectories after doing ablation study. We use permutation augmentation [4] for weights during training. We’re inspired by [5] to use behavioral loss and extend it to DAgger-style loss for robotics tasks [6]. The DAgger-style loss in particular targets the weight sensitivity problem outlined in the introduction, as it exhibits better regret scaling with horizon length.
  3. Evolution module: we use a transformer-based architecture for the evolution module design inspired by [7]. We use pair-wise checkpoint sampling including intermediate checkpoints for training to favour iterative improvement.

[1] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." CVPR 2022.

[2] Schürholt, Konstantin, et al. "Hyper-representations as generative models: Sampling unseen neural network weights." NeurIPS 2022.

[3] Liang, Yongyuan, et al. "Make-an-agent: A generalizable policy network generator with behavior-prompted diffusion." NeurIPS 2024.

[4] Konstantin Schürholt, Dimche Kostadinov, Damian Borth. "Hyper-Representations: Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction." NeurIPS 2021.

[5] Meynent, Léo, et al. "Structure is not enough: Leveraging behavior for neural network weight reconstruction." arXiv 2025.

[6] Ross, Stéphane, Geoffrey Gordon, and Drew Bagnell. "A reduction of imitation learning and structured prediction to no-regret online learning." AISTATS 2011.

[7] Peebles, William, et al. "Learning to learn with generative models of neural network checkpoints." arXiv 2022.

AuthorsOpenReview ↗

Response to Reviewer HYtg (2/2)

Also, why GOL improves more on OOD settings rather than ID setting? What experiment results can support the claim?

We have performed a new analysis to show that the advantage of our approach out of distribution stems from to its training procedure. The optimization process of prior work focuses on reward optimization, i.e., maximizing the sum of total rewards (in distribution). In contrast, we train GOL to learn-to-learn, by predicting higher-performance policies from lesser-performant ones. To ablate the effect of the optimization, we train GOL to achieve the highest reward in distribution. To do this, we directly train GOL with reinforcement learning. This mimics the optimization performed by DR-H, RMA, GENTLE, UNICORN + CERTAIN, and RAPTOR* baselines. The details of this process can be found in appendix G.2.

The results of this experiment are shown in the table below. We find that, when optimized for reward maximization alone, GOL exhibits the same failure case observed in the baselines: high performance in distribution and poor generalization out of distribution. Moreover, this ablation of GOL achieves a score OOD which closely resembles some of the baselines. Overall, this analysis justifies the claim that learning-to-learn is key to domain adaptation. We have added this discussion in Fig. 5 at line 426 in the manuscript.

IDOOD
GOL113.5594.32
GOL-RL116.6662.98
Reviewer_HYtgOpenReview ↗

Follow-up questions and reply

I thank again to the authors for their effort on this work and rebuttal. However, my concerns about narrow evaluation still remains. I admit that Mujoco environment may seem to be simpler on embodiments and dynamics. But the current experiments with only three main evaluation environment constrains its generalization. Also, I note the explanation and follow up experiments provided by authors about training procedures. But it still can not explain why this difference on training setting can influence its performance. I am sorry to retain my score.

Reviewer_obUROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Generative On-Policy Learners (GOLs), a conditional hypernetwork that generates updated policy weights from the current policy parameters, a short interaction trajectory, and a desired performance level. It combines a behavior-aligned autoencoder with a transformer trained to predict later RL checkpoints in latent space. The method enables adaptation without target-domain rewards or dynamics parameters, with experiments on locomotion, drone racing, and real-world drone transfer.

Critical strengths
  1. Practical adaptation setting: Adapting from short state–action trajectories without target-domain rewards or privileged dynamics information is useful for robotics.
  2. Encouraging empirical results: GOL outperforms the evaluated baselines on OOD quadruped locomotion and drone racing, while the real-drone experiment demonstrates adaptation from initially unsuccessful policies.
  3. Well-motivated training design: Behavioral reconstruction and checkpoint-evolution training address relevant challenges in policy-weight generation, with ablations supporting several design choices.
Critical weaknesses
  1. Limited statistical validation: Figure 3 reports results from only one independently trained GOL model, whereas baselines use four or sixteen training seeds. Evaluating multiple initial policies, domains, and rollouts does not capture variability across generator-training runs, limiting confidence in the robustness of the reported improvements.
  2. Unequal source-training resources: GOL relies on 500 specialist-training runs per task, followed by substantial generator training. Some baselines receive considerably smaller source-training budgets. Although this discrepancy is acknowledged, the experiments do not sufficiently separate the benefits of the proposed method from those of greater training resources.
  3. Incomplete validation of the adaptation mechanism: The ablations examine removing input weights but do not directly isolate the contribution of target-domain trajectory information. Consequently, it remains unclear how much improvement comes from domain-specific adaptation versus a strong policy-improvement prior learned from the checkpoint corpus.
  4. Limited evidence for real-world reliability and superiority: The hardware experiment considers three initial policies on one drone and one track. Recovering initially unsuccessful policies is encouraging, but the lap-time improvement over RAPTOR* is modest, and the small evaluation does not establish a robust comparative advantage.
Questions for authors
  1. Can the authors provide results from additional independent GOL training seeds and clarify whether the reported error bars reflect variability across domains, initial policies, rollouts, or independently trained models?
  2. What are the total source-environment interactions and training costs for each method, including specialist generation? Does GOL retain its advantage in a focused comparison using more comparable source-training resources?
  3. How many independent adaptation sequences and evaluation flights were conducted in the hardware experiment?
3 comments
AuthorsOpenReview ↗

Response to Reviewer obUR (1/2)

Can…independent GOL training seeds…models?

We provide the results of GOL and all the other baselines using 4 different training seeds in the following table. The reported standard deviation reflect the variation across different trained models, similarly as the other baselines. We have updated the results in Fig. 3.

MethodAnt IDAnt OODLegged robot IDLegged robot OODDrone racing IDDrone racing OOD
GOL (ours)5180 ± 811906 ± 1432.33 ± 0.0411.93 ± 0.21111.0 ± 3.091.9 ± 1.7
HyPoGen5320 ± 24935 ± 31832.28 ± 0.095.95 ± 1.8586.3 ± 2.428.3 ± 6.6
DR-H6404 ± 752357 ± 14525.49 ± 0.199.79 ± 0.97116.7 ± 0.774.0 ± 31.7
RMA5206 ± 1411175 ± 27831.82 ± 0.369.33 ± 2.01113.4 ± 2.415.7 ± 3.8
RAPTOR*5405 ± 131659 ± 22631.19 ± 0.285.13 ± 0.25112.0 ± 1.674.0 ± 7.6
GENTLE5132 ± 3781653 ± 18728.49 ± 5.784.37 ± 1.3872.3 ± 22.758.5 ± 28.3
UNICORN+CERTAIN3799 ± 1309716 ± 51013.11 ± 3.782.07 ± 0.5040.4 ± 2.329.3 ± 13.0

What are the total source-environment interactions and training costs for each method, including specialist generation? Does GOL retain its advantage in a focused comparison using more comparable source-training resources?

We refer the reviewer to Reviewer R9nX answer to Q2. There we provide the following: (1) a detailed breakdown of the total source-environment interactions for our method, including specialist generation, autoencoder and evolution module training, using the drone racing task; (2) a set of GOL variants with lower specialist generation, autoencoder and evolution module budgets, yet on par performance; (3) Two tables comparing the performance of GOL and its variants to online and offline RL methods using matched source-environment interactions. From these results we confirm that GOL still retains its advantages compared other baseline methods. This is in spite of small performance improvements of certain baseline given the higher budget. We have added the results and discussion in Sec. 4.4.

We report the training wall clock time of all methods with aligned training data budget in the following table (added to the manuscript in Appendix E). We benchmark the training costs on a TPU v4-8 pod with 4 chips and 32GB TPU memory per chip. Since some of the baseline methods are not optimized for parallelized training across multiple chips, we compute the chip hours instead of the pod hours. For GOL, it takes 32 chip hours for autoencoder training, and ~156 chip hours for the evolution module training. It takes much longer to train evolution module since we use transformer-based architecture, and the training speed is bottlenecked by gradient updates, not simulation time.

MethodTraining data (env steps)Chip-hours
Ours3.3e9189
HyPoGen3.3e93.5
CERTAIN3.3e94.2
RAPTOR3.3e98.3
GENTLE3.3e920

For the offline RL baseline methods, they all use small architectures and train much faster. However, it takes 620×8.75/60×4=362620 \times 8.75/60\times 4 = 362 chip hours to collect the training dataset with 620 domains, 8.75 minutes for each specialist training and evaluation.

MethodData collectionTrainingTotal
Ours12188200
RMA–1111
DR-H–284284

For the online RL baseline methods, RMA trains much faster than GOL, but DR-H trains slower due to large number of gradient updates.

AuthorsOpenReview ↗

Response to Reviewer obUR (2/2)

Incomplete validation of the adaptation mechanism: The ablations examine removing input weights but do not directly isolate the contribution of target-domain trajectory information. Consequently, it remains unclear how much improvement comes from domain-specific adaptation versus a strong policy-improvement prior learned from the checkpoint corpus.

We thank the reviewer for pointing out this missing ablation. We have added these results to the main manuscript (Fig. 5) and included them in the table below. Overall, removing the trajectory from the model's input while leaving everything else unchanged causes a substantial performance degradation, confirming that policy rollouts provide essential information for identifying the target domain and inferring policy improvements.

MethodIDOOD
Ours113.694.4
Ours w/o trajectory30.250.5

Limited evidence for real-world reliability and superiority: The hardware experiment considers three initial policies on one drone and one track. Recovering initially unsuccessful policies is encouraging, but the lap-time improvement over RAPTOR* is modest, and the small evaluation does not establish a robust comparative advantage. How many…in the hardware experiment?

While we appreciate the concern of the reviewer diversity of real-world experiments, we would like to point out that, to the best of our knowledge, it is the first time that weight generation has been demonstrated capable of effective few shot sim-to-real transfer. The observation that, on the task of drone racing (where track layouts are fixed and known in advance [1]), GOL outperforms established baselines could inspire interesting future work on sim-to-real transfer (which is out of scope for our manuscript).

To clarify, for RAPTOR* and DR-H we collect 18 lap times in the real world, comprising samples from two training seeds over three flights per seed, with each flight consisting of three laps. For GOL we collect 9 laps per optimization step, collected from three policies over one flight with three laps per flight. We find that the policies generated by GOL at the second adaptation iteration (7.81 ± 0.30 s) are on average comparable or faster than the ones by RAPTOR* (8.35 ± 0.13 s) and DR-H (12.07 ± 2.79 s).

[1] Kaufmann, Bauersfeld, Loquercio, Müller, Koltun, Scaramuzza, “Champion-level drone racing using deep reinforcement learning”, Nature 2023.

Thank you for the additional experiments and clarifications. The results across four independently trained models substantially address my concern about training variability. The significant performance drop when removing trajectory input also provides useful evidence that the interaction trajectory contributes to adaptation. Overall, the rebuttal meaningfully strengthens the empirical evidence and addresses several central concerns. I therefore raise my rating from 2: Weak rejection to 3: Weak acceptance.

Reviewer_E9nXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A significant challenge in reinforcement learning is presented by the gap between the training domain (source) and the domain in deployment (target), in which the target may vary in its physical structure (payload or onboard sensors) or its objective. To this end, adaptation of traditional, on-policy RL parameters may be required to deploy in the real world. To this end, the authors propose generative on-policy learners (GOL), which are generative latent space models that use (i) a set of initial policy parameters and (ii) a single target trajectory to generate new weights adapted from the initial trained policy. Further, GOL need not have access to the rewards associated with the target domain, as measuring these rewards may not be practical in many circumstances.

The developed approach is validated in simulated environments (MuJoCo continuous control tasks in which the source domain differs from the target domain in morphology, such as shortened leg lengths), and on a sim-to-real transfer in a drone racing scenario. GOL is shown to yield similar performance to existing baselines on less dynamic tasks, such as in the ant baseline, but offers significantly improved performance in the drone racing task. Specifically, GOL leads to a faster lap time on the drone racing task than the two other strongest baselines after 3 successive rounds of adaptation.

Critical strengths
  1. Selection of baseline methods is appropriate in both breadth across different types of adaptation approaches and depth within each approach.

  2. The sim-to-real transfer demonstrates the true applicability of the developed method in real-world robotics applications. Specifically, the ability to adapt using a single trajectory and successfully execute the racing task with faster lap times than the baseline is very impressive.

Critical weaknesses
  1. The role of source training resources in the comparative gains remains unclear. GOL uses 500 RL runs per task with 200 checkpoints each, substantially more source data than the baselines, and Figure 5(b) shows that reduced training domains preserve much performance. However, this does not establish how the strongest baselines compare under similar source training resources. The cost-performance trade-off of using additional source training to reduce target interaction needs a clearer comparison.

  2. The analysis for the dependence on checkpoint training algorithms exceeds the supporting evidence. Figure 7 attributes the better results from SAC checkpoints to a smoother optimization landscape using effective rank and gradient projection statistics, but these do not directly establish robustness to the parameter errors produced by GOL. The discussion later calls this a hypothesis, while the figure caption and preceding analysis make stronger claims. The observed SAC–PPO difference is informative, but the proposed account of which checkpoint properties favor weight generation remains insufficiently validated.

Questions for authors
  1. Can the authors repeat GOL training with independent seeds for the main out-of-distribution evaluations, using the same initial policies and target domains, and distinguish variation across generator training runs from variation across initial policies, target domains, and evaluation rollouts—do the advantages over the strongest baselines persist?

  2. Can the authors provide a consolidated comparison of source environment interactions and training computation, including checkpoint collection, autoencoder training, and evolution model training, and show that how GOL and the strongest baselines compare under similar source training budgets, particularly when GOL uses the reduced dataset from Figure 5(b)?

  3. On the existing Ant checkpoints, are SAC policies less sensitive than PPO policies to parameter perturbations at comparable policy performance and perturbation magnitude, and can the authors test perturbations representative of GOL’s reconstruction or generation errors to determine whether the resulting return degradation supports the explanation in Figure 7?

I would consider raising my score if the authors address these concerns through convincing clarification and supporting evidence.

4 comments
AuthorsOpenReview ↗

Response to Reviewer E9nX (1/3)

Can the authors repeat GOL training with independent seeds for the main out-of-distribution evaluations, using the same initial policies and target domains, and distinguish variation across generator training runs from variation across initial policies, target domains, and evaluation rollouts—do the advantages over the strongest baselines persist?

We provide the results of GOL and all the other baselines using 4 different training seeds in the following table. The reported error bars or standard deviation reflect the variation across different trained models, similarly as the other baselines. For the OOD domains in racing and legged robot task, GOL outperforms all the other baselines. The relative ranking for the OOD domains on the Ant task persists as well, i.e., GOL performs better than the other baselines except for the DR-H baseline. We have updated the results in Fig. 3.

MethodAnt IDAnt OODLegged robot IDLegged robot OODDrone racing IDDrone racing OOD
GOL (ours)5180 ± 811906 ± 1432.33 ± 0.0411.93 ± 0.21111.0 ± 3.091.9 ± 1.7
HyPoGen5320 ± 24935 ± 31832.28 ± 0.095.95 ± 1.8586.3 ± 2.428.3 ± 6.6
DR-H6404 ± 752357 ± 14525.49 ± 0.199.79 ± 0.97116.7 ± 0.774.0 ± 31.7
RMA5206 ± 1411175 ± 27831.82 ± 0.369.33 ± 2.01113.4 ± 2.415.7 ± 3.8
RAPTOR*5405 ± 131659 ± 22631.19 ± 0.285.13 ± 0.25112.0 ± 1.674.0 ± 7.6
GENTLE5132 ± 3781653 ± 18728.49 ± 5.784.37 ± 1.3872.3 ± 22.758.5 ± 28.3
UNICORN+CERTAIN3799 ± 1309716 ± 51013.11 ± 3.782.07 ± 0.5040.4 ± 2.329.3 ± 13.0
AuthorsOpenReview ↗

Response to Reviewer E9nX (2/3)

Can the authors...source environment interactions and training computation...GOL and the strongest baselines...reduced dataset...?

We compute how many source-domain transitions are used during data collection, autoencoder training, and evolution module training for our methods, and compare results in the drone racing task across 4 seeds against baselines with matched data budgets. In summary, we show that a GOL variant using 100 times fewer source environment transitions outperforms the strongest baselines under similar training budgets. We have added the following analysis in Sec. 4.4 and Appendix D.1 & E. Below are answer to the questions in detail.

In the current setting, GOL training uses 6.5×10116.5\times10^{11} transitions. However, we found it can still outperform other baselines with fewer transitions. We term GOL-R1 our method with fewer transitions during autoencoder and evolution module training, and GOL-R2 our method with fewer training data collection. GOL-R2 uses only 6.9×1096.9\times10^{9} interactions, about 100 times less. The interactions are broken down as follows.

Training data collection originally uses SAC policies trained on 200M transitions for a single training run, across 620 training runs. This yields a total of 200×106×620=1.24×1011200\times10^6\times620=1.24\times10^{11} transitions. However, training with SAC policies with only 26M transitions, across 152 domains yields competitive results with only 4.0×1094.0\times 10^9 transitions.

Autoencoder training originally runs 100 epochs, with 787787 batch updates per epoch, running 10241024 ground-truth policies and their reconstructions for 300300 steps per batch update. This yields 78,700×2×1024×300=4.8×101078{,}700\times2\times1024\times300=4.8\times10^{10} transitions. We shorten this to 12 epochs yielding only 2.9×1092.9\times 10^9 transitions.

Evolution module training runs 1000 epochs, with 98 batch updates per epoch, rolling out 16384 GT policies for 300 steps per batch update. This yields 98,000×16,384×300=4.8×101198{,}000\times16{,}384\times300=4.8\times10^{11} transitions. Instead of resampling during training, we precollect 10 trajectories per checkpoint and reuse them. This reduces transitions to 3.7×1083.7\times 10^8 transitions.

Rewards by policies adapted by GOL show little degradation as shown the table below:

GOL variantCollectionTraining (AE + evolution)IDOOD
Original1.24×10111.24\times10^{11}5.6×10115.6\times10^{11}111.0 ± 3.091.9 ± 1.7
Reduced training (GOL-R1)1.24×10111.24\times10^{11}3.3×1093.3\times10^{9}116.5 ± 1.596.3 ± 2.0
Reduced training + collection (GOL-R2)4.0×1094.0\times10^{9}2.9×1092.9\times10^{9}117.0 ± 0.193.5 ± 1.9

We report results for offline RL methods (RAPTOR, GENTLE, UNICORN+CERTAIN and HyPoGen ) that use trajectories by expert policies from the GOL training dataset (same source training budget) and online RL methods (DR-H and RMA ) that need no expert policies.

For the offline RL baselines, the data budget misalignment is the transitions sampled during online training. Therefore, we report results for the original training budget count (bottom table left), and matched training budget at to 3.3×1093.3\times10^9, (same as GOL-R1, bottom table right) and compare their performance. We see that under the same source training budget (used for data collection) and model training budget GOL-R1 outperforms offline RL methods.

MethodOriginal transitionsIDOODMatched transitionsIDOOD
GOL-R1 (ours)5.6×10115.6\times10^{11}111.0 ± 3.091.9 ± 1.73.3×1093.3\times10^{9}116.5 ± 1.596.3 ± 2.0
HyPoGen6.2×1076.2\times10^{7}86.3 ± 2.428.3 ± 6.63.3×1093.3\times10^{9}97.2 ± 2.250.9 ± 15.8
RAPTOR*2.0×1062.0\times10^{6}112.0 ± 1.674.0 ± 7.63.3×1093.3\times10^{9}107.9 ± 3.032.9 ± 7.0
GENTLE6.2×1076.2\times10^{7}72.3 ± 22.758.5 ± 28.33.3×1093.3\times10^{9}58.0 ± 24.876.9 ± 24.5
UNICORN+CERTAIN6.2×1076.2\times10^{7}40.4 ± 2.329.3 ± 13.03.3×1093.3\times10^{9}32.5 ± 1.936.0 ± 4.1

For the online RL methods, we increase the total training budget to 6.9×1096.9\times10^9, and we compare their performance with GOL-R2. For RMA , we further align the teacher training budget to our data collection budget, i.e., 4.0×1094.0\times10^9, and we use 2.9×1092.9\times10^9 training data budget for the student training. We report the results in the table below. We see that at matched budgets, GOL-R2 also outperforms online RL methods.

MethodOriginal transitionsIDOODMatched transitionsIDOOD
GOL-R2 (ours)5.6×10115.6\times10^{11}111.0 ± 3.091.9 ± 1.76.9×1096.9\times10^{9}117.0 ± 0.193.5 ± 1.9
RMA2.0×1082.0\times10^{8}113.4 ± 2.415.7 ± 3.86.9×1096.9\times10^{9}113.4 ± 0.918.8 ± 6.0
DR-H2.0×1082.0\times10^{8}116.7 ± 0.774.0 ± 31.76.9×1096.9\times10^{9}115.7 ± 1.573.6 ± 25.8
AuthorsOpenReview ↗

Response to Reviewer E9nX (3/3)

On the existing Ant checkpoints, are SAC policies less sensitive than PPO policies to parameter perturbations at comparable policy performance and perturbation magnitude, and can the authors test perturbations representative of GOL’s reconstruction or generation errors to determine whether the resulting return degradation supports the explanation in Figure 7?

As suggested, we study perturbations of SAC and PPO policies. From the held-out training dataset we select policies in the reward range [600, 800) and [1200, 1400), which are roughly 50% and 100% of the maximum rewards. We apply perturbations in the following way: we first normlize weights layer-wisely, then apply perturbation θ′=θ+σ ϵ\theta' = \theta + \sigma\,\epsilon with ϵ∼N(0,I)\epsilon \sim \mathcal{N}(0, I) and noise scale σ\sigma, and finnaly invert normalization. This accounts for varying weight scales across checkpoints. We evaluated the pertubation scales σ∈[0.0,0.02]\sigma \in [0.0, 0.02]. Each perturbed policy is then evaluated deterministically over 200-step episodes in 512 parallel environments. We report the mean relative reward reduction ∣E[J(θ′)]−E[J(θ)]∣/E[J(θ)]|\mathbb{E}[J(\theta')] - \mathbb{E}[J(\theta)]|/\mathbb{E}[J(\theta)]. The σ=0\sigma = 0 column re-evaluates the unperturbed policy under four separate reset seeds to show the noise in evaluation.

The results are shown below. For low-performing policies (reward range 600-800) SAC policies are more sensitive to weight perturbations than PPO policies. This coincides with our finding in Fig. 7 (middle), where early checkpoints show higher gradient projections (analogous to our relative reward reduction). As checkpoints are better trained, SAC remains less sensitive than PPO, coinciding with Fig. 7.

reward rangemethodσ = 00.0050.010.02
600-800SAC0.01190.01540.02420.0418
600-800PPO0.01340.01380.01440.0190
1200-1400SAC0.00490.00440.00590.0132
1200-1400PPO0.01070.01150.01230.0160

We turn to studying the sensitivity of our weight generation algorithm. To this end, we introduce latent-space perturbation on the order of the hypernetwork's latent prediction error on held-out checkpoint pairs (RMSE of 0.197 for SAC and 0.145 for PPO). We then encoded held-out policies θ\theta at several training fractions into latents μ\mu, and perturbed their latents as z~=μ+σe ϵ\tilde{z} = \mu + \sigma_e\,\epsilon with ϵ∼N(0,I)\epsilon \sim \mathcal{N}(0, I) and σe\sigma_e equal to the evolution module RMSE error, and decoded them back to weights θ~\tilde{\theta}. Finally, we report the relative drop in return Δ=1−E[J(θ~)]/E[J(θ)]\Delta = 1 - \mathbb{E}[J(\tilde{\theta})]/\mathbb{E}[J(\theta)] in the table below. We compute these numbers for checkpoints at different training stages. Results show that the performance of reconstructed PPO policies degrades more than that of reconstructed SAC policies at corresponding generation error magnitudes, independently of the training stage. This supports our hypothesis that the latent representations of checkpoints generated by SAC are more robust to prediction errors than the one generated by PPO.

Training Progress0.20.40.60.81.0
Ant SAC−8.3%7.2%10.9%19.3%25.7%
Ant PPO14.3%22.5%29.6%34.6%31.4%

We add these results in Fig. 8 and discussion in Line 487 and Appendix H.

Thank you for the thoughtful response and the additional experiments. I find the idea of adapting policy weights from a single target trajectory particularly interesting. The results across four training seeds and the drone racing comparisons under matched source interaction budgets give me greater confidence in the reported gains. I have raised my rating from 2 to 3.

Additional little request: In the camera ready version, I would appreciate a revision to the caption of Figure 7 that presents landscape smoothness as a possible explanation for GOL’s stronger performance with SAC checkpoints.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We would like to thank the reviewers for their insightful comments, and for finding our baseline selection appropriate and the real-world results impressive.

The reviewer’s proposed experiments helped us to deepen our understanding of our method’s sample efficiency, our design choices, and of why our method has an advantage over baselines out of distribution. We have updated the manuscript to reflect this deeper understanding of the approach, as well as the supporting experiments.

We have made the following major additions to the manuscript:

  1. We match the training budget of our approach to the different baselines, and introduce GOL variants which use up to 100x fewer environment interactions. Surprisingly, we find that the performance of our method is almost unchanged under this setting. When controlling for the training budget of environment steps, we retain a similar advantage to the baselines as reported in our original manuscript. (Sec. 4.4 and Appendix D.1 & E)
  2. We provide the results of GOL using 4 different training seeds (Fig. 3). The reported standard deviation reflects the variation across different trained models, similarly as the other baselines.
  3. We ablated the importance of the robot’s trajectory as an input to our model (Fig. 5). The results indicate that the trajectory is important to identify the domain and infer policy updates.
  4. We ran additional experiments to verify that policies trained with SAC are more robust to estimation noise than policies trained with PPO, both in the weight space and in the latent space. As suggested by the reviewer, we select the noise magnitude in the latent space to match our average training error. These results support our hypothesis that SAC policies are easier to fit for a weight-generation system. We have added these additional results to the manuscript (Fig. 8,lines 487, and Appendix H).
  5. We studied why our approach performs better than the baselines out of distribution, but comparably in distribution. This analysis shows that the main culprit is our training procedure. The optimization process of prior work focuses on reward optimization, i.e., maximizing the sum of total rewards (in distribution). In contrast, we train GOL to learn-to-learn, by predicting higher performance policies from lesser performant ones. To ablate the effect of the optimization, we train GOL to achieve the highest reward in distribution, either by direct optimization with reinforcement learning (as DR-H, RMA, GENTLE, UNICORN + CERTAIN, RAPTOR*). When optimized with this objective, GOL exhibits the same failure case observed in the baselines: high performance in distribution and poor generalization out of distribution. This supports the hypothesis that our learning target, i.e. the update between policies of increasing performance, is similar ID as OOD, supporting generalization. Overall, this analysis justifies the claim that learning-to-learn is key to domain adaptation. We have added this result in Fig.5 and discussion at line 426 in the manuscript.

Please see the attached revised paper with the changes highlighted in blue. We further answer to all reviewers’ questions below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.