acceptodds
Under review as a conference paper at ICLR 2027

Continual Robot Policy Learning via Variational Neural Dynamics

Abstract

Robots deployed in the real world rarely operate under a single fixed dynamics model: wind changes, payloads vary, batteries drain, and hardware wears. Yet most learning-based controllers are trained once and deployed as if learning were complete. This prevents the robot from using deployment experience to recursively improve task performance. In this work, we propose a continual learning framework that uses real-world experience to improve robot policies under hidden and time-varying dynamics. Our method learns a condition-aware dynamics model from real state-action data by combining an analytical physics prior with a neural residual for unmodeled effects. Crucially, our continual learning occurs during real-world deployment: the robot repeatedly collects new interaction data, updates its learned dynamics model, and improves the policy, in a continual loop of about 10 s. During execution, a recurrent encoder identifies the current hidden condition from recent interactions, allowing the policy to retain and reuse previously learned adaptations when conditions recur rather than adapting from scratch. Unlike meta-learning, our method can learn dynamics variations directly during deployment rather than from predefined, frozen training conditions. Through simulation studies and real-world experiments, we show that the framework improves policy performance across different platforms under a range of unobserved disturbances. In simulation under large disturbances, our method reduces the hover and tracking errors by 65.7% and 53.3% over state-of-the-art online adaptation approaches. On real quadrotor tracking task under changing disturbances, the policy recovers from recurring disturbances in roughly 1s, about 5× faster than online residual re-fitting.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, quadrotor control, real world learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Jr3eOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a continual policy learning framework for robots operating under hidden, time-varying dynamics. Its core, Variational Neural Dynamics (VND), combines an analytical physics prior with a FiLM-modulated neural residual conditioned on a latent variable inferred by a recurrent encoder from recent state-action history. Overall, the research topic is interesting and the proposed method is novel. The paper is also clearly written and easy to follow.

Critical strengths
  1. The problem is well motivated and practically important. Moving beyond the train-once, deploy-forever paradigm, and in particular reusing what was learned when a condition recurs, is a clear and valuable goal.

  2. The method is conceptually clean. Using one latent space for two roles, inferred at deployment and sampled during policy training, is an elegant way to learn the condition distribution from data rather than specifying it in simulation.

  3. The real-world validation is convincing. Closing the full loop on hardware in roughly 10 s per round is impressive, and the damaged-propeller experiment shows the method handles effects that the nominal model and standard randomization do not capture.

Critical weaknesses
  1. The buffer ssems to keep growing, but the paper doesn't say how it is managed. Over a long deployment, conditions seen early on will take up a smaller and smaller share of the data. Could forgetting creep back in? The hardware runs are only about 50 s with a few conditions, so a short discussion (or a longer run) would help here.

  2. In the introduction, "Once deployed, neither approach uses new data to update the underlying model or policy" is a bit too strong. Methods that pair offline learning with online adaptation do use deployment data in some sense, and this helps them generalize even though they don't learn continually. Neural-Fly (O'Connell et al., 2022) is already cited later in the paper but isn't reflected in this sentence; FORESEER (Jia et al., IJRR 2025) is another example. It would be good to cover this line of work. The same goes for "cannot expand this distribution as new conditions arise" in Related Work. Adaptation-plus-learning methods can handle some out-of-distribution cases, so I'd soften this.

  3. "VND models a distribution of residual dynamics ... so it can learn from new conditions while retaining what it learned from earlier ones." I don't see why modeling a distribution by itself prevents forgetting. My guess is that retention really comes from replaying the whole buffer and conditioning on the latent. This statement can be improved.

Questions for authors

The ~1 s recovery is highlighted in the abstract and introduction, but only compared with residual fitting method. How does it compare with classical online adaptive controllers, e.g. INDI-based ones (Sun et al., T-RO 2022)?

4 comments
AuthorsOpenReview ↗

Weakness 1 + Weakness 2

Thank you for the positive and constructive review and for recognizing the practical and methodological contributions of our work. We address each point below, with corresponding modifications highlighted in blue in the revised manuscript.


W1. "The buffer seems to keep growing, but the paper doesn't say how it is managed. [...] Could forgetting creep back in?"

Thank you for raising this important point.

We introduce three conditions sequentially: no wind (A) and two wind disturbances of 3 m/s² with directions of 45° (B) and 135° (C). We execute three full continual learning rounds per condition (A → A → A → B → B → B → C → C → C), resulting in nine rounds (1 minute each, so 9 minutes in total in simulation) of data collection for dynamics model learning and policy refinement. We then evaluate the final policy under all three conditions without further learning.

We compare VND against Recent-data-only, which uses only the most recent deployment data, and Simple Residual (Fig. 4), which retains all historical data but without latent conditioning.

MethodError on A after learning AFinal error on AFinal error on BFinal error on C
VND (ours)0.02 ± 0.010.04 ± 0.010.09 ± 0.040.03 ± 0.02
Recent-data-only0.02 ± 0.030.34 ± 0.070.49 ± 0.120.03 ± 0.02
Simple Residual0.02 ± 0.010.33 ± 0.090.42± 0.150.32 ± 0.09

Results are trajectory tracking errors [m] over five independent seeds (mean ± std). Recent-data-only largely forgets conditions A and B, while Simple Residual struggles to handle multiple conditions despite retaining all historical data. In contrast, VND retains good performance on previously learned conditions while matching Recent-data-only on the latest condition C. This demonstrates the importance of combining experience replay with latent-conditioned dynamics learning.

While these results are encouraging, our current experiments still cover a limited number of disturbance conditions and learning iterations, especially if we consider a lifelong-learning setup of the robot deployment. As the deployment buffer grows, earlier conditions may become increasingly underrepresented, which can affect both the learned dynamics representation and policy performance. We acknowledge this limitation and have added a discussion of long-term retention and scalable buffer management to the revised manuscript.

W2. "'Once deployed, neither approach uses new data to update the underlying model or policy' is a bit too strong. [...] Neural-Fly [...] FORESEER [...]"

We agree with the reviewer that our original statements were too restrictive. These methods effectively use deployment data for online adaptation and can generalize beyond their training conditions. The distinction we intend to emphasize is that VND continually updates the underlying dynamics representation and explicitly trains the policy on newly encountered dynamics, rather than relying solely on adaptation within an offline-learned representation.

We have revised the corresponding statements as follows:

  • Sec. 1: "Once deployed, these methods use new interaction data for online adaptation within a representation learned before deployment, while the underlying learned representation remains fixed."
  • Sec. 2: "These methods can generalize to some out-of-distribution conditions through online adaptation, but do not explicitly expand the learned dynamics representation through continual learning."

We have also expanded our discussion of Neural-Fly and included FORESEER in the related work.

AuthorsOpenReview ↗

Weakness 3 + Question 1

W3. "I don't see why modeling a distribution by itself prevents forgetting. My guess is that retention really comes from replaying the whole buffer and conditioning on the latent."

Your reading is correct, and we rewrote the sentence.

Old: "VND models a distribution of residual dynamics [...] so it can learn from new conditions while retaining what it learned from earlier ones."

New: "VND is retrained on the whole deployment buffer and conditions its residual on an inferred latent, so a new condition is fitted in a new region of the latent space while earlier conditions stay represented by their replayed data."

Q1. "The ~1 s recovery [...] How does it compare with classical online adaptive controllers, e.g. INDI-based ones (Sun et al., T-RO 2022)?"

Thank you for this suggestion. We have added comparisons against INDI controller (App. A.10). In simulation, wind conditions switch every 4 s, with 240 transitions per method (16 episodes × 5 switches × 3 seeds). Recovery time is measured as the time until the position error remains below 3 cm, with unsuccessful recoveries assigned 4 s.

MethodRecovery time [s] ↓Recovery rate ↑Steady-state error [m] ↓
VND (ours)1.23 ± 0.30100%0.018 ± 0.003
INDI2.28 ± 0.0299%0.0206 ± 0.0002

VND achieves faster recovery and lower steady-state errors. However, INDI reacts more quickly immediately after switching, while VND requires recent observations to infer the new condition (mean error during the first second: 0.07 m for INDI vs. 0.09 m for VND).

We view these approaches as complementary: classical adaptive controllers provide reactive disturbance rejection, while VND enables continual policy improvement from deployment experience. Combining VND with an INDI inner-loop controller is an interesting direction for future work.

Reviewer_Jr3eOpenReview ↗

Reply to author's responses

Thank you for the thorough response. I am very satisfied with the rebuttal, especially the new comparison with INDI, the additional forgetting experiment, and the updated discussion of adaptive control methods.

Although the growing replay burden of the buffer remains a minor limitation for long-term deployment, the overall contribution is solid. I have decided to raise my score!

We sincerely appreciate your timely and encouraging response, and thank you for taking the time to carefully consider our additional experiments and clarifications. Your constructive feedback has been very valuable in helping us improve both the experimental evaluation and the clarity of our work.

Reviewer_rnsSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a continual robot policy learning framework based on VND to adapt to changing dynamics during deployment. It combines residual dynamics modeling, latent conditioning, and recurrent context encoding. Experiments on quadrotors and simulated manipulation tasks demonstrate improved adaptation and recovery performance.

Critical strengths
  1. The paper addresses a relevant and practical problem in robot learning: continual policy improvement during real-world deployment.
  2. Using a shared latent space for dynamics modeling and policy adaptation is an interesting design, particularly for handling recurring conditions.
  3. The experiments include real-world quadrotor evaluations, multiple simulated tasks, and comparisons with representative baselines.
Critical weaknesses
  1. The method relies on replay and warm-started retraining, but does not directly evaluate knowledge retention or catastrophic forgetting. Experiments with recurring conditions and broader distribution shifts would strengthen the claims.
  2. The reported recovery gains mainly compare memory-based adaptation against retraining from scratch. Stronger memory-based baselines and more rigorous statistical evaluations are needed.
  3. MMD encourages distribution alignment but does not guarantee that sampled latents represent realistic dynamics. The paper lacks direct validation and policy-level ablations of this mechanism.
  4. Experiments rely on controlled dynamics, with manipulation evaluated only in simulation. Several implementation details and hyperparameters are also insufficiently specified.
Questions for authors
  1. Can the authors distinguish VND more clearly from latent-conditioned adaptation (RMA, Neural-Fly) and online residual learning (LOTF)? Which specific design choice is responsible for the claimed novelty? (I would consider increasing my score if this is clarified convincingly.)
  2. Could the authors evaluate sequentially introduced conditions (A→B→C→A), revisiting earlier ones, and compare against recent-data-only and replay-based baselines? Does the latent geometry of old conditions remain stable?
  3. How reliable is sampling from the MMD-aligned prior when observed conditions are sparse or imbalanced? Can the authors report policy performance without MMD, and with sampling from the empirical latent distribution instead?
  4. How does the method perform under rapidly changing or unseen hidden conditions and noisy onboard state estimates?
6 comments
AuthorsOpenReview ↗

Question 1 + Weakness 2

Thank you for the careful review. We address all of the listed concerns and the questions below. At the same time, we also listed all the modifications marked in blue in the revised PDF.


Q1. "Can the authors distinguish VND more clearly from latent-conditioned adaptation (RMA, Neural-Fly) and online residual learning (LOTF)? Which specific design choice is responsible for the claimed novelty?"

Thank you for this insightful question. The fundamental novelty of VND is that it continually expands the range of dynamics over which the policy is trained, rather than adapting within a fixed, offline-learned distribution.

  • RMA and Neural-Fly learn dynamics representations from offline data. RMA infers latent variables within a pretrained embedding space, and in Neural-Fly, the system adapts linear coefficients to a pretrained aerodynamic basis. When out-of-distribution conditions arise during deployment, both RMA and Neural-Fly must approximate the new dynamics using their existing representations and cannot learn new representations from deployment experience.
  • LOTF tries to address this limitation of offline learning by updating residual dynamics and the policy during deployment. However, it uses a simple MLP residual as external forces acting on the quadrotor. This design can neither model condition-specific dynamics nor generalize to other robotic systems beyond force disturbances. In contrast, VND learns latent-conditioned residuals directly in the state-transition space, enabling a more general formulation applicable beyond quadrotors, as demonstrated by our manipulation experiments.
  • Which specific design choice enables this capability? The key novelty is the construction of a continually learned, jointly inferable and sampleable latent dynamics space shared by the dynamics model and the policy. Unlike conventional latent-conditioned adaptation, we jointly optimize the history encoder and the latent-conditioned residual model through self-supervised dynamics prediction on real deployment data. This design enables two complementary capabilities: (i) previously unseen conditions are incorporated into the learned dynamics distribution and directly applied for policy optimization; and (ii) previously learned conditions can be recognized and handled through latent inference alone, rather than model fitting and policy optimization from scratch.

W2. "The reported recovery gains mainly compare memory-based adaptation against retraining from scratch. Stronger memory-based baselines and more rigorous statistical evaluations are needed."

Thank you for raising this point. To isolate the benefits of our latent representation beyond experience replay and further strengthen our statistical evaluation, we conducted additional experiments during the rebuttal period.

  • Memory architecture. We consistently use the same GRU-based recurrent encoder throughout our framework. Our existing ablation study (Fig. 5) shows that replacing the recurrent encoder with an MLP operating on the same history window degrades dynamics prediction, demonstrating the importance of temporal memory. Our contribution is not a new memory architecture, but how temporal memory is integrated into continual dynamics and policy learning. However, during preliminary development, we also explored LSTM and temporal convolutional networks (TCNs), which yielded comparable empirical performance in our setting.

  • Baselines and evaluation protocol. The Simple Residual baseline already included in Fig. 4 uses the same recurrent architecture and accumulated deployment data as VND, but without any latent conditioning. We additionally evaluate VND with the policy, the privileged GT-Dist. oracle trained on the true disturbance distribution (as an upper bound), a strong model-based adaptive controller, INDI, and Simple Residual and LOTF.

We switch between previously encountered wind conditions every 4 s and evaluate each method across three independent seeds, with 16 episodes and five switches per episode (240 transitions in total per method). Recovery time is measured as the time until the position error remains below a common threshold of 3 cm. Unsuccessful recoveries are assigned the maximum duration of 4 s.

MethodMemory mechanismRecovery time [s] ↓Recovery rate ↑
VND (ours)Learned latent dynamics1.23 ± 0.30100%
GT-Dist. oraclePrivileged training distribution1.17 ± 0.16100%
INDIReactive adaptation2.28 ± 0.0299%
Simple ResidualRecurrent policy without latent3.46 ± 0.2228%
LOTFSingle MLP Residual3.72 ± 0.4334%

We hope these additional results address the reviewer's concerns and provide a clearer basis for reconsidering our contribution. We would be happy to provide further clarification if needed.

AuthorsOpenReview ↗

Question 2 + Weakness 1

W1 / Q2. "Could the authors evaluate sequentially introduced conditions (A→B→C→A), revisiting earlier ones, and compare against recent-data-only and replay-based baselines? Does the latent geometry of old conditions remain stable?"

Thank you for this suggestion. We have now conducted these experiments to test the stability of handling recurring disturbances.

Experimental setting. We introduce three conditions sequentially: no wind (A) and two wind disturbances of 3 m/s² with directions of 45° (B) and 135° (C). At each stage, we execute the full continual learning loop: collecting new deployment data, updating the dynamics model, and refining the policy.

To evaluate whether previously learned conditions are retained as new conditions arrive, we follow the training sequence A → B → C and then evaluate the trained policy on all 3 conditions without further model or policy updates. Here we compare three methods:

  • VND (ours): learns latent-conditioned dynamics from accumulated deployment data.
  • Recent-data-only: follows the VND pipeline but uses only the most recent deployment data for learning.
  • Simple Residual (Fig. 4): uses the same accumulated deployment data as VND but only uses one recurrent network without the latent encoding.

Retention performance. We report trajectory tracking errors [m] over five independent seeds.

MethodError on A after learning AFinal error on AFinal error on BFinal error on C
VND (ours)0.02 ± 0.010.035 ± 0.010.09 ± 0.040.03 ± 0.02
Recent-data-only0.02 ± 0.030.341 ± 0.070.489 ± 0.120.03 ± 0.02
Simple Residual0.02 ± 0.010.326 ± 0.090.415 ± 0.150.321 ± 0.09

To summarize, all three methods achieve comparable performance after learning condition A. However, after learning B and C, Recent-data-only largely forgets A and B, while Simple Residual struggles to handle multiple conditions despite having access to all historical data. In contrast, VND retains its performance on A and B while matching Recent-data-only on the latest condition C. These results highlight that retaining historical data alone is insufficient; latent conditioning enables the model and policy to distinguish and reuse previously learned dynamics.

Regarding latent stability, the latent geometry is not necessarily fixed as new conditions are incorporated, since the encoder, dynamics model, and policy are jointly refined throughout deployment. Nevertheless, the consistently low tracking errors on previously learned conditions without further optimization indicate that the learned representation remains functionally stable. While this does not directly establish geometric stability, it demonstrates that changes in the latent space do not prevent the effective reuse of previously learned adaptations. The complete experimental results are provided in the new App. A.10.

AuthorsOpenReview ↗

Question 3 + Weakness 3

W3 / Q3. "MMD encourages distribution alignment but does not guarantee that sampled latents represent realistic dynamics. [...] Can the authors report policy performance without MMD, and with sampling from the empirical latent distribution instead?"

Thank you for raising this important point. To investigate its practical benefits and limitations, we conducted two additional experiments.

  • Does MMD improve the quality of sampled dynamics? We train the dynamics model on 17 wind conditions with magnitudes up to 3 m/s² and draw 1,000 latents from the Gaussian prior. For each latent, we evaluate the learned residual model at a fixed hovering state and convert its predicted velocity correction into an equivalent disturbance acceleration by dividing it by the simulation time step.

We then measure (i) the percentage of samples producing disturbance accelerations above 3.3 m/s² (10% beyond the maximum training disturbance), and (ii) the distance between sampled latents and their nearest encoded training latents, normalized by the typical spacing between training latents.

MetricWith MMDWithout MMD
Samples exceeding the maximum training wind magnitude by more than 10%6.8%23.8%
Normalized distance to the nearest encoded training latent2.65.7

These results demonstrate that MMD reduces extreme disturbance predictions and brings prior samples closer to the encoded training data. This allows us to train the policy on a distribution of dynamics that better reflects the conditions encountered during real-world deployment.

  • Does MMD improve policy performance? We compare our original approach (MMD with Gaussian prior sampling) against empirical sampling without MMD, where latent vectors are directly sampled from the encoded deployment data.

We evaluate both approaches across three independent seeds under three training data settings:

  • Balanced: data collected evenly across 17 wind conditions.
  • Sparse: data collected from only 4 wind conditions.
  • Imbalanced: 80% of the collected data corresponds to a single wind condition with a direction of 45°.

The following table reports hovering errors [m] across all 17 wind conditions and separately under an unseen, stronger wind of 4 m/s².

Training dataEvaluation conditionMMD + prior (ours)No MMD + empirical
BalancedAll winds0.0180.010
BalancedUnseen wind0.0370.068
SparseAll winds0.1030.106
SparseUnseen wind0.2630.262
ImbalancedAll winds0.0240.045
ImbalancedUnseen wind0.0480.077

These results show that our method achieves better generalization to unseen and imbalanced disturbances. This advantage is particularly important for continual learning, where disturbances are unknown, change over time, and cannot be assumed to occur with equal frequency.

Reviewer_4EnGOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a continual robot policy-learning framework for hidden, changing dynamics. A recurrent encoder infers a latent condition from recent state action history, and a FiLM-conditioned residual model augments nominal physics. Dynamics prediction, context reconstruction, and MMD regularization jointly train the representation. Policies are subsequently optimized under sampled latent dynamics, using BPTT for differentiable tasks and PPO for box pushing. Deployment alternates data collection, model learning, and policy improvement. The intended contribution is to combine learning new conditions during deployment with rapid inference when previously encountered conditions recur. Validation includes quadrotor and manipulation simulations and real quadrotor experiments with intermittent wind and a damaged propeller. The paper reports substantial tracking improvements and approximately fivefold faster recovery from recurring wind than LOTF.

Critical strengths
  1. Meaningful real-world validation of learning during operation. The quadrotor experiments demonstrate policy improvement while flight continues, including adaptation to propeller damage that is not explicitly incorporated into the nominal model. The measured update timings in Appendix A.6 make the approximately 10-second learning loop substantially more convincing than a claim based only on offline training throughput. This is the strongest practical contribution.
  2. A useful separation between acquiring and reusing adaptations. The recurring-wind experiment distinguishes learning new dynamics from recognizing previously learned conditions. Appendix A.5 explicitly pauses learning during this evaluation, so recovery measures inference-based reuse rather than another round of optimization. This supports an important capability for robots operating under recurring conditions.
Critical weaknesses
  1. The novelty and evidence for continual learning are not sufficiently isolated from established adaptation and model-based finetuning approaches. The related-work discussion omits relevant precedents: latent-space adaptation in Peng et al. [1], physics-informed model-based finetuning in LIFT [2], and real-data-driven simulator correction followed by policy finetuning in ASAP [3]. These methods are not equivalent to the submission, but they make it important to locate the contribution specifically in deployment-learned condition representations and their reuse.

  2. The manipulation dynamics are insufficiently specified to assess or reproduce the claimed extension. Appendix A.2 provides explicit quadrotor equations, but no comparably concrete specification for the Franka tasks. For box pushing, it remains unclear whether object pose and velocity enter the model state and history, how nominal object/contact dynamics are constructed, and which state variables receive the learned correction. Table 4 lists a 14-dimensional residual for both manipulation tasks without defining its components. Explaining that PPO replaces BPTT addresses the optimizer choice, but not these modeling questions. Since manipulation provides the main evidence beyond aerial robotics, this omission materially weakens the generality and reproducibility claims.

References

[1] Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Lee, Jie Tan, and Sergey Levine. Learning Agile Robotic Locomotion Skills by Imitating Animals. Robotics: Science and Systems (RSS), 2020.

[2] Weidong Huang, Zhehan Li, Hangxin Liu, Biao Hou, Yao Su, and Jingwen Zhang. Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Control. International Conference on Learning Representations (ICLR), 2026.

[3] Tairan He et al. ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills. 2025. arXiv:2502.01143.

Questions for authors
  1. Contribution relative to prior work — clarification. Could you explicitly distinguish the proposed method from Peng et al.’s latent-search adaptation, LIFT’s model-and-policy finetuning loop, and ASAP’s simulator correction? Which specific mechanism should be regarded as the main algorithmic advance, and which existing experiment isolates its contribution?
  2. Manipulation model — clarification. Please specify fpriorf_{\mathrm{prior}}, the model state, encoder inputs, action interface, and residual outputs for each Franka task. For pushing, where are box pose, velocity, mass, friction, and contact represented? What do the 14 residual outputs in Table 4 correct, and how is the next object state obtained?
  3. Does continual finetuning on the deployment buffer reduce generalization or robustness to conditions that are absent or underrepresented in that buffer? Please compare the base and refined policies on held-out dynamics and disturbances, including conditions outside the collected-data distribution. Beyond the state-estimation limitations discussed in Section 5, what are the method’s principal failure modes and operating limits—for example, when conditions change faster than the context window can identify them, or when newly encountered dynamics differ substantially from previously collected experience?
5 comments
AuthorsOpenReview ↗

Weakness 1 + Question 1

Thank you for the careful review. We address all of the listed concerns and the questions below. At the same time, we also listed all the modifications marked in blue in the revised PDF.


W1 / Q1. "Could you explicitly distinguish the proposed method from Peng et al.'s latent-search adaptation, LIFT's model-and-policy finetuning loop, and ASAP's simulator correction? Which specific mechanism should be regarded as the main algorithmic advance, and which existing experiment isolates its contribution?"

Our main contribution has two parts:

  1. Learning during deployment in a continual loop. The robot keeps learning its dynamics model and policy while it operates: collect data, update the model, update the policy, redeploy. This is closer to the recursive policy improvement setting instead of one-shot or offline improvement as done by Peng, at al, LIFT and ASAP. In fact, we carefully designed many system components to enable the system to learn the variational dynamics model and the policy quickly (~10s), e.g., training the policy using BPTT, so the policy can learn in the real world at runtime and handle diverse environmental conditions. This is a foundamental distinction from the existing approaches.

  2. More deeply, from a methodology perspective, we use one latent condition space that is learned from deployment data (where Peng et al 2020, learn a fixed one in simulation, which cannot handle real OOD cases) and used in two roles, it is inferred online to recognize a recurring condition, and it is sampled during policy optimization so the policy trains on the learned distribution of conditions, built by the deployment data. This property exists because of our proposed variational neural dynamics architecture, which can effectively learn a distribution from the real world and combine it with policy learning.

More details can be found in the table below

What is updated from real dataLearns during deployment?Continual loop or one-shot?Condition representationWhen a condition recurs
Peng et al. (2020)a latent vector, by search over real episodespolicy frozenone-shotlearned in simulationfinetune again
LIFTdynamics model and policyseparate finetuning stageone-shotnone (single fixed target environment)finetune again
ASAPfirst a delta-action model, then policy fine-tuning in the corrected simulatorno, offline learningone-shotnone (one correction)finetune again
Ourslatent-conditioned dynamics model and policyyescontinual loop, 9.7s/roundlearned from deployment datainferred online from 0.4 s of history

Experiments isolate this distinction. Baselines that adapt in this one-shot way already run inside our own original manuscript. A policy trained on one fixed condition only ("Single Dist.", which is close to LIFT's finetuning to one target environment) or on one residual correction without a condition latent ("Simple Residual", closest to ASAP) is far worse than ours in every task.

Task (Fig. 3, 4)OursSingle Dist.Simple Residual
Quadrotor tracking error [m] ↓0.0450.3120.311
Quadrotor hovering error [m] ↓0.0700.5640.318
Franka tracking error [mm] ↓7.7143.444.2
Box-pushing success ↑70%0%40%

On hardware, refitting one residual after every change needs 5.3 ± 0.6 s to recover when a learned wind returns, vs. 1.1 ± 0.2 s for ours (Tab. 2).

We do hope these can clarify the methodology distinction than the exsiting approaches.

AuthorsOpenReview ↗

Weakness 2 + Question 2

W2 / Q2. "Please specify f_prior, the model state, encoder inputs, action interface, and residual outputs for each Franka task. [...] What do the 14 residual outputs in Table 4 correct, and how is the next object state obtained?"

Thank the reviewer for the detailed questions. We agree that the details of this setup were underspecified, and this can cause confusion in understanding our main contribution. We have now added a full specification to the appendix (new App. A.11).

End-effector tracking:

  • f_prior: the nominal Franka Panda model in MuJoCo MJX, with the nominal hand mass (0.73 kg). Each 0.02 s control step runs 4 Euler substeps of 0.005 s.
  • Action interface: 7 joint torques, normalized to [−1, 1] and scaled to ±87 N·m (joints 1–4) and ±12 N·m (joints 5–7).
  • Model state (14-D): joint positions (7) and velocities (7), s=(q,q˙)∈R14\mathbf{s} = (\mathbf{q}, \dot{\mathbf{q}}) \in \mathbb{R}^{14}. With the action, this gives the 21-D state-action input of Tab. 4.
  • Residual outputs (14): Δq∈R7\Delta\mathbf{q} \in \mathbb{R}^7 and Δq˙∈R7\Delta\dot{\mathbf{q}} \in \mathbb{R}^7, defined as the observed next state minus the nominal MJX prediction from the same state and action.
  • Encoder input: 20 consecutive previous steps of the state information.
  • Hidden condition: a payload added to the hand body mass.

Box pushing

  • f_prior: the MJX simulation of the Panda push-cube task from MuJoCo Playground, control step 0.005 s.
  • Action interface: 7 joint torques same as before, scaled by 0.1 and clipped to the torque limits.
  • Model state (48-D): target pose (9), last action (7), joint positions and velocities (14), gripper pose (9), and box position and orientation (9). The box enters the model only through its pose in this state. We did not use contact or friction for the policy input.
  • Residual outputs (48): a correction to every entry of the next state; the next box pose is the nominal MJX step plus this correction.
  • Encoder input: 20 consecutive previous steps of the state information.
  • Next object state: MJX simulates the arm and the box together, including contact, for one control step; the next box position and orientation are taken from this simulated step and then corrected by the residual.
  • Hidden condition: the box mass.
AuthorsOpenReview ↗

Question 3

Q3 (1/2). "Does continual finetuning on the deployment buffer reduce generalization or robustness to conditions that are absent or underrepresented in that buffer? Please compare the base and refined policies on held-out dynamics and disturbances, including conditions outside the collected-data distribution."

Thanks for the great question. We have now conducted this new comparison (Appendix E7 in the revised manuscript). From the experiments we performed, our online refinement does not reduce robustness; instead, it improves every condition we tested, including effects never in the buffer. The base policy is trained on the nominal model only. The refined policy comes from our loop, whose buffer contains winds up to 3 m/s² only. Here we report the trajectory tracking error [m]:

Test conditionIn buffer?Base policyRefined policy (ours)
Held-out winds between buffer conditionsno0.334 ± 0.0620.014 ± 0.01
Wind 4 m/s²beyond range0.583 ± 0.080.067 ± 0.02
15% thrust lossnew type0.336 ± 0.040.074 ± 0.03
+30% massnew type0.370 ± 0.060.120 ± 0.03
Linear dragnew type0.312 ± 0.050.018 ± 0.02

We believe our better performance comes from two aspects:

  • We train our policy on a learned distribution, instead of the latest condition. Each round retrains the policy on latents sampled across the whole buffer. This precise coverage of environment domains makes the policy robust even when handling OOD cases.
  • Online inference still helps outside the buffer. Removing the latent consistently increases the error under unseen conditions. We attribute this to the encoder detecting that the dynamics differ and shifting the latent accordingly, which allows the policy to respond even to effects it was never trained on.

Q3 (2/2). "What are the method's principal failure modes and operating limits—for example, when conditions change faster than the context window can identify them, or when newly encountered dynamics differ substantially from previously collected experience?"

Thanks for the great question.

  • Conditions that change faster than the context window. We conducted the policy evaluation in simulation for the Figure 8 trajectory tracking task. The encoder reads 0.4 s of history, so the latent variable is lagging after each switch. Tracking error [m] of our policy:
Wind changes every0.4 s0.8 s1.6 s3.2 snever (constant)
Ours0.0900.0750.0480.0300.018
RMA0.3320.2850.2450.2020.208

As expected, performance degrades as the switching frequency increases. Nevertheless, even when conditions change at the timescale of the encoder's context window (0.4 s), VND maintains a tracking error below 0.1 m and substantially outperforms RMA across all switching frequencies. We believe this robustness comes from training the policy over a diverse, data-driven distribution of dynamics rather than a single condition. This likely enables the policy to tolerate temporary mismatches between the inferred latent and the actual dynamics.

  • Main failure mode: a nominal model too far from the real system. The loop starts from a policy trained on the nominal model alone, and everything it learns comes from the data this policy collects. If the nominal model is so wrong that the base policy cannot keep the robot near the task, the first data will not cover the conditions that matter, and the model and policy cannot improve from them.

The loop would then need another way to start, for example, human teleoperation. In our experiments, the base policy was good enough: even with a cut propeller, it still flew at 52 cm error. But we also believe that if we do not use the initial policy rollout to learn the dynamics model, our approach remains rather unchanged and cannot perform continual self-improvement.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their careful and constructive feedback. We appreciate that the reviewers recognize the importance of continual robot learning, the design of our shared latent dynamics space, and the real-world demonstration of continual policy improvement.

In response to the reviewers' suggestions, we conducted six additional experimental studies (E5–E10), introduced new baselines and ablations, and substantially revised the manuscript. All changes are highlighted in blue.

We summarize the main improvements below:

  • Methodological Contribution: We clarified what distinguishes VND from existing latent-conditioned adaptation, online residual learning, and model-based policy fine-tuning methods. We also incorporated the suggested references and expanded the related-work discussion.

  • Recurring and Long-horizon Environments: We conducted sequential learning experiments over nine learning rounds and five independent seeds. When revisiting the first condition, VND achieves a tracking error of 0.035 m, compared with 0.341 m for recent-data-only learning and 0.326 m for replay without latent conditioning. These results provide direct evidence of the importance of combining experience replay with latent conditioning (E5).

  • Additional Baselines and Statistics: We extended our recovery evaluation with memory-based baselines, a privileged oracle, and the classical adaptive controller INDI, covering 240 disturbance switches per method across three independent seeds. These comparisons further demonstrate the benefits of reusing previously learned adaptations (E10).

  • MMD and Latent Sampling: We conducted direct sampling diagnostics and policy-level experiments under balanced, sparse, and imbalanced deployment data. These results clarify when MMD-based prior sampling is beneficial and when empirical sampling provides a competitive alternative (E6).

  • Robustness and Limitations: We evaluated our method under unseen dynamics, rapidly changing disturbances, and simulated state-estimation noise. We also provide a more detailed discussion of failure modes, long-term buffer management, and the complementary strengths of classical adaptive control (E7–E9, Secs. 3 and 5).

  • Technical Clarity: We expanded the methodological descriptions and provided detailed specifications of the manipulation dynamics, including the nominal model, state and action representations, encoder inputs, and residual outputs (App. A.11).

All additional experimental studies are included in App. A.10 of the revised manuscript.

We believe the new experiments provide substantially stronger evidence for our central contribution: enabling robots to continually learn from deployment experience while retaining the ability to handle previously encountered dynamics.

We sincerely appreciate the reviewers' suggestions, which have helped us improve both the scientific evidence and the clarity of our work. We hope our additional results and clarifications address the reviewers' concerns, and we look forward to further discussions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.