- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a continual policy learning framework for robots operating under hidden, time-varying dynamics. Its core, Variational Neural Dynamics (VND), combines an analytical physics prior with a FiLM-modulated neural residual conditioned on a latent variable inferred by a recurrent encoder from recent state-action history. Overall, the research topic is interesting and the proposed method is novel. The paper is also clearly written and easy to follow.
Critical strengths
-
The problem is well motivated and practically important. Moving beyond the train-once, deploy-forever paradigm, and in particular reusing what was learned when a condition recurs, is a clear and valuable goal.
-
The method is conceptually clean. Using one latent space for two roles, inferred at deployment and sampled during policy training, is an elegant way to learn the condition distribution from data rather than specifying it in simulation.
-
The real-world validation is convincing. Closing the full loop on hardware in roughly 10 s per round is impressive, and the damaged-propeller experiment shows the method handles effects that the nominal model and standard randomization do not capture.
Critical weaknesses
-
The buffer ssems to keep growing, but the paper doesn't say how it is managed. Over a long deployment, conditions seen early on will take up a smaller and smaller share of the data. Could forgetting creep back in? The hardware runs are only about 50 s with a few conditions, so a short discussion (or a longer run) would help here.
-
In the introduction, "Once deployed, neither approach uses new data to update the underlying model or policy" is a bit too strong. Methods that pair offline learning with online adaptation do use deployment data in some sense, and this helps them generalize even though they don't learn continually. Neural-Fly (O'Connell et al., 2022) is already cited later in the paper but isn't reflected in this sentence; FORESEER (Jia et al., IJRR 2025) is another example. It would be good to cover this line of work. The same goes for "cannot expand this distribution as new conditions arise" in Related Work. Adaptation-plus-learning methods can handle some out-of-distribution cases, so I'd soften this.
-
"VND models a distribution of residual dynamics ... so it can learn from new conditions while retaining what it learned from earlier ones." I don't see why modeling a distribution by itself prevents forgetting. My guess is that retention really comes from replaying the whole buffer and conditioning on the latent. This statement can be improved.
Questions for authors
The ~1 s recovery is highlighted in the abstract and introduction, but only compared with residual fitting method. How does it compare with classical online adaptive controllers, e.g. INDI-based ones (Sun et al., T-RO 2022)?
4 comments
Weakness 3 + Question 1
W3. "I don't see why modeling a distribution by itself prevents forgetting. My guess is that retention really comes from replaying the whole buffer and conditioning on the latent."
Your reading is correct, and we rewrote the sentence.
Old: "VND models a distribution of residual dynamics [...] so it can learn from new conditions while retaining what it learned from earlier ones."
New: "VND is retrained on the whole deployment buffer and conditions its residual on an inferred latent, so a new condition is fitted in a new region of the latent space while earlier conditions stay represented by their replayed data."
Q1. "The ~1 s recovery [...] How does it compare with classical online adaptive controllers, e.g. INDI-based ones (Sun et al., T-RO 2022)?"
Thank you for this suggestion. We have added comparisons against INDI controller (App. A.10). In simulation, wind conditions switch every 4 s, with 240 transitions per method (16 episodes × 5 switches × 3 seeds). Recovery time is measured as the time until the position error remains below 3 cm, with unsuccessful recoveries assigned 4 s.
| Method | Recovery time [s] ↓ | Recovery rate ↑ | Steady-state error [m] ↓ |
|---|---|---|---|
| VND (ours) | 1.23 ± 0.30 | 100% | 0.018 ± 0.003 |
| INDI | 2.28 ± 0.02 | 99% | 0.0206 ± 0.0002 |
VND achieves faster recovery and lower steady-state errors. However, INDI reacts more quickly immediately after switching, while VND requires recent observations to infer the new condition (mean error during the first second: 0.07 m for INDI vs. 0.09 m for VND).
We view these approaches as complementary: classical adaptive controllers provide reactive disturbance rejection, while VND enables continual policy improvement from deployment experience. Combining VND with an INDI inner-loop controller is an interesting direction for future work.
Reply to author's responses
Thank you for the thorough response. I am very satisfied with the rebuttal, especially the new comparison with INDI, the additional forgetting experiment, and the updated discussion of adaptive control methods.
Although the growing replay burden of the buffer remains a minor limitation for long-term deployment, the overall contribution is solid. I have decided to raise my score!
We sincerely appreciate your timely and encouraging response, and thank you for taking the time to carefully consider our additional experiments and clarifications. Your constructive feedback has been very valuable in helping us improve both the experimental evaluation and the clarity of our work.
Weakness 1 + Weakness 2
Thank you for the positive and constructive review and for recognizing the practical and methodological contributions of our work. We address each point below, with corresponding modifications highlighted in blue in the revised manuscript.
Thank you for raising this important point.
We introduce three conditions sequentially: no wind (A) and two wind disturbances of 3 m/s² with directions of 45° (B) and 135° (C). We execute three full continual learning rounds per condition (A → A → A → B → B → B → C → C → C), resulting in nine rounds (1 minute each, so 9 minutes in total in simulation) of data collection for dynamics model learning and policy refinement. We then evaluate the final policy under all three conditions without further learning.
We compare VND against Recent-data-only, which uses only the most recent deployment data, and Simple Residual (Fig. 4), which retains all historical data but without latent conditioning.
Results are trajectory tracking errors [m] over five independent seeds (mean ± std). Recent-data-only largely forgets conditions A and B, while Simple Residual struggles to handle multiple conditions despite retaining all historical data. In contrast, VND retains good performance on previously learned conditions while matching Recent-data-only on the latest condition C. This demonstrates the importance of combining experience replay with latent-conditioned dynamics learning.
While these results are encouraging, our current experiments still cover a limited number of disturbance conditions and learning iterations, especially if we consider a lifelong-learning setup of the robot deployment. As the deployment buffer grows, earlier conditions may become increasingly underrepresented, which can affect both the learned dynamics representation and policy performance. We acknowledge this limitation and have added a discussion of long-term retention and scalable buffer management to the revised manuscript.
We agree with the reviewer that our original statements were too restrictive. These methods effectively use deployment data for online adaptation and can generalize beyond their training conditions. The distinction we intend to emphasize is that VND continually updates the underlying dynamics representation and explicitly trains the policy on newly encountered dynamics, rather than relying solely on adaptation within an offline-learned representation.
We have revised the corresponding statements as follows:
We have also expanded our discussion of Neural-Fly and included FORESEER in the related work.