- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SteerControl, a method for adapting a frozen reinforcement learning policy when the environment dynamics change during deployment. The method uses a one-step dynamics model trained under nominal conditions to predict the next state. By comparing the predicted and observed transitions, it estimates a correction to the policy's actions. Both the policy and dynamics model remain frozen, and the adaptation does not require reward feedback, fault labels, or additional training.
The authors evaluate the method on 12 continuous-control environments, including locomotion, robotic manipulation, and other control tasks. They consider both fixed disturbances and dynamics that change during an episode. The results show that SteerControl can recover a substantial amount of the performance lost under these changes. The authors also study the effect of fault severity, the correction smoothing parameter, and the assumptions behind their residual-based approach.
Critical strengths
1. A simple approach to online adaptation without updating the policy.
I think one of the strengths of the paper is that the proposed method is relatively simple and does not require additional training during deployment. The authors use prediction errors from a frozen dynamics model to adjust the actions of an existing policy. This is useful because retraining a policy or collecting reward feedback may not always be practical in real-world applications. I also appreciate that the method does not require knowing the type of fault or the changed physical parameters in advance.
2. Evaluation across different tasks and changing dynamics.
The experiments cover 12 continuous-control environments and several types of disturbances, which is a positive aspect of the paper. The results in Tables 1 and 2 show consistent improvements over the perturbed base policy in many settings.
Critical weaknesses
1. The methodological novelty is relatively limited.
My main concern is that the proposed method seems quite close to existing ideas in model-based control and online adaptation. Using the difference between predicted and observed transitions to estimate a dynamics mismatch, and then adjusting the control action to compensate for it, is already a well-established idea. The main contribution here seems to be applying this principle to a frozen reinforcement learning policy using a learned nominal dynamics model [1][2][3].
2. The experimental evaluation does not fully establish the advantage over stronger adaptation methods or under more realistic conditions.
Although the paper evaluates many environments, I have some concerns about the experimental comparisons. The main baselines are Online LMS, CMAC, and Cerebellar controllers, while several other relevant approaches to adaptive control and dynamics adaptation are discussed only in the related work. I understand that these methods do not all operate under exactly the same assumptions, but I would still like a clearer comparison against stronger alternatives that can use recent interaction history to estimate disturbances or adjust control actions, eg RL policy trained with domain randomization or some meta learning based method or even RMA[4] based method.
Another concern is that the method assumes the visited states remain close to the nominal training distribution and that the dynamics discrepancy is approximately constant across nearby actions and consecutive time steps. These assumptions are important to the correction mechanism, but they may not hold under more severe faults or rapidly changing dynamics. The fault-severity results in Figure 2 already show that the recovered performance drops considerably at larger perturbations in several tasks.
Finally, all experiments are conducted in simulation, and the paper does not evaluate the effect of noisy state estimates, delayed observations, or inaccurate dynamics predictions in a physical robot. Since the motivation is adaptation during real-world deployment, I think this limits how strongly the results support the practical claims.
[1] Online Dynamics Learning for Predictive Control with an Application to Aerial Robots [2] Learning on the Fly: Rapid Policy Adaptation via Differentiable Simulation [3] Learning Agile and Dynamic Motor Skills for Legged Robots [4] rma rapid motor adaptation for legged robots
Questions for authors
Clarification questions (answers could affect my score):
-
Could the authors clarify the main methodological novelty compared with existing model-mismatch compensation and disturbance-observer-based control methods?
-
How would SteerControl compare with more established controller/RL policies under similar assumptions?
-
How sensitive is the method to errors in the learned nominal dynamics model? Since these errors also contribute to the observed residual, how does SteerControl distinguish actual dynamics changes from inaccurate model predictions?
-
The method assumes that the dynamics discrepancy is approximately constant across nearby actions and consecutive time steps. How well does this assumption hold under more severe faults or rapidly changing dynamics? Could the authors clarify where the method starts to fail?
5 comments
Responses - Part 2/2
W2 and Q2 - Continued
We have kept these preliminary comparisons with other baselines outside the edited manuscript pending validation across tasks; their difficulties do not establish limitations of the original methods.
We agree that experiments on physical robots are important; however, we want to clarify that our paper does not claim successful real-world robotic deployment. The opening of the Introduction mentions this setting just as motivation. Our extensive evaluation instead examines adaptation across twelve simulated environments spanning locomotion and manipulation, compared with four environments in Cerebellar. These include physics-based MuJoCo tasks such as FetchReach, which models a 7-DoF robotic arm. While real-world robotic experiments are not feasible within the present work and are beyond its scope, we consider them valuable and encourage future work in this direction.
ℒ₁: Improving the Robustness of Reinforcement Learning Policies With ℒ₁ Adaptive Control, IEEE RA-L, 2022.
EVOLVER: EVOLVER: Online Learning and Prediction of Disturbances for Robot Control, IEEE T-RO, 2024.
DADO: Data-Driven Adaptive Disturbance Observers for Model-Free Trajectory Tracking Control of Maritime Autonomous Surface Ships, IEEE TNNLS, 2021.
W2, Q3, Q4 - Assumptions and analysis
Input-distribution assumption.
We clarify that the input-distribution assumption is not a strict limitation. We agree that submission line 270 stated nominal input coverage too strongly, and we have narrowed the stated reliance of our fault design on this assumption (Section 3.1). This assumption mainly reflects the generalization limits of neural networks: we use frozen neural policy and world models at inference, whose generalization outside their training distributions is not guaranteed. In practice, however, neural networks can tolerate small distribution shifts and modest extrapolation. Our fault-severity experiment (Figure 2) was intended to examine SteerControl’s applicability under severe faults, where this assumption may hold less closely.
We should also clarify that, in the fault-severity experiment, even a severity of 0.5 represents a substantial fault for a frozen model-free policy trained solely under nominal dynamics. SteerControl still provides effective compensation, although under very large faults, correction bounds may also limit the achievable compensation (submission line 361). In addition, Figure 2b (edited draft) quantifies drift in velocity means, showing that test-time input distributions do not necessarily remain aligned with nominal distributions. However, Figure 2c (edited draft) shows that the ratio of nominal-model error to the fault-induced transition change generally decreases over the milder fault range. This may help explain why SteerControl can still compensate for dynamics shifts that also change the input distributions of its frozen neural networks.
Locality assumption.
We clarify that our nonstationary fault experiments are specifically intended to show that SteerControl remains robust and applicable under abrupt, adversarial fault changes when sufficient intervals are available for recovery. Figure 3 shows that after a severe fault reversal in Pusher and Reacher, compensation rebounds sharply within a few steps and rapidly returns toward pre-switch performance. This responsiveness follows from SteerControl’s simple online compensation mechanism, which constructs a new counterfactual target and recomputes the correction after each observed transition. Such recovery on the scale of individual control steps is particularly valuable when comparing adaptation timescales in continual and meta-RL. Some adaptation protocols operate over multiple episodes or hundreds of steps, which can still be called rapid relative to RL training or fine-tuning: for example, RL² uses multi-episode trials, and PEARL reports test performance after two 200-step trajectories.
Figure 4a further examines dependence on the EMA weight. Smaller weights retain a longer history and therefore rely on locality over a longer temporal window. Nevertheless, SteerControl maintains positive recovered gains throughout the tested range of 0.1–1.0, showing that effective compensation does not depend on a specific, narrowly tuned EMA value in practice. Together with Figure 3, this supports practical flexibility in the temporal locality assumption.
Finally, a few abrupt changes within an episode are therefore compatible with SteerControl, as our experiments directly demonstrate. The relevant temporal limitation arises when changes occur so frequently that the correction cannot track the new dynamics and recover before the next switch. We make these points clear in the paper.
RL²: Fast Reinforcement Learning via Slow Reinforcement Learning, arXiv preprint, 2016.
PEARL: Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables, ICML, 2019.
Reviewer 6yKX: are you convinced, particularly as regards the novelty and the positioning.
Responses - Part 1/2
We thank the reviewer for their insightful feedback and for recognizing our method’s simplicity, its adaptation without deployment-time training, and the breadth of our evaluation across tasks and changing dynamics.
W1 and Q1
We thank the reviewer for requesting clearer positioning. We agree that mismatch compensation is an established control principle. The submitted Related Work discussed DOB methods (lines 736–745) and positioned SteerControl relative to them (759–762). We have expanded that discussion into two paragraphs covering additional methods and made the positioning more precise, with changes marked in blue.
The distinction concerns how compensation is constructed and what prior information it requires. Representative DOB and local inverse-control formulations use supplied plant inverses, mechanical or disturbance equations, or control-effectiveness maps; tracking variants also use reference trajectories. Learning-based methods relax some requirements but still retain some limitations. For example, Cerebellar (ICML 2026), a recent competitor evaluated in our paper, relies on nominal trajectories (instead of a trained world model) in its tracking variants, and our implementation showed its sensitivity to configuration choices across settings. The reviewer's references use different mechanisms: Jiahao et al. [1] and Pan et al. [2] combine physical models with online learning and a supplied MPC cost or differentiable task reward, respectively; Hwangbo et al. [3] use learned actuator dynamics and randomized policy training. SteerControl requires no additional deployment information beyond the observations and executed actions available in a standard RL evaluation protocol.
Our motivation is to complement strong, widely used model-free policies (lines 69–80) whose performance can deteriorate under unseen dynamics, as discussed in lines 704–713. Under unseen dynamics shifts, their performance can drop significantly, yet these nominal policies lack an explicit mechanism to detect that their behavior has degraded. SteerControl gives a nominal world model an online compensatory role by steering model-free actions through a simple counterfactual target, without relying on mechanical equations, an external tracking reference, or additional supervision at deployment. Across twelve simulated environments under stationary and nonstationary perturbations, our results demonstrate the effectiveness of this combination in enhancing the robustness of model-free agents to previously unseen, moderate changes in dynamics.
W2 and Q2 - Baseline comparisons
Our baseline selection reflects our nominal-only policy/model training and the fact that only observed state–action transitions are available as feedback during inference. Most related methods discussed here require training across anticipated dynamics information or variations (like suggested [3] and RMA [4]), analytical mechanical models, or a deployment task objective: an MPC cost in [1] or differentiable simulated rewards in [2]. As the reviewer recognizes, these additional requirements complicate direct comparison under our simple protocol; ignoring these differences would lead to an unfair assessment.
We already included a recent strong baseline, Cerebellar (ICML 2026), which closely matches our training and evaluation protocols. However, to obtain comparable results, we evaluated approximately 30 additional configurations per setting using calibration seeds disjoint from evaluation, allowed cross-episode memory when favored by calibration, and supplied reference or task-performance feedback beyond SteerControl's requirements.
Following the reviewer's suggestion, we added two domain randomization baselines (DR-SAC and an RMA-style PPO baseline) in Appendix E. Both train across the full tested actuator-bias range without receiving severity at evaluation. SteerControl achieves higher mean return than the RMA port in most evaluated settings; DR-SAC outperforms the RMA throughout the sweep. DR-SAC is strong on manipulation and outperforms SteerControl at larger locomotion biases, while SteerControl performs better at smaller biases. The RMA's weak privileged teacher on locomotion tasks limits conclusions about its adaptation mechanism.
Finally, after carefully screening additional controllers for compatibility, we also found and implemented ℒ₁, EVOLVER, and DADO. Preliminary recovered gains (%) under the same benchmark training and evaluation settings are:
These implementations were competitive on manipulation tasks but did not reliably train or compensate on Ant, Humanoid, and HalfCheetah; ℒ₁ nevertheless recovered 11.4 ± 0.7% on HalfCheetah and 13.2 ± 10.2% on Ant, while the other measured gains were negative or close to zero.