acceptodds
Under review as a conference paper at ICLR 2027

SteerControl: World Model-Guided Adaptation of Frozen Policies to Changing Dynamics

Abstract

Unexpected changes in dynamics are often unavoidable for intelligent agents operating in real-world settings. Policies trained under fixed dynamics can suffer substantial performance losses after deployment when the same action produces a different state transition. Many existing adaptation methods address such failures by training on anticipated variations or by relying on deployment rewards, reference trajectories, or parameter updates. We address this problem using a learned dynamics model that requires no supervision beyond what is available in a standard reinforcement learning (RL) setting. SteerControl pairs a frozen policy with a frozen one-step dynamics model, both trained only under nominal dynamics. It uses the difference between predicted and observed transitions to estimate a bounded correction to the policy’s action mean without requiring fault labels, external supervision, deployment rewards, or weight updates. Across twelve continuous-control environments with varied actuator faults and physical perturbations, SteerControl recovers return lost by the faulted base policy under constant and switching perturbations. These results suggest that a nominal dynamics model can help an existing model-free policy maintain more of its performance when faced with unseen shifts in action effects during deployment.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, online adaptation, dynamics shifts, world models, test-time adaptation, reward-free adaptation, continuous control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_6yKXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SteerControl, a method for adapting a frozen reinforcement learning policy when the environment dynamics change during deployment. The method uses a one-step dynamics model trained under nominal conditions to predict the next state. By comparing the predicted and observed transitions, it estimates a correction to the policy's actions. Both the policy and dynamics model remain frozen, and the adaptation does not require reward feedback, fault labels, or additional training.

The authors evaluate the method on 12 continuous-control environments, including locomotion, robotic manipulation, and other control tasks. They consider both fixed disturbances and dynamics that change during an episode. The results show that SteerControl can recover a substantial amount of the performance lost under these changes. The authors also study the effect of fault severity, the correction smoothing parameter, and the assumptions behind their residual-based approach.

Critical strengths

1. A simple approach to online adaptation without updating the policy.

I think one of the strengths of the paper is that the proposed method is relatively simple and does not require additional training during deployment. The authors use prediction errors from a frozen dynamics model to adjust the actions of an existing policy. This is useful because retraining a policy or collecting reward feedback may not always be practical in real-world applications. I also appreciate that the method does not require knowing the type of fault or the changed physical parameters in advance.

2. Evaluation across different tasks and changing dynamics.

The experiments cover 12 continuous-control environments and several types of disturbances, which is a positive aspect of the paper. The results in Tables 1 and 2 show consistent improvements over the perturbed base policy in many settings.

Critical weaknesses

1. The methodological novelty is relatively limited.

My main concern is that the proposed method seems quite close to existing ideas in model-based control and online adaptation. Using the difference between predicted and observed transitions to estimate a dynamics mismatch, and then adjusting the control action to compensate for it, is already a well-established idea. The main contribution here seems to be applying this principle to a frozen reinforcement learning policy using a learned nominal dynamics model [1][2][3].

2. The experimental evaluation does not fully establish the advantage over stronger adaptation methods or under more realistic conditions.

Although the paper evaluates many environments, I have some concerns about the experimental comparisons. The main baselines are Online LMS, CMAC, and Cerebellar controllers, while several other relevant approaches to adaptive control and dynamics adaptation are discussed only in the related work. I understand that these methods do not all operate under exactly the same assumptions, but I would still like a clearer comparison against stronger alternatives that can use recent interaction history to estimate disturbances or adjust control actions, eg RL policy trained with domain randomization or some meta learning based method or even RMA[4] based method.

Another concern is that the method assumes the visited states remain close to the nominal training distribution and that the dynamics discrepancy is approximately constant across nearby actions and consecutive time steps. These assumptions are important to the correction mechanism, but they may not hold under more severe faults or rapidly changing dynamics. The fault-severity results in Figure 2 already show that the recovered performance drops considerably at larger perturbations in several tasks.

Finally, all experiments are conducted in simulation, and the paper does not evaluate the effect of noisy state estimates, delayed observations, or inaccurate dynamics predictions in a physical robot. Since the motivation is adaptation during real-world deployment, I think this limits how strongly the results support the practical claims.

[1] Online Dynamics Learning for Predictive Control with an Application to Aerial Robots [2] Learning on the Fly: Rapid Policy Adaptation via Differentiable Simulation [3] Learning Agile and Dynamic Motor Skills for Legged Robots [4] rma rapid motor adaptation for legged robots

Questions for authors

Clarification questions (answers could affect my score):

  1. Could the authors clarify the main methodological novelty compared with existing model-mismatch compensation and disturbance-observer-based control methods?

  2. How would SteerControl compare with more established controller/RL policies under similar assumptions?

  3. How sensitive is the method to errors in the learned nominal dynamics model? Since these errors also contribute to the observed residual, how does SteerControl distinguish actual dynamics changes from inaccurate model predictions?

  4. The method assumes that the dynamics discrepancy is approximately constant across nearby actions and consecutive time steps. How well does this assumption hold under more severe faults or rapidly changing dynamics? Could the authors clarify where the method starts to fail?

5 comments
AuthorsOpenReview ↗

Responses - Part 1/2

We thank the reviewer for their insightful feedback and for recognizing our method’s simplicity, its adaptation without deployment-time training, and the breadth of our evaluation across tasks and changing dynamics.

W1 and Q1

We thank the reviewer for requesting clearer positioning. We agree that mismatch compensation is an established control principle. The submitted Related Work discussed DOB methods (lines 736–745) and positioned SteerControl relative to them (759–762). We have expanded that discussion into two paragraphs covering additional methods and made the positioning more precise, with changes marked in blue.

The distinction concerns how compensation is constructed and what prior information it requires. Representative DOB and local inverse-control formulations use supplied plant inverses, mechanical or disturbance equations, or control-effectiveness maps; tracking variants also use reference trajectories. Learning-based methods relax some requirements but still retain some limitations. For example, Cerebellar (ICML 2026), a recent competitor evaluated in our paper, relies on nominal trajectories (instead of a trained world model) in its tracking variants, and our implementation showed its sensitivity to configuration choices across settings. The reviewer's references use different mechanisms: Jiahao et al. [1] and Pan et al. [2] combine physical models with online learning and a supplied MPC cost or differentiable task reward, respectively; Hwangbo et al. [3] use learned actuator dynamics and randomized policy training. SteerControl requires no additional deployment information beyond the observations and executed actions available in a standard RL evaluation protocol.

Our motivation is to complement strong, widely used model-free policies (lines 69–80) whose performance can deteriorate under unseen dynamics, as discussed in lines 704–713. Under unseen dynamics shifts, their performance can drop significantly, yet these nominal policies lack an explicit mechanism to detect that their behavior has degraded. SteerControl gives a nominal world model an online compensatory role by steering model-free actions through a simple counterfactual target, without relying on mechanical equations, an external tracking reference, or additional supervision at deployment. Across twelve simulated environments under stationary and nonstationary perturbations, our results demonstrate the effectiveness of this combination in enhancing the robustness of model-free agents to previously unseen, moderate changes in dynamics.

W2 and Q2 - Baseline comparisons

Our baseline selection reflects our nominal-only policy/model training and the fact that only observed state–action transitions are available as feedback during inference. Most related methods discussed here require training across anticipated dynamics information or variations (like suggested [3] and RMA [4]), analytical mechanical models, or a deployment task objective: an MPC cost in [1] or differentiable simulated rewards in [2]. As the reviewer recognizes, these additional requirements complicate direct comparison under our simple protocol; ignoring these differences would lead to an unfair assessment.

We already included a recent strong baseline, Cerebellar (ICML 2026), which closely matches our training and evaluation protocols. However, to obtain comparable results, we evaluated approximately 30 additional configurations per setting using calibration seeds disjoint from evaluation, allowed cross-episode memory when favored by calibration, and supplied reference or task-performance feedback beyond SteerControl's requirements.

Following the reviewer's suggestion, we added two domain randomization baselines (DR-SAC and an RMA-style PPO baseline) in Appendix E. Both train across the full tested actuator-bias range without receiving severity at evaluation. SteerControl achieves higher mean return than the RMA port in most evaluated settings; DR-SAC outperforms the RMA throughout the sweep. DR-SAC is strong on manipulation and outperforms SteerControl at larger locomotion biases, while SteerControl performs better at smaller biases. The RMA's weak privileged teacher on locomotion tasks limits conclusions about its adaptation mechanism.

Finally, after carefully screening additional controllers for compatibility, we also found and implemented ℒ₁, EVOLVER, and DADO. Preliminary recovered gains (%) under the same benchmark training and evaluation settings are:

MethodReacherPusher
ℒ₁89.6±0.952.3±1.4
EVOLVER73.2±1.566.2±1.8
DADO92.2±1.279.9±1.5
SteerControl93.5±0.884.5±1.4

These implementations were competitive on manipulation tasks but did not reliably train or compensate on Ant, Humanoid, and HalfCheetah; ℒ₁ nevertheless recovered 11.4 ± 0.7% on HalfCheetah and 13.2 ± 10.2% on Ant, while the other measured gains were negative or close to zero.

AuthorsOpenReview ↗

Responses - Part 2/2

W2 and Q2 - Continued

We have kept these preliminary comparisons with other baselines outside the edited manuscript pending validation across tasks; their difficulties do not establish limitations of the original methods.

We agree that experiments on physical robots are important; however, we want to clarify that our paper does not claim successful real-world robotic deployment. The opening of the Introduction mentions this setting just as motivation. Our extensive evaluation instead examines adaptation across twelve simulated environments spanning locomotion and manipulation, compared with four environments in Cerebellar. These include physics-based MuJoCo tasks such as FetchReach, which models a 7-DoF robotic arm. While real-world robotic experiments are not feasible within the present work and are beyond its scope, we consider them valuable and encourage future work in this direction.

ℒ₁: Improving the Robustness of Reinforcement Learning Policies With ℒ₁ Adaptive Control, IEEE RA-L, 2022.

EVOLVER: EVOLVER: Online Learning and Prediction of Disturbances for Robot Control, IEEE T-RO, 2024.

DADO: Data-Driven Adaptive Disturbance Observers for Model-Free Trajectory Tracking Control of Maritime Autonomous Surface Ships, IEEE TNNLS, 2021.

W2, Q3, Q4 - Assumptions and analysis

Input-distribution assumption.

We clarify that the input-distribution assumption is not a strict limitation. We agree that submission line 270 stated nominal input coverage too strongly, and we have narrowed the stated reliance of our fault design on this assumption (Section 3.1). This assumption mainly reflects the generalization limits of neural networks: we use frozen neural policy and world models at inference, whose generalization outside their training distributions is not guaranteed. In practice, however, neural networks can tolerate small distribution shifts and modest extrapolation. Our fault-severity experiment (Figure 2) was intended to examine SteerControl’s applicability under severe faults, where this assumption may hold less closely.

We should also clarify that, in the fault-severity experiment, even a severity of 0.5 represents a substantial fault for a frozen model-free policy trained solely under nominal dynamics. SteerControl still provides effective compensation, although under very large faults, correction bounds may also limit the achievable compensation (submission line 361). In addition, Figure 2b (edited draft) quantifies drift in velocity means, showing that test-time input distributions do not necessarily remain aligned with nominal distributions. However, Figure 2c (edited draft) shows that the ratio of nominal-model error to the fault-induced transition change generally decreases over the milder fault range. This may help explain why SteerControl can still compensate for dynamics shifts that also change the input distributions of its frozen neural networks.

Locality assumption.

We clarify that our nonstationary fault experiments are specifically intended to show that SteerControl remains robust and applicable under abrupt, adversarial fault changes when sufficient intervals are available for recovery. Figure 3 shows that after a severe fault reversal in Pusher and Reacher, compensation rebounds sharply within a few steps and rapidly returns toward pre-switch performance. This responsiveness follows from SteerControl’s simple online compensation mechanism, which constructs a new counterfactual target and recomputes the correction after each observed transition. Such recovery on the scale of individual control steps is particularly valuable when comparing adaptation timescales in continual and meta-RL. Some adaptation protocols operate over multiple episodes or hundreds of steps, which can still be called rapid relative to RL training or fine-tuning: for example, RL² uses multi-episode trials, and PEARL reports test performance after two 200-step trajectories.

Figure 4a further examines dependence on the EMA weight. Smaller weights retain a longer history and therefore rely on locality over a longer temporal window. Nevertheless, SteerControl maintains positive recovered gains throughout the tested range of 0.1–1.0, showing that effective compensation does not depend on a specific, narrowly tuned EMA value in practice. Together with Figure 3, this supports practical flexibility in the temporal locality assumption.

Finally, a few abrupt changes within an episode are therefore compatible with SteerControl, as our experiments directly demonstrate. The relevant temporal limitation arises when changes occur so frequently that the correction cannot track the new dynamics and recover before the next switch. We make these points clear in the paper.

RL²: Fast Reinforcement Learning via Slow Reinforcement Learning, arXiv preprint, 2016.

PEARL: Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables, ICML, 2019.

Reviewer 6yKX: are you convinced, particularly as regards the novelty and the positioning.

Reviewer_d1KhOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SteerControl, a test-time adaptation method for frozen continuous-control policies under unseen dynamics changes. A one-step world model is trained only on nominal transitions. At deployment, the discrepancy between the predicted and observed transition is treated as a disturbance, and the method optimizes a bounded action correction intended to cancel this discrepancy on subsequent steps. Neither the policy nor the world model is updated, and no deployment reward, fault label, or reference trajectory is required. Experiments across twelve simulated control environments show substantial recovery under both stationary and switching perturbations.

Critical strengths
  1. The problem setting is practically relevant and fairly restrictive. Adapting a frozen policy without rewards, target-domain training, or parameter updates is useful for deployed controllers, and the proposed residual-based correction is simple and easy to understand.

  2. The empirical evaluation is broad. The paper covers twelve environments, several types and magnitudes of perturbations, and both stationary and nonstationary settings. SteerControl obtains the best point estimate in 24 of 30 stationary settings and 9 of 12 switching settings. The fault-magnitude and EMA analyses also help characterize when the method works.

Critical weaknesses
  1. The conceptual novelty and justification of the correction rule are somewhat limited. The method is closely related to a learned disturbance observer or local inverse-dynamics correction. More importantly, the correction is optimized using the previous state and then reused at the next state, assuming that both the disturbance and the model’s action sensitivity remain locally stable. The residual-cosine experiment provides partial evidence for one assumption, but does not fully validate this cross-state transfer. Stronger analysis and ablations against simpler residual-cancellation variants would clarify what is genuinely responsible for the gains.

  2. The empirical uncertainty is underestimated. Each environment uses only one trained SAC policy, apparently together with one trained world-model ensemble; the reported uncertainty therefore mainly reflects evaluation rollouts rather than policy/model training variability. This matters because some results, particularly Humanoid and several switching settings, have substantial variance or relatively small gains. Results over multiple independent policy and world-model training seeds, together with ablations on random-action augmentation, the ensemble, and model accuracy, are needed to support the generality of the conclusions.

Questions for authors

See Critical Weaknesses

6 comments

Authors: ideally, please submit a response to the review as this helps facilitate the paper discussion.

AuthorsOpenReview ↗

Responses - Part 1/2

We thank the reviewer for their insightful feedback and for recognizing the practical relevance of adapting frozen policies without deployment rewards, target-domain training, or parameter updates, as well as the simplicity of our correction mechanism and the breadth of our evaluation.

W1 - Conceptual novelty

We agree that compensating for dynamics mismatch is an established problem setting in RL and, particularly, control. The submitted Related Work already discussed various closely related works, including disturbance observers, model-based adaptation, and residual control. We further expanded the Related Work to cover additional control- and learning-based approaches and made our positioning more precise, with changes marked in blue.

Our contribution concerns how these components are combined to equip a frozen, nominally trained model-free policy with an online compensation mechanism without supplied mechanical equations or prior fault information. SteerControl gives a nominal world model an online compensatory role through counterfactual targets that steer actions toward effects familiar to the model-free policy. Both components are trained only under nominal dynamics, and online compensation uses only observations and executed actions, without deployment rewards, external references, supplied task costs, or weight updates. This combination fits a standard RL evaluation protocol and recovers a substantial share of lost performance in many stationary and switching settings across twelve simulated environments.

We also compare against Cerebellar (ICML 2026), a recent closely related method and one of the few we identified as feasible to evaluate under our experimental protocol.

W1 - Cross-state transfer

Since we rely only on actions and their observed next-state outcomes, we cannot causally access an action's actual outcome before executing it in the environment. Predicting that outcome under shifted dynamics is itself a counterfactual question and requires an estimate of the shift's effect, for example from prior knowledge of the fault or an estimator trained on episodes with the same dynamics. We chose not to use such fault-specific priors and accept a one-step-lagged update. The simplicity of action steering through counterfactual targets enables this online compensation, which we find effective across the tested continuous-control environments.

We agree that abrupt, severe fault changes can violate these conditions. However, SteerControl recomputes its correction after each observed transition and can adapt again once the new dynamics persist. Our nonstationary experiments were intended specifically to examine this behavior: Figure 3 shows how effectively SteerControl recovers performance within a few steps after abrupt changes. If faults change so frequently that the correction cannot track the new dynamics before the next change, recovery is limited. We make no claim for this rapidly fluctuating setting; adapting to unpredictable changes without prior information about their evolution is challenging for any method relying on transition feedback.

We clarify that Figure 4b provides only a complementary diagnostic of whether correction estimation needs dedicated unsteered probing or can accompany corrected execution. We now additionally compare probe- and execution-derived corrections P,EP,E, optimized from residuals at matched simulator states with the same incoming estimate. Cosine similarity measures direction; the vector agreement score 2P⊤E/(∥P∥22+∥E∥22)2P^\top E/(\|P\|_2^2+\|E\|_2^2) also accounts for magnitude. All defined scores for differing measurement actions are retained without a similarity cutoff. These same-state comparisons provide supporting evidence rather than full validation of cross-state transfer.

In addition to the vector analysis, behavioral comparisons show higher mean recovered gain with simultaneous estimation and execution in all six environments, including Ant actuator gain and Swimmer action rotation. At the same interaction budget, interleaved probing takes one unsteered measurement/update step per eight steps and corrects the other seven; simultaneous corrects and updates every step, using the interaction budget more effectively even if individual estimates are less accurate. The full results table and a paragraph describing the new experiments have been added in blue at the end of Section 3.4.

EnvironmentCosineVector agreementInterleaved probing gain (%)Simultaneous gain (%)
Reacher0.9990.99954.093.5
Pusher0.9740.97345.684.5
Ant0.8480.80625.237.1
HalfCheetah0.9480.92735.744.9
Walker2d0.9120.88436.252.3
Swimmer0.9820.9519.169.6

Regarding additional analysis, could the reviewer specify the intended baseline or additional experiment that would most directly address their requests?

AuthorsOpenReview ↗

Responses - Part 2/2

W2 - Training variability and model ablations

We thank the reviewer for emphasizing the importance of policy and world-model training variability and for suggesting valuable ablations. Our initial evaluation used one trained SAC checkpoint per environment because SteerControl steers the actions of a frozen, healthy policy independently of the algorithm used to train it. We therefore initially treated these training choices as indirect factors and prioritized expanding the evaluation across twelve continuous-control environments. For comparison, Cerebellar (ICML 2026) evaluated four environments.

Another reason for choosing SAC was its more stable nominal performance in our initial experiments, whereas PPO was more vulnerable to underperformance in certain environments, particularly locomotion tasks. We therefore used SAC for consistency of our reports across environments. We also acknowledge the reviewer's point about Humanoid, where a fall can terminate the episode. Consequently, a fixed fault severity need not be equally recoverable across training seeds: a dynamics shift that allows compensation for one policy may trigger an unrecoverable fall for another. We therefore initially selected one healthy policy to study our post-hoc method.

Following the reviewer's suggestion, we trained SAC and PPO, two widely used model-free RL algorithms, on two locomotion tasks (Ant and HalfCheetah) and two manipulation tasks (Pusher and Reacher), retaining five independently trained seeds with healthy nominal policies per environment and algorithm and excluding underperforming agents. We collected fresh nominal data and trained a world-model ensemble from scratch for each policy. The results below follow the Table 1 evaluation protocol and have been added in blue in Appendix F. SteerControl improves mean return over the faulted base policy in all eight environment–algorithm combinations.

EnvironmentPolicyClean returnPerturbed returnSteerControl returnRecovered gain (%)
AntPPO2008.01447.61655.027.6 ± 18.5
AntSAC4407.42410.83098.740.8 ± 16.4
HalfCheetahPPO6676.6421.32253.629.1 ± 11.9
HalfCheetahSAC10832.04652.66746.135.0 ± 10.2
PusherPPO−29.0−73.2−45.462.3 ± 5.6
PusherSAC−23.3−68.3−29.686.0 ± 3.5
ReacherPPO−6.07−14.80−6.9190.3 ± 2.2
ReacherSAC−3.50−17.06−4.5292.3 ± 1.2

Regarding ensembling and random-action augmentation, we chose these strategies based on common practice for improving neural network robustness under distribution shifts. Following the reviewer's suggestion, we ablated both factors on the same four environments; recovered gains (%) are shown below. The results support both choices, particularly random-action augmentation, with the full configuration achieving the highest mean recovery in all four tasks. We have added this analysis in blue in Appendix E.

Environment50% augmentation + ensemble50% augmentation + single memberNo augmentation + ensembleNo augmentation + single member
Ant37.135.028.926.2
HalfCheetah45.043.043.642.6
Reacher93.592.387.985.2
Pusher84.583.149.746.1

Finally, revised Figure 2c also compares ensemble and individual-model error ratios, showing slightly lower ratios for the ensemble.

Comments on the paper

6 comments
Area_Chair_7cC9OpenReview ↗

From AC.

Dear Authors,

Given the deadline on 18 November, please submit your responses to the reviews to facilitate a productive discussion with the reviewers.

Thanks, AC

AuthorsOpenReview ↗

Responses - Part 1

We thank the reviewer for their insightful feedback and for recognizing the practical relevance of adapting frozen policies without deployment rewards, as well as our evaluation across diverse control tasks and perturbations. Their careful reading also helped us correct inconsistencies in the method description and experimental setup and express our assumptions more cautiously.

W1 and Q2

We thank the reviewer for identifying this inconsistency. The original equations were not fully aligned with the implementation: they mixed the local offset with persistent compensation, yielding the stated accumulation. The implementation already averages local estimates directly. We have corrected Sections 2.3–2.4 and Algorithm 1, with changes marked in blue.

The revised description centers the candidate on the already corrected policy mean, μθ(st)+cˉt−1+c\mu_\theta(s_t)+\bar c_{t-1}+c, so at(0)=ata_t(0)=a_t. We explicitly specify five normalized projected gradient steps from zero at every transition. Only the resulting local estimate enters the EMA, initialized with cˉ0=0\bar c_0=0; it is not first added to the previous compensation.

In the scalar additive-action example with inactive bounds, the corrected loss is (c+b)2(c+b)^2, independent of the previous compensation. An exact solve gives c^t=−b\hat c_t=-b for every cˉt−1\bar c_{t-1}, so cˉt+b=(1−α)(cˉt−1+b)\bar c_t+b=(1-\alpha)(\bar c_{t-1}+b): the correction converges to −b-b geometrically at rate 1−α1-\alpha. The five-step solver may leave estimation error, but this error is averaged rather than accumulated because its estimate is independent of cˉt−1\bar c_{t-1} in this example.

Moreover, projection onto the convex set C\mathcal C and EMA with 0<α≤10<\alpha\leq1 ensure cˉt∈C\bar c_t\in\mathcal C for all tt, starting from cˉ0=0∈C\bar c_0=0\in\mathcal C. This guarantees bounded compensation; general closed-loop state stability requires additional conditions.

W2, Q5 and Q6

We acknowledge that some implementation details were not stated explicitly, although the core training architectures and settings are documented in Appendix B, and correction parameters in Appendix C and the main text (submission Tables 3–4; revised Tables 4–5). Fault types, severities, and switching settings are provided in Tables 1–2, their captions, and Appendix D, while Section 3 discusses setting-specific steering choices. Section 3.1 also specifies five held-out seed batches of eight streams per setting, yielding at least 40 complete episodes per method and evaluating one trained policy per environment (submission lines 299–302).

To strengthen the comparison, Cerebellar received approximately 30 additional configurations per setting, selected using calibration seeds disjoint from evaluation (292–295), covering learning rate, gain, clipping, reference lookup, gates, traces, memory retention, and goal-directed error (893–896). We also allowed corrective memory to carry across episodes, adding the assumption that the same fault persists across episodes in stationary settings. These allowances provide additional privileges and access to prior-episode information, whereas SteerControl requires no cross-episode fault-persistence assumption (295–298).

We thank the reviewer for highlighting the ambiguity in Q5, which arose from an obsolete sentence about single-reference reuse (282–284), now removed from Section 3.1. The immediately following sentence (284–285) and Appendix C (893–896) already state that Cerebellar's search allowed error relative to the current episode's goal in FetchReach and PandaReach, providing goal-dependent targets rather than restricting all candidates to a single stored trajectory. In Table 1, Cerebellar's baseline (†) has a higher recovery point estimate in PandaReach: 87.7% versus SteerControl's 85.5%, which would be difficult to achieve using references associated with unrelated goals.

To address the additional implementation details requested in Q6, we have clarified Section 2.3, Appendix C, and Table 5 to improve reproducibility, with changes marked in blue. Table 5 now includes zero initialization and a normalized L2L_2 correction radius of 1. Appendix C specifies the correction scales, normalization of gradients by their L2L_2 norms, conversion of standardized model outputs back to state-change units, and standardized steering loss. Online steering uses only observed transitions and does not receive rewards.

AuthorsOpenReview ↗

Responses - Part 2

Q1

We agree that mismatch compensation is an established control principle. The submitted Related Work discussed DOB methods (lines 736–745) and model-based approaches (lines 742 and 748), and positioned SteerControl relative to them in its final paragraph (lines 759–767). We have expanded these discussions and made the positioning more explicit, with changes marked in blue. Many prior approaches require additional physical priors or structural assumptions for compensation, such as prescribed plant or disturbance equations, or rely on external tracking references or online dynamics updates coupled with a supplied task cost or reward. SteerControl aims to keep training and evaluation close to a standard RL protocol: both components are trained under a single nominal dynamics setting, and test-time compensation uses only observations and executed actions.

Our motivation is to equip strong, nominally trained model-free policies with an explicit mechanism to make them robust to dynamics shifts that change the effects of their actions(Introduction, lines 69–80). SteerControl turns a nominal world model into an online compensator: prediction residuals generate a counterfactual target that provides action compensation toward a regime familiar to the model-free policy, enabling it to continue operating effectively. These corrections require no explicitly specified inverse model, control-affine parameterization, or separate actuator-effectiveness map. Both components are trained exclusively under nominal dynamics, and online compensation uses observed transitions without deployment rewards, external tracking references, task-specific planning objectives, or weight updates. Across twelve simulated environments, this combination recovers a substantial share of the performance lost under unseen stationary and switching perturbations in many settings (Section 3).

Q3

Figure 4b provides a complementary diagnostic of whether correction estimation needs dedicated unsteered probing or can accompany corrected execution. We now compare probe- and execution-derived corrections P,EP,E, optimized from residuals at matched simulator states with the same incoming estimate. Cosine similarity measures direction; the vector agreement score 2P⊤E/(∥P∥22+∥E∥22)2P^\top E/(\|P\|_2^2+\|E\|_2^2) also accounts for magnitude. All defined scores for differing measurement actions are retained without a similarity cutoff.

In addition to the vector analysis, behavioral comparisons show higher mean recovered gain with simultaneous estimation and execution in all six environments, including Ant actuator gain and Swimmer action rotation. At the same interaction budget, interleaved probing takes one unsteered measurement/update step per eight steps and corrects the other seven; simultaneous corrects and updates every step, using the compensation budget more effectively even if individual estimates are less accurate. The full results table and a paragraph describing the new experiments have been added in blue at the end of Section 3.4.

EnvironmentCosineVector agreementInterleaved probing gainSimultaneous gain
Reacher0.9990.99954.093.5
Pusher0.9740.97345.684.5
Ant0.8480.80625.237.1
HalfCheetah0.9480.92735.744.9
Swimmer0.9820.9519.169.6

Figure 4b's original 0.96 plotting window was chosen solely for better visualization, with bin heights decreasing outside the displayed range. Normalization is likewise a histogram display choice; we chose density within the displayed range instead of raw bin counts. Without restricting the x-axis, narrow peaks near +1 dominated the full-range view and reduced readability. This plotting cutoff is not used by the controller.

Q4

We clarify that the input-distribution assumption is not a strict limitation. We agree that submission line 270 stated nominal input coverage too strongly, and we have narrowed down the reliance of our fault design on this assumption (Section 3.1). In fact, this assumption is directly related to neural networks. We use frozen neural policy and world models at inference, and their generalization outside their training distribution is not guaranteed. In practice, however, neural networks can tolerate small distribution shifts and modest extrapolation. Our fault-severity experiment (Figure 2) was intended to demonstrate SteerControl's applicability under severe faults, which may be less consistent with this assumption.

Motivated by the reviewer, we added Figure 2b–c and analysis in Section 3.2, marked in blue, to examine distribution shift and the error ratio. In the velocity dimensions, distribution shift increases with fault severity, showing that test-time input distributions do not necessarily remain aligned with nominal evaluation distributions. However, the ratio of model error to fault-induced environment error generally decreases over the milder fault range, which may help explain why SteerControl can still compensate for dynamics shifts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.