acceptodds
Under review as a conference paper at ICLR 2027

Robustifying Decision Transformers at Test Time with Direct Compensation

Abstract

Policies trained offline, such as the Decision Transformer (DT), degrade when the deployment transition dynamics differ from those seen during training. adaptive control offers a test-time remedy, but its cancellation step is built for control-affine dynamics, so applying it to a learned model has meant affinizing that model and tuning when to refresh the affinization. We propose direct compensation, an -inspired compensator that uses the learned next-state model directly with no affinization or tolerance needed. Analytically, direct cancellation's undamped first step is a regularized affine inversion, and when the learned model is control-affine and no regularizer is used, the direct cancellation coincides with the affine inversion. On MuJoCo locomotion under matched disturbances, direct overall achieves better compensation than affine for DT policies trained across diverse environments, and datasets. Ablation studies locate the primary source of the gain in the removal of the stale affinization action, which is made possible by direct 's non-interrupted state predictor and filter updates. The approach is agnostic to the base policy, so it applies beyond transformer-based policies.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Decision Transformer, Test-Time Adaptation, Robust Adaptive Control, Disturbance Rejection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VW7iOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Direct L1\mathcal {L}_1, a test-time compensator for offline policies instantiated by a Decision Transformer with an auxiliary next-state prediction head. At deployment, the compensator estimates the delta action to offset prediction errors of state. Then a estimator will propagate state to next timestep. The learned dynamics model and policy remains frozen.

Critical strengths
  1. This method proposed a deployment-time correction mechanism that do not require online retraining. Using a correction action head to inject delta action into original action to offset the errors is a practical way.
  2. The ablation part is informative that the author controls different parts of Direct L1\mathcal {L}_1 then tests these different variants. The results show that different components contribute together to the improvements.
Critical weaknesses
  1. The comparison experiments do not prove the advantage of proposed Direct L1\mathcal{L}_1. The paper only compares Direct L1\mathcal{L}_1 with two baselines DT and affine L1\mathcal{L}_1 on three environments. These results can not support the claim that Direct L1\mathcal{L}_1, with relatively complex design, really performs better than current baselines and alternatives.
  2. The distrubance evaluation is too restricted to support general deployment-time dynamics robustness. All experiments use action noise under accumulation error hypothesis. The discrepency of action is injected into the same channel which is used to correct it. There are no tests of changes to transition model. Also, the experiment results do not show robustness against unpredictable or rapid change to environment.
Questions for authors
  1. Can the author provide results with a learned residual controller or history-conditioned adaptive policy? Can this comparison be conducted under the main Table 1 configurations? Also, what advantage does the adaptive law provide over a simple proportional/PID style model residual controller?
  2. What if we change the embodiments, friction coefficient of environment in the experiments?The author may refer to DADP[1], Meta-DT[2] about the possible extensive experiments design. Also, what happens if we inject completely IID errors into actions? Can the proposed method still performs well under these settings? [1]Wang, P., Liu, Q., Lin, H., Li, Y., Zhan, G., Tomizuka, M., & Wang, Y. (2026). DADP: Domain Adaptive Diffusion Policy. arXiv preprint arXiv:2602.04037. [2]Wang, Z., Zhang, L., Wu, W., Zhu, Y., Zhao, D., & Chen, C. (2024). Meta-DT: Offline meta-RL as conditional sequence modeling with world model disentanglement. Advances in Neural Information Processing Systems, 37, 44845-44870.
Reviewer_g9Y7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies test-time action correction in the context of offline reinforcement learning with a decision transformer (DT). It proposes a method inspired by the optimal control L1 compensation method to improve the choice of the action. The method works by estimating a state prediction difference using last time-step action, and finding a good action perturbation at current time-step to compensate the effect at the next time-step. This leverages the native DT capabilities, autoregressive on (state, action, rtg) trajectories. The method evaluates on continuous control task with fixed open source reference datasets. It shows uniformly improved performance (of a modest magnitude).

Critical strengths
  • The idea is natural and fairly simple to apply. It works with fixed-weights and standard datasets.
Critical weaknesses
  • I found the paper really hard to read and understand. I found the phrasing (over)-complicated, especially when it comes to technical details and ablations. I think it could use a more pedagogical tone and simpler language. Especially for the broader ICLR community which is not necessarily expert in continuous control.

  • I don't think the contribution is significant enough for ICLR. It demonstrates a modest improvement on 3 tasks and 3 datasets but it does not compare to other methods that would use the predictive model of DT to improve the predictions (expect for the affine L1 baseline). Besides, most reported improvement has overlapping confidence intervals with the baselines.

Questions for authors
  • How does this method compares to other search-based methods in the prediction space of the DT?

  • What is the test overhead of the direct L1 correction compared to the vanilla DT inference ?

  • Can you clarify how to interpret the ablations conducted in Table 3?

Reviewer_vPPSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes "direct L1 compensation", a test-time module that makes an offline-trained DT more robust to deployment-time disturbances. The DT is trained with an auxiliary next-state head, which is used as a learned one-step dynamics model. At test time, an L1-adaptive-control state predictor and adaptive law estimate a lumped residual from the one-step prediction error; an action correction is then obtained by directly minimizing a regularized objective through the nonlinear state head, low-pass filtered, and added to the DT action. Compared with the affine L1 approach of Sung et al. (2024), this removes the affinization, its tolerance, and the associated estimator resets. Two propositions show that the first undamped Gauss-Newton step equals a ridge-regularized affine inversion, and that it reduces exactly to the classical matched cancellation when the model is control-affine and unregularized. Experiments on D4RL HalfCheetah/Hopper/Walker2d under random-walk additive action noise show 2-8 point gains over DT and affine L1, and ablations attribute most of the gain to removing the stale linearization point.

Critical strengths

Simple, plug-in test-time correction with careful ablations. The method requires no retraining of the policy, adds roughly the same per-step cost as affine L1, and the component ablation (carry / regularizer / linearization point) gives a clear picture of where the gain over the affine baseline comes from.

Critical weaknesses
  1. The state-prediction capability and the correction mechanism are not explained or validated.

i) The whole method relies on the auxiliary next-state head being an accurate dynamics model, yet its one-step prediction accuracy, in particular under disturbance, where states leave the data support, is never reported. The residual estimate lumps together the true disturbance and the learned model’s own error, and the paper does not show how these are separated or why the resulting correction points in the right direction. The authors’ own diagnostics suggest it often does not. The “parallel” component (fraction of the disturbance actually cancelled) is only 0.10-0.25, while the “orthogonal” component (correction in directions the disturbance does not occupy) is of similar size. Without this analysis, it is unclear why the correction works and when it would fail.

ii) The DT is not a natural host for this idea. A standard DT does not predict states, so a state head must be bolted on. Policies that explicitly predict future states and recover actions via inverse dynamics, such as Decision Diffuser (Ajay et al., 2023), would be a more natural and more convincing testbed; the claim that the approach is “agnostic to the base policy” is not supported by any experiment beyond DT.

  1. The experiments do not test genuinely different dynamics. The paper is motivated by deployment dynamics that differ from training, but the only disturbance evaluated is additive random-walk noise on the executed action, which is slowly varying and, as the authors note, “action-compensable by construction”. This is the setting most favorable to the method. There are no experiments with changed physical parameters (mass, friction, damping, gravity), contact changes, or unmatched/state-dependent disturbances, so the central claim of robustness to dynamics shift is not substantiated.
Questions for authors
  1. What is the one-step prediction error of the state head on clean and disturbed rollouts, and how does it compare with the magnitude of the injected disturbance? How much of the estimated residual is model error rather than disturbance?

  2. How does the method perform under genuine dynamics shifts, e.g., changed body mass, friction, or joint damping, where the residual is state-dependent and not necessarily matched to the action channel?

  3. Can the compensator be applied to a policy that natively predicts future states, such as Decision Diffuser, and to standard non-sequence policies (e.g., TD3+BC, IQL) with a separately trained dynamics model, to support the base-policy-agnostic claim?

  4. How does the method compare with robust DT variants (RDT, ARDT) or other robust offline RL methods under the same disturbances?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.