- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether offline RL policies trained with independent errors remain effective when errors persist across consecutive steps at test time. UDT extends RDT with a learned score that controls attention to each input token. The same score also weights prediction errors during training. The authors evaluate UDT under Drift-Attack, which generates persistent errors. UDT achieves the highest average score across nine D4RL tasks.
Critical strengths
- The experiments cover nine tasks from three benchmark families. UDT outperforms DT and RDT on average after training with either random or adversarial corruption.
Critical weaknesses
-
The paper lacks a comparison with independent test errors at the same frequency and size. The burst experiment helps, but longer bursts also corrupt more tokens. Keeping the amount of corruption fixed would help distinguish an advantage against persistent errors from better noise tolerance in general.
-
The experiment without routing is difficult to interpret. The paper states that uncertainty scores are trained through attention, while gradients are blocked through the loss weights. Under this description, removing routing appears to leave the scores untrained even though they still weight prediction errors. The lower performance could then come from using untrained scores in the loss. Please clarify how these scores are obtained in this experiment.
Questions for authors
-
How does UDT compare with RDT when test errors occur independently but have the same frequency and size as Drift-Attack?
-
Are the uncertainty scores trained from scratch in the experiment without routing? If so, how are they updated when gradients are blocked through the loss weights?
4 comments
Clustered and scattered errors at a fixed corruption budget (W1)
2. Clustered and scattered errors at a fixed corruption budget (W1)
We keep the corruption budget fixed to see how clustering changes the predicted action. For each trained policy, we start from the same clean history and apply the same offsets to ten of the last 20 steps. We place the errors either in one block or spread them through the history. The latest corrupted step is the same in both layouts, and the current observation is clean. We measure how far the prediction moves from the action for a clean history, then divide that distance by the square root of the action dimension.
Table R2. Action deviation under clustered and scattered errors at the same corruption budget. Deviations (×10⁻³) are measured relative to predictions from clean histories. Change is the deviation under clustered errors minus the deviation under scattered errors.
| Corrupted input | Scattered / RDT | Scattered / UDT | Clustered / RDT | Clustered / UDT | Change / RDT | Change / UDT |
|---|---|---|---|---|---|---|
| Observation | 34.90 | 28.63 | 40.53 | 31.89 | 5.63 | 3.26 |
| Action | 40.06 | 32.74 | 51.33 | 41.20 | 11.27 | 8.47 |
| Reward | 0.141 | 0.191 | 0.060 | 0.121 | -0.081 | -0.071 |
With the same corruption budget, clustering observation or action errors increases action deviation less for UDT than for RDT. The increase in action deviation is 42.1% smaller for observations and 24.8% smaller for actions. UDT has the smaller increase in 29 of 36 observation and action settings, across both random and adversarial training corruption. The scattered layout also places some errors farther back in the history.
For reward errors, UDT's action deviation is larger than RDT's in both layouts. Clustering the errors reduces the deviation for both methods. Moving the same raw reward offsets to different steps can change the return-to-go tokens in different ways.
Do untrained loss weights explain the drop when routing is removed? (W2, Q2)
3. Do untrained loss weights explain the drop when routing is removed? (W2, Q2)
In the original w/o Routing ablation, we initialize the uncertainty branch from scratch and it receives no gradient from the prediction loss. In full UDT, that gradient reaches the branch through attention. In w/o Routing, the uncertainty scores still weight the loss, and a stop-gradient blocks learning through those weights. That leaves untrained loss weights in the ablation, which could contribute to the drop.
We trained w/o Both to test whether the untrained weights account for the gap to full UDT. This variant keeps routing disabled and drops the uncertainty-based loss weights. The comparison with w/o Routing tests those weights. The comparison with full UDT tests routing and reweighting together.
Table R3. Effect of removing untrained loss weights with routing disabled. D4RL normalized returns under Drift-Attack (mean ± SD across three training seeds).
| Training variant | Overall |
|---|---|
| Full UDT | 63.5 ± 0.6 |
| w/o Routing | 62.3 ± 0.3 |
| w/o Both | 62.4 ± 0.5 |
Removing the untrained weights leaves almost the same gap to full UDT. With routing disabled, the overall score goes from 62.3 to 62.4 when we remove those weights. Full UDT scores 63.5.
To measure the effect of routing at inference, we switch it on and off in the same trained model. We pick the checkpoint and target return with routing on, using one evaluation seed. We then run both modes under Drift-Attack on a different seed, keeping the checkpoint and target return fixed.
Table R4. Routing on vs. off under Drift-Attack with the same trained checkpoint. D4RL normalized returns. Paired differences are mean ± SD across three training-seed averages.
| Routing | Overall | Observation | Action | Reward |
|---|---|---|---|---|
| On | 56.5 | 48.9 | 52.5 | 68.1 |
| Off | 51.8 | 41.1 | 48.2 | 66.1 |
| On minus off | 4.7 ± 0.9 | 7.8 ± 3.1 | 4.3 ± 4.0 | 2.0 ± 1.7 |
Switching on routing improves returns with the same checkpoint and learned uncertainty encoder. The gain is 4.7 ± 0.9 points overall and is positive on average for all three training seeds. Routing gives a higher score in 95 of 162 comparisons across settings and seeds. We will clarify the original ablation and report both comparisons.
Looking forward to your feedback (Urgent)
We hope that our rebuttal has effectively addressed all your concerns. As the deadline for the discussion period draws near (less than 3 hours), we would greatly appreciate it if you could re-evaluate our manuscript and provide any feedback. Thank you for your time and consideration.
How do UDT and RDT compare under independent errors matched to Drift-Attack? (W1, Q1)
1. How do UDT and RDT compare under independent errors matched to Drift-Attack? (W1, Q1)
We compare Uncertainty-Guided Decision Transformer (UDT) and Robust Decision Transformer (RDT) under independent and persistent test errors with similar rates and magnitudes. For IID-Matched, we sample magnitudes from Drift-Attack independently at each step. The samples include zeros, so the error rate and magnitude distribution stay the same in expectation. DeFog (ICLR 2023, Fig. 13) compares independent frame drops with Markov bursts, and GRAD (ICLR 2024, Figs. 3–4) compares temporally coupled and decoupled perturbations. We use the submitted Drift-Attack scores for the same 18 settings on Walker2d, Kitchen-Mixed, and Relocate.
Table R1. UDT and RDT under independent and persistent errors of similar frequency and size. D4RL normalized returns. The ± values are standard deviations across three training seeds. The Drift-Attack overall scores average the 18 setting means from Tables 7 and 8 of the submission. Parentheses indicate the training corruption.
At similar error rates and magnitudes, UDT leads RDT on average under both independent and persistent errors. The lead is 4.93 points under IID-Matched and about 3.50 points under Drift-Attack. On Relocate, RDT leads under IID-Matched and UDT leads under Drift-Attack. On Kitchen-Mixed, UDT leads under IID-Matched and RDT leads under Drift-Attack. The rankings are descriptive comparisons across two separate evaluations.
On Walker2d, the two tests produce observation errors of similar frequency and size. Under Drift-Attack and IID-Matched, the error rates are 24.70% and 24.59%, and the root-mean-square magnitudes are 0.4187 and 0.4168, respectively. The correlation between consecutive magnitudes is 0.7408 under Drift-Attack and 0.0008 under IID-Matched.