- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a new architecture to address data corruption in offline reinforcement learning. In specific, the paper disentangle the part that is markovian in the sense that the next step is determined by the current step and current action with the part that is noise. They do so by separately embed the corruption and causal. Then they add penalty term to make the two embeddings disentangled. Numerically, the authors show they achieve the highest score on 47 tasks out of 54 tasks.
Critical strengths
The idea of separating the part that follows markovian property and believing the rest is corrupted noise is interesting.
Critical weaknesses
1, The empirical improvement is minor, almost all tasks' scores confidence interval are overlapping and the average improvement over RDT is only 4-8 percent. This questions the real benefits of the specific way of implementing the idea.
2, The authors argue that the noise embedding and causal embedding and disentangled by showing that the embeddings are not similar. However, disentanglement in representation learning means changing the corruption would only affect corruption without affecting the causal embedding [1]. This further questions whether the proposed method really separate the causal part and the corruption part.
[1] Representation Learning: A Review and New Perspectives. Yoshua Bengio, Aaron Courville, and Pascal Vincent
Questions for authors
1, Could the authors provide more evidence that the causal and corruption embeddings are disentangled?
2, Could the authors show additional experiments where the difference between proposed method and RDT are well separated?
8 comments
1. Reliability of the empirical improvements (W1, Q2) (Part 1/2)
CDA-DT achieves the best mean score on 47 of 54 (87%) tasks, indicating a consistent improvement across corruption types and datasets. Our empirical improvements extend across a larger proportion of tasks than those reported in many RL studies. LSDT (ICLR 2025) achieves the highest reported mean on two of nine (22.2%) locomotion datasets. CURL (ICML 2020) reports state-of-the-art scores on seven of 26 (26.9%) Atari100k games. EDAC (NeurIPS 2021) achieves the highest mean in its EDAC column on six of 18 (33.3%) Gym datasets. SPOT (NeurIPS 2022) achieves the highest mean on eight of 15 (53.3%) offline RL datasets. PEER (CVPR 2023) reports state-of-the-art scores on 19 of 26 (73.1%) Atari100k games.
The number of improved tasks is a valid and widely used evaluation metric in RL. RDT (ICLR 2025) highlights the best scores on 24 of 27 random-corruption settings. CURL (ICML 2020) highlights improvements over Efficient Rainbow on 19 of 26 Atari games. EfficientZero V2 (ICML 2024) highlights outperforming DreamerV3 on 50 of 66 tasks. MOSDT (NeurIPS 2025) highlights the highest safe returns on 14 of 18 tasks. PEER (CVPR 2023) similarly highlights that adding its representation regularizer improves CURL on 19 of 26 Atari games and DrQ on 16 of 26. Following these works, we use the number of tasks improved as a primary source of feedback when developing our components.
The number of improved tasks helps assess a component's applicability and generality across environments. Each task contributes one vote, preventing a few extreme gains or losses from dominating the count. The mean is sensitive to both extreme changes and base scores: the same percentage change produces a larger absolute effect on higher-scoring tasks. The two metrics serve different reporting needs. We therefore emphasize improved-task counts in the paper's detailed analysis to show consistency across environments, and mean scores for concise comparisons in the rebuttal.
Compared with RDT, the strongest baseline by overall score, CDA-DT has higher mean scores in all six corruption types and increases the overall score from 66.63 to 70.51, corresponding to a 5.82% gain. Gains of this magnitude have direct precedents in offline RL. Decision ConvFormer (ICLR 2024) reports a locomotion mean of 82.2 versus 81.0 for ODT, the strongest baseline on that benchmark, corresponding to a 1.48% gain. Decision Mixer (ICML 2025) reports a Gym mean of 84.7 versus 82.4 for CGDT, its strongest baseline on that benchmark, yielding a 2.79% gain. RORL (NeurIPS 2022) reports a Gym mean of 85.7 versus 82.9 for EDAC, its strongest baseline by average score, yielding a 3.38% gain. QT (ICML 2024) reports an average of 76.5 versus 73.7 for Diffusion-QL, the strongest baseline on its four AntMaze datasets, yielding a 3.80% gain. EDAC (NeurIPS 2021) reports a Gym mean of 85.2 versus 84.5 for SAC-N, the strongest reference configuration introduced in the same paper, yielding a 0.83% gain. CDA-DT's 5.82% gain over RDT is larger than the improvements reported in these published studies.
1. Reliability of the empirical improvements (W1, Q2) (Part 2/2)
The reported error bars are sample standard deviations, rather than confidence intervals. They show how much each method's score varies across training seeds. Overlapping error bars alone do not tell us whether CDA-DT's improvement over RDT is statistically significant.
Overlapping error bars are also common in RL studies. In ReBRAC (NeurIPS 2023), the error bars of ReBRAC and SAC-RND, the strongest baseline by average score, overlap on five of six AntMaze tasks. In SPOT (NeurIPS 2022), the error bars of SPOT and IQL, identified as the strongest baseline in its offline experiments, also overlap on five of six AntMaze tasks. In PEER (CVPR 2023), the error bars of PEER+CURL and DrQ overlap on all six DMControl500k tasks.
To further assess the consistency of these gains, we repeat the comparison with two additional random seeds across all 54 tasks. Results are shown in Table R1.
Table R1. Comparison with RDT over two additional random seeds. Each score is averaged over all nine datasets and the two new seeds. All other training and evaluation settings are identical to those in the main experiments (Table 1). Bold marks the highest score in each row.
| Corruption | RDT | CDA-DT (ours) |
|---|---|---|
| Adversarial action | 57.54 | 62.59 |
| Adversarial state | 64.05 | 72.04 |
| Adversarial reward | 72.94 | 75.53 |
| Random action | 63.48 | 75.33 |
| Random state | 65.09 | 66.23 |
| Random reward | 69.86 | 80.60 |
| Overall | 65.49 | 72.05 |
Table R2 combines the existing four-seed results in the submitted paper's Table 1 with the results under two additional seeds in Table R1, providing a six-seed comparison.
Table R2. Comparison with RDT combining the four original seeds in Table 1 with the two additional seeds in Table R1. Each score averages all nine datasets and six seeds. All other training and evaluation settings are identical to those in the main experiments (Table 1). Bold marks the highest score in each row.
| Corruption | RDT | CDA-DT (ours) |
|---|---|---|
| Adversarial action | 56.71 | 60.66 |
| Adversarial state | 66.49 | 72.42 |
| Adversarial reward | 72.18 | 76.06 |
| Random action | 66.05 | 71.84 |
| Random state | 63.70 | 67.38 |
| Random reward | 72.36 | 77.79 |
| Overall | 66.25 | 71.02 |
The six-seed results retain CDA-DT's advantage in all six corruption types, with an overall mean of 71.02 versus 66.25 for RDT. We have added these new results in the revised version.
Changes in the revised manuscript (blue). Section 5.1 clarifies the error bars. Section 5.2 distinguishes strict wins from ties and summarizes additional-seed results. Section 5.3 explains the task-count metric. Appendix A.6 gives the metric rationale. Appendix A.18 reports the additional-seed and six-seed results (Tables 16–17). Appendix A.19 contextualizes the gains and overlapping error bars with published studies.
We thank you for recognizing the dynamics-based motivation. Below, we address each of your concerns and questions.