- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AC/DT, an actor critic framework for offline to online reinforcement learning that replaces the behavior cloning objective of Decision Transformer with direct actor critic optimization. The method combines a sequence conditioned stochastic Transformer actor, lightweight Markovian critics, context truncated lambda returns, and source aware conservative value regularization. The empirical results on D4RL are strong, with particularly good performance on AntMaze and competitive overall offline and online results. However, the methodological novelty appears to be incremental because several key components and the overall offline to online actor critic direction have close precedents. More importantly, the central claim that removing behavior cloning is beneficial is not directly isolated by the experiments.
Critical strengths
-
The paper presents a clear and practically motivated combination of sequence modeling and explicit value based policy improvement. Replacing the standard behavior cloning objective with actor critic optimization is conceptually interesting for Decision Transformer style policies.
-
The empirical results are strong. AC/DT performs very well across the D4RL benchmark, achieving the best offline result on most of the evaluated tasks and strong online performance, with especially notable improvements on AntMaze.
-
The paper includes several useful component ablations, including the Transformer actor, Markovian versus Transformer critics, context truncated lambda returns, source aware conservative regularization, and context length. These experiments help clarify the contribution of individual design choices.
Critical weaknesses
-
The methodological novelty is somewhat limited. Actor critic methods for offline to online reinforcement learning and value guided Decision Transformer variants have already been studied in prior work such as Online Decision Transformer, Q value Regularized Transformer, and Actor Critic Alignment. The main contribution therefore appears to be the particular combination of a sequence conditioned actor, actor critic optimization, context truncated lambda returns, and source aware conservative regularization rather than a fundamentally new learning principle.
-
The central claim that fully replacing behavior cloning with actor critic optimization is beneficial is not directly isolated. The current comparisons change multiple components at the same time, making it difficult to determine whether the improvement comes specifically from removing behavior cloning. A much more convincing experiment would keep the same AC/DT architecture, critic, return estimation, replay strategy, and conservative regularization, while optimizing an objective of the form L_actor = L_AC + beta L_BC with several values of beta, including beta equal to zero. This would directly measure the effect of gradually removing behavior cloning.
-
The state matched distributional analysis does not fully support the claim that the learned policy moves beyond the behavior policy in a state dependent manner. The analysis compares marginal action distributions using kernel density estimation and an averaged per dimension KL divergence. Such marginal statistics discard the state action correspondence and cross dimension action correlations. Two policies can have similar global action marginals while producing substantially different actions at individual states. Moreover, this analysis is only presented for HalfCheetah. A stronger evaluation would compare state conditional action distributions or directly measure policy differences at matched states across multiple tasks.
-
There is a potential mismatch between the history conditioned actor and the Markovian critic. The actor conditions on a sequence of observations and actions, whereas the critic only observes the current state and action. Consequently, the critic is effectively approximating the value of a history dependent policy with a Markovian value function. The paper acknowledges this design choice, but does not provide a theoretical characterization or a sufficiently thorough empirical analysis of when this approximation is reliable. It would be useful to evaluate the sensitivity of AC/DT to different degrees of context aliasing or compare the Markovian critic with a history conditioned critic under otherwise identical settings.
-
Some of the comparisons with strong offline to online baselines rely on previously reported numbers rather than a fully controlled common experimental protocol. This makes it harder to determine whether the observed improvements come from the proposed method itself or from differences in implementation, interaction budgets, or evaluation procedures. In particular, comparisons with methods such as Cal QL, VDT, CPQL, and PQL would be more convincing if they were reproduced under exactly the same online interaction budget and evaluation protocol.
-
The paper currently does not provide implementation code. It only states that the code will be released upon acceptance, which limits independent verification of the reported results and implementation details.
Questions for authors
-
Can the authors provide a controlled experiment using L_actor = L_AC + beta L_BC with multiple values of beta, while keeping all other components fixed? This would directly test whether the benefit comes from removing behavior cloning rather than from the other proposed components.
-
Can the authors provide a state conditional analysis of the learned policy beyond marginal action distributions, preferably across multiple D4RL tasks? How does the proposed policy differ from the behavior policy at the same states?
-
What are the conditions under which the Markovian critic provides a sufficiently accurate learning signal for the history conditioned actor? How sensitive is AC/DT to state aliasing and context length?
-
Can the authors provide same protocol comparisons with the strongest offline to online baselines, using the same interaction budget, evaluation procedure, and implementation setting?
9 comments
Weakness 1. Novelty with respect to ODT, QT and Actor-Critic Alignment.
Weakness 1. Novelty with respect to ODT, QT and Actor-Critic Alignment.
None of the cited methods trains a DT-style policy without a dataset-action likelihood or behavior-cloning term in both offline pretraining and online fine-tuning. ODT has no critic and trains the Transformer by action likelihood with entropy regularization; QT adds a Q-value term to a behavior-cloning objective and is offline only; Actor-Critic Alignment (Yu & Zhang, 2023) uses MLP policies without sequence modeling (we will cite it). AC/DT directly optimizes the stochastic Transformer through a soft actor-critic objective in both phases, with a critic trained on position-wise context-truncated λ-returns built from the same Transformer context. Table 4 shows the importance of multi-step propagation, particularly during online fine-tuning on AntMaze, and the controlled ablation in our response to Question 1 directly tests the effect of removing the behavior-cloning term (removing it improves the final online return on every tested task). Training a DT-style policy without likelihood supervision in both phases, made possible by a critic built from the Transformer context, is therefore the core idea that distinguishes AC/DT from the cited methods, rather than a combination of their components, and both elements contribute to the reported gains. We therefore respectfully disagree with the assessment that the methodological novelty is incremental.
Weakness 2–Weakness 5. These weaknesses correspond to Questions 1–4 and are addressed in our answers below.
Weakness 2–Weakness 5. These weaknesses correspond to Questions 1–4 and are addressed in our answers below.
We thank the reviewer for the review and for recognizing our strong empirical results, particularly on AntMaze, and the usefulness of our component ablations. We address each point below. We hope the new results and clarifications resolve the concerns raised and help the reviewer reassess the paper’s contributions. We would be happy to discuss any remaining questions.