acceptodds
Under review as a conference paper at ICLR 2027

AC/DT: Actor-Critic Decision Transformer For Offline-to-Online Reinforcement Learning

Abstract

We introduce the Actor‑Critic Decision Transformer (AC/DT), an offline‑to‑online (O2O) reinforcement learning transformer-based method that replaces the behavior‑cloning objective of Decision‑Transformer‑style models with a value‑based actor‑critic optimization over a sequence‑conditioned stochastic Transformer policy. AC/DT combines long-horizon sequence conditioning with explicit policy improvement through twin Markovian critics. To exploit the temporal information available within each sampled context, the critics are trained position-wise using context-truncated -returns, providing multi-step value targets while retaining lightweight Markovian value functions. During offline pretraining, we mitigate out-of-distribution overestimation through conservative value regularization. During online fine-tuning, conservative regularization is restricted to offline experience. On D4RL benchmarks, AC/DT moves beyond the offline data distribution and achieves competitive performance against action‑likelihood‑based Transformer baselines such as ODT, hybrid variants like ODT+TD3 and other state-of-the-art O2O methods. A state-matched distributional analysis further shows that AC/DT reduces adherence to the dataset action marginals relative to likelihood-based ODT, providing evidence of a consistent shift in the policy-induced action marginals toward an expert reference.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Offline-to-Online Reinforcement Learning, Transformer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2tXROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AC/DT, an actor critic framework for offline to online reinforcement learning that replaces the behavior cloning objective of Decision Transformer with direct actor critic optimization. The method combines a sequence conditioned stochastic Transformer actor, lightweight Markovian critics, context truncated lambda returns, and source aware conservative value regularization. The empirical results on D4RL are strong, with particularly good performance on AntMaze and competitive overall offline and online results. However, the methodological novelty appears to be incremental because several key components and the overall offline to online actor critic direction have close precedents. More importantly, the central claim that removing behavior cloning is beneficial is not directly isolated by the experiments.

Critical strengths
  1. The paper presents a clear and practically motivated combination of sequence modeling and explicit value based policy improvement. Replacing the standard behavior cloning objective with actor critic optimization is conceptually interesting for Decision Transformer style policies.

  2. The empirical results are strong. AC/DT performs very well across the D4RL benchmark, achieving the best offline result on most of the evaluated tasks and strong online performance, with especially notable improvements on AntMaze.

  3. The paper includes several useful component ablations, including the Transformer actor, Markovian versus Transformer critics, context truncated lambda returns, source aware conservative regularization, and context length. These experiments help clarify the contribution of individual design choices.

Critical weaknesses
  1. The methodological novelty is somewhat limited. Actor critic methods for offline to online reinforcement learning and value guided Decision Transformer variants have already been studied in prior work such as Online Decision Transformer, Q value Regularized Transformer, and Actor Critic Alignment. The main contribution therefore appears to be the particular combination of a sequence conditioned actor, actor critic optimization, context truncated lambda returns, and source aware conservative regularization rather than a fundamentally new learning principle.

  2. The central claim that fully replacing behavior cloning with actor critic optimization is beneficial is not directly isolated. The current comparisons change multiple components at the same time, making it difficult to determine whether the improvement comes specifically from removing behavior cloning. A much more convincing experiment would keep the same AC/DT architecture, critic, return estimation, replay strategy, and conservative regularization, while optimizing an objective of the form L_actor = L_AC + beta L_BC with several values of beta, including beta equal to zero. This would directly measure the effect of gradually removing behavior cloning.

  3. The state matched distributional analysis does not fully support the claim that the learned policy moves beyond the behavior policy in a state dependent manner. The analysis compares marginal action distributions using kernel density estimation and an averaged per dimension KL divergence. Such marginal statistics discard the state action correspondence and cross dimension action correlations. Two policies can have similar global action marginals while producing substantially different actions at individual states. Moreover, this analysis is only presented for HalfCheetah. A stronger evaluation would compare state conditional action distributions or directly measure policy differences at matched states across multiple tasks.

  4. There is a potential mismatch between the history conditioned actor and the Markovian critic. The actor conditions on a sequence of observations and actions, whereas the critic only observes the current state and action. Consequently, the critic is effectively approximating the value of a history dependent policy with a Markovian value function. The paper acknowledges this design choice, but does not provide a theoretical characterization or a sufficiently thorough empirical analysis of when this approximation is reliable. It would be useful to evaluate the sensitivity of AC/DT to different degrees of context aliasing or compare the Markovian critic with a history conditioned critic under otherwise identical settings.

  5. Some of the comparisons with strong offline to online baselines rely on previously reported numbers rather than a fully controlled common experimental protocol. This makes it harder to determine whether the observed improvements come from the proposed method itself or from differences in implementation, interaction budgets, or evaluation procedures. In particular, comparisons with methods such as Cal QL, VDT, CPQL, and PQL would be more convincing if they were reproduced under exactly the same online interaction budget and evaluation protocol.

  6. The paper currently does not provide implementation code. It only states that the code will be released upon acceptance, which limits independent verification of the reported results and implementation details.

Questions for authors
  1. Can the authors provide a controlled experiment using L_actor = L_AC + beta L_BC with multiple values of beta, while keeping all other components fixed? This would directly test whether the benefit comes from removing behavior cloning rather than from the other proposed components.

  2. Can the authors provide a state conditional analysis of the learned policy beyond marginal action distributions, preferably across multiple D4RL tasks? How does the proposed policy differ from the behavior policy at the same states?

  3. What are the conditions under which the Markovian critic provides a sufficiently accurate learning signal for the history conditioned actor? How sensitive is AC/DT to state aliasing and context length?

  4. Can the authors provide same protocol comparisons with the strongest offline to online baselines, using the same interaction budget, evaluation procedure, and implementation setting?

9 comments

We thank the reviewer for the review and for recognizing our strong empirical results, particularly on AntMaze, and the usefulness of our component ablations. We address each point below. We hope the new results and clarifications resolve the concerns raised and help the reviewer reassess the paper’s contributions. We would be happy to discuss any remaining questions.

AuthorsOpenReview ↗

Weakness 1. Novelty with respect to ODT, QT and Actor-Critic Alignment.

Weakness 1. Novelty with respect to ODT, QT and Actor-Critic Alignment.

None of the cited methods trains a DT-style policy without a dataset-action likelihood or behavior-cloning term in both offline pretraining and online fine-tuning. ODT has no critic and trains the Transformer by action likelihood with entropy regularization; QT adds a Q-value term to a behavior-cloning objective and is offline only; Actor-Critic Alignment (Yu & Zhang, 2023) uses MLP policies without sequence modeling (we will cite it). AC/DT directly optimizes the stochastic Transformer through a soft actor-critic objective in both phases, with a critic trained on position-wise context-truncated λ-returns built from the same Transformer context. Table 4 shows the importance of multi-step propagation, particularly during online fine-tuning on AntMaze, and the controlled ablation in our response to Question 1 directly tests the effect of removing the behavior-cloning term (removing it improves the final online return on every tested task). Training a DT-style policy without likelihood supervision in both phases, made possible by a critic built from the Transformer context, is therefore the core idea that distinguishes AC/DT from the cited methods, rather than a combination of their components, and both elements contribute to the reported gains. We therefore respectfully disagree with the assessment that the methodological novelty is incremental.

AuthorsOpenReview ↗

Weakness 2–Weakness 5. These weaknesses correspond to Questions 1–4 and are addressed in our answers below.

Weakness 2–Weakness 5. These weaknesses correspond to Questions 1–4 and are addressed in our answers below.

Reviewer_wwWEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Actor-Critic Decision Transformer (AC/DT), an offline-to-online reinforcement learning method that replaces the action-likelihood objective of Decision Transformer with a soft actor-critic objective. The method combines a sequence-conditioned stochastic Transformer actor with twin state-action critics, trained at each context position using context-truncated λ-returns. Conservative value regularization is applied during offline pretraining and restricted to offline-origin samples during online fine-tuning. Experiments on ten D4RL locomotion and AntMaze tasks report competitive offline and online performance against the evaluated baselines. Ablation studies investigate actor and critic architectures, multi-step targets, conservative regularization, and context length. An additional state-matched analysis compares the policy-induced action marginals of AC/DT and ODT with medium and expert dataset references.

Critical strengths
  1. Competitive empirical results. AC/DT obtains the highest mean final online score on nine of the ten tasks in the main comparison, with results reported over five random seeds. Its performance on sparse-reward AntMaze tasks provides evidence that the approach is effective in the evaluated offline-to-online setting.

  2. Ablations covering the main components. The paper studies actor and critic architectures, multi-step targets, conservative regularization, and context length. The MLP-actor comparison keeps the remaining training components fixed, and the λ ablation examines the contribution of multi-step targets. These experiments help assess the contribution of the proposed design choices.

Critical weaknesses

AC/DT combines a Transformer actor, a soft actor-critic objective, conservative regularization, and multi-step targets. Although the ablations support several components, they do not directly establish the benefit of fully removing behavior cloning over retaining it under otherwise identical settings. Moreover, varying context length changes both the actor’s history and the critic’s backup horizon, leaving their respective contributions unclear. These gaps make it difficult to assess the specific advance over existing value-guided Transformer and multi-step actor-critic methods.

Questions for authors

1.Do you have a controlled comparison with a variant that retains a behavior-cloning term while keeping the architecture, critic targets, replay strategy, and training budget unchanged? This would help establish whether removing behavior cloning contributes to the reported gains.

2.In mixed replay batches, is the conservative penalty averaged over the entire batch, with online samples contributing zero, or only over the offline subset? Equation (10) appears to specify the latter, which does not by itself imply that the effective regularization weight decreases as the offline-data proportion declines.

3.Could you verify �0��0� in Table 4 and �1��1� in Table 2? If these summarize five seed-level scores bounded between 0 and 100, the reported standard deviations exceed the corresponding mathematical upper bounds. Please clarify the number of seeds, the variability measure, and whether these entries contain typographical errors.

6 comments

We thank the reviewer for the review and for recognizing our competitive results, including on sparse-reward AntMaze, and the ablations covering the main components. We address each point below. We hope the new results and clarifications resolve the concerns raised and help the reviewer reassess the paper’s contributions. We would be happy to discuss any remaining questions.

AuthorsOpenReview ↗

Weakness. Isolating the benefit of removing behavior cloning, and the separate roles of actor history and critic backup horizon.

Weakness. Isolating the benefit of removing behavior cloning, and the separate roles of actor history and critic backup horizon.

'The ablations do not directly establish the benefit of fully removing behavior cloning over retaining it under otherwise identical settings'. We address the comparison with a retained likelihood term, and the resulting advance over existing methods, in Question 1. In short, reintroducing the likelihood term lowers the final online return on all four tested tasks, and all runs with behavior cloning fail on antmaze-large-diverse. Since existing value-guided Transformer methods such as QT, VDT and ODT+TD3 retain a likelihood term, this ablation also isolates the specific advance of AC/DT over them.

'Varying context length changes both the actor’s history and the critic’s backup horizon, leaving their respective contributions unclear'. Varying K changes both factors by design. In AC/DT, the critic has no context length of its own: its multi-step targets are built from the same context window that the Transformer actor receives, which is precisely our second contribution. A single K therefore sets both the actor history and the maximum critic backup horizon, and the K ablation (Appendix G) evaluates this shared design parameter, we can clarify this in the paper. We nevertheless respectfully disagree that their respective contributions are unclear, because they are isolated by dedicated ablations. Appendix A.2 replaces the Transformer actor with an MLP, removing the actor history, while retaining the same critic-learning procedure (the Transformer actor is higher on all four tasks, e.g., 128.2 vs. 96.1 on walker-medium). Table 4 in Appendix C varies the contribution of multi-step backups while keeping the actor architecture and context length fixed (with one-step targets, the final online return on antmaze-large-diverse drops from 96.2 to 0.7). Appendix A.1 and Appendix B additionally examine critic architecture and source-aware regularization. Both factors therefore contribute, and each can be assessed separately from these ablations.

The K ablation addresses a different question: the sensitivity of the complete method to its shared context length. As noted above, our critic-learning contribution explicitly reuses the Transformer context for position-wise multi-step supervision, so this experiment preserves that coupling. Appendix G shows that increasing the shared context length does not yield monotonic performance gains, and that K = 5 is a robust operating point.

AuthorsOpenReview ↗

Question 1. Controlled comparison with a variant that retains a behavior-cloning term, with everything else unchanged.

Question 1. Controlled comparison with a variant that retains a behavior-cloning term, with everything else unchanged.

We thank the reviewer for suggesting this experiment, which directly tests our central claim. Reintroducing ODT’s action-likelihood supervision reduces mean final online performance on all four tasks. We optimize L_actor = L_AC/DT + β L_BC, with β ∈ {0, 0.1, 0.5, 1} during both offline pretraining and online fine-tuning, where β=0\beta=0 is the original AC/DT without the behavior-cloning term. L_BC is the same supervised action-likelihood term used by stochastic ODT. Actor and critic architectures, critic-target construction, conservative regularization, replay strategy, and training budget are otherwise unchanged. The original AC/DT achieves the highest mean online return for every task, and increasing the BC weight progressively lowers these returns on all three MuJoCo tasks. On walker-medium, the return drops from 128.2 without BC to 91.9 at β = 1. On antmaze-large-diverse, AC/DT reaches 96.2, whereas all nine BC runs (three values of β, three seeds each) finish at zero. At β = 0.1, this failure occurs despite an improved offline return of 45.7 versus 23.6. This answers the question directly: with all other components fixed, removing the behavior-cloning term improves the final online return on every task, most strongly on sparse-reward AntMaze.

Taskβ = 0 Off.β = 0 On.β = 0.1 Off.β = 0.1 On.β = 0.5 Off.β = 0.5 On.β = 1 Off.β = 1 On.
hopper-medium86.4 ± 4.0103.7 ± 3.283.2 ± 1.097.9 ± 3.080.3 ± 9.995.8 ± 3.885.8 ± 2.671.7 ± 40.1
walker-medium91.2 ± 1.2128.2 ± 4.785.3 ± 0.1122.9 ± 3.285.4 ± 0.1112.9 ± 2.182.9 ± 1.391.9 ± 1.9
halfcheetah-medium52.9 ± 0.399.5 ± 1.652.2 ± 0.197.4 ± 0.451.2 ± 0.197.3 ± 1.749.3 ± 0.394.5 ± 0.4
antmaze-large-diverse23.6 ± 6.996.2 ± 1.345.7 ± 12.50.0 ± 0.021.7 ± 12.50.0 ± 0.012.3 ± 6.00.0 ± 0.0

Results are mean D4RL normalized scores ± standard deviation across seeds. β = 0 reports the original proposed AC/DT from Table 1 using five seeds; each configuration with β ∈ {0.1, 0.5, 1} uses three seeds. Bold indicates the highest mean for each task and phase.

Reviewer_9af2OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AC/DT for offline-to-online reinforcement learning by replacing the behavior-cloning objective of Decision Transformers with a soft actor-critic objective. Specifically, it employs a stochastic Transformer actor trained with twin MLP critics using context-truncated multi-step returns. A CQL regularizer is applied only to offline samples, gradually reducing conservatism as online data accumulates. Experiments on ten D4RL tasks show that AC/DT achieves the best performance on 9 out of 10 tasks compared with IQL, CQL, AWAC, ODT, and ODT+TD3. The paper also provides extensive ablation studies on network architectures, multi-step returns, context length, and regularization.

Critical strengths

1. Fully replacing likelihood training with actor-critic optimization in a DT-style policy is simple and well motivated, and the method is precisely specified (Eqs. 5–11, Figs. 1–2).

2.* With 5 seeds and IQM aggregation, AC/DT improves substantially over Transformer baselines (e.g., halfcheetah-medium online 99.5 vs. 41.9 for ODT) and reaches about 96 on AntMaze-large.

3. Extensive, candid ablations. The ablations cover actor/critic architecture, λ, context length and the regularization schedule. The divergence of Transformer critics (App. A.1) is an informative negative result, and the authors openly report cases where alternatives do better.

Critical weaknesses

1. gains seem to come from the multi-step critic rather than the DT design. The method combines SAC, CQL, uncorrected multi-step returns and a Transformer actor. With λ=0, the AntMaze online IQM drops from 97.4 to 0.3 (Table 4), far below one-step CQL. Yet the closest method, conservative Peng's Q(λ) (CPQL/PQL), is not compared under the same protocol, and its published results are higher on several MuJoCo tasks (Table 8).

2. Without the likelihood loss, nothing ties actions to return-to-go, yet evaluation and online RTGs are tuned per task (Table 7), and no ablation removes the RTG tokens. The Transformer actor's advantage over an MLP actor (Table 2) rests on two comparisons in which the MLP variant has very high variance (96.1±26.6, 63.7±55.1).

3.

  • The authors' own CQL runs fall well below the published numbers they cite (hopper-medium online 64.8 vs. 99.0; walker-medium 84.9 vs. 104.0).
  • AC/DT is tuned per task with seven hyperparameters, while baseline tuning is not described; Cal-QL and RLPD are absent from the main table.
  • Some reported ± values cannot be standard deviations of scores bounded in [0, 100] (e.g., 28.7±67.9 in Table 4).
Questions for authors

See weaknesses and:

1. The actor objective (Eq. 5) does not involve return-to-go, yet evaluation and online RTGs are tuned per task (Table 7). How does AC/DT perform without RTG tokens, or with a fixed RTG at evaluation?

2. With λ=0, the AntMaze online IQM drops to 0.3 (Table 4), whereas one-step CQL reaches about 95. What causes this collapse? How does AC/DT compare, under your protocol, with CQL using the same λ-returns, or with CPQL/PQL?

3. In Table 4, antmaze-medium-play (λ=0, offline) reports 28.7±67.9, which cannot be a standard deviation of scores in [0, 100]. The same holds for 63.7±55.1 in Table 2 if 5 seeds were used. What does ± denote, and how many seeds are used in the ablations?

8 comments

We thank the reviewer for the review and for recognizing that fully replacing likelihood training with actor-critic optimization is simple and well motivated, and that our ablations are extensive and candid. We address each point below. We hope the new results and clarifications resolve the concerns raised and help the reviewer reassess the paper’s contributions. We would be happy to discuss any remaining questions.

AuthorsOpenReview ↗

Weakness 1 Do the gains come from the multi-step critic rather than the DT design, and how does AC/DT compare with CPQL/PQL?

Weakness 1 Do the gains come from the multi-step critic rather than the DT design, and how does AC/DT compare with CPQL/PQL?

We agree with the reviewer that multi-step propagation is central to the performance of AC/DT, especially on AntMaze. However, we respectfully disagree that this separates the gains from the DT design, because the multi-step critic is itself part of that design.

'AC/DT does not simply attach a generic multi-step critic to a DT actor'. Our second contribution is Position-wise Context-Truncated λ\lambda-Return Critic Learning (Sec. 4.4). The same sampled Transformer window that conditions the actor also provides critic supervision at every decision position. Concretely, with a context of K = 5 steps, the first position in the window receives a λ\lambda-return that combines backups of up to five rewards observed in the context, the second position up to four, and the last position a one-step target. Each backup whose final transition is nonterminal ends with a bootstrap value computed from actions sampled by the Transformer actor (Eq. 6). In other words, the critic targets are built from the same context windows that the Transformer actor uses, and the bootstrap uses actions sampled from the Transformer policy and evaluated by the target critics. The λ\lambda ablation demonstrates the importance of multi-step propagation within this design, particularly on sparse-reward tasks, and therefore supports one of the paper’s explicit contributions. It should not be interpreted as evidence against the AC/DT design, of which this critic-learning scheme is an integral part. The comparison between λ=0\lambda = 0 and one-step CQL is discussed in our answer to Question 2.

'The actor ablation directly tests whether the Transformer contributes beyond the multi-step critic'. To separate the contribution of the Transformer actor from that of the multi-step critic, we performed a controlled ablation. In Appendix A.2, we replace the Transformer actor with an MLP actor that sees only the current state, while retaining the same critic architecture, position-wise λ\lambda-return construction, conservative regularization, replay strategy, and training protocol. Both variants therefore receive exactly the same multi-step critic signal, and only the actor differs. If the gains came from the critic alone, the two variants would perform similarly. Tab 2 reports higher mean final online returns for the Transformer across all four evaluated tasks (hopper-medium 103.7 vs. 99.6, walker-medium 128.2 vs. 96.1, antmaze-large-play 96.4 vs. 95.3, antmaze-large-diverse 96.2 vs. 63.7), with a lower seed standard deviation on all four. The high variance of the MLP variant, which the reviewer notes in Weakness 2, reflects the instability that the Transformer actor avoids. Fig 5 also shows differences in fine-tuning stability: on antmaze-large-play, the final returns are similar, but the MLP undergoes substantial intermediate degradation during the online phase before recovering, whereas the Transformer actor reaches high returns early and sustains them through most of fine-tuning. . This shows that the improvement does not come from the multi-step critic alone.

'CPQL/PQL reproduction'. The released CPQL code implements offline training but does not provide the online PQL fine-tuning stage required for the offline-to-online comparison; the replay buffer’s method for adding online transitions is explicitly left unimplemented in the official code. We attempted to implement the missing online stage following the paper, but obtained lower scores than those reported by the authors. Presenting those runs as CPQL/PQL results in the main comparison could therefore misrepresent the method. Instead, we include the authors’ published scores in Tab 8, recognizing CPQL/PQL as a relevant baseline while distinguishing published reference results from methods we could evaluate reliably under our common protocol.

'Tab 8 does not show an overall advantage for CPQL/PQL in final online performance'. The CPQL/PQL scores in Tab 8 are published results obtained under a different protocol, not a controlled comparison. Even taken at face value, AC/DT obtains a final online score higher than or equal to CPQL/PQL on seven of the ten tasks, including all four AntMaze tasks, and a higher average online score. On the challenging large AntMaze tasks, AC/DT reaches 96.4 and 96.2, versus 85.4 and 82.0 for CPQL/PQL. AC/DT’s final online score is at least 96.2 on every task, while CPQL/PQL’s drops to 82.0. CPQL/PQL’s higher final online scores are confined to three MuJoCo locomotion tasks (hopper-medium, hopper-medium-replay, walker-medium-replay). Therefore, while we agree that CPQL/PQL is relevant baseline, its published results don't indicate that it outperforms AC/DT: on the 6 locomotion tasks each method is higher on 3, and on sparse-reward AntMaze AC/DT is clearly stronger, with a gap of 11 to 14 points on the large mazes.

AuthorsOpenReview ↗

Weakness 2 RTG conditioning, and the Transformer actor’s advantage over an MLP actor (Table 2)

'Weakness 2 RTG conditioning, and the Transformer actor’s advantage over an MLP actor (Table 2)'

'Without the likelihood loss, nothing ties actions to return-to-go, yet evaluation and online RTGs are tuned per task (Table 7), and no ablation removes the RTG tokens'. We address this point in detail in our answer to Question 1. In short, the RTG remains a conditioning input of the Transformer under the actor-critic objective, the RTG values follow the standard ODT and ODT+TD3 protocols rather than a per-task search, and removing the RTG tokens lowers offline and final online performance on all tested tasks (Table T1 Below). We therefore respectfully disagree that actions are disconnected from the RTG in AC/DT.

'The Transformer actor’s advantage over an MLP actor (Table 2) rests on two comparisons in which the MLP variant has very high variance'. We agree with the reviewer that the MLP variant shows very high variance on walker-medium and antmaze-large-diverse. However, we respectfully disagree that this weakens the comparison: the advantage is not limited to two tasks, and the high variance is itself evidence of the instability that the Transformer actor avoids. With the critic-learning procedure held fixed, the Transformer actor achieves a higher mean final online return and a lower standard deviation across seeds on all four tasks in Table 2 (3.2 vs. 4.4, 4.7 vs. 26.6, 1.7 vs. 5.0, and 1.3 vs. 45.0). Since the two variants share the same critic, replay strategy and training protocol, this higher variability is attributable to the actor: across seeds, the MLP actor does not fine-tune reliably, while the Transformer actor does. On walker-medium and antmaze-large-diverse, the two tasks highlighted in the review, the Transformer achieves both the largest gains in mean final return (128.2 vs. 96.1 and 96.2 vs. 63.7) and the largest reductions in variability across seeds. Figure 5 shows a further advantage on antmaze-large-play, where the final returns are close. The Transformer reaches a high return early and sustains it through most of fine-tuning, while the MLP degrades sharply before recovering. The Transformer actor’s advantage is therefore supported by the consistent results across all four tasks in Table 2 and by the fine-tuning dynamics in Figure 5.

Comments on the paper

1 comment

We thank all the reviewers and the Area Chair for their time, constructive feedback, and careful consideration of our rebuttal responses.

We have uploaded a revised version of the manuscript incorporating the clarifications and analyses presented in our rebuttal responses. All changes are highlighted in red for ease of reference.

The main text remains within the 10-page limit as indicated in the guidelines for submission and rebuttal, with additional experimental results and analyses provided in the appendix.

We appreciate the reviewers' feedback and their reconsideration of our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.